barebrowse 0.2.0 → 0.2.2

package/.claude/stash/barebrowse-research-2026-02-22.md

@@ -1,49 +0,0 @@
-# Stash: barebrowse Research & Repo Creation
-**Timestamp:** 2026-02-22
-**Session focus:** Research steipete repos, design shared browsing layer
-
-## Key Decisions
-
-1. **Researched steipete/sweetlink** — daemon + WebSocket + CDP architecture for driving real browser tabs. Not headless — opposite approach. Good ideas, overcomplicated for our needs.
-
-2. **Researched steipete/sweet-cookie** — TS library extracting cookies from real browser profiles (Chrome/FF/Safari). Direct fit for mcprune's Cloudflare/auth-wall problem.
-
-3. **Identified the parallel** — mcprune (headless DOM reading) and Multis (interactive browsing) share the same underlying need: get a browser context that loads any URL, authenticated as the user, without being blocked.
-
-4. **Designed `barebrowse`** — a unified browsing layer with 3 modes:
-   - `headless` — Playwright + cookie injection + stealth (mcprune)
-   - `headed` — CDP connect to running Chrome (Multis)
-   - `hybrid` — try headless, auto-fallback to headed
-
-5. **Key simplification over sweetlink** — skip daemon entirely, use CDP directly. Playwright wraps CDP already. No WebSocket bridge, no in-page runtime injection.
-
-6. **Named it `barebrowse`** — follows bare- ecosystem (bareagent, barebrowse, mcprune, multis)
-
-## Work Completed
-
-- Created repo: `~/PycharmProjects/barebrowse/` (git init, main branch)
-- Moved research doc: `mcprune/docs/peter-repos.md` → `barebrowse/docs/prd.md`
-- PRD includes: architecture diagrams, API sketches, 4-phase POC plan, repos to study, package structure
-
-## Repos to Borrow From
-
-| Repo | Take |
-|---|---|
-| steipete/sweet-cookie | Cookie extraction (TS) |
-| steipete/sweetlink | Selector discovery, click patterns, CDP dual-channel |
-| steipete/canvas | Stealth/anti-detection config |
-
-## Next Steps
-
-- [ ] Start barebrowse POC Phase 1 (headless + cookies)
-- [ ] Wire mcprune to use barebrowse instead of direct Playwright
-- [ ] Design Multis headed-mode integration
-
-## Bare Ecosystem Map
-
-```
-bareagent  → agent orchestration
-barebrowse → browser access layer (NEW)
-mcprune    → DOM snapshot pruning
-multis     → personal assistant
-```

package/.claude/stash/phase3-interactions-complete.md

@@ -1,69 +0,0 @@
-# Phase 3 Interactions — Complete
-
-**Date:** 2026-02-22
-**Session:** Real-world interaction testing + fixes
-
-## What Was Done
-
-### Fixes to `src/interact.js`
-1. **scrollIntoView before click** — `DOM.scrollIntoViewIfNeeded` in `getCenter()` before `DOM.getBoxModel`
-2. **`press(key)` function** — 14-key KEY_MAP (Enter, Tab, Escape, Backspace, Delete, arrows, Home/End, PageUp/Down, Space). Enter has `text: '\r'`, Tab has `text: '\t'` for proper form submission.
-3. **`type({ clear: true })`** — Ctrl+A select-all + Backspace before typing to replace pre-filled content
-4. **Key event text field** — Without `text: '\r'` on Enter keyDown, form `onsubmit` never fires
-
-### Additions to `src/index.js`
-- `page.press(key)` — wired to `cdpPress`
-- `page.waitForNavigation(timeout)` — `Page.loadEventFired` promise
-
-### Tests: 54/54 passing
-- `test/integration/interact.test.js` (15 tests, 8 suites):
-  - Round 1: data: URL fixture (7 tests — click, type, clear, offscreen scroll, Enter submit, unknown key error, link navigation)
-  - Round 2: Google Search (consent handling, type+Enter+nav — bot-blocked but flow works)
-  - Round 3: Wikipedia (article link click + navigation)
-  - Round 4: GitHub (SPA navigation with settle time)
-  - Round 5: DuckDuckGo (search + results — headless-friendly)
-  - Round 6: Hacker News (story link click + navigation)
-  - Round 7: Reddit/old.reddit.com (with fallback to www.reddit.com)
-  - Round 8: Firefox cookie injection (extractCookies → injectCookies → Network.getAllCookies verification)
-
-### `examples/headed-demo.js`
-10-step demo: Wikipedia → click link → DuckDuckGo → type query → Enter → results. Requires `chromium-browser --remote-debugging-port=9222`.
-
-### YouTube Demo (ad-hoc, in /tmp)
-Proved the full loop: Firefox cookies extracted → injected into headed Chromium → YouTube consent bypassed → searched "Family Portrait Pink" → clicked and played the video. User watched it happen live.
-
-## Key Findings
-
-### Real-world breakage patterns
-- **Cookie consent walls** block everything (Google, YouTube). Need locale-aware button matching or cookie injection to bypass.
-- **Bot detection** (Google, Reddit) blocks headless. Headed mode with real cookies is the fix.
-- **ARIA roles vary wildly** — DuckDuckGo search box is `LabelText` not `textbox`, Google's is `combobox` with `[expanded=false]` between role and ref.
-- **`Page.loadEventFired` doesn't fire** for data: URL → data: URL navigation or SPA transitions.
-- **`Page.frameNavigated` fires too early** — page DOM not ready yet, snapshots return empty.
-- **Example.com link text changed** from "More information..." to "Learn more" — real sites change.
-- **IANA page about example domains** contains "Example Domain" text — can't assert absence.
-- **Act mode prunes links** on simple pages like example.com — need browse mode to find clickable links.
-
-### Firefox → Chromium cookie flow (proven)
-```
-Firefox cookies.sqlite (plaintext) → extractCookies({ browser: 'firefox', domain })
-  → injectCookies(session, cookies) via Network.setCookie
-  → headless/headed Chromium uses user's sessions
-```
-Works for YouTube (8 cookies), Google (35 cookies), GitHub, etc.
-
-## Docs Updated
-- `docs/poc-plan.md` — Phase 3 DoD checkboxes checked, test section expanded
-- `docs/blueprint.md` — interactions section updated, "Real-world testing" marked DONE, wait strategies partially done
-- `docs/prd.md` — wait strategies and cookie auth decision updated
-
-## Commits
-1. `6017113` — feat(interact): add scrollIntoView, press(), clear, waitForNavigation
-2. `41bed12` — test: add real-site rounds + headed demo, update docs
-
-## What's Next (not started)
-- Phase 4: Hybrid mode (headless → headed fallback)
-- Stealth patches (`navigator.webdriver`, etc.)
-- More wait strategies (network idle, element presence)
-- Shopping/checkout flows, login forms, dropdowns
-- Screenshot capture for visual verification

package/.claude/stash/phase3-prep.md

@@ -1,88 +0,0 @@
-# Stash: barebrowse Phase 3 Prep
-**Timestamp:** 2026-02-22
-**Session focus:** Built Phase 1 + Phase 2 of barebrowse POC
-
-## What barebrowse Is
-Vanilla JS library — CDP-direct browsing for autonomous agents. URL in → pruned ARIA snapshot out. No Playwright, no bundled browser. Uses user's installed Chromium.
-
-## Completed Work
-
-### Phase 1 — CDP + ARIA Foundation (DONE)
-- `src/chromium.js` (142 lines) — Find/launch any Chromium browser, parse CDP WebSocket URL from stderr
-- `src/cdp.js` (148 lines) — Vanilla WebSocket CDP client with flattened session support (sessionId at top level)
-- `src/aria.js` (69 lines) — Format ARIA tree as YAML-like text, skip InlineTextBox/LineBreak noise
-- `src/index.js` (223 lines) — `browse(url)` and `connect(opts)` public API
-
-### Phase 2 — Auth + Prune (DONE)
-- `src/auth.js` (279 lines) — Cookie extraction from Chromium (AES-128-CBC + KWallet/GNOME keyring) and Firefox (plaintext). Injection via CDP `Network.setCookie`. Uses `node:sqlite` with `immutable=1` URI to read live DBs.
-- `src/prune.js` (472 lines) — Full port of mcprune's 9-step pruning pipeline adapted for CDP ARIA tree format. Two modes: `act` (actions only) and `browse` (keeps content).
-
-### Tests — 39/39 passing
-- `test/unit/prune.test.js` — 16 tests (pruning logic, pure function)
-- `test/unit/auth.test.js` — 7 tests (cookie extraction from Firefox)
-- `test/unit/cdp.test.js` — 5 tests (CDP client, browser launch, ARIA tree)
-- `test/integration/browse.test.js` — 11 tests (end-to-end pipeline)
-
-### Docs
-- `docs/prd.md` — Comprehensive PRD with all decisions and rationale
-- `docs/poc-plan.md` — 4-phase plan with DoD, test instructions, repo study reference
-- `CLAUDE.md` — Project dev rules stub
-
-## Key Architecture Decisions (settled)
-- CDP direct, not Playwright (no 200MB download)
-- ARIA tree, not DOM (semantic, token-efficient)
-- Pruning built-in from mcprune (not optional)
-- Three modes: headless/headed/hybrid (one flag, same CDP code)
-- Chromium-only (CDP constraint, Firefox later via BiDi)
-- Vanilla JS, ES modules, Node >= 22, zero required deps
-- sweet-cookie not on npm — wrote our own auth.js
-- Unique temp dirs per browser instance (`/tmp/barebrowse-{pid}-{timestamp}`)
-
-## Test Results
-- HN: 51,932 chars raw → 27,312 pruned (47% reduction on minimal site)
-- example.com: browse mode keeps paragraphs, act mode keeps only heading
-- Cookie extraction: 181 Firefox cookies across 54 domains
-- GitHub cookies found but `logged_in=no` (user not logged in via Firefox)
-
-## System Details
-- OS: Fedora Linux, KDE Plasma, Wayland
-- Node: 22.22.0 (built-in WebSocket, experimental node:sqlite)
-- Browser: Firefox default, Chromium installed via `sudo dnf install chromium`
-- Chromium binary: `/usr/bin/chromium-browser`
-- KWallet running with Chromium Safe Storage key available
-
-## What's Next — Phase 3 (Headed + Interaction)
-
-**Goal:** Connect to user's running browser, click/type on pages.
-
-**Files to create/update:**
-- Update `src/chromium.js` — `connect()` mode for running browser on debug port
-- `src/interact.js` — Click (`Input.dispatchMouseEvent`), type (`Input.dispatchKeyEvent`), scroll. Resolve ARIA nodeId to DOM coordinates.
-- Update `src/index.js` — Add click/type to connect() page handle
-
-**Key challenge:** Mapping ARIA nodeId to screen coordinates for click targeting. CDP `DOM.getBoxModel` + `Accessibility.getFullAXTree` node→backendDOMNodeId mapping needed.
-
-**Prerequisite:** User launches browser with `--remote-debugging-port=9222`
-
-## Phase 4 — Hybrid + bareagent Integration
-- `src/stealth.js` — Anti-detection patches via `Runtime.evaluate`
-- Hybrid mode: try headless, detect CF/403, fall back to headed
-- Wire as bareagent tool functions
-
-## Repos Studied
-- steipete/sweet-cookie — concept only (not on npm, wrote our own)
-- steipete/sweetlink — CDP-direct concept, skip daemon/WS bloat
-- steipete/canvas — stealth patterns noted for Phase 4
-- mcprune (own) — pruning logic fully ported
-
-## File Inventory
-```
-src/index.js      223 lines  — Public API
-src/chromium.js   142 lines  — Browser find/launch
-src/cdp.js        148 lines  — CDP WebSocket client
-src/aria.js        69 lines  — ARIA tree formatter
-src/auth.js       279 lines  — Cookie extraction + injection
-src/prune.js      472 lines  — ARIA tree pruning
-                 1333 total source
-                  576 total tests
-```

package/docs/poc-plan.md

@@ -1,230 +0,0 @@
-# barebrowse — POC Plan
-
-**Date:** 2026-02-22
-**Goal:** Prove that an autonomous agent can get an authenticated, pruned ARIA snapshot of any web page via CDP — no Playwright, no bundled browser.
-
----
-
-## Repo Structure
-
-```
-barebrowse/
-├── src/
-│   ├── index.js          # Public API: browse(), connect()
-│   ├── chromium.js        # Find/launch/connect to Chromium browsers
-│   ├── cdp.js             # Vanilla WebSocket CDP client
-│   ├── aria.js            # Accessibility.getFullAXTree → structured tree
-│   ├── auth.js            # Cookie extraction + CDP Network.setCookie injection
-│   ├── prune.js           # ARIA tree pruning (ported from mcprune)
-│   ├── interact.js        # Click, type, scroll via CDP Input domain
-│   ├── consent.js         # Auto-dismiss cookie consent dialogs
-│   └── stealth.js         # Anti-detection patches via Runtime.evaluate (Phase 4)
-├── test/
-│   ├── integration/
-│   │   ├── browse.test.js     # End-to-end: URL → pruned snapshot
-│   │   ├── auth.test.js       # Cookie injection → authenticated page
-│   │   └── interact.test.js   # Click/type on a live page
-│   └── unit/
-│       ├── cdp.test.js        # CDP client message handling
-│       ├── aria.test.js       # ARIA tree formatting
-│       └── prune.test.js      # Pruning logic on sample trees
-├── docs/
-│   ├── prd.md             # Product requirements (comprehensive)
-│   └── poc-plan.md        # This file
-├── package.json
-└── CLAUDE.md
-```
-
-**No build step.** Vanilla JS, ES modules, runs directly with Node.js >= 22.
-
----
-
-## Phases
-
-### Phase 1 — CDP + ARIA Foundation
-
-**Prove:** Get an ARIA tree from any page via CDP, no Playwright.
-
-**Files:**
-- `src/chromium.js` — Find installed Chromium browsers on the system (Chrome, Chromium, Brave, Edge). Launch headless with `--headless=new --remote-debugging-port=<port>`. Parse CDP WebSocket URL from stderr output.
-- `src/cdp.js` — Vanilla WebSocket client that speaks CDP. Send JSON commands, receive responses and events. Handle command IDs, promises, event subscriptions. ~100 lines.
-- `src/aria.js` — Call `Accessibility.getFullAXTree` via CDP. Transform the raw CDP response (flat array of AXNodes with parentId references) into a nested tree structure. Format as readable output.
-- `src/index.js` — Wire chromium → cdp → aria into `browse(url)` function. Minimal, just the pipeline.
-
-**Test:**
-```bash
-node -e "import { browse } from './src/index.js'; console.log(await browse('https://example.com'))"
-```
-
-**DoD:**
-- [x] `chromium.js` finds and launches at least one Chromium browser on Fedora Linux
-- [x] `cdp.js` connects via WebSocket, sends commands, receives responses
-- [x] `aria.js` returns a structured ARIA tree for any public page
-- [x] `browse(url)` works end-to-end with zero external dependencies
-- [x] Headless Chrome process is cleaned up on close
-
-### Phase 2 — Auth + Prune
-
-**Prove:** Authenticated, pruned ARIA snapshot of a Cloudflare-protected page.
-
-**Files:**
-- `src/auth.js` — Extract cookies from user's browser profile (use sweet-cookie or implement minimal extraction from Chrome's Cookies SQLite DB + Linux keyring decryption via `secret-tool`). Inject via CDP `Network.setCookie` before navigation.
-- `src/prune.js` — Port mcprune's pruning logic as a pure function. Input: raw ARIA tree. Output: pruned ARIA tree. Role-based: keep landmarks + interactive elements, drop noise/structural wrappers.
-- Update `src/index.js` — Add cookie injection and pruning to the `browse()` pipeline.
-
-**Test:**
-```bash
-# Should return authenticated content, not a login wall or CF challenge
-node -e "import { browse } from './src/index.js'; console.log(await browse('https://some-cf-protected-site.com'))"
-```
-
-**DoD:**
-- [x] `auth.js` extracts cookies from Firefox profile on Linux (also supports Chromium when installed)
-- [x] Cookies injected via CDP before navigation
-- [ ] CF-protected page returns real content, not challenge page (needs active session to test)
-- [x] `prune.js` reduces ARIA tree by 47%+ on HN (minimal site — heavier sites will see 70%+)
-- [x] Pruned output preserves all interactive elements and landmarks
-- [x] `browse(url)` returns pruned, authenticated snapshot by default
-
-### Phase 3 — Headed Mode + Interaction
-
-**Prove:** Connect to user's running browser and interact with a logged-in page.
-
-**Files:**
-- Update `src/chromium.js` — Add `connect()` mode: connect to an already-running browser's debug port instead of launching a new one. Detect running browsers with debug ports.
-- `src/interact.js` — Click (`Input.dispatchMouseEvent`), type (`Input.dispatchKeyEvent`), scroll. Resolve ARIA node IDs to DOM coordinates for click targets.
-- Update `src/index.js` — Add `connect()` export for long-lived sessions. Add `mode: 'headed'` option.
-
-**Prerequisite:** User must launch their browser with `--remote-debugging-port=9222` flag.
-
-**Test:**
-```bash
-# User has Chrome open with debug port, logged into GitHub
-node -e "
-  import { connect } from './src/index.js';
-  const page = await connect({ mode: 'headed' });
-  await page.goto('https://github.com/notifications');
-  console.log(await page.snapshot());
-"
-```
-
-**DoD:**
-- [x] `connect()` attaches to a running Chromium browser via CDP
-- [x] Same ARIA + prune pipeline works on headed browser
-- [x] `click()` and `type()` send real input events via CDP
-- [x] `press()` sends special keys (Enter, Tab, Escape, arrows) — triggers form submit
-- [x] `scrollIntoView` before click ensures off-screen elements are reachable
-- [x] `type({ clear: true })` replaces pre-filled input content
-- [x] `waitForNavigation()` waits for page load after link clicks
-- [x] Interactions tested against real sites: Wikipedia, GitHub, Google, Hacker News, DuckDuckGo, YouTube
-- [x] Browser stays open after barebrowse disconnects
-- [x] Cookie injection via `page.injectCookies()` for headed mode (Firefox → Chromium)
-- [x] Permission prompts suppressed via launch flags + CDP `Browser.setPermission`
-- [x] Cookie consent dialogs auto-dismissed across 16+ sites in 7 languages
-- [x] YouTube end-to-end: Firefox cookies → search → click → video playback in headed mode
-
-### Phase 4 — Hybrid + bareagent Integration
-
-**Prove:** Agent autonomously browses the web using barebrowse tools.
-
-**Files:**
-- Update `src/chromium.js` — Add `mode: 'hybrid'`. Try headless first. If navigation returns a CF challenge or 403, automatically retry in headed mode.
-- `src/stealth.js` — Basic anti-detection: patch `navigator.webdriver`, `navigator.plugins`, `window.chrome`. Applied via `Runtime.evaluate` on new page.
-- Update `src/index.js` — Final API surface: `browse()`, `connect()`.
-
-**Test:**
-```js
-import { Loop } from 'bare-agent';
-import { browse } from './src/index.js';
-
-const tools = [
-  { name: 'browse', execute: ({ url }) => browse(url) },
-];
-
-const loop = new Loop({ provider });
-await loop.run([
-  { role: 'user', content: 'Go to hacker news and tell me the top 3 stories' }
-], tools);
-```
-
-**DoD:**
-- [ ] Hybrid mode automatically falls back when headless is blocked
-- [ ] Stealth patches reduce headless detection on common sites
-- [ ] bareagent can use `browse()` as a tool in its think/act/observe loop
-- [ ] Agent successfully completes a multi-page research task autonomously
-
----
-
-## Definition of Done — Full POC
-
-The POC is complete when ALL of these are true:
-
-1. **`browse(url)` works end-to-end** — URL in, pruned ARIA snapshot out, authenticated as the user
-2. **Zero heavy deps** — no Playwright, no Puppeteer. Only deps: `ws` (WebSocket client, if Node's built-in isn't sufficient) and optionally `sweet-cookie`
-3. **Three modes work** — headless (default), headed (connect to running browser), hybrid (auto-fallback)
-4. **Works on Fedora Linux** — finds Chrome/Chromium/Brave, launches headless, connects headed
-5. **Token-efficient output** — pruned ARIA tree is 70%+ smaller than raw tree
-6. **Clean process management** — headless browser spawned and killed cleanly, no orphan processes
-7. **Under 1,000 lines total** for core src/ (excluding tests)
-8. **Documented** — PRD captures all decisions, this file captures all phases
-
-## What the POC is NOT
-
-- Not production-ready. No error recovery, no retry logic, no edge case handling beyond happy path.
-- Not cross-platform tested. Linux first (Fedora). macOS/Windows later.
-- Not an MCP server. That's a future wrapper.
-- Not a published npm package. Local development only.
-
----
-
-## Running Tests
-
-```bash
-# All tests (47+ tests)
-node --test test/unit/*.test.js test/integration/*.test.js
-
-# Unit tests only (fast, no network)
-node --test test/unit/prune.test.js    # 16 tests — pruning logic
-node --test test/unit/auth.test.js     # 7 tests — cookie extraction (2 fail when Chromium locked)
-node --test test/unit/cdp.test.js      # 5 tests — CDP client + browser launch
-
-# Integration tests (needs network + Chromium)
-node --test test/integration/browse.test.js    # 11 tests — end-to-end pipeline
-node --test test/integration/interact.test.js  # 15 tests — interactions on real sites
-
-# Quick smoke test
-node -e "import { browse } from './src/index.js'; console.log(await browse('https://example.com'))"
-
-# Headed mode demos (requires: chromium-browser --remote-debugging-port=9222)
-node examples/headed-demo.js    # Wikipedia → DuckDuckGo search
-node examples/yt-demo.js        # YouTube: Firefox cookies → search → play video
-```
-
----
-
-## Repos Studied — What We Borrowed vs Built
-
-| steipete repo | What we studied | What we used | Why not more |
-|---|---|---|---|
-| **sweet-cookie** | Cookie extraction (SQLite + keyring) | **Concept only** — wrote `auth.js` ourselves | Not on npm (different package). Our version is simpler, tailored, vanilla JS |
-| **sweetlink** | CDP dual-channel, selector discovery, daemon | **CDP-direct concept only** | Daemon + WebSocket bridge + in-page runtime = bloat. CDP direct is 100 lines vs ~2,000 |
-| **canvas** | Stealth/anti-detection patterns | **Noted for Phase 4** `stealth.js` | Not needed yet — headless + real cookies handles most cases |
-| **mcprune (own)** | ARIA pruning pipeline | **Full port** — `prune.js` (472 lines) | Proven code, adapted node format from Playwright YAML to CDP tree objects |
-
-### What to explore in later phases
-
-- **Selector discovery** (sweetlink) — crawl ARIA tree, score interactive elements, rank action targets. Phase 3/4.
-- **Stealth patches** (canvas) — `navigator.webdriver`, plugins, chrome object spoofing. Phase 4.
-- **In-page JS execution** (sweetlink) — `Runtime.evaluate` for complex interactions. Phase 3.
-- **Screenshot + visual grounding** — `Page.captureScreenshot` for multimodal agents. Post-POC.
-
----
-
-## Dev Rules (from AGENT_RULES.md)
-
-- **Vanilla JS only.** No TypeScript, no build step, no transpilation.
-- **Dependency hierarchy:** vanilla → stdlib → external. Write it yourself if <50 lines.
-- **Simple > clever.** Readable code a junior can follow.
-- **POC first.** Validate logic before designing. Never ship the POC — rewrite it.
-- **Test behavior, not implementation.** Integration tests over unit tests.
-- **No speculative code.** Every line must have a purpose.