cdpilot 0.9.0 → 0.9.1

package/README.md

@@ -76,7 +76,7 @@ npx cdpilot <command>
 npm i -g cdpilot
 ```
 
-**Requirements:** Node.js 18+ and one of: Brave Browser, Google Chrome, or Chromium.
+**Requirements:** Node.js 18+, Python 3.10+, and one of: Brave Browser, Google Chrome, or Chromium. (The Python `websockets` module is auto-installed by the pre-flight wizard on first launch.)
 
 ### First-time setup
 
@@ -562,20 +562,16 @@ on macOS — spot degradation before your automation silently stalls.
 ### Scaling & Workstation Use
 
 ```bash
-CDPILOT_POOL_SIZE=4 cdpilot launch   # 4 separate browser processes
 CDPILOT_OFFSCREEN=1 cdpilot launch   # headed, but no window on your screen
 ```
 
-- **Multi-instance pool** (`CDPILOT_POOL_SIZE=N`) starts N independent browser
-  processes and dispatches work to the least-loaded one, for `N × per-instance`
-  parallelism. Default is `1` — single instance, no change for existing users.
 - **Off-screen mode** (`CDPILOT_OFFSCREEN=1`) keeps the browser headed (real
   rendering, no headless fingerprint) but positions the window where it can't
   steal focus — meant for automating on a workstation you're also using.
-- **Docker + Xvfb harness** (`cdpilot-bench/docker/`) runs headed-in-Xvfb so
-  bench/automation runs never pop a window on the host display. CI-ready. Note
-  that software rendering (no GPU) lowers anti-bot scores versus native — it's
-  an isolated environment for reproducibility, not the headline configuration.
+- **Multi-instance pool** (`CDPILOT_POOL_SIZE`) — planned, not shipped yet.
+  See [Roadmap](#coming-soon).
+- **Docker + Xvfb harness** — used in a separate, private benchmark
+  repo (`cdpilot-bench`), not part of this package.
 
 ## Use with AI Agents
 
@@ -624,7 +620,6 @@ print(result.stdout)
 | `BROWSER_SESSION` | Auto | Session identifier |
 | `CDPILOT_MODE` | `regular` | Stealth tier override (`regular`/`stealth`/`undetected`) |
 | `CDPILOT_OFFSCREEN` | `0` | Headed but render off-screen — no window steals focus |
-| `CDPILOT_POOL_SIZE` | `1` | N separate browser processes, least-loaded dispatch |
 
 ## How It Works
 
@@ -636,7 +631,7 @@ print(result.stdout)
        │                                        │
        │  Zero dependencies                     │  Isolated profile
        │  Pure HTTP + WebSocket                 │  Separate from your
-       │  ~2500 lines, single file              │  personal browser
+       │  single file, stdlib + websockets      │  personal browser
        └────────────────────────────────────────┘
 ```
 
@@ -694,8 +689,8 @@ The full adaptive layer is bench-neutral vs baseline (30/80 vs 30/80) for Stealt
 
 | Feature | cdpilot | Puppeteer | Playwright | Selenium |
 |---------|-----------|-----------|------------|----------|
-| Install size | **~50KB** | 400MB+ | 200MB+ | 100MB+ |
-| Dependencies | **0** | 50+ | 30+ | Java + drivers |
+| Install size | **one Python file + a small Node launcher, no node_modules** | 400MB+ | 200MB+ | 100MB+ |
+| Dependencies | **0 npm** (Python: `websockets`, auto-installed) | 50+ | 30+ | Java + drivers |
 | Setup time | **instant** | minutes | minutes | painful |
 | AI-agent ready | **yes** | manual | manual | manual |
 | Browser download | **no** | yes (Chromium) | yes (3 browsers) | no |
@@ -753,7 +748,8 @@ The only browser MCP with built-in test assertions. Here's what we've shipped an
 - [x] 70+ CLI commands (navigate, click, fill, screenshot, PDF, console, network, video understanding, friction ladder...)
 - [x] MCP server for AI agent integration (Claude Code, Cursor, etc.)
 - [x] **10 built-in test assertions** — assert, assert-url, assert-title, assert-count, assert-value, assert-attr, assert-visible/hidden, wait-for, check (batch), screenshot-diff
-- [x] **Accessibility tree snapshot** (`a11y-snapshot`) — structured data with @ref references, 500x fewer tokens than screenshots
+- [x] **`test` runner + `trace` viewer** — `cdpilot test [--watch]` runs `*.cdpt.js` files; `cdpilot trace open <run>` opens a time-travel trace viewer for a run
+- [x] **Accessibility tree snapshot** (`a11y-snapshot`) — structured text with @ref handles the agent can act on directly, no vision model needed; 1.4–42× smaller than the page's raw HTML on the four pages we measured (2026-09-27). Not always cheaper than a screenshot: link-heavy pages produce more text than a small screenshot costs
 - [x] **Token-efficient screenshots** — element-level crop (13x smaller), JPEG quality control, format selection
 - [x] **Vision fallback** (`describe`) — a11y + screenshot + text in one call
 - [x] **Annotated screenshots** — @N badge overlays on interactive elements
@@ -780,7 +776,7 @@ The only browser MCP with built-in test assertions. Here's what we've shipped an
 - [x] **Three-tier stealth mode** (`mode regular|stealth|undetected`) — single switch over fingerprint surface, with per-host adaptive tier learning
 - [x] **Press-and-hold solver** (`press-hold`) — humanized press → hold → release gesture (Gaussian 3–7s + micro-jitter) for PerimeterX/HUMAN behavioral challenges; `captcha-solve` auto-routes here
 - [x] **Image CAPTCHA + profile warming** — offline Amazon image OCR (optional `amazoncaptcha` lib) + BYOK image-to-text (capsolver/2captcha); `profile warm` ages the profile for reCAPTCHA v3 score
-- [x] **Multi-instance pool + off-screen mode** (`CDPILOT_POOL_SIZE`, `CDPILOT_OFFSCREEN`) — N parallel browser processes; headed rendering without stealing window focus
+- [x] **Off-screen mode** (`CDPILOT_OFFSCREEN`) — headed rendering without stealing window focus
 - [x] **Data extraction** (`extract`) — structured DOM data in text, JSON, or list format
 - [x] **Page observation** (`observe`) — list all interactive elements with available actions
 - [x] **Script runner** (`run`) — execute `.cdp` script files with pass/fail reporting
@@ -788,6 +784,7 @@ The only browser MCP with built-in test assertions. Here's what we've shipped an
 ### Coming Soon
 
 - [ ] **iframe** support — interact with elements inside iframes (Shadow DOM traversal already shipped in smart commands)
+- [ ] **Multi-instance pool** (`CDPILOT_POOL_SIZE`) — N independent browser processes with least-loaded dispatch
 - [ ] **Session recording & replay** — record browser sessions and replay them deterministically
 - [ ] **Stealth mode** *(Pro)* — human-like mouse/typing, anti-fingerprint, CAPTCHA solving
 - [ ] **cdpilot Cloud** — hosted browser sessions API, REST + WebSocket MCP endpoint

package/bin/cdpilot.js

@@ -620,7 +620,7 @@ if (cmd === 'status') {
   // Delegate to Python
   const python = findPython();
   if (!python) {
-    console.error('Error: Python 3.8+ required. Install: https://www.python.org/downloads/');
+    console.error('Error: Python 3.10+ required. Install: https://www.python.org/downloads/');
     process.exit(1);
   }
 

package/package.json

@@ -1,6 +1,7 @@
 {
   "name": "cdpilot",
-  "version": "0.9.0",
+  "version": "0.9.1",
+  "mcpName": "io.github.mehmetnadir/cdpilot",
   "description": "Zero-dependency browser automation CLI. Raw CDP, 70+ commands, friction ladder, 3-tier stealth, CAPTCHA + press-and-hold solvers, proxy pools, MCP server. No Puppeteer/Playwright/Selenium.",
   "bin": {
     "cdpilot": "./bin/cdpilot.js",
@@ -48,15 +49,18 @@
     "shadow-dom",
     "perimeterx",
     "press-and-hold",
-    "progressive-resilience",
-    "multi-instance"
+    "progressive-resilience"
   ],
   "author": "Mehmet Nadir",
   "license": "MIT",
+  "homepage": "https://cdpilot.ndr.ist",
   "repository": {
     "type": "git",
     "url": "https://github.com/mehmetnadir/cdpilot"
   },
+  "bugs": {
+    "url": "https://github.com/mehmetnadir/cdpilot/issues"
+  },
   "engines": {
     "node": ">=18.0.0"
   },

package/src/cdpilot.py

@@ -14,7 +14,7 @@ Environment:
   CDPILOT_PROFILE      Isolated browser profile directory
 """
 
-__version__ = "0.9.0"
+__version__ = "0.9.1"
 
 import asyncio
 import atexit
@@ -35,13 +35,16 @@ import secrets
 import glob
 import datetime
 import concurrent.futures
+import tempfile
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 # ─── Project-Based Multi-Instance Configuration ───
 # Each project directory (cwd) gets its own browser instance with
 # a unique CDP port and isolated profile directory. Zero-config.
 
-CDPILOT_HOME = os.path.expanduser("~/.cdpilot")
+# CDPILOT_HOME override lets tests (and advanced users) point the registry
+# at an isolated directory instead of the real ~/.cdpilot.
+CDPILOT_HOME = os.environ.get("CDPILOT_HOME") or os.path.expanduser("~/.cdpilot")
 REGISTRY_FILE = os.path.join(CDPILOT_HOME, "registry.json")
 CDPILOT_PORT_RANGE_START = 9222
 CDPILOT_PORT_RANGE_END = 9322
@@ -65,10 +68,19 @@ def _get_project_id():
 
 
 def _is_port_free(port):
-    """Check if a port is available for binding."""
+    """Check if a port is available for binding.
+
+    On Windows, SO_REUSEADDR lets a socket bind on top of a port that another
+    socket is actively listening on, so a probe using it called every port
+    free (and registry cleanup pruned live entries). Exclusive use fails when
+    anything holds the port. POSIX SO_REUSEADDR only skips TIME_WAIT.
+    """
     with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
         try:
-            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            if os.name == "nt":
+                s.setsockopt(socket.SOL_SOCKET, socket.SO_EXCLUSIVEADDRUSE, 1)
+            else:
+                s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
             s.bind(("127.0.0.1", port))
             return True
         except OSError:
@@ -86,10 +98,26 @@ def _load_registry():
 
 
 def _save_registry(projects):
-    """Write the global project registry."""
-    os.makedirs(os.path.dirname(REGISTRY_FILE), exist_ok=True)
-    with open(REGISTRY_FILE, 'w') as f:
-        json.dump({"version": 1, "projects": projects}, f, indent=2)
+    """Write the global project registry atomically.
+
+    Writes to a temp file in the same directory and os.replace()s it into
+    place, so a concurrent cdpilot invocation reading registry.json always
+    sees either the fully-old or fully-new content — never a torn/partial
+    write (os.replace is atomic on both POSIX and Windows).
+    """
+    reg_dir = os.path.dirname(REGISTRY_FILE)
+    os.makedirs(reg_dir, exist_ok=True)
+    fd, tmp_path = tempfile.mkstemp(prefix='.registry-', suffix='.tmp', dir=reg_dir)
+    try:
+        with os.fdopen(fd, 'w') as f:
+            json.dump({"version": 1, "projects": projects}, f, indent=2)
+        os.replace(tmp_path, REGISTRY_FILE)
+    except BaseException:
+        try:
+            os.remove(tmp_path)
+        except OSError:
+            pass
+        raise
 
 
 def _register_project(project_id, port, profile_dir, pid=None):
@@ -108,25 +136,83 @@ def _register_project(project_id, port, profile_dir, pid=None):
     _save_registry(registry)
 
 
+def _pid_alive(pid):
+    """True if a process with this pid exists. Never signals it.
+
+    Sending signal 0 is a harmless existence probe on POSIX, but on Windows
+    Python maps every signal other than CTRL_C/CTRL_BREAK to TerminateProcess:
+    the "probe" kills the process. Registry pids are stale by nature and may
+    have been reused by an unrelated program, so on Windows ask the kernel
+    through OpenProcess/GetExitCodeProcess instead. Defined this early because
+    it runs at import time (_resolve_project_config -> _allocate_port).
+    """
+    try:
+        pid = int(pid)
+    except (TypeError, ValueError):
+        return False
+    if pid <= 0:
+        return False
+    if os.name == "nt":
+        import ctypes
+        kernel32 = ctypes.windll.kernel32
+        handle = kernel32.OpenProcess(0x1000, False, pid)  # QUERY_LIMITED_INFORMATION
+        if not handle:
+            return False
+        try:
+            code = ctypes.c_ulong()
+            if not kernel32.GetExitCodeProcess(handle, ctypes.byref(code)):
+                return False
+            return code.value == 259  # STILL_ACTIVE
+        finally:
+            kernel32.CloseHandle(handle)
+    try:
+        os.kill(pid, 0)
+    except ProcessLookupError:
+        return False
+    except PermissionError:
+        return True  # exists, just owned by another user
+    except OSError:
+        return False
+    return True
+
+
+def _is_registry_entry_dead(info):
+    """A registry entry is dead only when we can PROVE nothing is using it.
+
+    Dead = nothing is listening on its port AND (if a pid was recorded)
+    that process is no longer alive. If the port is currently bound, we
+    never call it dead even if our own bookkeeping (status/pid) looks
+    stale — it may be another project's live browser instance, and a
+    live entry must never be pruned.
+    """
+    port = info.get("port")
+    if port and not _is_port_free(port):
+        return False
+    pid = info.get("pid")
+    if pid and _pid_alive(pid):
+        return False
+    return True
+
+
 def _cleanup_registry():
-    """Update status for dead processes and return cleaned registry."""
+    """Prune dead entries from the registry and return what's left.
+
+    Called before every port allocation (see _allocate_port) so the
+    registry never accumulates stale entries that starve the
+    9222-9322 port range — the root cause of the "No free port" failure
+    documented in .claude/docs/vaat-denetimi-2026-09-27.md.
+    """
     registry = _load_registry()
-    changed = False
-    for pid_key, info in registry.items():
-        if info.get("status") == "running":
-            port = info.get("port")
-            if port and _is_port_free(port):
-                info["status"] = "stopped"
-                info["pid"] = None
-                changed = True
-    if changed:
-        _save_registry(registry)
-    return registry
+    alive = {pid_key: info for pid_key, info in registry.items()
+             if not _is_registry_entry_dead(info)}
+    if len(alive) != len(registry):
+        _save_registry(alive)
+    return alive
 
 
 def _allocate_port(project_id):
     """Find a free port for the given project."""
-    registry = _load_registry()
+    registry = _cleanup_registry()
 
     # Reuse existing port if still free
     if project_id in registry:
@@ -2528,7 +2614,16 @@ def cmd_launch():
     sys.exit(1)
 
 
-def cmd_tabs():
+def cmd_tabs(reap=False, max_tabs=1, include_real=False, dry_run=False):
+    if reap:
+        result = _reap_tabs(max_tabs=max_tabs, include_real=include_real,
+                            dry_run=dry_run)
+        verb = "would close" if result.get("dry_run") else "closed"
+        print(f"{verb} {len(result['closed'])} tab(s), kept {result['kept']}"
+              f" (blank {result.get('blank', 0)}, real {result.get('real', 0)})")
+        if result.get("reason"):
+            print(f"  {result['reason']}")
+        return
     tabs = get_tabs()
     pages = [t for t in tabs if t.get("type") == "page"]
     for i, p in enumerate(pages):
@@ -4527,6 +4622,87 @@ def _is_chrome_internal_url(url):
             or u.startswith("vivaldi://"))
 
 
+def _close_target(target_id):
+    """Close one CDP target. Returns True when the browser accepted it."""
+    try:
+        urllib.request.urlopen(f"{CDP_BASE}/json/close/{target_id}", timeout=2)
+        return True
+    except Exception:
+        return False
+
+
+def _reap_tabs(max_tabs=1, include_real=False, dry_run=False):
+    """Enforce a tab budget on the connected browser.
+
+    A long-lived browser accumulates abandoned tabs: every client that opens a
+    target and exits without closing it leaks one, and nothing collects them.
+    Measured on srv21 2026-08-30 — 513 pages, 506 of them about:blank, 12.7 GB
+    across 529 renderer processes, growing ~6/hour for three days on a box with
+    47 GB total.
+
+    Blank tabs go first: they hold nothing a caller could still want. Pages with
+    a real URL are never touched unless include_real is set, because on a shared
+    browser they may belong to somebody else. The session's own tab and the last
+    remaining page always survive.
+    """
+    tabs = cdp_get("/json", no_cache=True)
+    if not tabs:
+        return {"closed": [], "kept": 0, "reason": "no CDP connection"}
+    pages = [t for t in tabs if t.get("type") == "page"]
+    if len(pages) <= 1:
+        return {"closed": [], "kept": len(pages), "reason": "nothing to reap"}
+
+    session_id = _get_session_window_target_id()
+    real, blank = [], []
+    for t in pages:
+        (blank if _is_chrome_internal_url(t.get("url")) else real).append(t)
+
+    keep_ids = []
+    if session_id and any(t.get("id") == session_id for t in pages):
+        keep_ids.append(session_id)
+    if not include_real:
+        keep_ids += [t["id"] for t in real if t["id"] not in keep_ids]
+    for t in blank:
+        if len(keep_ids) >= max_tabs:
+            break
+        if t["id"] not in keep_ids:
+            keep_ids.append(t["id"])
+    if not keep_ids:  # never leave the browser with zero pages
+        keep_ids.append(pages[0]["id"])
+
+    doomed = [t for t in pages if t["id"] not in keep_ids]
+    closed = []
+    for t in doomed:
+        if dry_run or _close_target(t["id"]):
+            closed.append(t["id"])
+    if closed and not dry_run:
+        cdp_cache_invalidate()
+        owned = _load_owned_tabs()
+        if owned & set(closed):
+            _save_owned_tabs(owned - set(closed))
+    return {"closed": closed, "kept": len(keep_ids),
+            "blank": len(blank), "real": len(real), "dry_run": dry_run}
+
+
+def _enforce_tab_budget():
+    """Opt-in post-command reaping via CDPILOT_MAX_TABS.
+
+    Off unless the variable is set: silently closing tabs would be hostile on a
+    desktop where the browser is also the user's. On an unattended server it is
+    the difference between one tab and five hundred.
+    """
+    raw = os.environ.get("CDPILOT_MAX_TABS", "").strip()
+    if not raw.isdigit() or int(raw) < 1:
+        return
+    try:
+        result = _reap_tabs(max_tabs=int(raw))
+    except Exception:
+        return
+    if result.get("closed"):
+        print(f"[cdpilot] tab budget: closed {len(result['closed'])} idle tab(s)",
+              file=sys.stderr)
+
+
 async def _browser_close_graceful():
     """Ask the browser to shut down gracefully via the CDP Browser.close
     command (cleanest cross-platform path — flushes state, no orphaned procs).
@@ -7421,9 +7597,13 @@ def _stop_browser_on_port(port):
     """Stop the browser process listening on the given port."""
     import signal
     try:
+        # Bounded: on some hosts/sandboxes `lsof`/`pkill` can hang instead of
+        # erroring (observed while testing the port-registry fix — a stuck
+        # `lsof -ti :<port>` blocked the whole /v1/sessions DELETE request).
+        # A timeout keeps this best-effort call from ever wedging the caller.
         result = subprocess.run(
             ["lsof", "-ti", f":{port}"],
-            capture_output=True, text=True
+            capture_output=True, text=True, timeout=5
         )
         pids = [p.strip() for p in result.stdout.strip().split("\n") if p.strip()]
         if pids:
@@ -7433,7 +7613,7 @@ def _stop_browser_on_port(port):
         else:
             subprocess.run(
                 ["pkill", "-f", f"remote-debugging-port={port}"],
-                capture_output=True, text=True
+                capture_output=True, text=True, timeout=5
             )
             return True
     except Exception:
@@ -7514,7 +7694,11 @@ def _api_release_session(session_id: str) -> bool:
     if not sess:
         return False
     port = sess.get('port')
-    if port:
+    # In TEST_MODE, _api_create_session never actually launched a browser on
+    # `port` (see the sibling check there) — nothing is listening, so skip
+    # the lsof/pkill round trip entirely instead of paying its cost for a
+    # no-op every time.
+    if port and os.environ.get('CDPILOT_API_TEST_MODE') != '1':
         _stop_browser_on_port(port)
     sess['status'] = 'STOPPED'
     del _api_session_store[session_id]
@@ -11999,13 +12183,8 @@ def _watch_clear_state():
 
 
 def _watch_pid_alive(pid):
-    if not pid:
-        return False
-    try:
-        os.kill(int(pid), 0)
-        return True
-    except (OSError, ValueError):
-        return False
+    # os.kill(pid, 0) terminates the process on Windows — see _pid_alive.
+    return _pid_alive(pid)
 
 
 def _watch_parse_timecode(spec):
@@ -12777,7 +12956,12 @@ if __name__ == "__main__":
 
     sync_cmds = {
         'launch': cmd_launch,
-        'tabs': cmd_tabs,
+        'tabs': lambda: cmd_tabs(
+            reap='--reap' in args,
+            max_tabs=next((int(a.split('=')[1]) for a in args
+                           if a.startswith('--max=') and a.split('=')[1].isdigit()), 1),
+            include_real='--all' in args,
+            dry_run='--dry-run' in args),
         'extensions': cmd_extensions,
         'stop': cmd_stop,
         'version': cmd_version,
@@ -12857,6 +13041,7 @@ if __name__ == "__main__":
 
     if cmd in sync_cmds:
         sync_cmds[cmd]()
+        _enforce_tab_budget()
         sys.exit(0)
 
     def require_args(n, usage):
@@ -13001,6 +13186,7 @@ if __name__ == "__main__":
         if cmd in NO_CONTROL_CMDS:
             asyncio.run(async_map[cmd]())
             _update_session_timestamp()
+            _enforce_tab_budget()
         else:
             async def _wrapped():
                 ws_url = None
@@ -13021,6 +13207,7 @@ if __name__ == "__main__":
                                 await _control_end(ws_url)
             asyncio.run(_wrapped())
             _update_session_timestamp()
+            _enforce_tab_budget()
     else:
         print(f"Unknown command: {cmd}", file=sys.stderr)
         all_cmds = sorted(set(list(sync_cmds.keys()) + list(async_map.keys())))