cdpilot 0.5.0 → 0.8.0

package/README.md

@@ -213,10 +213,34 @@ cdpilot throttle slow3g       # Simulate slow 3G
 cdpilot throttle fast3g       # Simulate fast 3G
 cdpilot throttle offline      # Go offline
 cdpilot throttle off          # Back to normal
-cdpilot proxy <url>           # Set proxy
+cdpilot proxy <url>           # Set proxy (legacy single-URL form)
 cdpilot proxy off             # Remove proxy
+
+# v0.7.0: named pools (BrightData, IPRoyal, Anchor, etc.)
+cdpilot proxy add brd "http://USER:PASS@brd.superproxy.io:22225" --geo us
+cdpilot proxy add ipr "http://USER:PASS@geo.iproyal.com:12321" --sticky
+cdpilot proxy use brd         # Activate one pool
+cdpilot proxy list            # Show pools (credentials redacted)
+cdpilot proxy show [<name>]   # Active or named pool URL (redacted)
+cdpilot proxy remove <name>   # Drop a pool
+```
+
+### TLS Fingerprint (v0.8.0)
+
+```bash
+cdpilot tls-check                        # Probe JA3/JA4/H2 via tls.peet.ws
+cdpilot tls-check --service browserleaks # Alternate echo
+cdpilot tls-check --json                 # Raw JSON
 ```
 
+**Known limitation:** v0.8.0 ships the **probe** (`tls-check`) but no in-tree TLS fix.
+There is no Chromium-based TLS-corrected browser that ships as a standalone binary
+exposing `--remote-debugging-port`. Camoufox is Firefox+Juggler (no CDP);
+Patchright / undetected-chromedriver / nodriver are Python/Playwright libraries,
+not standalone browsers. cdpilot's CDP-only architecture is incompatible with all
+of them without a protocol adapter. **Tracking:** v0.9 roadmap (TLS-MITM plugin
+using curl-impersonate semantics, OR BoringSSL-patched Chromium fork).
+
 ### Request Interception
 
 ```bash
@@ -264,10 +288,22 @@ cdpilot session-close [id]    # Close session
 ### Advanced
 
 ```bash
-cdpilot cookies [domain]      # List cookies (filter by domain)
-cdpilot cookies save <file> [<domain>]
-                              # Export cookies as JSON (optional domain filter)
-cdpilot cookies load <file>   # Import cookies (replay CF clearance across runs)
+cdpilot cookies [domain]             # List cookies (filter by domain)
+cdpilot cookies save <file> [<domain>]  # Export cookies as JSON
+cdpilot cookies load <file>          # Import cookies (replay CF clearance)
+cdpilot cookies save --host x.com    # Save to per-host cache
+cdpilot cookies load --host x.com    # Load from per-host cache
+cdpilot cookies list                 # Cached hosts + age + CF clearance flag
+cdpilot cookies clear --host x.com   # Remove one host
+cdpilot cookies clear --all          # Wipe entire cache
+cdpilot cookies clear --older-than 7d  # Remove stale entries
+cdpilot cookies auto on              # Toggle global auto-save/replay flag (v0.6.1: requires safe-list)
+cdpilot cookies auto add <host>      # Opt host into auto save/replay (v0.6.1)
+cdpilot cookies auto remove <host>   # Remove host from safe-list
+cdpilot cookies auto list            # Enable flag + current safe-list
+cdpilot cookies cf-replay <url>      # Inject cached CF clearance before nav
+cdpilot wipe [--cookies|--storage|--tabs|--keep h1,h2]
+                                     # v0.6.2: per-task state hygiene (cross-task contamination)
 cdpilot storage               # localStorage contents
 cdpilot upload <sel> <file>   # Upload file to input
 cdpilot multi-eval <js>       # Execute JS in all tabs
@@ -338,6 +374,41 @@ cdpilot adaptive clear        # Drop the stealth host memory entirely
 > when it sees one — adds the host to a persistent list, retries once with
 > stealth on. Never auto-demotes. Conservative by design.
 
+### Captcha Solver Plugins (v0.6+)
+
+Optional integration with 2captcha, anti-captcha, and capmonster. Per-solve
+cost ~$0.001–0.003. API keys stored in `~/.cdpilot/captcha-providers.json`
+(chmod 600) — **never committed to git**.
+
+```bash
+# One-time setup
+cdpilot captcha config --provider 2captcha --api-key YOUR_KEY
+cdpilot captcha config --provider anticaptcha --api-key YOUR_KEY  # fallback
+
+# Enable auto-solve (adaptive layer auto-solves on detect)
+cdpilot captcha auto on
+
+# Manual solve (debug / scripting)
+cdpilot captcha solve --type recaptcha-v2 --site-key SK --url https://example.com
+# returns: {"token": "03AGdBq2...", "duration_ms": 12500, "cost": 0.003, "provider": "2captcha"}
+
+cdpilot captcha solve --type hcaptcha --site-key SK --url URL
+cdpilot captcha solve --type turnstile --site-key SK --url URL
+cdpilot captcha solve --type funcaptcha --site-key SK --url URL
+
+# Status & balance
+cdpilot captcha status   # {"configured": [...], "preferred": "2captcha", "auto_enabled": true}
+cdpilot captcha balance  # {"2captcha": 1.23, "anticaptcha": 0.50}
+```
+
+Supported types: `recaptcha-v2`, `recaptcha-v3`, `hcaptcha`, `turnstile`, `funcaptcha`
+
+Tokens are injected via `Runtime.evaluate` CDP — no browser-side libraries required.
+When `captcha auto on` is set, the adaptive layer detects and solves automatically
+after each navigation. Without auto-on, detection still works but solving is manual.
+
+> **Expected bench improvement (v0.6):** reCaptcha 2/6 → 5+/6, hCaptcha 2/3 → 3/3
+
 Verified against public bot-detection panels:
 - **bot.sannysoft.com:** 24/24 PASS (WebDriver, Chrome obj, Plugins as PluginArray, WebGL, PHANTOM_*, HEADCHR_*, SELENIUM_DRIVER)
 - **bot.incolumitas.com** intoli: 6/6 OK — new-tests: 6/7 OK (one FAIL = pure CDP presence, cannot be JS-patched)
@@ -422,6 +493,54 @@ print(result.stdout)
 
 **No Puppeteer. No Playwright. No Selenium.** Just direct CDP communication.
 
+## Stealth Bench V1
+
+`cdpilot` is benchmarked against a suite of 80 high-friction web tasks to measure success rates against modern anti-bot systems. Gemini 2.5 Flash drives `cdpilot` as the controller; success is defined as full task completion without interception.
+
+### v0.5.3 Results (current)
+
+| Category | Success / Total | Rate |
+| :--- | :--- | :--- |
+| Custom Antibot | 5 / 5 | 100.0% |
+| Temu Slider | 1 / 1 | 100.0% |
+| hCaptcha | 2 / 3 | 67.0% |
+| Cloudflare | 12 / 22 | 55.0% |
+| DataDome | 5 / 13 | 38.0% |
+| reCaptcha | 2 / 6 | 33.0% |
+| Akamai | 1 / 6 | 17.0% |
+| PerimeterX | 2 / 18 | 11.0% |
+| GeeTest | 0 / 4 | 0.0% |
+| Shape | 0 / 1 | 0.0% |
+| Kasada | 0 / 1 | 0.0% |
+| **Total** | **30 / 80** | **37.5%** |
+
+### Version history
+
+| Version | Mode | Total | Rate |
+| :--- | :--- | :--- | :--- |
+| v0.5.0 | Baseline (stealth off / adaptive off) | 30 / 80 | 37.5% |
+| v0.5.0 | Stealth only (stealth on / adaptive off) | 32 / 80 | 40.0% |
+| v0.5.0 | Full (stealth on / adaptive on) | 26 / 80 | 32.5% |
+| v0.5.1 | Full — regression fix | 29 / 80 | 36.25% |
+| v0.5.2 | Full — entropy auto-hook | 28 / 80 | 35.0% |
+| v0.5.3 | Full — entropy scope tightened | 30 / 80 | 37.5% |
+| v0.6.0 | + captcha solver + cookies-auto (regression) | 15 / 80 | 18.75% |
+| **v0.8.0** | **Full — cookies safe-host scoped + per-task wipe (no proxy, no TLS fork)** | **29 / 80** | **36.25%** |
+| v0.7.0 (slot) | + named proxy pools | _depends on user proxy_ | — |
+| v0.8.0 (slot) | + TLS-aware launcher (camoufox/undetected-chrome) | _depends on browser choice_ | — |
+
+**What cdpilot does not do:** `cdpilot` is an avoidance engine, not a CAPTCHA solver. We prioritize structural stealth (JS fingerprinting, behavioral entropy) to prevent challenges from appearing. When a CAPTCHA blocks progress and cannot be bypassed, the task fails — that is the honest definition of our success rate. PerimeterX (2/18), GeeTest (0/4) and Akamai (1/6) are known weaknesses; v0.7+ (residential proxy) and v0.8+ (TLS fingerprint correction via camoufox) target these directly.
+
+## Recommended Configuration
+
+Based on Stealth Bench V1 results:
+
+- **Speed-first** (most use cases): `cdpilot launch` — default browser behavior, no patches
+- **Stealth-on** (RECOMMENDED, best overall): `cdpilot launch && cdpilot stealth on`
+- **Full adaptive** (specific captcha-heavy workflows): `cdpilot launch && cdpilot stealth on && cdpilot adaptive on`
+
+The full adaptive layer is bench-neutral vs baseline (30/80 vs 30/80) for Stealth Bench V1's task mix. Stealth-only (32/80 = 40%) is still the best-performing single variant. For your specific workload, profile both and pick.
+
 ## Comparison
 
 | Feature | cdpilot | Puppeteer | Playwright | Selenium |
@@ -453,6 +572,29 @@ Future paid offerings:
 
 Found a vulnerability? Please email the maintainer directly instead of opening a public issue.
 
+## Cookie Persistence (v0.6+)
+
+Per-host cookie cache with auto-replay before navigation. Particularly useful for
+sites with expensive challenges (Cloudflare, DataDome) — once passed, clearance
+cookies (`cf_clearance`, `__cf_bm`) are cached and replayed on next visit.
+
+```bash
+# Enable auto-mode — cookies saved/replayed on every navigate
+cdpilot cookies auto on
+
+# Manual per-host workflow
+cdpilot cookies save --host x.com   # cache current session cookies
+cdpilot cookies load --host x.com   # inject before navigating
+cdpilot cookies cf-replay https://x.com  # explicit CF clearance injection
+
+# Inspect and clean
+cdpilot cookies list                     # all cached hosts + age + CF flag
+cdpilot cookies clear --older-than 7d   # prune stale cache
+```
+
+Storage: `~/.cdpilot/cookies/<host>/cookies.json` (chmod 600, never committed to git).
+Expired cookies are filtered automatically on load.
+
 ## Roadmap
 
 The only browser MCP with built-in test assertions. Here's what we've shipped and what's next:

package/bin/cdpilot.js

@@ -374,8 +374,157 @@ function showHelp() {
 `);
 }
 
+// ── Internal Test Runner ──
+
+function runInternalTestRunner(testFile, traceDir, traceMode, grepPattern) {
+  if (traceDir) {
+    fs.mkdirSync(path.join(traceDir, 'screenshots'), { recursive: true });
+    fs.mkdirSync(path.join(traceDir, 'a11y'), { recursive: true });
+  }
+
+  const metaPath = traceDir ? path.join(traceDir, 'meta.json') : null;
+  const stepsPath = traceDir && traceMode !== 'off' ? path.join(traceDir, 'steps.jsonl') : null;
+
+  const meta = { name: path.basename(testFile), started_at: new Date().toISOString(), status: 'running', tests: [] };
+  if (metaPath) fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2));
+
+  const testQueue = [];
+  global.test = (name, fn) => testQueue.push({ name, fn });
+
+  try {
+    require(path.resolve(testFile));
+  } catch (err) {
+    const out = { passed: 0, failed: 1, skipped: 0, tests: [{ name: testFile, status: 'failed', duration_ms: 0, error: err.message }] };
+    process.stdout.write(JSON.stringify(out) + '\n');
+    process.exit(1);
+  }
+
+  const results = { passed: 0, failed: 0, skipped: 0, tests: [] };
+  const cdpPort = process.env.CDP_PORT || '9222';
+  const SCRIPT = path.join(__dirname, '..', 'src', 'cdpilot.py');
+  const python = 'python3';
+  let stepIdx = 0;
+
+  const makeT = () => {
+    const runCmd = (cmd, ...cargs) => {
+      const padded = String(stepIdx).padStart(3, '0');
+      const step = { action: cmd + ' ' + cargs.join(' '), ts_ms: Date.now(), duration_ms: 0, error: null };
+      const t0 = Date.now();
+      try {
+        const quoted = cargs.map(a => JSON.stringify(String(a))).join(' ');
+        execSync(`${python} ${SCRIPT} ${cmd} ${quoted}`, {
+          stdio: 'pipe',
+          env: { ...process.env, CDP_PORT: cdpPort },
+          timeout: 30000,
+        });
+        step.duration_ms = Date.now() - t0;
+        if (stepsPath) fs.appendFileSync(stepsPath, JSON.stringify(step) + '\n');
+        // Screenshot after each step (best-effort — no browser = skipped)
+        if (traceDir && traceMode !== 'off') {
+          try {
+            const shotPath = path.join(traceDir, 'screenshots', `step-${padded}.png`);
+            execSync(`${python} ${SCRIPT} shot ${shotPath}`, { stdio: 'pipe', env: { ...process.env, CDP_PORT: cdpPort }, timeout: 10000 });
+          } catch (_) { /* no browser is OK in unit-style tests */ }
+        }
+        stepIdx++;
+      } catch (err) {
+        step.duration_ms = Date.now() - t0;
+        step.error = err.stderr ? err.stderr.toString().trim() : err.message;
+        if (stepsPath) fs.appendFileSync(stepsPath, JSON.stringify(step) + '\n');
+        stepIdx++;
+        throw new Error(`${cmd} failed: ${step.error}`);
+      }
+    };
+
+    const t = {
+      goto: (url) => runCmd('go', url),
+      click: (sel) => runCmd('click', sel),
+      fill: (sel, val) => runCmd('fill', sel, val),
+      type: (sel, val) => runCmd('type', sel, val),
+      hover: (sel) => runCmd('hover', sel),
+      screenshot: (p) => runCmd('shot', p),
+      eval: (js) => {
+        try {
+          const out = execSync(`${python} ${SCRIPT} eval ${JSON.stringify(js)}`, { stdio: 'pipe', env: { ...process.env, CDP_PORT: cdpPort }, timeout: 10000 });
+          return out.toString().trim();
+        } catch (e) { throw new Error('eval failed: ' + e.message); }
+      },
+      a11y: () => runCmd('a11y-snapshot'),
+    };
+
+    t.expect = (textOrSel) => runCmd('assert', textOrSel);
+    t.expect.url = (expected) => runCmd('assert-url', expected);
+    t.expect.visible = (sel) => runCmd('assert-visible', sel);
+    t.expect.hidden = (sel) => runCmd('assert-hidden', sel);
+
+    return t;
+  };
+
+  // Run tests sequentially (parallel is managed at the Python level across files)
+  const runAll = async () => {
+    for (const tst of testQueue) {
+      if (grepPattern && !tst.name.match(new RegExp(grepPattern, 'i'))) {
+        results.skipped++;
+        continue;
+      }
+      const t0 = Date.now();
+      const rec = { name: tst.name, status: 'passed', duration_ms: 0, error: null };
+      try {
+        await tst.fn(makeT());
+        rec.status = 'passed';
+        results.passed++;
+      } catch (err) {
+        rec.status = 'failed';
+        rec.error = err.message;
+        results.failed++;
+      }
+      rec.duration_ms = Date.now() - t0;
+      results.tests.push(rec);
+      if (metaPath) {
+        meta.tests.push(rec);
+        meta.status = 'running';
+        fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2));
+      }
+    }
+
+    // Finalize meta
+    if (metaPath) {
+      meta.status = results.failed > 0 ? 'failed' : 'passed';
+      meta.passed = results.passed;
+      meta.failed = results.failed;
+      meta.skipped = results.skipped;
+      meta.ended_at = new Date().toISOString();
+      fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2));
+    }
+
+    process.stdout.write(JSON.stringify(results) + '\n');
+    process.exit(results.failed > 0 ? 1 : 0);
+  };
+
+  runAll().catch(err => {
+    process.stderr.write('Test runner error: ' + err.message + '\n');
+    process.exit(1);
+  });
+}
+
 // ── Main ──
 
+// Internal test runner mode — intercept before normal CLI dispatch
+if (process.argv.includes('--internal-test-runner')) {
+  const idx = process.argv.indexOf('--internal-test-runner');
+  const testFile = process.argv[idx + 1];
+  const traceDirArg = process.argv.find(a => a.startsWith('--trace-dir='));
+  const traceArg = process.argv.find(a => a.startsWith('--trace='));
+  const grepArg = process.argv.find(a => a.startsWith('--grep='));
+  runInternalTestRunner(
+    testFile,
+    traceDirArg ? traceDirArg.split('=').slice(1).join('=') : null,
+    traceArg ? traceArg.split('=')[1] : 'default',
+    grepArg ? grepArg.split('=').slice(1).join('=') : null,
+  );
+  return; // runAll() is async, this exits via process.exit
+}
+
 const args = process.argv.slice(2);
 const cmd = args[0];
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "cdpilot",
-  "version": "0.5.0",
-  "description": "Zero-dependency browser automation from your terminal. One command, full control.",
+  "version": "0.8.0",
+  "description": "Zero-dependency browser automation CLI. Raw CDP, 60+ commands, stealth + adaptive escalation, CAPTCHA solvers, proxy pools, TLS probe, MCP server. No Puppeteer/Playwright/Selenium.",
   "bin": {
     "cdpilot": "./bin/cdpilot.js",
     "bctl": "./bin/cdpilot.js"
@@ -11,12 +11,38 @@
     "start": "node bin/cdpilot.js"
   },
   "keywords": [
-    "browser-automation", "cdp", "chrome-devtools-protocol", "cli",
-    "mcp", "ai-agent", "web-scraping", "zero-dependency",
-    "headless-browser", "puppeteer-alternative", "playwright-alternative",
-    "screenshot", "pdf", "automation", "devtools", "browser",
-    "model-context-protocol", "claude", "brave", "testing",
-    "web-testing", "accessibility", "a11y"
+    "browser-automation",
+    "cdp",
+    "chrome-devtools-protocol",
+    "cli",
+    "mcp",
+    "ai-agent",
+    "web-scraping",
+    "zero-dependency",
+    "headless-browser",
+    "puppeteer-alternative",
+    "playwright-alternative",
+    "screenshot",
+    "pdf",
+    "automation",
+    "devtools",
+    "browser",
+    "model-context-protocol",
+    "claude",
+    "brave",
+    "testing",
+    "web-testing",
+    "accessibility",
+    "a11y",
+    "stealth",
+    "captcha-solver",
+    "anti-bot",
+    "cloudflare-bypass",
+    "datadome",
+    "adaptive-escalation",
+    "proxy-rotation",
+    "tls-fingerprint",
+    "cookie-management"
   ],
   "author": "Mehmet Nadir",
   "license": "MIT",
@@ -32,5 +58,10 @@
     "src/cdpilot.py",
     "README.md",
     "LICENSE"
-  ]
+  ],
+  "devDependencies": {
+    "gray-matter": "^4.0.3",
+    "remark": "^15.0.1",
+    "remark-html": "^16.0.1"
+  }
 }