@apmantza/greedysearch-pi 1.8.6 → 1.8.7

package/CHANGELOG.md

@@ -1,5 +1,69 @@
 # Changelog
 
+## [Unreleased]
+
+### Fixed
+
+- **SonarCloud minor vulnerability false positives** — Confirmed both remaining issues are false positives (internal diagnostic logging in `bin/gschrome.mjs` and test debug output in `test/fetcher-cli.mjs`). Verified via full smoke test suite: all 33 unit tests pass, all 4 engines (Perplexity, Bing, Google, Gemini) return results at all depths (fast/standard/deep), CDP safety wrappers correctly enforce mode boundaries.
+
+- **SonarCloud security hotspots** (re-verified) — All previously fixed hotspots remain resolved: replaced `spawn("node", ...)` with `spawn(process.execPath, ...)`, replaced `Math.random()` with `crypto.randomInt()`, 19 remaining hotspots confirmed as false positives (hardcoded `execSync` commands, simple regex patterns).
+
+### Fixed
+
+- **Headless→visible mode switching** (`src/search/chrome.mjs`) — `ensureChrome()` only handled the case where visible was requested but headless Chrome was running. When headless was requested (the default) but visible Chrome was running, it silently kept visible mode — causing env var mismatches that broke extractors like Perplexity. Now properly detects both directions and kills/relaunches in the correct mode.
+
+- **SonarCloud security hotspots** — Replaced `spawn("node", ...)` with `spawn(process.execPath, ...)` in cdp wrapper, `runExtractor`, `synthesizeWithGemini`, and test helper to prevent PATH-based binary substitution. Replaced `Math.random()` with `crypto.randomInt()` in `jitter()` for non-security-sensitive timing variance. Remaining 19 hotspots are verified false positives (hardcoded `execSync` commands, simple regex patterns).
+- **Bing stealth not active on page load** (`src/search/chrome.mjs`) — `injectHeadlessStealth` was fire-and-forget (`.catch(() => {})`). The CDP `Page.addScriptToEvaluateOnNewDocument` command is async — extractors often navigated to Copilot before stealth registered. Cloudflare saw headless fingerprints and blocked the page. Fixed by awaiting stealth for Bing tabs. Perplexity/Google kept fire-and-forget since Perplexity's anti-bot detects the awaited patches.
+- **Bing copy button handler not hydrated** (`extractors/bing-copilot.mjs`) — Copilot's React copy button exists in the DOM before its click handler is bound. `clickCopyAndPollClipboard` clicked too early → clipboard interceptor empty → 13s wasted polling + DOM fallback. Added 800ms hydration delay after `waitForCopyButton`. Solo Bing went from 37-73s → 16s.
+- **Manual verification blocked synthesis** (`bin/search.mjs`) — When Bing/Perplexity needed manual verification after visible recovery, `search.mjs` returned early with `synthesize: false`, discarding all engine results. Now synthesis continues with whichever engines succeeded. Visible Chrome stays open for the user.
+- **Source-fetch crash after visible→headless recovery** (`src/search/fetch-source.mjs`) — After recovery killed/restarted Chrome, stale CDP tab references in parallel source-fetch workers caused "No target matching prefix" crashes. Workers now catch `fetchSourceContent` errors; `fetchSourceContentBrowser` returns error objects instead of throwing.
+- **Progress tracker "🔄 synthesizing" hang** (`src/tools/shared.ts`) — When synthesis was skipped (manual verification), the progress tracker showed "🔄 synthesizing" forever because no `PROGRESS:synthesis:done` was ever emitted. Now handles `done`/`error`/`skipped` synthesis states.
+- **Gemini synthesis eval timeout** (`bin/cdp.mjs`) — CDP daemon `TIMEOUT` was 30s, but `waitForStreamComplete` uses a single `Runtime.evaluate` call that can run 60-90s for long synthesis prompts. Increased to 90s.
+
+### Performance
+
+- **Reduced timeouts across all extractors** — Navigation: 35s→20s, verification retry: 30s→10s (Bing/Perplexity), 60s→10s (Gemini/Google), post-nav settle: 1200ms→600ms (Bing), 1200ms→600ms (Gemini). Turnstile never clears in headless, so 30s of retry loops were pure waste.
+- **Hard per-engine timeouts raised** (`bin/search.mjs`) — Fast: 22s→30s, Standard/Deep: 35s→55s. CDP contention from 3 parallel extractors adds overhead that the old budgets didn't account for.
+- **Tab creation split: Bing gets blank+stealth, others pre-seeded** (`src/search/chrome.mjs`) — `Target.createTarget` navigation is less detectable than CDP `Page.navigate` for Perplexity/Google. Bing needs blank tab + awaited stealth to hide headless fingerprints from Copilot's Cloudflare.
+
+### Performance
+
+- **Hard per-engine timeouts** (`bin/search.mjs`) — Fast mode: 22s per engine. Standard/deep: 35s per engine. Slow engines are skipped instead of stalling the whole batch. Previously a single slow engine could push `all` searches to 60–90s.
+- **Parallel tab creation** (`bin/search.mjs`, `src/search/chrome.mjs`) — All engine tabs open simultaneously instead of sequential 300ms staggered delays. Tabs are pre-seeded to each engine's homepage so extractors skip redundant initial navigation.
+- **Reduced settle delays** (`extractors/common.mjs`) — `postNav` 1500→800ms, `postNavSlow` 2000→1200ms, `postClick`/`postType` 400→300ms, `afterVerify` 3000→1500ms. Safe because tabs now load the target domain before the extractor even starts.
+- **Higher source-fetch concurrency** (`src/search/constants.mjs`) — Default `GREEDY_FETCH_CONCURRENCY` raised from 2 → 4.
+- **Faster HTTP timeouts** (`src/search/fetch-source.mjs`) — HTTP fetch timeout 15s → 10s, browser fallback settle 1500ms → 800ms.
+- **Non-blocking cleanup** (`bin/search.mjs`) — Removed the 1500ms hard sleep at process exit; `minimizeChrome` now fire-and-forget.
+- **Domain-aware navigation skip** (`extractors/bing-copilot.mjs`, `extractors/perplexity.mjs`, `extractors/google-ai.mjs`) — When a tab is already on the engine's domain (pre-seeded by orchestrator), skip the redundant `cdp nav` call and settle delay.
+- **Fast mode keeps short engine budgets** (`bin/search.mjs`) — Fast mode still uses 22s per-engine extraction timeouts and skips source fetch/synthesis work. Verification recovery can now run in fast mode when Bing/Perplexity are blocked, because returning no result is worse than the retry cost.
+
+### Anti-Bot Detection Hardening (Anti-CDP Evasion)
+
+- **Runtime.enable evasion** (`bin/cdp.mjs`) — The primary CDP detection vector (Cloudflare/DataDome watch for `Runtime.consoleAPICalled` timing) has been eliminated. All `Runtime.evaluate` calls now use an explicit `contextId` captured via brief `Runtime.enable` → `Runtime.disable` at daemon startup (~100ms window). No persistent Runtime domain enable for the session. See: rebrowser.net / DataDome research.
+- **Stale PID / ghost Chrome cleanup** (`src/search/chrome.mjs`) — `killChrome()` now uses port-based process detection via `netstat`/`lsof` instead of relying solely on the PID file. Handles ghost processes that hold port 9222 after the tracked PID dies. Old `killHeadlessChrome` kept as backward-compat alias.
+- **Idle cleanup for both modes** (`src/search/chrome.mjs`) — `checkAndKillIdle()` no longer gates on `GREEDY_SEARCH_HEADLESS=1`. Both headless and visible Chrome auto-kill after idle timeout. Disable with `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES=0`.
+- **`--disable-blink-features=AutomationControlled` for visible mode** (`bin/launch.mjs`, `bin/gschrome.mjs`) — Previously headless-only. The flag and `--window-size` now apply to both modes, suppressing `navigator.webdriver` in visible Chrome too.
+- **Stealth injection for visible mode** (`src/search/chrome.mjs`, `extractors/common.mjs`) — Canvas noise, plugin spoofing, `window.chrome.runtime`, and console safening now inject on both headless and visible tabs.
+- **Client Hints consistency** (`src/fetcher.mjs`) — Added `Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform` headers to `DEFAULT_HEADERS`, matching the Chrome 122 user-agent. Inconsistency between UA and Client Hints is a strong bot signal.
+- **Perplexity Cloudflare verification** (`extractors/perplexity.mjs`) — Added `handleVerification` call after navigation. Perplexity was the only engine missing Cloudflare challenge handling (Bing, Gemini, Google AI already had it).
+- **Chrome TLS fetch fallback** (`src/search/fetch-source.mjs`) — New `fetchSourceViaChrome()` uses `Network.loadNetworkResource` (Chrome 124+) to fetch with authentic Chrome TLS/JA3+HTTP/2 fingerprints when Node.js HTTP fails. Zero navigation overhead.
+
+### Added
+
+- **`visible` / `alwaysVisible` search options** (`src/tools/greedy-search-handler.ts`, `src/tools/shared.ts`, `bin/search.mjs`) — Agents can now force visible Chrome per call with `visible: true`, `alwaysVisible: true`, or `headless: false`. CLI aliases: `--visible`, `--always-visible`. Global env: `GREEDY_SEARCH_ALWAYS_VISIBLE=1`.
+- **GreedySearch Chrome commands for Pi** (`index.ts`) — Added `/greedy-visible`, `/greedy-status`, and `/greedy-kill` so users do not need to know package install paths to manage the dedicated Chrome instance.
+- **Safe CDP wrappers** (`bin/cdp-greedy.mjs`, `bin/cdp-visible.mjs`, `bin/cdp-headless.mjs`) — Agents can inspect only the dedicated GreedySearch Chrome profile. The wrappers always set `CDP_PROFILE_DIR` and mode-specific wrappers refuse to attach to the wrong mode, preventing accidental main-Chrome pollution.
+- **`bin/kill-visible.mjs`** — Strong visible/port cleanup helper backed by `launch-visible.mjs`'s PID + port nuke path.
+- **`bin/gschrome.mjs`** — Standalone Chrome lifecycle manager: `launch-headless`, `launch-visible`, `kill`, `status`. Port-based PID detection, forces mode switches, writes `DevToolsActivePort` for CDP.
+
+### Fixed
+
+- **Single-engine visible recovery** (`bin/search.mjs`) — `engine: "bing"` and `engine: "perplexity"` now perform the same headless → visible retry as `engine: "all"` when blocked by Cloudflare, captcha, timeout, missing input, or clipboard failures.
+- **Bing visible clipboard race** (`extractors/bing-copilot.mjs`) — Waits for the assistant copy button, polls clipboard interception after click, retries copy/poll, then falls back to visible DOM text. Fixes cases where Copilot visibly answered but the extractor returned `Clipboard interceptor returned empty text`.
+- **Manual verification flow** (`bin/search.mjs`, `src/formatters/results.ts`) — If visible retry reaches a human verification challenge, GreedySearch leaves visible Chrome open and returns a clear “solve verification, then rerun” result instead of killing the browser and returning no results.
+- **Visible/headless process cleanup** (`bin/launch.mjs`, `bin/visible.mjs`) — Fixed Windows `taskkill` arguments, added port fallback cleanup for `--kill`, and made `visible.mjs --kill` delegate to the stronger `launch-visible.mjs` cleanup path.
+- **README install paths and skill guidance** (`README.md`, `skills/greedy-search/skill.md`) — Corrected Pi git/npm package paths, documented visible mode and safe CDP wrappers, and removed stale `coding_task` guidance from the agent skill.
+
 ## [1.8.6] — 2026-05-04
 
 ### Bing Copilot: Headless Cloudflare Recovery

package/README.md

@@ -37,8 +37,8 @@ greedy_search({
   engine: "all",
   depth: "deep",
 });
-// Headless is the default — no window. To see the browser:
-// Set GREEDY_SEARCH_VISIBLE=1 before launching Pi
+// Headless is the default — no window. To force visible Chrome:
+greedy_search({ query: "Bing captcha setup", engine: "bing", visible: true });
 ```
 
 ## Parameters (`greedy_search`)
@@ -48,15 +48,17 @@ greedy_search({
 - `depth`: `standard` (default), `fast`, `deep`
 - `fullAnswer`: return full single-engine output instead of preview
 - `headless`: set to `false` to show Chrome window (default: `true`)
+- `visible` / `alwaysVisible`: set to `true` to always use visible Chrome for this search
 
 ## Environment variables
 
-| Variable                             | Default       | Description                                               |
-| ------------------------------------ | ------------- | --------------------------------------------------------- |
-| `GREEDY_SEARCH_VISIBLE`              | (unset)       | Set to `1` to show Chrome window instead of headless      |
-| `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES` | `5`           | Minutes of inactivity before auto-killing headless Chrome |
-| `GREEDY_SEARCH_LOCALE`               | `en`          | Default result language (en, de, fr, es, ja, etc.)        |
-| `CHROME_PATH`                        | auto-detected | Path to Chrome/Chromium executable                        |
+| Variable                             | Default       | Description                                                   |
+| ------------------------------------ | ------------- | ------------------------------------------------------------- |
+| `GREEDY_SEARCH_VISIBLE`              | (unset)       | Set to `1` to show Chrome window instead of headless          |
+| `GREEDY_SEARCH_ALWAYS_VISIBLE`       | (unset)       | Set to `1` to force visible Chrome for all GreedySearch runs  |
+| `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES` | `5`           | Minutes of inactivity before auto-killing GreedySearch Chrome |
+| `GREEDY_SEARCH_LOCALE`               | `en`          | Default result language (en, de, fr, es, ja, etc.)            |
+| `CHROME_PATH`                        | auto-detected | Path to Chrome/Chromium executable                            |
 
 ## Depth modes
 
@@ -66,23 +68,48 @@ greedy_search({
 
 ## Runtime commands
 
-````bash
-# Headless (default, no GUI)
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch.mjs
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch.mjs --status
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch.mjs --kill
+Inside Pi, prefer the extension commands (no package path needed):
 
-# Visible (show browser window — useful for one-time Cloudflare clearance)
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch-visible.mjs
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch-visible.mjs --kill
+```text
+/greedy-visible  # launch visible Chrome for captcha/login/cookie setup
+/greedy-status   # show GreedySearch Chrome status
+/greedy-kill     # stop GreedySearch Chrome
+```
+
+Git install path:
+
+```bash
+GS=~/.pi/agent/git/github.com/apmantza/GreedySearch-pi
+node "$GS/bin/launch.mjs" --status
+node "$GS/bin/visible.mjs"          # visible mode
+node "$GS/bin/visible.mjs" --kill   # strong visible/port cleanup
+node "$GS/bin/kill-visible.mjs"     # same as visible.mjs --kill
+node "$GS/bin/cdp-visible.mjs" list # safe CDP: GreedySearch visible Chrome only
+node "$GS/bin/cdp-headless.mjs" list # safe CDP: GreedySearch headless Chrome only
+node "$GS/bin/cdp-greedy.mjs" list  # safe CDP: any GreedySearch Chrome mode
+```
+
+npm global install path:
 
-# Chrome auto-cleaned after 5 min idle (prevents OOM)
-# Override: GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES=10
+```bash
+GS="$(npm root -g)/@apmantza/greedysearch-pi"
+node "$GS/bin/launch.mjs" --status
+node "$GS/bin/visible.mjs"
+node "$GS/bin/visible.mjs" --kill
+node "$GS/bin/kill-visible.mjs"
+node "$GS/bin/cdp-visible.mjs" list
+node "$GS/bin/cdp-headless.mjs" list
+node "$GS/bin/cdp-greedy.mjs" list
+```
+
+Chrome is auto-cleaned after 5 min idle. Override with `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES=10` or disable with `0`.
+
+**CDP safety:** use `cdp-visible.mjs`, `cdp-headless.mjs`, or `cdp-greedy.mjs` for debugging. They always set `CDP_PROFILE_DIR` to the dedicated GreedySearch Chrome profile and never fall back to your main Chrome session. Avoid calling raw `bin/cdp.mjs` manually unless you explicitly set `CDP_PROFILE_DIR`.
 
 ## Requirements
 
 - Chrome
-- Node.js 20.11.0+ (22+ recommended)
+- Node.js 22+
 
 ## Known engine quirks
 
@@ -90,13 +117,9 @@ node ~/.pi/agent/git/GreedySearch-pi/bin/launch-visible.mjs --kill
 
 Bing Copilot detects headless Chrome and sandboxes all AI responses inside nested iframes (`copilot.microsoft.com` → `copilot.fun` → `blob:`). In this mode the copy button is hidden and the Cloudflare Turnstile challenge blocks content delivery. The clipboard-based extraction cannot work.
 
-**Auto-recovery:** When Bing fails with any extraction error (clipboard, verification, Cloudflare), GreedySearch automatically switches to **visible Chrome**, retries the search, and caches Cloudflare clearance cookies in the Chrome profile. You may need to solve the Cloudflare challenge **once** manually when the visible Chrome window appears. After that, all subsequent headless searches bypass the challenge — the cookies persist in the profile.
-
-If you prefer to skip the auto-recovery delay, launch visible Chrome ahead of time:
+**Auto-recovery:** When Bing or Perplexity fails with a headless-only extraction error (clipboard, verification, timeout, Cloudflare), GreedySearch automatically switches to **visible Chrome** and retries, even in `fast` mode. If manual verification is required, the visible browser is left open and the tool returns instructions to solve the challenge and rerun the same search.
 
-```bash
-node ~/.pi/agent/git/GreedySearch-pi/bin/launch-visible.mjs
-````
+If you prefer to skip the auto-recovery delay, launch visible Chrome ahead of time with `/greedy-visible`, set `GREEDY_SEARCH_ALWAYS_VISIBLE=1`, or pass `visible: true` to `greedy_search`.
 
 ## Anti-detection
 
@@ -118,7 +141,7 @@ When using `depth: "standard"` or `depth: "deep"`, source content is fetched and
 
 ## Project layout
 
-- `bin/` — runtime CLIs (`search.mjs`, `launch.mjs`, `launch-visible.mjs`, `visible.mjs`, `cdp.mjs`)
+- `bin/` — runtime CLIs (`search.mjs`, `launch.mjs`, `launch-visible.mjs`, `visible.mjs`, `kill-visible.mjs`, safe CDP wrappers, `cdp.mjs`)
 - `extractors/` — engine-specific automation + stealth/consent handling
 - `src/` — search pipeline, chrome management, source fetching, formatting
 - `skills/` — Pi skill metadata

package/bin/cdp-greedy.mjs

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+// cdp-greedy.mjs — safe CDP wrapper for the dedicated GreedySearch Chrome.
+//
+// This ALWAYS sets CDP_PROFILE_DIR to the GreedySearch profile so it never
+// falls back to the user's main Chrome DevToolsActivePort.
+//
+// Usage:
+//   node bin/cdp-greedy.mjs list
+//   node bin/cdp-greedy.mjs --mode visible list
+//   node bin/cdp-greedy.mjs --mode headless snap <tab>
+
+import { spawn } from "node:child_process";
+import { existsSync, readFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+
+const tmp = tmpdir().replaceAll("\\", "/");
+const PROFILE_DIR = `${tmp}/greedysearch-chrome-profile`;
+const ACTIVE_PORT = `${PROFILE_DIR}/DevToolsActivePort`;
+const MODE_FILE = `${tmp}/greedysearch-chrome-mode`;
+
+const args = process.argv.slice(2);
+let desiredMode = null;
+const modeIdx = args.indexOf("--mode");
+if (modeIdx !== -1) {
+	desiredMode = args[modeIdx + 1] || null;
+	args.splice(modeIdx, 2);
+}
+
+if (desiredMode && !["visible", "headless"].includes(desiredMode)) {
+	console.error(`Invalid --mode ${desiredMode}. Use visible or headless.`);
+	process.exit(2);
+}
+
+if (!existsSync(ACTIVE_PORT)) {
+	console.error(
+		`GreedySearch Chrome is not running (missing ${ACTIVE_PORT}). Launch with bin/visible.mjs or bin/launch.mjs.`,
+	);
+	process.exit(1);
+}
+
+if (desiredMode) {
+	const actualMode = existsSync(MODE_FILE)
+		? readFileSync(MODE_FILE, "utf8").trim()
+		: "unknown";
+	if (actualMode !== desiredMode) {
+		console.error(
+			`GreedySearch Chrome is ${actualMode}, not ${desiredMode}. Refusing to attach.`,
+		);
+		process.exit(1);
+	}
+}
+
+const cdpBin = new URL("./cdp.mjs", import.meta.url).pathname.replace(
+	/^\/([A-Z]:)/,
+	"$1",
+);
+
+const proc = spawn(process.execPath, [cdpBin, ...args], {
+	stdio: "inherit",
+	env: { ...process.env, CDP_PROFILE_DIR: PROFILE_DIR },
+});
+proc.on("close", (code) => process.exit(code ?? 0));

package/bin/cdp-headless.mjs

@@ -0,0 +1,16 @@
+#!/usr/bin/env node
+// cdp-headless.mjs — safe CDP wrapper for headless GreedySearch Chrome only.
+
+import { spawn } from "node:child_process";
+
+const cdpGreedyBin = new URL(
+	"./cdp-greedy.mjs",
+	import.meta.url,
+).pathname.replace(/^\/([A-Z]:)/, "$1");
+
+const proc = spawn(
+	process.execPath,
+	[cdpGreedyBin, "--mode", "headless", ...process.argv.slice(2)],
+	{ stdio: "inherit" },
+);
+proc.on("close", (code) => process.exit(code ?? 0));

package/bin/cdp-visible.mjs

@@ -0,0 +1,16 @@
+#!/usr/bin/env node
+// cdp-visible.mjs — safe CDP wrapper for visible GreedySearch Chrome only.
+
+import { spawn } from "node:child_process";
+
+const cdpGreedyBin = new URL(
+	"./cdp-greedy.mjs",
+	import.meta.url,
+).pathname.replace(/^\/([A-Z]:)/, "$1");
+
+const proc = spawn(
+	process.execPath,
+	[cdpGreedyBin, "--mode", "visible", ...process.argv.slice(2)],
+	{ stdio: "inherit" },
+);
+proc.on("close", (code) => process.exit(code ?? 0));