@apmantza/greedysearch-pi 2.1.2 → 2.1.4

package/docs/runtime.md

@@ -0,0 +1,65 @@
+# Runtime and Chrome
+
+GreedySearch uses a dedicated Chrome profile and debug port. It must not attach
+to your normal Chrome profile.
+
+- Profile: OS temp directory `greedysearch-chrome-profile`
+- Port: `9222`
+- Default mode: headless
+
+## Pi Commands
+
+```text
+/greedy-visible      # launch visible Chrome for captcha/login/cookie setup
+/greedy-status       # show GreedySearch Chrome status
+/greedy-kill         # stop GreedySearch Chrome
+/set-greedy-locale   # set default result language
+```
+
+## Environment Variables
+
+- `GREEDY_SEARCH_VISIBLE` — set `1` to show Chrome instead of headless.
+- `GREEDY_SEARCH_ALWAYS_VISIBLE` — set `1` to force visible mode for all runs.
+- `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES` — headless idle cleanup timeout;
+  default `5`.
+- `GREEDY_SEARCH_LOCALE` — default result language; default `en`.
+- `CHROME_PATH` — Chrome/Chromium executable path; auto-detected by default.
+
+## Runtime Helpers
+
+Git install path:
+
+```bash
+GS=~/.pi/agent/git/github.com/apmantza/GreedySearch-pi
+node "$GS/bin/launch.mjs" --status
+node "$GS/bin/visible.mjs"
+node "$GS/bin/visible.mjs" --kill
+node "$GS/bin/kill-visible.mjs"
+node "$GS/bin/cdp-visible.mjs" list
+node "$GS/bin/cdp-headless.mjs" list
+node "$GS/bin/cdp-greedy.mjs" list
+```
+
+npm global install path:
+
+```bash
+GS="$(npm root -g)/@apmantza/greedysearch-pi"
+node "$GS/bin/launch.mjs" --status
+node "$GS/bin/visible.mjs"
+node "$GS/bin/visible.mjs" --kill
+node "$GS/bin/kill-visible.mjs"
+node "$GS/bin/cdp-visible.mjs" list
+node "$GS/bin/cdp-headless.mjs" list
+node "$GS/bin/cdp-greedy.mjs" list
+```
+
+## CDP Safety
+
+Use only the safe wrappers for manual debugging:
+
+- `bin/cdp-visible.mjs` — refuses unless GreedySearch Chrome is visible.
+- `bin/cdp-headless.mjs` — refuses unless GreedySearch Chrome is headless.
+- `bin/cdp-greedy.mjs` — attaches only to the GreedySearch Chrome profile.
+
+Avoid raw `bin/cdp.mjs` unless `CDP_PROFILE_DIR` explicitly points at the
+GreedySearch profile.

package/docs/source-fetching.md

@@ -0,0 +1,33 @@
+# Source Fetching
+
+For `engine: "all"`, GreedySearch ranks discovered sources and fetches top source
+content by default. `synthesize: true` then asks the configured synthesizer to
+combine engine answers with fetched evidence.
+
+## Supported Sources
+
+- **PDFs** — direct PDF links are parsed into markdown text.
+- **Semantic Scholar** — academic paper URLs and direct PDFs are preferred when
+  the engine is used directly or opted into the fan-out.
+- **Reddit** — public `.json` API for posts and comments.
+- **GitHub** — REST API for repos, READMEs, file trees, and raw file content.
+- **General web** — Readability extraction with browser fallback.
+
+## Metadata
+
+Fetched sources include the best available:
+
+- title;
+- final URL;
+- status/content type;
+- byline and site name;
+- publish or modified date;
+- language;
+- excerpt/snippet;
+- trimmed markdown/plain text content.
+
+## Security
+
+Fetchers reject private/internal URLs and re-check the final redirected URL to
+avoid SSRF bypasses. GitHub and Reddit URLs use dedicated fetchers where
+possible to avoid fragile page scraping.

package/docs/usage.md

@@ -0,0 +1,69 @@
+# Usage Guide
+
+GreedySearch registers one Pi tool: `greedy_search`.
+
+## Common Calls
+
+```js
+greedy_search({ query: "React 19 changes" });
+greedy_search({ query: "React 19 changes", synthesize: true });
+greedy_search({ query: "Prisma vs Drizzle", engine: "perplexity" });
+greedy_search({ query: "What is Redis?", depth: "research", maxSources: 5 });
+```
+
+## Parameters
+
+### Common
+
+- `query` — required search/research question.
+- `fullAnswer` — return complete engine output instead of a preview.
+- `headless` — defaults to `true`; set to `false` to show Chrome.
+- `visible` / `alwaysVisible` — force visible Chrome for this run.
+
+### Normal Search
+
+- `engine` — `all` by default. Supported individual engines:
+  `perplexity`, `google`, `chatgpt`, `gemini`, `semantic-scholar`,
+  `logically`, and `bing`.
+- `synthesize` — for `engine: "all"`, synthesize fetched sources with the
+  configured synthesizer.
+- `synthesizer` — `gemini` or `chatgpt`.
+- `depth` — legacy `fast` / `standard` / `deep` aliases are still accepted;
+  prefer `synthesize` for normal search.
+
+### Research
+
+- `depth: "research"` — run the iterative research workflow.
+- `breadth` — actions per round, 1-5.
+- `iterations` — rounds, 1-3.
+- `maxSources` — fetched source cap, 3-12.
+- `researchOutDir` — custom bundle directory.
+- `writeResearchBundle` — write the bundle to disk; default `true`.
+
+## Search Modes
+
+- **Individual engine** — calls one engine and returns its answer/sources.
+- **Grounded all-engine search** — fans out to configured engines, ranks
+  sources, fetches top source content, and returns confidence metadata.
+- **All + synthesis** — adds a synthesis pass over engine answers and fetched
+  source evidence.
+- **Research** — runs planning, searches, source fetching, learning extraction,
+  citation audit, and bundle writing.
+
+## Configuration
+
+Configure all-engine fan-out and synthesis in `~/.pi/greedyconfig`:
+
+```json
+{
+  "engines": ["perplexity", "google", "chatgpt", "gemini"],
+  "synthesizer": "gemini"
+}
+```
+
+`semantic-scholar` and `logically` are opt-in research engines. They remain
+available as individual engines and can be added to `engines` when you want them
+in normal `engine: "all"` fan-out.
+
+Deep research child searches reuse the configured `engines` list and pass query
+text through stdin to avoid leaking prompts in process arguments.

package/extractors/chatgpt.mjs

@@ -25,7 +25,6 @@ import {
 	prepareArgs,
 	validateQuery,
 	waitForSelector,
-	waitForStreamComplete,
 } from "./common.mjs";
 import { dismissConsent, handleVerification } from "./consent.mjs";
 
@@ -131,14 +130,96 @@ const CHATGPT_RESPONSE_SELECTOR = String.raw`(() => {
  *                      and resolve quickly because the entire response
  *                      actually has finished.
  */
-async function waitForResponse(tab, timeoutMs = 20000) {
-	return waitForStreamComplete(tab, {
-		timeout: timeoutMs,
-		interval: 600,
-		stableRounds: 6,
-		minLength: 1,
-		selector: CHATGPT_RESPONSE_SELECTOR,
-	});
+async function bringToFront(tab, { required = false } = {}) {
+	try {
+		await cdp(["evalraw", tab, "Page.bringToFront", "{}"], 5000);
+		return true;
+	} catch (error) {
+		console.error(`[chatgpt] bringToFront failed: ${error.message}`);
+		if (required) throw error;
+		return false;
+	}
+}
+
+async function waitForResponse(tab, timeoutMs = 35000) {
+	await bringToFront(tab, { required: true });
+	let foregroundAttempts = 0;
+	let foregroundInFlight = false;
+	const keepForeground = setInterval(() => {
+		if (foregroundInFlight) return;
+		if (foregroundAttempts >= 8) {
+			console.error("[chatgpt] stopping foreground keepalive after 8 attempts");
+			clearInterval(keepForeground);
+			return;
+		}
+		foregroundAttempts++;
+		foregroundInFlight = true;
+		bringToFront(tab).finally(() => {
+			foregroundInFlight = false;
+		});
+	}, 4000);
+	try {
+		const code = String.raw`
+		new Promise((resolve, reject) => {
+			const _deadline = Date.now() + ${timeoutMs};
+			const _baseInterval = 700;
+			const _stableRounds = 5;
+			let _lastLen = -1;
+			let _stableCount = 0;
+
+			function _jitter(ms) {
+				return Math.max(50, ms + (Math.random() * ms * 0.4 - ms * 0.2));
+			}
+
+			function _assistantAfterLastUser() {
+				const all = document.querySelectorAll('[data-message-author-role]');
+				let lastUserIdx = -1;
+				for (let i = 0; i < all.length; i++) {
+					if (all[i].getAttribute('data-message-author-role') === 'user') lastUserIdx = i;
+				}
+				if (lastUserIdx < 0) return null;
+				let assistant = null;
+				for (let i = lastUserIdx + 1; i < all.length; i++) {
+					if (all[i].getAttribute('data-message-author-role') === 'assistant') assistant = all[i];
+				}
+				return assistant;
+			}
+
+			function _poll() {
+				try {
+					const el = _assistantAfterLastUser();
+					const text = (el?.innerText || '').trim();
+					const len = text.length;
+					const streaming = !!el?.querySelector('.streaming-animation,[data-is-streaming="true"]') ||
+						!!document.querySelector('button[data-testid="stop-button"], button[aria-label="Stop generating"], button[aria-label*="Stop"]');
+					if (len >= 1 && !streaming) {
+						if (len === _lastLen) {
+							_stableCount++;
+							if (_stableCount >= _stableRounds) { resolve(len); return; }
+						} else {
+							_lastLen = len;
+							_stableCount = 0;
+						}
+					} else if (len !== _lastLen) {
+						_lastLen = len;
+						_stableCount = 0;
+					}
+					if (Date.now() < _deadline) setTimeout(_poll, _jitter(_baseInterval));
+					else if (_lastLen >= 1 && !streaming) resolve(_lastLen);
+					else reject(new Error('ChatGPT response did not finish streaming within ${timeoutMs}ms'));
+				} catch (e) { reject(e); }
+			}
+			_poll();
+		})`;
+		const lenStr = await cdp(["eval", tab, code], timeoutMs + 10000);
+		const len = parseInt(lenStr, 10) || 0;
+		if (len >= 1) return len;
+		throw new Error(
+			`ChatGPT response did not finish streaming within ${timeoutMs}ms`,
+		);
+	} finally {
+		clearInterval(keepForeground);
+	}
 }
 
 /**
@@ -211,6 +292,8 @@ async function extractAnswerFromDom(tab) {
 					skipped: 'no-assistant-response',
 				});
 			}
+			const streaming = !!assistant.querySelector('.streaming-animation,[data-is-streaming="true"]') ||
+				!!document.querySelector('button[data-testid="stop-button"], button[aria-label="Stop generating"], button[aria-label*="Stop"]');
 			const answer = (assistant.innerText || assistant.textContent || '').trim();
 			const seen = new Set();
 			const sources = [];
@@ -222,13 +305,16 @@ async function extractAnswerFromDom(tab) {
 				sources.push({ title, url });
 				if (sources.length >= 10) break;
 			}
-			return JSON.stringify({ answer, sources });
+			return JSON.stringify({ answer, sources, streaming });
 		})()
 	`,
 	]);
 	try {
 		return JSON.parse(raw);
-	} catch {
+	} catch (error) {
+		console.error(
+			`[chatgpt] DOM fallback JSON parse failed (${raw.length} chars): ${error.message}`,
+		);
 		return { answer: "", sources: [], skipped: "parse-error" };
 	}
 }
@@ -307,6 +393,9 @@ async function extractAnswer(tab, env) {
 	let domFallback = null;
 	if (!answer) {
 		domFallback = await extractAnswerFromDom(tab);
+		if (domFallback.streaming) {
+			return { answer: "", sources: [], skipped: "still-streaming" };
+		}
 		answer = domFallback.answer;
 		env.fallbackUsed = answer ? "dom" : null;
 	}
@@ -335,15 +424,25 @@ async function extractAnswer(tab, env) {
 			trimmed.length <= 50 &&
 			/\s|[.,!?;:]/.test(trimmed);
 		const looksLikeLongAnswer = trimmed.length > 50;
-		if (!looksLikeShortAnswer && !looksLikeLongAnswer) {
+		const words = trimmed.split(/\s+/).filter(Boolean);
+		const domainRepeats = (
+			trimmed.match(/\b[a-z0-9-]+\.(?:com|org|net|dev|io)\b/gi) || []
+		).length;
+		const looksLikeCitationStub =
+			domainRepeats >= 4 &&
+			domainRepeats >= Math.max(3, Math.floor(words.length / 3));
+		if (
+			(!looksLikeShortAnswer && !looksLikeLongAnswer) ||
+			looksLikeCitationStub
+		) {
 			console.error(
-				`[chatgpt] DOM fallback answer suspiciously short (${trimmed.length} chars: ${JSON.stringify(trimmed.slice(0, 80))}) — returning empty for caller to retry`,
+				`[chatgpt] DOM fallback answer suspicious (${trimmed.length} chars, domainRepeats=${domainRepeats}: ${JSON.stringify(trimmed.slice(0, 80))}) — returning empty for caller to retry`,
 			);
 			env.fallbackUsed = null;
 			return {
 				answer: "",
 				sources: [],
-				skipped: "header-stub",
+				skipped: looksLikeCitationStub ? "citation-stub" : "header-stub",
 			};
 		}
 	}
@@ -518,15 +617,20 @@ async function main() {
 		if (!answer) {
 			env.blockedBy = "no-response";
 			env.skipped = skipped || null;
-			throw new Error(
-				skipped === "no-user-message"
-					? "ChatGPT still on homepage — query was not submitted"
-					: skipped === "no-assistant-response"
-						? "ChatGPT did not return an assistant response after submit"
-						: skipped === "header-stub"
-							? "ChatGPT response appeared to be a header stub after 3 retries — assistant never rendered the body"
-							: "ChatGPT returned no answer — assistant never responded",
-			);
+			let message = "ChatGPT returned no answer — assistant never responded";
+			if (skipped === "no-user-message") {
+				message = "ChatGPT still on homepage — query was not submitted";
+			} else if (skipped === "no-assistant-response") {
+				message = "ChatGPT did not return an assistant response after submit";
+			} else if (
+				skipped === "header-stub" ||
+				skipped === "citation-stub" ||
+				skipped === "still-streaming"
+			) {
+				message =
+					"ChatGPT response did not finish rendering after 3 retries — assistant never rendered the body";
+			}
+			throw new Error(message);
 		}
 		logStage(env, "done", startTime);
 

package/extractors/gemini.mjs

@@ -66,6 +66,17 @@ async function typeIntoGemini(tab, text) {
 	}
 }
 
+async function bringToFront(tab, { required = false } = {}) {
+	try {
+		await cdp(["evalraw", tab, "Page.bringToFront", "{}"], 5000);
+		return true;
+	} catch (error) {
+		console.error(`[gemini] bringToFront failed: ${error.message}`);
+		if (required) throw error;
+		return false;
+	}
+}
+
 async function scrollToBottom(tab) {
 	await cdp([
 		"eval",
@@ -93,7 +104,8 @@ async function extractAnswerFromDom(tab) {
 			new Promise((resolve) => {
 				const _deadline = Date.now() + 6000;
 				function _tryExtract() {
-					const resp = document.querySelector('model-response');
+					const responses = Array.from(document.querySelectorAll('model-response'));
+					const resp = responses[responses.length - 1];
 					if (resp) {
 						const text = (resp.innerText || resp.textContent || '').trim();
 						const idx = text.indexOf('\n');
@@ -126,11 +138,32 @@ async function extractAnswerFromDom(tab) {
 	);
 	try {
 		return JSON.parse(raw);
-	} catch {
+	} catch (error) {
+		console.error(
+			`[gemini] DOM fallback JSON parse failed (${raw.length} chars): ${error.message}`,
+		);
 		return { answer: "", sources: [] };
 	}
 }
 
+async function clickLatestModelResponseCopy(tab) {
+	return cdp([
+		"eval",
+		tab,
+		`(() => {
+			const responses = Array.from(document.querySelectorAll('model-response'));
+			const resp = responses[responses.length - 1];
+			if (!resp) return 'no-model-response';
+			const buttons = Array.from(resp.querySelectorAll('${S.copyButton}'));
+			const nonCodeButtons = buttons.filter((btn) => !btn.closest('pre, code, code-block, .code-block, .code-container'));
+			const btn = nonCodeButtons[nonCodeButtons.length - 1] || buttons[buttons.length - 1];
+			if (!btn) return 'no-copy-button';
+			btn.click();
+			return 'clicked';
+		})()`,
+	]);
+}
+
 async function extractAnswer(tab, query = "") {
 	const queryNorm = query.toLowerCase().trim();
 
@@ -151,7 +184,8 @@ async function extractAnswer(tab, query = "") {
 			"eval",
 			tab,
 			String.raw`(() => {
-				const r = document.querySelector('model-response');
+				const responses = Array.from(document.querySelectorAll('model-response'));
+				const r = responses[responses.length - 1];
 				if (!r) return false;
 				const t = (r.innerText || '').trim();
 				// Must have content beyond the locale-specific label
@@ -174,18 +208,7 @@ async function extractAnswer(tab, query = "") {
 	// not the absolute last copy button on the page. The page has many
 	// copy icons (copy link, copy code, etc.) and the last one is not
 	// always the assistant's response copy button.
-	await cdp([
-		"eval",
-		tab,
-		`(() => {
-			const resp = document.querySelector('model-response');
-			if (!resp) return 'no-model-response';
-			const btn = resp.querySelector('${S.copyButton}');
-			if (!btn) return 'no-copy-button';
-			btn.click();
-			return 'clicked';
-		})()`,
-	]);
+	await clickLatestModelResponseCopy(tab);
 	await new Promise((r) => setTimeout(r, 600));
 
 	let answer = await cdp(["eval", tab, `window.${GLOBAL_VAR} || ''`]);
@@ -200,35 +223,32 @@ async function extractAnswer(tab, query = "") {
 	) {
 		console.error("[gemini] Clipboard echoed query, retrying in 2s...");
 		await new Promise((r) => setTimeout(r, 2000));
-		await cdp([
-			"eval",
-			tab,
-			`(() => {
-				const resp = document.querySelector('model-response');
-				if (!resp) return 'no-model-response';
-				const btn = resp.querySelector('${S.copyButton}');
-				if (!btn) return 'no-copy-button';
-				btn.click();
-				return 'clicked';
-			})()`,
-		]);
+		await clickLatestModelResponseCopy(tab);
 		await new Promise((r) => setTimeout(r, 600));
 		answer = await cdp(["eval", tab, `window.${GLOBAL_VAR} || ''`]);
 	}
 
-	// DOM fallback: if the clipboard is empty or still echoes the query,
-	// read the model-response innerText directly. This handles the case
-	// where the copy button never rendered (response never appeared) or
-	// the click didn't fire.
+	// DOM fallback: if clipboard is empty, echoes the query, or looks like a
+	// short snippet, read the model-response innerText directly. Skip the
+	// fallback when native copy already returned a substantial full answer.
 	let domFallback = null;
-	if (
-		!answer ||
-		(queryNorm &&
-			(answer.toLowerCase().trim() === queryNorm ||
-				answer.trim().length < queryNorm.length))
-	) {
+	const answerText = answer?.trim() || "";
+	const clipboardLooksEchoed =
+		queryNorm &&
+		(answerText.toLowerCase() === queryNorm ||
+			answerText.length < queryNorm.length);
+	const clipboardLooksComplete =
+		answerText.length > 500 && !clipboardLooksEchoed;
+	if (!clipboardLooksComplete) {
 		domFallback = await extractAnswerFromDom(tab);
-		if (domFallback.answer) {
+		const domExtendsClipboard =
+			answerText &&
+			domFallback.answer.includes(answerText) &&
+			domFallback.answer.length > answerText.length * 2;
+		if (
+			domFallback.answer &&
+			(!answerText || clipboardLooksEchoed || domExtendsClipboard)
+		) {
 			answer = domFallback.answer;
 		}
 	}
@@ -304,6 +324,7 @@ async function main() {
 		await waitForSelector(tab, S.input, 8000, TIMING.inputPoll);
 		await new Promise((r) => setTimeout(r, jitter(TIMING.postClick)));
 
+		await bringToFront(tab, { required: true });
 		await injectClipboardInterceptor(tab, GLOBAL_VAR);
 		await typeIntoGemini(tab, query);
 		await new Promise((r) => setTimeout(r, jitter(TIMING.postType)));
@@ -316,8 +337,10 @@ async function main() {
 
 		// Wait for Gemini's response to finish streaming before extracting.
 		// Periodic scrolling keeps lazy-loaded content triggered in the viewport.
+		await bringToFront(tab);
 		let pollTick = 0;
 		const scrollInterval = setInterval(() => {
+			bringToFront(tab).catch(() => null);
 			if (++pollTick % 10 === 0) scrollToBottom(tab).catch(() => null);
 		}, 6000);
 		try {
@@ -325,6 +348,8 @@ async function main() {
 				timeout: 45000,
 				stableRounds: 5,
 				minLength: 60,
+				selector:
+					"(() => { const responses = Array.from(document.querySelectorAll('model-response')); return responses[responses.length - 1]; })()",
 			});
 		} finally {
 			clearInterval(scrollInterval);

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@apmantza/greedysearch-pi",
-	"version": "2.1.2",
+	"version": "2.1.4",
 	"description": "Headless multi-engine AI search (Perplexity, Google AI, ChatGPT, Gemini) via browser automation. NO API KEYS needed. Grounded all-engine search fetches sources by default, with optional configurable synthesis and deep research.",
 	"type": "module",
 	"keywords": [
@@ -22,7 +22,11 @@
 		"test:bash:quick": "./test.sh quick",
 		"test:bash:smoke": "./test.sh smoke",
 		"lint": "node scripts/lint.mjs",
-		"check:lockfile": "node scripts/check-lockfile.mjs"
+		"check:lockfile": "node scripts/check-lockfile.mjs",
+		"changelog:check": "node scripts/changelog-release.mjs --check",
+		"changelog:release": "node scripts/changelog-release.mjs",
+		"changelog:extract": "node scripts/changelog-extract.mjs",
+		"release:backfill-notes": "node scripts/backfill-github-releases.mjs"
 	},
 	"engines": {
 		"node": ">=20.11.0"
@@ -32,29 +36,26 @@
 		"test.mjs",
 		"bin/",
 		"src/",
-		"skills/",
 		"extractors/",
+		"scripts/",
 		"CHANGELOG.md",
 		"README.md",
-		"docs/banner.png"
+		"docs/"
 	],
 	"pi": {
 		"extensions": [
 			"./index.ts"
-		],
-		"skills": [
-			"./skills"
 		]
 	},
 	"dependencies": {
 		"@mozilla/readability": "^0.6.0",
-		"@sinclair/typebox": "^0.34.48",
+		"@sinclair/typebox": "^0.34.49",
 		"jsdom": "^29.1.1",
 		"pdf-parse": "^2.4.5",
 		"turndown": "^7.1.2"
 	},
 	"peerDependencies": {
-		"@earendil-works/pi-coding-agent": "*"
+		"@earendil-works/pi-coding-agent": "^0.79.0"
 	},
 	"peerDependenciesMeta": {
 		"@earendil-works/pi-coding-agent": {