@apmantza/greedysearch-pi 1.4.0 → 1.4.2

package/README.md

@@ -4,14 +4,6 @@ Pi extension that adds a `greedy_search` tool — fans out queries to Perplexity
 
 Forked from [GreedySearch-claude](https://github.com/apmantza/GreedySearch-claude).
 
-## What's New (v1.4.0)
-
-- **Grounded synthesis** — Gemini now receives a normalized source registry with stable source IDs, agreement summaries, caveats, and cited claims
-- **Real deep research** — top sources are fetched before synthesis so deep research answers are grounded in fetched evidence, not just engine summaries
-- **Richer source metadata** — source output now includes canonical URLs, domains, source types, per-engine attribution, and confidence metadata
-- **Cleaner tab lifecycle** — temporary Perplexity, Bing, and Google tabs are closed after each fan-out search, and synthesis finishes on the Gemini tab
-- **Isolated Chrome targeting** — GreedySearch now refuses to fall back to your normal Chrome session, preventing stray remote-debugging prompts
-
 ## Install
 
 ```bash
@@ -70,15 +62,7 @@ For complex research questions, use `synthesize: true` with `engine: "all"`:
 greedy_search({ query: "best auth patterns for SaaS in 2026", engine: "all", synthesize: true })
 ```
 
-This deduplicates sources across engines, builds a normalized source registry, and feeds that context to Gemini for one clean synthesized answer. Adds ~30s but now returns agreement summaries, caveats, key claims, and better-labeled top sources.
-
-For the most grounded mode, use deep research from the CLI:
-
-```bash
-node search.mjs all "best auth patterns for SaaS in 2026" --deep-research
-```
-
-Deep research fetches top source pages before synthesis and reports source confidence metadata such as agreement level, fetched-source success rate, and source mix.
+This deduplicates sources across engines, builds a normalized source registry, and feeds that context to Gemini for one clean synthesized answer. Adds ~30s but returns agreement summaries, caveats, key claims, and better-labeled top sources.
 
 **Use synthesis when:**
 - You need one definitive answer, not multiple perspectives
@@ -100,21 +84,25 @@ greedy_search({ query: "explain the React compiler", engine: "perplexity", fullA
 ## Examples
 
 **Quick technical lookup:**
+
 ```
 greedy_search({ query: "How to use async await in Python", engine: "perplexity" })
 ```
 
 **Compare tools (see where engines agree/disagree):**
+
 ```
 greedy_search({ query: "Prisma vs Drizzle in 2026", engine: "all" })
 ```
 
 **Research with synthesis:**
+
 ```
 greedy_search({ query: "Best practices for monorepo structure", engine: "all", synthesize: true })
 ```
 
 **Debug an error:**
+
 ```
 greedy_search({ query: "Error: Cannot find module 'react-dom/client' Next.js 15", engine: "all" })
 ```
@@ -163,32 +151,37 @@ Tests verify:
 ## Troubleshooting
 
 ### "Chrome not found"
+
 Set the path explicitly:
+
 ```bash
 export CHROME_PATH="/path/to/chrome"
 ```
 
 ### "CDP timeout" or "Chrome may have crashed"
+
 Restart GreedySearch Chrome:
+
 ```bash
 node ~/.pi/agent/git/GreedySearch-pi/launch.mjs --kill
 node ~/.pi/agent/git/GreedySearch-pi/launch.mjs
 ```
 
 ### Google / Bing "verify you're human"
-The extension auto-clicks verification buttons and Cloudflare Turnstile challenges. For hard CAPTCHAs (image puzzles), solve manually in the Chrome window that opens.
+
+The extension auto-clicks verification buttons and Cloudflare Turnstile challenges using broad keyword matching — resilient to variations like "Verify you are human" or localised button text. For hard CAPTCHAs (image puzzles), solve manually in the Chrome window that opens.
 
 ### Parallel searches failing
-Earlier versions shared Chrome tabs between concurrent searches, causing `ERR_ABORTED` errors. Version 1.2.0+ creates fresh tabs for each search, allowing safe parallel execution.
+
+Each search creates a fresh isolated browser tab that is closed after completion, allowing safe parallel execution without tab state conflicts.
 
 ### Search hangs
+
 Chrome may be unresponsive. Restart it with `launch.mjs --kill` then `launch.mjs`.
 
-### Sources are junk links
-This was a known issue with Gemini sources. If you're on an older version, update:
-```bash
-pi install npm:@apmantza/greedysearch-pi
-```
+### Sources are empty or junk links
+
+Sources are now extracted by regex-parsing Markdown links (`[title](url)`) from the clipboard text captured after each engine responds — not from DOM selectors that break when the engine's UI updates. If sources are empty, the engine's clipboard copy didn't include formatted links (Bing Copilot currently falls into this category).
 
 ## How It Works
 
@@ -199,6 +192,28 @@ pi install npm:@apmantza/greedysearch-pi
 - `cdp.mjs` — Chrome DevTools Protocol CLI for browser automation
 - `skills/greedy-search/SKILL.md` — skill file that guides the model on when/how to use greedy_search
 
+## Changelog
+
+### v1.4.2 (2026-03-25)
+
+- **Fresh isolated tabs** — each search now always creates a new `about:blank` tab via `Target.createTarget` and refreshes the CDP page cache immediately after, preventing SPA navigation failures and stale DOM state from prior queries
+- **Regex-based citation extraction** — all extractors (Perplexity, Bing, Gemini) now parse sources from clipboard Markdown links (`[title](url)`) instead of DOM selectors that break on UI updates
+- **Relaxed verification detection** — `consent.mjs` now uses broad keyword matching (`includes('verify')`, `includes('human')`) instead of anchored regexes, correctly catching button text variants like "Verify you are human" across Cloudflare, Microsoft, and generic modals
+
+---
+
+### v1.4.1
+
+- **Fixed parallel synthesis** — multiple `greedy_search` calls with `synthesize: true` now run safely in parallel. Each search creates a fresh Gemini tab that gets cleaned up after synthesis, preventing tab conflicts and "Uncaught" errors.
+
+### v1.4.0
+
+- **Grounded synthesis** — Gemini now receives a normalized source registry with stable source IDs, agreement summaries, caveats, and cited claims
+- **Real deep research** — top sources are fetched before synthesis so deep research answers are grounded in fetched evidence, not just engine summaries
+- **Richer source metadata** — source output now includes canonical URLs, domains, source types, per-engine attribution, and confidence metadata
+- **Cleaner tab lifecycle** — temporary Perplexity, Bing, and Google tabs are closed after each fan-out search, and synthesis finishes on the Gemini tab
+- **Isolated Chrome targeting** — GreedySearch now refuses to fall back to your normal Chrome session, preventing stray remote-debugging prompts
+
 ## License
 
 MIT

package/extractors/bing-copilot.mjs

@@ -1,204 +1,195 @@
-#!/usr/bin/env node
-// extractors/bing-copilot.mjs
-// Navigate copilot.microsoft.com, wait for answer to complete, return clean answer + sources.
-//
-// Usage:
-//   node extractors/bing-copilot.mjs "<query>" [--tab <prefix>]
-//
-// Output (stdout): JSON { answer, sources, query, url }
-// Errors go to stderr only — stdout is always clean JSON for piping.
-
-import { readFileSync, existsSync } from 'fs';
-import { spawn } from 'child_process';
-import { tmpdir } from 'os';
-import { join, dirname } from 'path';
-import { fileURLToPath } from 'url';
-import { dismissConsent, handleVerification } from './consent.mjs';
-import { SELECTORS } from './selectors.mjs';
-
-const __dir = dirname(fileURLToPath(import.meta.url));
-const CDP = join(__dir, '..', 'cdp.mjs');
-const PAGES_CACHE = `${tmpdir().replace(/\\/g, '/')}/cdp-pages.json`;
-
-const COPY_POLL_INTERVAL = 700;
-const COPY_TIMEOUT = 60000;
-
-const S = SELECTORS.bing;
-
-// ---------------------------------------------------------------------------
-
-function cdp(args, timeoutMs = 30000) {
-  return new Promise((resolve, reject) => {
-    const proc = spawn('node', [CDP, ...args], { stdio: ['ignore', 'pipe', 'pipe'] });
-    let out = '';
-    let err = '';
-    proc.stdout.on('data', d => out += d);
-    proc.stderr.on('data', d => err += d);
-    const timer = setTimeout(() => { proc.kill(); reject(new Error(`cdp timeout: ${args[0]}`)); }, timeoutMs);
-    proc.on('close', code => {
-      clearTimeout(timer);
-      if (code !== 0) reject(new Error(err.trim() || `cdp exit ${code}`));
-      else resolve(out.trim());
-    });
-  });
-}
-
-async function getOrOpenTab(tabPrefix) {
-  if (tabPrefix) return tabPrefix;
-
-  if (existsSync(PAGES_CACHE)) {
-    const pages = JSON.parse(readFileSync(PAGES_CACHE, 'utf8'));
-    const existing = pages.find(p => p.url.includes('copilot.microsoft.com'));
-    if (existing) return existing.targetId.slice(0, 8);
-  }
-
-  const list = await cdp(['list']);
-  const firstLine = list.split('\n')[0];
-  if (!firstLine) throw new Error('No Chrome tabs found. Is Chrome running with --remote-debugging-port=9222?');
-  return firstLine.slice(0, 8);
-}
-
-async function injectClipboardInterceptor(tab) {
-  await cdp(['eval', tab, `
-    window.__bingClipboard = null;
-    const _origWriteText = navigator.clipboard.writeText.bind(navigator.clipboard);
-    navigator.clipboard.writeText = function(text) {
-      window.__bingClipboard = text;
-      return _origWriteText(text);
-    };
-    const _origWrite = navigator.clipboard.write.bind(navigator.clipboard);
-    navigator.clipboard.write = async function(items) {
-      try {
-        for (const item of items) {
-          if (item.types && item.types.includes('text/plain')) {
-            const blob = await item.getType('text/plain');
-            window.__bingClipboard = await blob.text();
-            break;
-          }
-        }
-      } catch(e) {}
-      return _origWrite(items);
-    };
-  `]);
-}
-
-async function waitForCopyButton(tab) {
-  const deadline = Date.now() + COPY_TIMEOUT;
-  while (Date.now() < deadline) {
-    await new Promise(r => setTimeout(r, COPY_POLL_INTERVAL));
-    const found = await cdp(['eval', tab,
-      `!!document.querySelector('${S.copyButton}')`
-    ]).catch(() => 'false');
-    if (found === 'true') return;
-  }
-  throw new Error(`Copilot copy button did not appear within ${COPY_TIMEOUT}ms`);
-}
-
-async function extractAnswer(tab) {
-  await cdp(['eval', tab, `document.querySelector('${S.copyButton}')?.click()`]);
-  await new Promise(r => setTimeout(r, 400));
-
-  const answer = await cdp(['eval', tab, `window.__bingClipboard || ''`]);
-  if (!answer) throw new Error('Clipboard interceptor returned empty text');
-
-  const raw = await cdp(['eval', tab, `
-    (function() {
-      var sources = Array.from(document.querySelectorAll('${S.sourceLink}'))
-        .map(a => ({ url: a.href, title: a.innerText?.trim().split('\\n')[0] || a.title || '' }))
-        .filter(s => s.url && !s.url.includes('${S.sourceExclude}'))
-        .filter((v, i, arr) => arr.findIndex(x => x.url === v.url) === i)
-        .slice(0, 10);
-      return JSON.stringify(sources);
-    })()
-  `]).catch(() => '[]');
-  const sources = JSON.parse(raw);
-
-  return { answer: answer.trim(), sources };
-}
-
-// ---------------------------------------------------------------------------
-
-async function main() {
-  const args = process.argv.slice(2);
-  if (!args.length || args[0] === '--help') {
-    process.stderr.write('Usage: node extractors/bing-copilot.mjs "<query>" [--tab <prefix>]\n');
-    process.exit(1);
-  }
-
-  const short = args.includes('--short');
-  const rest  = args.filter(a => a !== '--short');
-  const tabFlagIdx = rest.indexOf('--tab');
-  const tabPrefix = tabFlagIdx !== -1 ? rest[tabFlagIdx + 1] : null;
-  const query = tabFlagIdx !== -1
-    ? rest.filter((_, i) => i !== tabFlagIdx && i !== tabFlagIdx + 1).join(' ')
-    : rest.join(' ');
-
-  try {
-    await cdp(['list']);
-    const tab = await getOrOpenTab(tabPrefix);
-
-    // Navigate to Copilot homepage and use the chat input
-    await cdp(['nav', tab, 'https://copilot.microsoft.com/'], 35000);
-    await new Promise(r => setTimeout(r, 2000));
-    await dismissConsent(tab, cdp);
-    
-    // Handle verification challenges (Cloudflare Turnstile, Microsoft auth, etc.)
-    const verifyResult = await handleVerification(tab, cdp, 90000);
-    if (verifyResult === 'needs-human') {
-      throw new Error('Copilot verification required — please solve it manually in the browser window');
-    }
-    
-    // After verification, page may have redirected or reloaded — wait for it to settle
-    if (verifyResult === 'clicked') {
-      await new Promise(r => setTimeout(r, 3000));
-      
-      // Re-navigate if we got redirected
-      const currentUrl = await cdp(['eval', tab, 'document.location.href']).catch(() => '');
-      if (!currentUrl.includes('copilot.microsoft.com')) {
-        await cdp(['nav', tab, 'https://copilot.microsoft.com/'], 35000);
-        await new Promise(r => setTimeout(r, 2000));
-        await dismissConsent(tab, cdp);
-      }
-    }
-
-    // Wait for React app to mount input (up to 15s, longer after verification)
-    const inputDeadline = Date.now() + 15000;
-    while (Date.now() < inputDeadline) {
-      const found = await cdp(['eval', tab, `!!document.querySelector('${S.input}')`]).catch(() => 'false');
-      if (found === 'true') break;
-      await new Promise(r => setTimeout(r, 500));
-    }
-    await new Promise(r => setTimeout(r, 300));
-
-    // Verify input is actually there before proceeding
-    const inputReady = await cdp(['eval', tab, `!!document.querySelector('${S.input}')`]).catch(() => 'false');
-    if (inputReady !== 'true') {
-      throw new Error('Copilot input not found — verification may have failed or page is in unexpected state');
-    }
-
-    await injectClipboardInterceptor(tab);
-    await cdp(['click', tab, S.input]);
-    await new Promise(r => setTimeout(r, 400));
-    await cdp(['type', tab, query]);
-    await new Promise(r => setTimeout(r, 400));
-
-    // Submit with Enter (most reliable across locales and Chrome instances)
-    await cdp(['eval', tab,
-      `document.querySelector('${S.input}')?.dispatchEvent(new KeyboardEvent('keydown',{key:'Enter',bubbles:true,keyCode:13})), 'ok'`
-    ]);
-
-    await waitForCopyButton(tab);
-
-    const { answer, sources } = await extractAnswer(tab);
-    if (!answer) throw new Error('No answer extracted — Copilot may not have responded');
-    const out = short ? answer.slice(0, 300).replace(/\s+\S*$/, '') + '…' : answer;
-
-    const finalUrl = await cdp(['eval', tab, 'document.location.href']).catch(() => '');
-    process.stdout.write(JSON.stringify({ query, url: finalUrl, answer: out, sources }, null, 2) + '\n');
-  } catch (e) {
-    process.stderr.write(`Error: ${e.message}\n`);
-    process.exit(1);
-  }
-}
-
-main();
+#!/usr/bin/env node
+// extractors/bing-copilot.mjs
+// Navigate copilot.microsoft.com, wait for answer to complete, return clean answer + sources.
+//
+// Usage:
+//   node extractors/bing-copilot.mjs "<query>" [--tab <prefix>]
+//
+// Output (stdout): JSON { answer, sources, query, url }
+// Errors go to stderr only — stdout is always clean JSON for piping.
+
+import { readFileSync, existsSync } from 'fs';
+import { spawn } from 'child_process';
+import { tmpdir } from 'os';
+import { join, dirname } from 'path';
+import { fileURLToPath } from 'url';
+import { dismissConsent, handleVerification } from './consent.mjs';
+import { SELECTORS } from './selectors.mjs';
+
+const __dir = dirname(fileURLToPath(import.meta.url));
+const CDP = join(__dir, '..', 'cdp.mjs');
+const PAGES_CACHE = `${tmpdir().replace(/\\/g, '/')}/cdp-pages.json`;
+
+const COPY_POLL_INTERVAL = 700;
+const COPY_TIMEOUT = 60000;
+
+const S = SELECTORS.bing;
+
+// ---------------------------------------------------------------------------
+
+function cdp(args, timeoutMs = 30000) {
+  return new Promise((resolve, reject) => {
+    const proc = spawn('node', [CDP, ...args], { stdio: ['ignore', 'pipe', 'pipe'] });
+    let out = '';
+    let err = '';
+    proc.stdout.on('data', d => out += d);
+    proc.stderr.on('data', d => err += d);
+    const timer = setTimeout(() => { proc.kill(); reject(new Error(`cdp timeout: ${args[0]}`)); }, timeoutMs);
+    proc.on('close', code => {
+      clearTimeout(timer);
+      if (code !== 0) reject(new Error(err.trim() || `cdp exit ${code}`));
+      else resolve(out.trim());
+    });
+  });
+}
+
+async function getOrOpenTab(tabPrefix) {
+  if (tabPrefix) return tabPrefix;
+  // Always open a fresh tab to avoid SPA navigation issues
+  const list = await cdp(['list']);
+  const anchor = list.split('\n')[0]?.slice(0, 8);
+  if (!anchor) throw new Error('No Chrome tabs found. Is Chrome running with --remote-debugging-port=9222?');
+  const raw = await cdp(['evalraw', anchor, 'Target.createTarget', '{"url":"about:blank"}']);
+  const { targetId } = JSON.parse(raw);
+  await cdp(['list']); // refresh cache
+  return targetId.slice(0, 8);
+}
+
+async function injectClipboardInterceptor(tab) {
+  await cdp(['eval', tab, `
+    window.__bingClipboard = null;
+    const _origWriteText = navigator.clipboard.writeText.bind(navigator.clipboard);
+    navigator.clipboard.writeText = function(text) {
+      window.__bingClipboard = text;
+      return _origWriteText(text);
+    };
+    const _origWrite = navigator.clipboard.write.bind(navigator.clipboard);
+    navigator.clipboard.write = async function(items) {
+      try {
+        for (const item of items) {
+          if (item.types && item.types.includes('text/plain')) {
+            const blob = await item.getType('text/plain');
+            window.__bingClipboard = await blob.text();
+            break;
+          }
+        }
+      } catch(e) {}
+      return _origWrite(items);
+    };
+  `]);
+}
+
+async function waitForCopyButton(tab) {
+  const deadline = Date.now() + COPY_TIMEOUT;
+  while (Date.now() < deadline) {
+    await new Promise(r => setTimeout(r, COPY_POLL_INTERVAL));
+    const found = await cdp(['eval', tab,
+      `!!document.querySelector('${S.copyButton}')`
+    ]).catch(() => 'false');
+    if (found === 'true') return;
+  }
+  throw new Error(`Copilot copy button did not appear within ${COPY_TIMEOUT}ms`);
+}
+
+async function extractAnswer(tab) {
+  await cdp(['eval', tab, `document.querySelector('${S.copyButton}')?.click()`]);
+  await new Promise(r => setTimeout(r, 400));
+
+  const answer = await cdp(['eval', tab, `window.__bingClipboard || ''`]);
+  if (!answer) throw new Error('Clipboard interceptor returned empty text');
+
+  // Regex parse Markdown links from clipboard — robust against DOM changes
+  const sources = Array.from(answer.matchAll(/\[([^\]]+)\]\((https?:\/\/[^\s\)]+)\)/g))
+    .map(m => ({ title: m[1], url: m[2] }))
+    .filter((v, i, arr) => arr.findIndex(x => x.url === v.url) === i)
+    .slice(0, 10);
+
+  return { answer: answer.trim(), sources };
+}
+
+// ---------------------------------------------------------------------------
+
+async function main() {
+  const args = process.argv.slice(2);
+  if (!args.length || args[0] === '--help') {
+    process.stderr.write('Usage: node extractors/bing-copilot.mjs "<query>" [--tab <prefix>]\n');
+    process.exit(1);
+  }
+
+  const short = args.includes('--short');
+  const rest  = args.filter(a => a !== '--short');
+  const tabFlagIdx = rest.indexOf('--tab');
+  const tabPrefix = tabFlagIdx !== -1 ? rest[tabFlagIdx + 1] : null;
+  const query = tabFlagIdx !== -1
+    ? rest.filter((_, i) => i !== tabFlagIdx && i !== tabFlagIdx + 1).join(' ')
+    : rest.join(' ');
+
+  try {
+    await cdp(['list']);
+    const tab = await getOrOpenTab(tabPrefix);
+
+    // Navigate to Copilot homepage and use the chat input
+    await cdp(['nav', tab, 'https://copilot.microsoft.com/'], 35000);
+    await new Promise(r => setTimeout(r, 2000));
+    await dismissConsent(tab, cdp);
+    
+    // Handle verification challenges (Cloudflare Turnstile, Microsoft auth, etc.)
+    const verifyResult = await handleVerification(tab, cdp, 90000);
+    if (verifyResult === 'needs-human') {
+      throw new Error('Copilot verification required — please solve it manually in the browser window');
+    }
+    
+    // After verification, page may have redirected or reloaded — wait for it to settle
+    if (verifyResult === 'clicked') {
+      await new Promise(r => setTimeout(r, 3000));
+      
+      // Re-navigate if we got redirected
+      const currentUrl = await cdp(['eval', tab, 'document.location.href']).catch(() => '');
+      if (!currentUrl.includes('copilot.microsoft.com')) {
+        await cdp(['nav', tab, 'https://copilot.microsoft.com/'], 35000);
+        await new Promise(r => setTimeout(r, 2000));
+        await dismissConsent(tab, cdp);
+      }
+    }
+
+    // Wait for React app to mount input (up to 15s, longer after verification)
+    const inputDeadline = Date.now() + 15000;
+    while (Date.now() < inputDeadline) {
+      const found = await cdp(['eval', tab, `!!document.querySelector('${S.input}')`]).catch(() => 'false');
+      if (found === 'true') break;
+      await new Promise(r => setTimeout(r, 500));
+    }
+    await new Promise(r => setTimeout(r, 300));
+
+    // Verify input is actually there before proceeding
+    const inputReady = await cdp(['eval', tab, `!!document.querySelector('${S.input}')`]).catch(() => 'false');
+    if (inputReady !== 'true') {
+      throw new Error('Copilot input not found — verification may have failed or page is in unexpected state');
+    }
+
+    await injectClipboardInterceptor(tab);
+    await cdp(['click', tab, S.input]);
+    await new Promise(r => setTimeout(r, 400));
+    await cdp(['type', tab, query]);
+    await new Promise(r => setTimeout(r, 400));
+
+    // Submit with Enter (most reliable across locales and Chrome instances)
+    await cdp(['eval', tab,
+      `document.querySelector('${S.input}')?.dispatchEvent(new KeyboardEvent('keydown',{key:'Enter',bubbles:true,keyCode:13})), 'ok'`
+    ]);
+
+    await waitForCopyButton(tab);
+
+    const { answer, sources } = await extractAnswer(tab);
+    if (!answer) throw new Error('No answer extracted — Copilot may not have responded');
+    const out = short ? answer.slice(0, 300).replace(/\s+\S*$/, '') + '…' : answer;
+
+    const finalUrl = await cdp(['eval', tab, 'document.location.href']).catch(() => '');
+    process.stdout.write(JSON.stringify({ query, url: finalUrl, answer: out, sources }, null, 2) + '\n');
+  } catch (e) {
+    process.stderr.write(`Error: ${e.message}\n`);
+    process.exit(1);
+  }
+}
+
+main();