@bacnh85/pi-web 0.7.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/README.md +60 -2
- package/extensions/index.ts +63 -0
- package/extensions/lib/gemini.ts +339 -0
- package/package.json +6 -3
- package/skills/pi-web/SKILL.md +8 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,28 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.0 (2026-09-13)
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- **`web_research` tool** — AI-synthesized web research via Gemini's web tier
|
|
8
|
+
(gemini.google.com), cookie-authed with `__Secure-1PSID`.
|
|
9
|
+
`mode: "ask"` returns a quick grounded answer with extracted source links
|
|
10
|
+
(works in guest mode without any cookie, Flash-only);
|
|
11
|
+
`mode: "research"` runs Gemini **Deep Research** — an autonomous agent that
|
|
12
|
+
browses the web for minutes and returns a comprehensive report (requires the
|
|
13
|
+
cookie and a Gemini Advanced subscription; default timeout 10 min, cap 30).
|
|
14
|
+
- New `lib/gemini.ts` wrapper over the `gemini-reverse` npm package (lazy
|
|
15
|
+
dynamic import, injectable client for tests, one AuthError retry that
|
|
16
|
+
re-absorbs rotated Set-Cookies). Sources are extracted from markdown links
|
|
17
|
+
in the answer/report text (the web protocol exposes no structured citations).
|
|
18
|
+
- `web_status` now reports `geminiWeb` (configured/cookieSource/proxy).
|
|
19
|
+
- Env config: `GEMINI_WEB_SECURE_1PSID` (required for authed/research mode),
|
|
20
|
+
optional `GEMINI_WEB_PROXY` (escape hatch if Google blocks the IP).
|
|
21
|
+
- **Header-cap fix (authed mode)** — Google ships ~25 KB of response headers on
|
|
22
|
+
Gemini pages (giant `content-security-policy`), over Node's default 16 KB
|
|
23
|
+
parser cap; the wrapper injects a per-request `maxHeaderSize` for
|
|
24
|
+
gemini.google.com hosts only (lazy, no global flag needed).
|
|
25
|
+
|
|
3
26
|
## 0.7.1 (2026-09-12)
|
|
4
27
|
|
|
5
28
|
### Fixed
|
package/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# @bacnh85/pi-web
|
|
2
2
|
|
|
3
|
-
Pi extension for **unified web search, content extraction, site crawling, and
|
|
3
|
+
Pi extension for **unified web search, content extraction, site crawling, page capture, and Gemini web-tier research**.
|
|
4
4
|
|
|
5
|
-
Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, Crawl4AI, and agy (Gemini/Claude, when installed) — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
|
|
5
|
+
Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, Crawl4AI, and agy (Gemini/Claude, when installed) — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave. `web_research` adds AI-synthesized research with citations via your gemini.google.com session.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -30,10 +30,13 @@ Variables:
|
|
|
30
30
|
| `FIRECRAWL_API_KEY` | No (2) | — | Required for hosted Firecrawl |
|
|
31
31
|
| `CRAWL4AI_API_URL` | No | `http://127.0.0.1:11235` | Self-hosted Crawl4AI |
|
|
32
32
|
| `CRAWL4AI_API_TOKEN` | No (3) | — | Required if Crawl4AI auth enabled |
|
|
33
|
+
| `GEMINI_WEB_SECURE_1PSID` | No (4) | — | `__Secure-1PSID` cookie from gemini.google.com — enables authed `web_research` (Deep Research) |
|
|
34
|
+
| `GEMINI_WEB_PROXY` | No | — | Proxy URL for Gemini web calls (escape hatch if Google blocks the IP) |
|
|
33
35
|
|
|
34
36
|
> (1) At least one search backend (SearXNG, Brave, or Firecrawl) must be configured for `web_search`.
|
|
35
37
|
> (2) Required for hosted Firecrawl; optional for self-hosted instances without auth.
|
|
36
38
|
> (3) Required for Crawl4AI v0.9+ default config.
|
|
39
|
+
> (4) Without it `web_research mode=ask` still works in guest mode (Flash-only); `mode=research` errors with setup steps.
|
|
37
40
|
|
|
38
41
|
Secrets are never printed; `web_status` reports only presence/source.
|
|
39
42
|
|
|
@@ -197,10 +200,65 @@ Typical output:
|
|
|
197
200
|
"health": { "status": "healthy", "version": "0.5.0", ... }
|
|
198
201
|
},
|
|
199
202
|
"agy": { "installed": true },
|
|
203
|
+
"geminiWeb": { "configured": true, "cookieSource": "process.env", "proxy": false },
|
|
200
204
|
"localChrome": { "path": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" }
|
|
201
205
|
}
|
|
202
206
|
```
|
|
203
207
|
|
|
208
|
+
### `web_research` — Gemini web-tier research
|
|
209
|
+
|
|
210
|
+
AI-synthesized research through your gemini.google.com session (powered by the
|
|
211
|
+
[`gemini-reverse`](https://github.com/rynn-k/Gemini-Reverse) client, lazy-loaded).
|
|
212
|
+
|
|
213
|
+
```
|
|
214
|
+
web_research(query="compare the top 3 cloud providers' AI offerings", mode="research", timeout_ms=600000)
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
- **`mode: "ask"`** (default) — quick grounded answer (Gemini auto-grounds with
|
|
218
|
+
Google Search). Works **without any cookie** in guest mode (Flash-only).
|
|
219
|
+
Sent as a temporary chat so your Gemini history stays clean.
|
|
220
|
+
- **`mode: "research"`** — full Gemini **Deep Research**: plan → autonomous web
|
|
221
|
+
browsing (minutes) → cited report. Requires the cookie **and a Gemini
|
|
222
|
+
Advanced subscription** on the account. Default timeout 600 s, cap 1 800 000.
|
|
223
|
+
|
|
224
|
+
Both modes return the text plus **Sources** — URLs extracted from the
|
|
225
|
+
answer/report markdown (the web protocol exposes no structured citations field).
|
|
226
|
+
|
|
227
|
+
Setup (authed mode):
|
|
228
|
+
|
|
229
|
+
1. Sign in at [gemini.google.com](https://gemini.google.com/).
|
|
230
|
+
2. `F12` → **Application** → **Cookies** → `https://gemini.google.com`.
|
|
231
|
+
3. Copy the `__Secure-1PSID` value into `~/.pi/agent/.env.local`:
|
|
232
|
+
|
|
233
|
+
```bash
|
|
234
|
+
GEMINI_WEB_SECURE_1PSID=your-cookie-value
|
|
235
|
+
# optional, if Google blocks your IP:
|
|
236
|
+
# GEMINI_WEB_PROXY=http://host:port
|
|
237
|
+
```
|
|
238
|
+
4. Restart pi; `web_status` shows `geminiWeb.configured: true`.
|
|
239
|
+
|
|
240
|
+
Live verification script (also proves the header-cap patch end-to-end — an
|
|
241
|
+
authed failure would surface `HPE_HEADER_OVERFLOW`):
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
npx tsx extensions/scripts/gemini-smoke.ts "test query" # ask (authed or guest)
|
|
245
|
+
npx tsx extensions/scripts/gemini-smoke.ts "topic" research # Deep Research
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
⚠️ **Unofficial, at your own risk.** Cookie auth uses your real Google session
|
|
249
|
+
against gemini.google.com's internal web API and may not comply with Google's
|
|
250
|
+
ToS; the protocol can break when Google changes it. `ask` mode errors map to
|
|
251
|
+
actionable steps (expired cookie → re-copy; IP block → set `GEMINI_WEB_PROXY`).
|
|
252
|
+
|
|
253
|
+
Troubleshooting:
|
|
254
|
+
|
|
255
|
+
- *"cookie expired or invalid"* — re-copy `__Secure-1PSID` (it rotates).
|
|
256
|
+
- *"temporarily blocked this IP"* — set `GEMINI_WEB_PROXY`.
|
|
257
|
+
- *research mode: "Unknown API error: 1184"* — on this account Deep Research
|
|
258
|
+
was rejected; usually means no Gemini Advanced subscription on the account
|
|
259
|
+
(Deep Research is Advanced-only), or Google changed the protocol. `ask`
|
|
260
|
+
mode is unaffected.
|
|
261
|
+
|
|
204
262
|
## Library structure
|
|
205
263
|
|
|
206
264
|
| Module | Contents |
|
package/extensions/index.ts
CHANGED
|
@@ -21,6 +21,12 @@ import {
|
|
|
21
21
|
formatUnifiedSearchResults,
|
|
22
22
|
} from "./lib/format";
|
|
23
23
|
import { searchWithDiagnostics } from "./lib/search";
|
|
24
|
+
import {
|
|
25
|
+
loadGeminiWebConfig,
|
|
26
|
+
geminiAsk,
|
|
27
|
+
geminiResearch,
|
|
28
|
+
describeGeminiError,
|
|
29
|
+
} from "./lib/gemini";
|
|
24
30
|
import { extractWithDiagnostics, type ExtractMode } from "./lib/extract";
|
|
25
31
|
import { firecrawlRequest, type FirecrawlResult } from "./lib/firecrawl";
|
|
26
32
|
import {
|
|
@@ -77,6 +83,7 @@ const WEB_ROUTING_GUIDANCE = `## Web Tool Routing (pi-web)
|
|
|
77
83
|
- **web_map** — discover site URLs (Firecrawl Map).
|
|
78
84
|
- **web_crawl** — multi-page crawl: \`mode: "light"\` (Firecrawl, url) or \`mode: "full"\` (Crawl4AI, urls[]).
|
|
79
85
|
- **web_screenshot** / **web_pdf** — page capture (Crawl4AI).
|
|
86
|
+
- **web_research** — AI-synthesized research via Gemini web (mode "ask" = grounded answer, guest OK; mode "research" = Deep Research report, needs cookie + Gemini Advanced, takes minutes).
|
|
80
87
|
- **web_status** — provider config + health.
|
|
81
88
|
|
|
82
89
|
Rules: Firecrawl Search is weak on domain-specific queries — prefer SearXNG/Brave; Firecrawl Scrape fails on bot-protected sites — use Crawl4AI (\`mode: "full"\`) then agy (\`mode: "agy"\`); cite source URLs.`;
|
|
@@ -440,6 +447,58 @@ export default function piWebExtension(pi: ExtensionAPI) {
|
|
|
440
447
|
},
|
|
441
448
|
});
|
|
442
449
|
|
|
450
|
+
// ── web_research ─────────────────────────────────────────────────────
|
|
451
|
+
pi.registerTool({
|
|
452
|
+
name: "web_research",
|
|
453
|
+
label: "Web Research (Gemini)",
|
|
454
|
+
description:
|
|
455
|
+
"AI-synthesized web research via Gemini (gemini.google.com web tier, cookie auth). Mode 'ask' returns a quick grounded answer with source links (works guest-mode, Flash only). Mode 'research' runs Gemini Deep Research — an autonomous agent browses the web for minutes and returns a comprehensive cited report (requires GEMINI_WEB_SECURE_1PSID cookie and a Gemini Advanced subscription).",
|
|
456
|
+
promptSnippet: "AI-synthesized research with citations",
|
|
457
|
+
promptGuidelines: [
|
|
458
|
+
"Use for AI-synthesized research with sources (mode ask = quick grounded answer; mode research = multi-minute Deep Research report). NOT for URL-list searches (web_search) or single-URL extraction (web_extract). Cite the returned source URLs.",
|
|
459
|
+
],
|
|
460
|
+
parameters: Type.Object({
|
|
461
|
+
query: Type.String({ description: "Research question or topic." }),
|
|
462
|
+
mode: Type.Optional(Type.Union(
|
|
463
|
+
[Type.Literal("ask"), Type.Literal("research")],
|
|
464
|
+
{ default: "ask", description: "ask = quick grounded answer (guest OK); research = full Deep Research report (cookie + Gemini Advanced required, takes minutes)." },
|
|
465
|
+
)),
|
|
466
|
+
model: Type.Optional(Type.String({ description: "Gemini model for ask mode (e.g. gemini-3-flash). Discovered from the account by default." })),
|
|
467
|
+
...sharedControlSchema,
|
|
468
|
+
}),
|
|
469
|
+
async execute(_id: string, params: Record<string, unknown>, signal: AbortSignal, _onUpdate: unknown, ctx: any) {
|
|
470
|
+
const config = loadGeminiWebConfig(cwdFromContext(ctx), includeProjectEnv(ctx));
|
|
471
|
+
const mode = (params.mode as string) || "ask";
|
|
472
|
+
const query = params.query as string;
|
|
473
|
+
try {
|
|
474
|
+
if (mode === "research") {
|
|
475
|
+
const timeoutMs = Math.min(Math.max((params.timeout_ms as number) ?? 600_000, 30_000), 1_800_000);
|
|
476
|
+
const result = await geminiResearch(query, { config, timeoutMs, signal });
|
|
477
|
+
const meta = [
|
|
478
|
+
"Mode: research (Gemini Deep Research)",
|
|
479
|
+
result.title ? `Title: ${result.title}` : null,
|
|
480
|
+
result.eta ? `ETA: ${result.eta}` : null,
|
|
481
|
+
].filter(Boolean).join("\n");
|
|
482
|
+
const sources = result.sources.length ? result.sources.map((s, i) => `${i + 1}. ${s}`).join("\n") : "(none found in report text)";
|
|
483
|
+
const text = `${meta}\n\n${result.text}\n\n--- Sources (extracted from report) ---\n${sources}`;
|
|
484
|
+
return { content: [{ type: "text" as const, text: truncateText(text) }], details: { mode, ...result } };
|
|
485
|
+
}
|
|
486
|
+
const askTimeoutMs = Math.min(Math.max((params.timeout_ms as number) ?? 120_000, 30_000), 600_000);
|
|
487
|
+
const result = await geminiAsk(query, { config, model: params.model as string | undefined, timeoutMs: askTimeoutMs, signal });
|
|
488
|
+
const meta = [
|
|
489
|
+
"Mode: ask",
|
|
490
|
+
`Model: ${result.model ?? "unknown"}`,
|
|
491
|
+
result.guest ? "Guest mode (no cookie — Flash only; set GEMINI_WEB_SECURE_1PSID for full access)" : "Cookie auth",
|
|
492
|
+
].join("\n");
|
|
493
|
+
const sources = result.sources.length ? result.sources.map((s, i) => `${i + 1}. ${s}`).join("\n") : "(none found in answer text)";
|
|
494
|
+
const text = `${meta}\n\n${result.text}\n\n--- Sources (extracted from answer) ---\n${sources}`;
|
|
495
|
+
return { content: [{ type: "text" as const, text: truncateText(text) }], details: { mode, ...result } };
|
|
496
|
+
} catch (err) {
|
|
497
|
+
throw new Error(describeGeminiError(err));
|
|
498
|
+
}
|
|
499
|
+
},
|
|
500
|
+
});
|
|
501
|
+
|
|
443
502
|
// ── web_status ───────────────────────────────────────────────────────
|
|
444
503
|
pi.registerTool({
|
|
445
504
|
name: "web_status",
|
|
@@ -484,6 +543,10 @@ export default function piWebExtension(pi: ExtensionAPI) {
|
|
|
484
543
|
apiTokenSource: c4aiToken.value ? c4aiToken.source : "not set",
|
|
485
544
|
},
|
|
486
545
|
agy: { installed: isAgyInstalled() },
|
|
546
|
+
geminiWeb: (() => {
|
|
547
|
+
const cfg = loadGeminiWebConfig(cwd, trusted);
|
|
548
|
+
return { configured: Boolean(cfg.psid), cookieSource: cfg.psidSource, proxy: Boolean(cfg.proxy) };
|
|
549
|
+
})(),
|
|
487
550
|
localChrome: { path: findChromeBinary() ?? "not found" },
|
|
488
551
|
};
|
|
489
552
|
|
|
@@ -0,0 +1,339 @@
|
|
|
1
|
+
// Gemini web-tier (gemini.google.com) research backend, cookie-authed via the
|
|
2
|
+
// __Secure-1PSID cookie. Thin wrapper over the `gemini-reverse` npm package
|
|
3
|
+
// (CJS), lazily dynamic-imported so pi startup pays zero cost when unused.
|
|
4
|
+
|
|
5
|
+
import http from "node:http";
|
|
6
|
+
import https from "node:https";
|
|
7
|
+
import { urlToHttpOptions } from "node:url";
|
|
8
|
+
import { findEnvValue } from "./config";
|
|
9
|
+
|
|
10
|
+
// ---------------------------------------------------------------------------
|
|
11
|
+
// Config
|
|
12
|
+
// ---------------------------------------------------------------------------
|
|
13
|
+
|
|
14
|
+
export interface GeminiWebConfig {
|
|
15
|
+
psid?: string;
|
|
16
|
+
psidSource: string;
|
|
17
|
+
proxy?: string;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export function loadGeminiWebConfig(cwd = process.cwd(), includeCwdEnv = false): GeminiWebConfig {
|
|
21
|
+
const psid = findEnvValue("GEMINI_WEB_SECURE_1PSID", cwd, includeCwdEnv);
|
|
22
|
+
const proxy = findEnvValue("GEMINI_WEB_PROXY", cwd, includeCwdEnv);
|
|
23
|
+
return { psid: psid.value, psidSource: psid.value ? psid.source : "not set", proxy: proxy.value };
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
// ---------------------------------------------------------------------------
|
|
27
|
+
// Minimal structural view of the gemini-reverse client (keeps tests injectable
|
|
28
|
+
// and decouples us from upstream type drift).
|
|
29
|
+
// ---------------------------------------------------------------------------
|
|
30
|
+
|
|
31
|
+
export interface GeminiOutputLike {
|
|
32
|
+
text?: string | null;
|
|
33
|
+
model?: string;
|
|
34
|
+
candidates?: Array<{ text?: string | null }>;
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
export interface GeminiResearchLike {
|
|
38
|
+
text?: string | null;
|
|
39
|
+
plan?: { title?: string | null; eta_text?: string | null };
|
|
40
|
+
statuses?: Array<Record<string, unknown>>;
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
export interface GeminiClientLike {
|
|
44
|
+
ask(prompt: string, opts?: Record<string, unknown>): Promise<GeminiOutputLike>;
|
|
45
|
+
research(
|
|
46
|
+
prompt: string,
|
|
47
|
+
opts?: { wait?: boolean; pollInterval?: number; timeout?: number; onStatus?: (s: Record<string, unknown>) => void },
|
|
48
|
+
): Promise<GeminiResearchLike>;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
export type GeminiClientFactory = (
|
|
52
|
+
opts: { secure_1psid?: string; proxy?: string },
|
|
53
|
+
) => GeminiClientLike | Promise<GeminiClientLike>;
|
|
54
|
+
|
|
55
|
+
// Cached per config (psid|proxy) so a config change re-creates the client.
|
|
56
|
+
let cached: { key: string; client: GeminiClientLike } | null = null;
|
|
57
|
+
|
|
58
|
+
// Google ships ~25KB of response headers on every Gemini page (a 19.7KB
|
|
59
|
+
// content-security-policy + 3.9KB reporting-endpoints) — over Node's default
|
|
60
|
+
// 16KB parser cap, so the http parser rejects them (HPE_HEADER_OVERFLOW; the
|
|
61
|
+
// same cap exists in undici, but gemini-reverse uses axios's node http adapter).
|
|
62
|
+
// Node honors a per-request maxHeaderSize override, but axios doesn't forward
|
|
63
|
+
// it — so lazily patch http.request/https.request to inject it for
|
|
64
|
+
// gemini.google.com hosts only.
|
|
65
|
+
// ponytail: process-wide patch, scoped to one hostname; if it ever misbehaves,
|
|
66
|
+
// revert to launching pi with NODE_OPTIONS=--max-http-header-size=262144.
|
|
67
|
+
|
|
68
|
+
/**
|
|
69
|
+
* Returns the request options to pass through with the cap injected when the
|
|
70
|
+
* target host is gemini.google.com, or null when the call must pass through
|
|
71
|
+
* untouched. Normalizes all http.request input forms (options object, string,
|
|
72
|
+
* URL) — string/URL forms become a fresh options object.
|
|
73
|
+
*
|
|
74
|
+
* @internal exported for tests
|
|
75
|
+
*/
|
|
76
|
+
export function injectGeminiHeaderCap(options: unknown): Record<string, unknown> | null {
|
|
77
|
+
let opts: Record<string, unknown>;
|
|
78
|
+
if (typeof options === "string") {
|
|
79
|
+
opts = urlToHttpOptions(new URL(options)) as Record<string, unknown>;
|
|
80
|
+
} else if (options instanceof URL) {
|
|
81
|
+
opts = urlToHttpOptions(options) as Record<string, unknown>;
|
|
82
|
+
} else if (options && typeof options === "object") {
|
|
83
|
+
opts = options as Record<string, unknown>;
|
|
84
|
+
} else {
|
|
85
|
+
return null;
|
|
86
|
+
}
|
|
87
|
+
const host = String(opts.hostname ?? opts.host ?? "").split(":")[0];
|
|
88
|
+
if (host !== "gemini.google.com" || opts.maxHeaderSize) return null;
|
|
89
|
+
opts.maxHeaderSize = 256 * 1024;
|
|
90
|
+
return opts;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
let headerCapPatched = false;
|
|
94
|
+
function patchHeaderCap(): void {
|
|
95
|
+
if (headerCapPatched) return;
|
|
96
|
+
headerCapPatched = true;
|
|
97
|
+
for (const mod of [http, https]) {
|
|
98
|
+
const real = mod.request as unknown as (...args: unknown[]) => unknown;
|
|
99
|
+
const patched = function (this: unknown, options: unknown, ...rest: unknown[]) {
|
|
100
|
+
try {
|
|
101
|
+
const override = injectGeminiHeaderCap(options);
|
|
102
|
+
if (override) return real.call(this, override, ...rest);
|
|
103
|
+
} catch { /* malformed input — let the real request surface the error */ }
|
|
104
|
+
return real.call(this, options, ...rest);
|
|
105
|
+
} as typeof mod.request;
|
|
106
|
+
mod.request = patched;
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
/** @internal exported for tests — resolves the real gemini-reverse module */
|
|
111
|
+
export async function loadDefaultFactory(): Promise<GeminiClientFactory> {
|
|
112
|
+
patchHeaderCap();
|
|
113
|
+
const mod = (await import("gemini-reverse")) as unknown as Record<string, unknown>;
|
|
114
|
+
// CJS interop: named export usually works, but fall back to default.Gemini.
|
|
115
|
+
const Gemini = (mod.Gemini ?? (mod as { default?: Record<string, unknown> }).default?.Gemini) as
|
|
116
|
+
| (new (opts: Record<string, unknown>) => GeminiClientLike)
|
|
117
|
+
| undefined;
|
|
118
|
+
if (typeof Gemini !== "function") {
|
|
119
|
+
throw new Error("gemini-reverse: Gemini export not found (unexpected package shape)");
|
|
120
|
+
}
|
|
121
|
+
// ponytail: generous per-request cap (covers research); per-mode ask/research
|
|
122
|
+
// timeouts are enforced by raceGuard below.
|
|
123
|
+
return (opts) => new Gemini({ secure_1psid: opts.secure_1psid, proxy: opts.proxy ?? null, timeout: 1_800_000 });
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
async function getClient(config: GeminiWebConfig, factory?: GeminiClientFactory): Promise<GeminiClientLike> {
|
|
127
|
+
const key = `${config.psid ?? ""}|${config.proxy ?? ""}`;
|
|
128
|
+
if (cached?.key === key) return cached.client;
|
|
129
|
+
const make = factory ?? (await loadDefaultFactory());
|
|
130
|
+
const client = await make({ secure_1psid: config.psid, proxy: config.proxy });
|
|
131
|
+
cached = { key, client };
|
|
132
|
+
return client;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/** @internal test hook */
|
|
136
|
+
export function __resetGeminiClientCache(): void {
|
|
137
|
+
cached = null;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
function errorName(err: unknown): string {
|
|
141
|
+
const e = err as { name?: string; constructor?: { name?: string } } | null;
|
|
142
|
+
const name = e?.name;
|
|
143
|
+
// Subclasses that don't set this.name inherit the generic "Error" — prefer
|
|
144
|
+
// the constructor name in that case (how we detect upstream error classes).
|
|
145
|
+
if (name && name !== "Error") return name;
|
|
146
|
+
return e?.constructor?.name ?? name ?? "";
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
function isAuthError(err: unknown): boolean {
|
|
150
|
+
return errorName(err) === "AuthError";
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
// One AuthError retry: re-creating the client re-runs init, which absorbs the
|
|
154
|
+
// rotated Set-Cookies (incl. __Secure-1PSIDTS) Google hands back.
|
|
155
|
+
export async function withGeminiClient<T>(
|
|
156
|
+
config: GeminiWebConfig,
|
|
157
|
+
run: (client: GeminiClientLike) => Promise<T>,
|
|
158
|
+
factory?: GeminiClientFactory,
|
|
159
|
+
): Promise<T> {
|
|
160
|
+
try {
|
|
161
|
+
return await run(await getClient(config, factory));
|
|
162
|
+
} catch (err) {
|
|
163
|
+
if (isAuthError(err) && config.psid) {
|
|
164
|
+
cached = null;
|
|
165
|
+
return run(await getClient(config, factory));
|
|
166
|
+
}
|
|
167
|
+
throw err;
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
// ponytail: gemini-reverse polls aren't cancellable — abort/timeout rejects the
|
|
172
|
+
// tool call promptly, but the underlying client poll finishes/times out in the
|
|
173
|
+
// background (ceiling; real cancellation needs upstream AbortSignal support).
|
|
174
|
+
function raceGuard<T>(
|
|
175
|
+
promise: Promise<T>,
|
|
176
|
+
opts: { signal?: AbortSignal; timeoutMs?: number; label: string },
|
|
177
|
+
): Promise<T> {
|
|
178
|
+
const { signal, timeoutMs, label } = opts;
|
|
179
|
+
if (!signal && !timeoutMs) return promise;
|
|
180
|
+
return new Promise<T>((resolve, reject) => {
|
|
181
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
182
|
+
const onAbort = () => {
|
|
183
|
+
clearTimeout(timer);
|
|
184
|
+
const err = new Error(`${label} aborted`);
|
|
185
|
+
err.name = "AbortError";
|
|
186
|
+
reject(err);
|
|
187
|
+
};
|
|
188
|
+
const onTimeout = () => {
|
|
189
|
+
signal?.removeEventListener("abort", onAbort);
|
|
190
|
+
const err = new Error(`${label} timed out after ${timeoutMs}ms`);
|
|
191
|
+
err.name = "TimeoutError";
|
|
192
|
+
reject(err);
|
|
193
|
+
};
|
|
194
|
+
if (signal) {
|
|
195
|
+
if (signal.aborted) return onAbort();
|
|
196
|
+
signal.addEventListener("abort", onAbort, { once: true });
|
|
197
|
+
}
|
|
198
|
+
if (timeoutMs) timer = setTimeout(onTimeout, timeoutMs);
|
|
199
|
+
promise.then(
|
|
200
|
+
(v) => {
|
|
201
|
+
signal?.removeEventListener("abort", onAbort);
|
|
202
|
+
clearTimeout(timer);
|
|
203
|
+
resolve(v);
|
|
204
|
+
},
|
|
205
|
+
(e) => {
|
|
206
|
+
signal?.removeEventListener("abort", onAbort);
|
|
207
|
+
clearTimeout(timer);
|
|
208
|
+
reject(e);
|
|
209
|
+
},
|
|
210
|
+
);
|
|
211
|
+
});
|
|
212
|
+
}
|
|
213
|
+
|
|
214
|
+
// ---------------------------------------------------------------------------
|
|
215
|
+
// Source extraction (parsed web output has no structured citations field —
|
|
216
|
+
// URLs are pulled from the answer/report markdown text)
|
|
217
|
+
// ---------------------------------------------------------------------------
|
|
218
|
+
|
|
219
|
+
const MD_LINK_RE = /\[[^\]]*\]\((https?:\/\/[^)\s]+)\)/g;
|
|
220
|
+
const BARE_URL_RE = /https?:\/\/[^\s<>()\[\]{}"'`]+/g;
|
|
221
|
+
|
|
222
|
+
export function extractSources(text: string, cap = 30): string[] {
|
|
223
|
+
if (!text) return [];
|
|
224
|
+
const seen = new Set<string>();
|
|
225
|
+
const sources: string[] = [];
|
|
226
|
+
const push = (raw: string) => {
|
|
227
|
+
const url = raw.replace(/[.,;:!?)\]]+$/, "");
|
|
228
|
+
if (!seen.has(url)) {
|
|
229
|
+
seen.add(url);
|
|
230
|
+
sources.push(url);
|
|
231
|
+
}
|
|
232
|
+
};
|
|
233
|
+
for (const m of text.matchAll(MD_LINK_RE)) push(m[1]);
|
|
234
|
+
for (const m of text.matchAll(BARE_URL_RE)) push(m[0]);
|
|
235
|
+
return sources.slice(0, cap);
|
|
236
|
+
}
|
|
237
|
+
|
|
238
|
+
// ---------------------------------------------------------------------------
|
|
239
|
+
// Error mapping
|
|
240
|
+
// ---------------------------------------------------------------------------
|
|
241
|
+
|
|
242
|
+
export function describeGeminiError(err: unknown): string {
|
|
243
|
+
switch (errorName(err)) {
|
|
244
|
+
case "AuthError":
|
|
245
|
+
return "Gemini web cookie expired or invalid. Re-copy __Secure-1PSID from gemini.google.com (F12 → Application → Cookies) into GEMINI_WEB_SECURE_1PSID in ~/.pi/agent/.env.local, then restart pi.";
|
|
246
|
+
case "UsageLimitExceeded":
|
|
247
|
+
return "Gemini web usage limit reached. Try again later or pick a different model.";
|
|
248
|
+
case "TemporarilyBlocked":
|
|
249
|
+
return "Gemini web temporarily blocked this IP. Wait a while or set GEMINI_WEB_PROXY (e.g. http://host:port).";
|
|
250
|
+
case "ModelInvalid":
|
|
251
|
+
return "Gemini model unavailable for this account. Try another model or drop the model parameter.";
|
|
252
|
+
case "AbortError":
|
|
253
|
+
case "TimeoutError":
|
|
254
|
+
return err instanceof Error ? err.message : String(err);
|
|
255
|
+
default: {
|
|
256
|
+
const msg = err instanceof Error ? err.message : String(err);
|
|
257
|
+
if (/Unknown API error/.test(msg)) {
|
|
258
|
+
return `Gemini web rejected the request (${msg}). In research mode this usually means the account lacks a Gemini Advanced subscription (Deep Research is Advanced-only) or the web protocol changed. mode=ask still works.`;
|
|
259
|
+
}
|
|
260
|
+
return `Gemini web error: ${msg}`;
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
// ---------------------------------------------------------------------------
|
|
266
|
+
// Public API
|
|
267
|
+
// ---------------------------------------------------------------------------
|
|
268
|
+
|
|
269
|
+
export interface GeminiAskResult {
|
|
270
|
+
text: string;
|
|
271
|
+
model?: string;
|
|
272
|
+
guest: boolean;
|
|
273
|
+
sources: string[];
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
export async function geminiAsk(
|
|
277
|
+
query: string,
|
|
278
|
+
opts: { config: GeminiWebConfig; model?: string; timeoutMs?: number; signal?: AbortSignal; factory?: GeminiClientFactory },
|
|
279
|
+
): Promise<GeminiAskResult> {
|
|
280
|
+
const out = await raceGuard(
|
|
281
|
+
withGeminiClient(
|
|
282
|
+
opts.config,
|
|
283
|
+
(client) => client.ask(query, { temporary: true, ...(opts.model ? { model: opts.model } : {}) }),
|
|
284
|
+
opts.factory,
|
|
285
|
+
),
|
|
286
|
+
{ signal: opts.signal, timeoutMs: opts.timeoutMs ?? 120_000, label: "web_research ask" },
|
|
287
|
+
);
|
|
288
|
+
const text = String(out?.text ?? out?.candidates?.[0]?.text ?? "");
|
|
289
|
+
return { text, model: out?.model, guest: !opts.config.psid, sources: extractSources(text) };
|
|
290
|
+
}
|
|
291
|
+
|
|
292
|
+
export interface GeminiResearchResult {
|
|
293
|
+
text: string;
|
|
294
|
+
title?: string | null;
|
|
295
|
+
eta?: string | null;
|
|
296
|
+
guest: boolean;
|
|
297
|
+
sources: string[];
|
|
298
|
+
}
|
|
299
|
+
|
|
300
|
+
export async function geminiResearch(
|
|
301
|
+
query: string,
|
|
302
|
+
opts: {
|
|
303
|
+
config: GeminiWebConfig;
|
|
304
|
+
timeoutMs?: number;
|
|
305
|
+
signal?: AbortSignal;
|
|
306
|
+
onStatus?: (s: Record<string, unknown>) => void;
|
|
307
|
+
factory?: GeminiClientFactory;
|
|
308
|
+
},
|
|
309
|
+
): Promise<GeminiResearchResult> {
|
|
310
|
+
if (!opts.config.psid) {
|
|
311
|
+
throw new Error(
|
|
312
|
+
"Deep Research requires GEMINI_WEB_SECURE_1PSID (gemini.google.com cookie) in ~/.pi/agent/.env.local — guest mode does not support it. Deep Research also needs a Gemini Advanced subscription on the account.",
|
|
313
|
+
);
|
|
314
|
+
}
|
|
315
|
+
const timeoutMs = opts.timeoutMs ?? 600_000;
|
|
316
|
+
const result = await raceGuard(
|
|
317
|
+
withGeminiClient(
|
|
318
|
+
opts.config,
|
|
319
|
+
(client) =>
|
|
320
|
+
client.research(query, {
|
|
321
|
+
wait: true,
|
|
322
|
+
pollInterval: 10_000,
|
|
323
|
+
timeout: timeoutMs,
|
|
324
|
+
...(opts.onStatus ? { onStatus: opts.onStatus } : {}),
|
|
325
|
+
}),
|
|
326
|
+
opts.factory,
|
|
327
|
+
),
|
|
328
|
+
// +5s grace so the client's own poll-timeout (better semantics) fires first.
|
|
329
|
+
{ signal: opts.signal, timeoutMs: timeoutMs + 5_000, label: "web_research research" },
|
|
330
|
+
);
|
|
331
|
+
const text = String(result?.text ?? "");
|
|
332
|
+
return {
|
|
333
|
+
text,
|
|
334
|
+
title: result?.plan?.title ?? null,
|
|
335
|
+
eta: result?.plan?.eta_text ?? null,
|
|
336
|
+
guest: false,
|
|
337
|
+
sources: extractSources(text),
|
|
338
|
+
};
|
|
339
|
+
}
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bacnh85/pi-web",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Pi extension for web search, page extraction, Firecrawl scraping/crawling,
|
|
3
|
+
"version": "0.8.0",
|
|
4
|
+
"description": "Pi extension for web search, page extraction, Firecrawl scraping/crawling, Crawl4AI headless browser crawling, and Gemini web-tier research.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"publishConfig": {
|
|
@@ -21,7 +21,9 @@
|
|
|
21
21
|
"firecrawl",
|
|
22
22
|
"crawl4ai",
|
|
23
23
|
"scraping",
|
|
24
|
-
"crawling"
|
|
24
|
+
"crawling",
|
|
25
|
+
"gemini",
|
|
26
|
+
"deep-research"
|
|
25
27
|
],
|
|
26
28
|
"scripts": {
|
|
27
29
|
"test": "cd extensions && npx mocha"
|
|
@@ -44,6 +46,7 @@
|
|
|
44
46
|
},
|
|
45
47
|
"dependencies": {
|
|
46
48
|
"@mozilla/readability": "^0.6.0",
|
|
49
|
+
"gemini-reverse": "^2.1.0",
|
|
47
50
|
"jsdom": "^27.0.1",
|
|
48
51
|
"turndown": "^7.2.2",
|
|
49
52
|
"turndown-plugin-gfm": "^1.0.2"
|
package/skills/pi-web/SKILL.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: pi-web
|
|
3
|
-
description: Web search, content extraction, site crawling,
|
|
3
|
+
description: Web search, content extraction, site crawling, page capture, and Gemini web-tier research via the pi-web extension. Use when the user needs current web search results, documentation lookup, factual research, AI-synthesized research with sources (Gemini Deep Research), source discovery, URL-to-markdown extraction, JSON extraction from websites, site URL discovery, site crawling, or page screenshots/PDFs. Use when the user mentions searching the web, finding docs, looking something up, researching deeply, scraping/extracting content from a URL, or capturing a page.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# pi-web — Unified Web Tools
|
|
7
7
|
|
|
8
|
-
Use the **
|
|
8
|
+
Use the **8 unified tools** from the `pi-web` extension for all web-related tasks. These tools automatically select the best backend from SearXNG, Brave Search, Firecrawl, Crawl4AI, and agy (when installed) — you don't need to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive queries and inline content prefer Brave.
|
|
9
9
|
|
|
10
10
|
## Quick Reference
|
|
11
11
|
|
|
@@ -17,6 +17,7 @@ Use the **7 unified tools** from the `pi-web` extension for all web-related task
|
|
|
17
17
|
| `web_crawl` | Crawl multiple pages from a site | Light (Firecrawl) or Full (Crawl4AI) |
|
|
18
18
|
| `web_screenshot` | Capture page screenshot as PNG | Crawl4AI daemon (public URLs) or local headless Chrome (localhost/LAN/file URLs — auto-detected) |
|
|
19
19
|
| `web_pdf` | Generate page PDF | Crawl4AI daemon (public URLs) or local headless Chrome (localhost/LAN/file URLs — auto-detected) |
|
|
20
|
+
| `web_research` | AI-synthesized research with sources | Gemini web tier: ask = grounded answer (guest OK); research = Deep Research report (cookie + Gemini Advanced) |
|
|
20
21
|
| `web_status` | Check provider configuration and health | — |
|
|
21
22
|
|
|
22
23
|
## Decision Tree
|
|
@@ -53,6 +54,11 @@ What do you need?
|
|
|
53
54
|
├── Printable/archivable PDF of a page
|
|
54
55
|
│ → web_pdf
|
|
55
56
|
│
|
|
57
|
+
├── AI-synthesized research with sources (synthesis, comparisons, reports)
|
|
58
|
+
│ → web_research
|
|
59
|
+
│ ├─ quick grounded answer: mode=ask (default; guest OK, Flash-only)
|
|
60
|
+
│ └─ multi-minute Deep Research report: mode=research (needs cookie + Gemini Advanced)
|
|
61
|
+
│
|
|
56
62
|
└── Check what web tools are configured
|
|
57
63
|
→ web_status
|
|
58
64
|
```
|