pi-web-access 0.4.2

package/CHANGELOG.md

@@ -0,0 +1,96 @@
+# Pi Web Access - Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.4.2] - 2026-01-27
+
+### Fixed
+
+- Single-URL fetches now store content for retrieval via `get_search_content` (previously only multi-URL)
+- Corrected `get_search_content` usage syntax in fetch_content help messages
+
+### Changed
+
+- Increased inline content limit from 10K to 30K chars (larger content truncated but fully retrievable)
+
+### Added
+
+- Banner image for README
+
+## [0.4.1] - 2026-01-26
+
+### Changed
+- Added `pi` manifest to package.json for pi v0.50.0 package system compliance
+- Added `pi-package` keyword for npm discoverability
+
+## [0.4.0] - 2026-01-19
+
+### Added
+
+- PDF extraction via `unpdf` - fetches PDFs from URLs and saves as markdown to `~/Downloads/`
+  - Extracts text, metadata (title, author), page count
+  - Supports PDFs up to 20MB (vs 5MB for HTML)
+  - Handles arxiv URLs with smart title fallback
+
+### Fixed
+
+- Plain text URL detection now uses hostname check instead of substring match
+
+## [0.3.0] - 2026-01-19
+
+### Added
+
+- RSC (React Server Components) content extraction for Next.js App Router pages
+  - Parses flight data from `<script>self.__next_f.push([...])</script>` tags
+  - Reconstructs markdown with headings, tables, code blocks, links
+  - Handles chunk references and nested components
+  - Falls back to RSC extraction when Readability fails
+- Content-type validation rejects binary files (images, PDFs, audio, video, zip)
+- 5MB response size limit (checked via Content-Length header) to prevent memory issues
+
+### Fixed
+
+- `fetch_content` now handles plain text URLs (raw.githubusercontent.com, gist.githubusercontent.com, any text/plain response) instead of failing with "Could not extract readable content"
+
+## [0.2.0] - 2026-01-11
+
+### Added
+
+- Activity monitor widget (`Ctrl+Shift+O`) showing live request/response activity
+  - Displays last 10 API calls and URL fetches with status codes and timing
+  - Shows rate limit usage and reset countdown
+  - Live updates as requests complete
+  - Auto-clears on session switch
+
+### Changed
+
+- Refactored activity tracking into dedicated `activity.ts` module
+
+## [0.1.0] - 2026-01-06
+
+Initial release. Designed for pi v0.37.3.
+
+### Added
+
+- `web_search` tool - Search via Perplexity AI with synthesized answers and citations
+  - Single or batch queries (parallel execution)
+  - Recency filter (day/week/month/year)
+  - Domain filter (include or exclude)
+  - Optional async content fetching with agent notification
+- `fetch_content` tool - Fetch and extract readable content from URLs
+  - Single URL returns content directly
+  - Multiple URLs store for retrieval via `get_search_content`
+  - Concurrent fetching (3 max) with 30s timeout
+- `get_search_content` tool - Retrieve stored search results or fetched content
+  - Access by response ID, URL, query, or index
+- `/search` command - Interactive browser for stored results
+- TUI rendering with progress bars, URL lists, and expandable previews
+- Session-aware storage with 1-hour TTL
+- Rate limiting (10 req/min for Perplexity API)
+- Config file support (`~/.pi/web-search.json`)
+- Content extraction via Readability + Turndown (max 10k chars)
+- Proper session isolation - pending fetches abort on session switch
+- URL validation before fetch attempts
+- Defensive JSON parsing for API responses

package/README.md

@@ -0,0 +1,179 @@
+<p>
+  <img src="banner.png" alt="pi-web-access" width="1100">
+</p>
+
+# Pi Web Access
+
+An extension for [Pi coding agent](https://github.com/badlogic/pi-mono/) that gives Pi web capabilities: search via Perplexity AI, fetch and extract content from URLs, and read PDFs.
+
+```typescript
+web_search({ query: "TypeScript best practices 2025" })
+fetch_content({ url: "https://docs.example.com/guide" })
+```
+
+## Install
+
+```bash
+# Clone to extensions directory
+git clone https://github.com/nicobailon/pi-web-access ~/.pi/agent/extensions/pi-web-access
+cd ~/.pi/agent/extensions/pi-web-access
+npm install
+```
+
+Add your Perplexity API key:
+
+```bash
+# Option 1: Environment variable
+export PERPLEXITY_API_KEY="pplx-..."
+
+# Option 2: Config file
+echo '{"perplexityApiKey": "pplx-..."}' > ~/.pi/web-search.json
+```
+
+Get a key at https://perplexity.ai/settings/api
+
+**Requires:** Pi v0.37.3+
+
+## Tools
+
+### web_search
+
+Search the web via Perplexity AI. Returns synthesized answer with source citations.
+
+```typescript
+// Single query
+web_search({ query: "rust async programming" })
+
+// Multiple queries (parallel)
+web_search({ queries: ["query 1", "query 2"] })
+
+// With options
+web_search({
+  query: "latest news",
+  numResults: 10,              // Default: 5, max: 20
+  recencyFilter: "week",       // day, week, month, year
+  domainFilter: ["github.com"] // Prefix with - to exclude
+})
+
+// Fetch full page content (async)
+web_search({ query: "...", includeContent: true })
+```
+
+When `includeContent: true`, sources are fetched in the background. Agent receives notification when ready.
+
+### fetch_content
+
+Fetch URL(s) and extract readable content as markdown.
+
+```typescript
+// Single URL - returns content directly (also stored for retrieval)
+fetch_content({ url: "https://example.com/article" })
+
+// Multiple URLs - returns summary (content stored for retrieval)
+fetch_content({ urls: ["url1", "url2", "url3"] })
+
+// PDFs - extracted and saved to ~/Downloads/
+fetch_content({ url: "https://arxiv.org/pdf/1706.03762" })
+// → "PDF extracted and saved to: ~/Downloads/arxiv-170603762.md"
+```
+
+**PDF handling:** When fetching a PDF URL, the extension extracts text and saves it as a markdown file in `~/Downloads/`. The agent can then use `read` to access specific sections without loading 200K+ chars into context.
+
+### get_search_content
+
+Retrieve stored content from previous searches or fetches.
+
+```typescript
+// By response ID (from web_search or fetch_content)
+get_search_content({ responseId: "abc123", urlIndex: 0 })
+
+// By URL
+get_search_content({ responseId: "abc123", url: "https://..." })
+
+// By query (for search results)
+get_search_content({ responseId: "abc123", query: "original query" })
+```
+
+## Features
+
+### Activity Monitor (Ctrl+Shift+O)
+
+Toggle live request/response activity:
+
+```
+─── Web Search Activity ────────────────────────────────────
+  API  "typescript best practices"     200    2.1s ✓
+  GET  docs.example.com/article        200    0.8s ✓
+  GET  blog.example.com/post           404    0.3s ✗
+  GET  news.example.com/latest         ...    1.2s ⋯
+────────────────────────────────────────────────────────────
+Rate: 3/10 (resets in 42s)
+```
+
+### RSC Content Extraction
+
+Next.js App Router pages embed content as RSC (React Server Components) flight data in script tags. When Readability fails, the extension parses these JSON payloads directly, reconstructing markdown with headings, tables, code blocks, and links.
+
+### TUI Rendering
+
+Tool calls render with real-time progress:
+
+```
+┌─ search "TypeScript best practices 2025" ─────────────────────────┐
+│ [████████░░] searching                                            │
+└───────────────────────────────────────────────────────────────────┘
+```
+
+## Commands
+
+### /search
+
+Browse stored search results interactively.
+
+## How It Works
+
+```
+Agent Request → Perplexity API → Synthesized Answer + Citations
+                                         ↓
+                              [if includeContent: true]
+                                         ↓
+                              Background Fetch (3 concurrent)
+                                         ↓
+                        ┌────────────────┼────────────────┐
+                        ↓                ↓                ↓
+                       PDF          HTML/Text          RSC
+                        ↓                ↓                ↓
+                   unpdf →        Readability →    RSC Parser →
+                 Save to file      Markdown          Markdown
+                        ↓                ↓                ↓
+                        └────────────────┼────────────────┘
+                                         ↓
+                              Agent Notification (triggerTurn)
+```
+
+## Rate Limits
+
+- **Perplexity API**: 10 requests/minute (enforced client-side)
+- **Content Fetch**: 3 concurrent requests, 30s timeout per URL
+- **Cache TTL**: 1 hour
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `index.ts` | Extension entry, tool definitions, commands, widget |
+| `perplexity.ts` | Perplexity API client, rate limiting |
+| `extract.ts` | URL fetching, content extraction routing |
+| `pdf-extract.ts` | PDF text extraction, saves to markdown |
+| `rsc-extract.ts` | RSC flight data parser for Next.js pages |
+| `storage.ts` | Session-aware result storage |
+| `activity.ts` | Activity tracking for observability widget |
+
+## Limitations
+
+- Content extraction works best on article-style pages
+- Heavy JS sites may not extract well (no browser rendering), though Next.js App Router pages with RSC flight data are supported
+- PDFs are extracted as text (no OCR for scanned documents)
+- Max response size: 20MB for PDFs, 5MB for HTML
+- Max inline content: 30,000 chars per URL (larger content stored for retrieval via get_search_content)
+- Requires Pi restart after config file changes

package/activity.ts

@@ -0,0 +1,102 @@
+// Types
+export interface ActivityEntry {
+	id: string;
+	type: "api" | "fetch";
+	startTime: number;
+	endTime?: number;
+
+	// For API calls
+	query?: string;
+
+	// For URL fetches
+	url?: string;
+
+	// Result - status is number (HTTP code) or null (pending/network error)
+	status: number | null;
+	error?: string;
+}
+
+export interface RateLimitInfo {
+	used: number;
+	max: number;
+	oldestTimestamp: number | null;
+	windowMs: number;
+}
+
+export class ActivityMonitor {
+	private entries: ActivityEntry[] = [];
+	private readonly maxEntries = 10;
+	private listeners = new Set<() => void>();
+	private rateLimitInfo: RateLimitInfo = { used: 0, max: 10, oldestTimestamp: null, windowMs: 60000 };
+	private nextId = 1;
+
+	logStart(partial: Omit<ActivityEntry, "id" | "startTime" | "status">): string {
+		const id = `act-${this.nextId++}`;
+		const entry: ActivityEntry = {
+			...partial,
+			id,
+			startTime: Date.now(),
+			status: null,
+		};
+		this.entries.push(entry);
+		if (this.entries.length > this.maxEntries) {
+			this.entries.shift();
+		}
+		this.notify();
+		return id;
+	}
+
+	logComplete(id: string, status: number): void {
+		const entry = this.entries.find((e) => e.id === id);
+		if (entry) {
+			entry.endTime = Date.now();
+			entry.status = status;
+			this.notify();
+		}
+	}
+
+	logError(id: string, error: string): void {
+		const entry = this.entries.find((e) => e.id === id);
+		if (entry) {
+			entry.endTime = Date.now();
+			entry.error = error;
+			this.notify();
+		}
+	}
+
+	getEntries(): readonly ActivityEntry[] {
+		return this.entries;
+	}
+
+	getRateLimitInfo(): RateLimitInfo {
+		return this.rateLimitInfo;
+	}
+
+	updateRateLimit(info: RateLimitInfo): void {
+		this.rateLimitInfo = info;
+		this.notify();
+	}
+
+	onUpdate(callback: () => void): () => void {
+		this.listeners.add(callback);
+		return () => this.listeners.delete(callback);
+	}
+
+	clear(): void {
+		this.entries = [];
+		this.rateLimitInfo = { used: 0, max: 10, oldestTimestamp: null, windowMs: 60000 };
+		this.notify();
+	}
+
+	private notify(): void {
+		for (const cb of this.listeners) {
+			try {
+				cb();
+			} catch {
+				/* ignore */
+			}
+		}
+	}
+}
+
+export const activityMonitor = new ActivityMonitor();

package/extract.ts

@@ -0,0 +1,189 @@
+import { Readability } from "@mozilla/readability";
+import { parseHTML } from "linkedom";
+import TurndownService from "turndown";
+import pLimit from "p-limit";
+import { activityMonitor } from "./activity.js";
+import { extractRSCContent } from "./rsc-extract.js";
+import { extractPDFToMarkdown, isPDF } from "./pdf-extract.js";
+
+const DEFAULT_TIMEOUT_MS = 30000;
+const CONCURRENT_LIMIT = 3;
+
+const turndown = new TurndownService({
+	headingStyle: "atx",
+	codeBlockStyle: "fenced",
+});
+
+const fetchLimit = pLimit(CONCURRENT_LIMIT);
+
+export interface ExtractedContent {
+	url: string;
+	title: string;
+	content: string;
+	error: string | null;
+}
+
+export async function extractContent(
+	url: string,
+	signal?: AbortSignal,
+	timeoutMs: number = DEFAULT_TIMEOUT_MS,
+): Promise<ExtractedContent> {
+	if (signal?.aborted) {
+		return { url, title: "", content: "", error: "Aborted" };
+	}
+
+	try {
+		new URL(url);
+	} catch {
+		return { url, title: "", content: "", error: "Invalid URL" };
+	}
+
+	const activityId = activityMonitor.logStart({ type: "fetch", url });
+
+	const controller = new AbortController();
+	const timeoutId = setTimeout(() => controller.abort(), timeoutMs);
+
+	const onAbort = () => controller.abort();
+	signal?.addEventListener("abort", onAbort);
+
+	try {
+		const response = await fetch(url, {
+			signal: controller.signal,
+			headers: {
+				"User-Agent": "Mozilla/5.0 (compatible; pi-agent/1.0)",
+				Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+			},
+		});
+
+		if (!response.ok) {
+			activityMonitor.logComplete(activityId, response.status);
+			return {
+				url,
+				title: "",
+				content: "",
+				error: `HTTP ${response.status}: ${response.statusText}`,
+			};
+		}
+
+		// Check content length to avoid memory issues with huge responses
+		const contentLengthHeader = response.headers.get("content-length");
+		const contentType = response.headers.get("content-type") || "";
+		const isPDFContent = isPDF(url, contentType);
+		const maxResponseSize = isPDFContent ? 20 * 1024 * 1024 : 5 * 1024 * 1024; // 20MB for PDFs, 5MB otherwise
+		if (contentLengthHeader) {
+			const contentLength = parseInt(contentLengthHeader, 10);
+			if (contentLength > maxResponseSize) {
+				activityMonitor.logComplete(activityId, response.status);
+				return {
+					url,
+					title: "",
+					content: "",
+					error: `Response too large (${Math.round(contentLength / 1024 / 1024)}MB)`,
+				};
+			}
+		}
+
+		// Handle PDFs - extract and save to markdown file
+		if (isPDFContent) {
+			try {
+				const buffer = await response.arrayBuffer();
+				const result = await extractPDFToMarkdown(buffer, url);
+				activityMonitor.logComplete(activityId, response.status);
+				return {
+					url,
+					title: result.title,
+					content: `PDF extracted and saved to: ${result.outputPath}\n\nPages: ${result.pages}\nCharacters: ${result.chars}`,
+					error: null,
+				};
+			} catch (err) {
+				const message = err instanceof Error ? err.message : String(err);
+				activityMonitor.logError(activityId, message);
+				return { url, title: "", content: "", error: `PDF extraction failed: ${message}` };
+			}
+		}
+		
+		// Reject binary/non-text content types
+		if (contentType.includes("application/octet-stream") ||
+			contentType.includes("image/") ||
+			contentType.includes("audio/") ||
+			contentType.includes("video/") ||
+			contentType.includes("application/zip")) {
+			activityMonitor.logComplete(activityId, response.status);
+			return {
+				url,
+				title: "",
+				content: "",
+				error: `Unsupported content type: ${contentType.split(";")[0]}`,
+			};
+		}
+		
+		// Return plain text directly without Readability
+		const urlHostname = new URL(url).hostname;
+		const isPlainText = contentType.includes("text/plain") || 
+			urlHostname === "raw.githubusercontent.com" ||
+			urlHostname === "gist.githubusercontent.com";
+
+		const text = await response.text();
+
+		if (isPlainText) {
+			activityMonitor.logComplete(activityId, response.status);
+			const content = text;
+			// Extract filename from URL as title
+			const urlPath = new URL(url).pathname;
+			const title = urlPath.split("/").pop() || url;
+			return { url, title, content, error: null };
+		}
+
+		const html = text;
+		const { document } = parseHTML(html);
+
+		const reader = new Readability(document as unknown as Document);
+		const article = reader.parse();
+
+		if (!article) {
+			// Fallback: Try extracting from RSC flight data (Next.js App Router)
+			const rscResult = extractRSCContent(html);
+			if (rscResult) {
+				activityMonitor.logComplete(activityId, response.status);
+				return { url, title: rscResult.title, content: rscResult.content, error: null };
+			}
+			
+			activityMonitor.logComplete(activityId, response.status);
+			return {
+				url,
+				title: "",
+				content: "",
+				error: "Could not extract readable content",
+			};
+		}
+
+		const markdown = turndown.turndown(article.content);
+
+		activityMonitor.logComplete(activityId, response.status);
+		return {
+			url,
+			title: article.title || "",
+			content: markdown,
+			error: null,
+		};
+	} catch (err) {
+		const message = err instanceof Error ? err.message : String(err);
+		if (message.toLowerCase().includes("abort")) {
+			activityMonitor.logComplete(activityId, 0);
+		} else {
+			activityMonitor.logError(activityId, message);
+		}
+		return { url, title: "", content: "", error: message };
+	} finally {
+		clearTimeout(timeoutId);
+		signal?.removeEventListener("abort", onAbort);
+	}
+}
+
+export async function fetchAllContent(
+	urls: string[],
+	signal?: AbortSignal,
+	timeoutMs?: number,
+): Promise<ExtractedContent[]> {
+	return Promise.all(urls.map((url) => fetchLimit(() => extractContent(url, signal, timeoutMs))));
+}