@tduyng/pi-web 0.0.1 → 0.0.3

package/README.md

@@ -1,39 +1,62 @@
 # @tduyng/pi-web
 
-Web search and fetch for pi. The extension registers `web_search` and `web_fetch`, and registers the keyless Exa MCP server through pi's own MCP client: `pi.registerMcpServer('exa', ...)`, exposure `codemode`, listed in `/mcp`. The tools call that server's two tools, `web_search_exa` and `web_fetch_exa`, through `ctx.executeTool` and carry no MCP SDK of their own. The server takes a 60 second request timeout, which covers the connect handshake, and pi retries only HTTP 408, 429 and 5xx: a handshake that overruns marks the server failed for the session and `/mcp` shows why.
+Web search and fetch for pi that needs no API key, no setup, and no MCP tool pile in your context. Two tools, 326 tokens a request, done.
 
-## Backends
+## What it costs
 
-`web_search` takes an array of up to six queries, one call each, and groups the results under the query that produced them. The primary backend is keyless Exa MCP, with DuckDuckGo HTML behind it. The Exa leg goes through the registered `exa` server, so an unconnected or `--no-mcp` session lands in the same DuckDuckGo fallback. x.com and twitter.com URLs go to Fixupx instead, which answers a page X refuses. It returns a post together with the whole long-form Article it links. An `x.com/i/article/` URL carries an article id rather than a post id, so the reply names the post URL to fetch.
+`web_search` is 219 tokens a request and `web_fetch` is 107. That is the whole bill. The Exa MCP server is registered with exposure `codemode`, so its two tools never enter the prompt: a codemode script calls them through `ctx.executeTool`, and the model that never writes such a script never pays for them.
 
-`source: arxiv` searches the arXiv Atom API instead, sorted by submission date, and returns abstracts with authors and dates. A miss falls back to the open web, because arXiv indexes preprints and nothing else.
+Cold start is the same 326 tokens on every session. No key to paste, no account, no handshake, so the first search of a session is as fast as the last.
 
-`source: hn` searches Hacker News through Algolia's index, which needs no account, and returns each thread's points and comment count, the only measure of how much discussion a subject has. A comment hit is reported as the thread that holds it, with the comment's first 300 characters. A miss falls back to the open web.
+## What you get
 
-Search returns excerpts, so the result ends with the ranked URLs to read. Without that list the model answers from a snippet.
+`web_search` takes up to six queries in one call and groups the results under the query that produced them. Three backends answer, in order, and a backend that returns nothing hands over to the next one:
+
+| Backend | When | What it returns |
+| --- | --- | --- |
+| Exa MCP | always, keyless | prose answer plus highlights |
+| DuckDuckGo | Exa is unconnected, `--no-mcp`, or empty | result list |
+| arXiv Atom API | `source: arxiv` | abstracts, authors, dates, newest first |
+| Hacker News (Algolia) | `source: hn` | threads with points and comment count |
+
+`web_fetch` reads a URL as Markdown. `x.com` and `twitter.com` links go to Fixupx, which answers a page X refuses, and it returns the post together with the long-form Article it links to.
 
 ## Typed results
 
-`web_search` declares an output schema, so a codemode script gets the answer as data in `structuredContent.groups`, one entry per query carrying `title`, `url` and `snippet`, plus `published` and `authors` where the backend has them, and `points` and `comments` on Hacker News. A script can compare values across results without parsing text. Exa's answer is prose: `Title:`, `URL:`, `Published:`, `Author:` and `Highlights:` blocks with a rule between them, which `parseExa` reads back into that shape, so the typed view holds for every backend. The model still receives the text, and the data is not sent to it a second time.
+`web_search` declares an output schema, so a codemode script gets the answer as data in `structuredContent.groups`: `title`, `url`, `snippet`, plus `published` and `authors` where the backend has them, and `points` and `comments` on Hacker News. Compare values across results without parsing text. Exa's answer is prose, and `parseExa` reads it back into the same shape, so the typed view holds for every backend. The text still reaches the model and the data is not sent twice.
+
+## Search is a step, not an answer
 
-A backend that answers with nothing is a miss, and the next one runs. Before, an empty Exa answer ended the search and DuckDuckGo never got a turn.
+Results are excerpts, so the reply ends with the ranked URLs to read. Without that list the model answers from a snippet. Reddit and X have no keyless search: measured on 2026-09-27, `reddit.com/search.json` answers 403 on `www`, `old`, and `api`, with a browser user agent and a plain one. DuckDuckGo's `site:` operator does reach both, so name the forum in the query and the backend finds the thread.
 
 ## DuckDuckGo queue
 
-DuckDuckGo answers a burst of requests from one address with a bot check, and keeps that address locked out for 10 to 15 minutes, with each retry extending it. Measured on 2026-10-26: six requests at once got three pages of results and three challenge pages at 14 KB apiece, and the challenge arrives as HTTP 202 as often as a 403. Searches therefore run one at a time, two seconds apart. A bot check pauses the provider for five minutes, doubling to an hour while it keeps answering that way, and a paused search fails at once instead of waiting, so the batch reports that query as failed rather than holding the tool open.
+DuckDuckGo answers a burst from one address with a bot check and keeps that address locked out for 10 to 15 minutes, with each retry extending it. Measured on 2026-10-26: six requests at once got three pages of results and three challenge pages at 14 KB apiece, and the challenge arrives as HTTP 202 as often as 403.
+
+So searches run one at a time, two seconds apart. A bot check pauses the provider for five minutes, doubling to an hour while it keeps answering that way. A paused search fails at once instead of holding the tool open, and the batch reports that query as failed.
 
 ## Links in a fetched page
 
-Exa's extraction returns plain text. Measured on 2026-09-27, `arxiv.org/abs/1706.03762` came back with no links at all, so a citation cannot be read out of the page that cites it. Jina Reader returns Markdown with the links intact, 295 of them on a Hacker News thread, but it blocks a domain that hammered it: `arxiv.org` answered 403 until 2026-09-27. Exa answers first for that reason, and the links survive only when Jina does. Follow a citation by searching for the paper's title.
+Exa's extraction returns plain text. Measured on 2026-09-27, `arxiv.org/abs/1706.03762` came back with no links at all, so a citation cannot be read out of the page that cites it. Jina Reader returns Markdown with links intact, 295 of them on a Hacker News thread, but it blocks a domain that hammered it: `arxiv.org` answered 403 until 2026-09-27. Exa answers first for that reason, and links survive only when Jina does. Follow a citation by searching for the paper's title.
 
 ## arXiv query form
 
-arXiv ANDs nothing on its own. Measured on the live API: `typed decision model` returns 1,470,125 rows because the words are read as alternatives, and `all:"typed decision model"` returns 3 because the phrase is too strict. So a plain query has its words ANDed one at a time, which is the only form that ranks usefully. A query already written with arXiv field prefixes (`ti:`, `cat:`, `au:`, and the rest) is passed through untouched.
-
-## Reddit and X
+arXiv ANDs nothing on its own. Measured on the live API: `typed decision model` returns 1,470,125 rows because the words are read as alternatives, and `all:"typed decision model"` returns 3 because the phrase is too strict. A plain query therefore has its words ANDed one at a time, which is the only form that ranks usefully. A query already written with arXiv field prefixes (`ti:`, `cat:`, `au:`, and the rest) passes through untouched.
 
-Neither has a keyless search. Measured on 2026-09-27: `reddit.com/search.json` answers 403 on `www`, `old` and `api`, with a browser user agent and with a plain one, and X has no search without a key. DuckDuckGo's `site:` operator does return both, so a query that names the forum reaches those threads. The web backend finds them when a query names them: `Hacker News and Reddit discussion of typed decision models` returned a `news.ycombinator.com` host, while the bare query did not.
+A miss on arXiv falls back to the open web, because arXiv indexes preprints and nothing else. A miss on Hacker News does the same.
 
 ## Quota
 
-The keyless Exa MCP endpoint needs no account. Set `EXA_API_KEY` to send your own key. The extension reads it at load and puts it on the registered server URL as the `exaApiKey` query parameter. Exa's free plan includes 1,000 requests per month. A batch of six queries costs six of them, so they run three at a time.
+The keyless Exa MCP endpoint needs no account. Set `EXA_API_KEY` to send your own key: the extension reads it at load and puts it on the registered server URL as the `exaApiKey` parameter. Exa's free plan includes 1,000 requests a month, so a batch of six queries costs six of them and runs three at a time.
+
+The server takes a 60 second request timeout, which covers the connect handshake. pi retries only HTTP 408, 429, and 5xx, so a handshake that overruns marks the server failed for the session and `/mcp` shows why.
+
+## Install
+
+```bash
+pi install npm:@tduyng/pi-web
+```
+
+Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-web`
+
+Nothing else to configure.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@tduyng/pi-web",
-  "version": "0.0.1",
-  "description": "Keyless web search and fetch for pi: batched queries, arXiv preprints, Exa MCP with DuckDuckGo and Jina Reader fallbacks, zero dependencies.",
+  "version": "0.0.3",
+  "description": "Keyless web search and fetch: batched queries, arXiv, Hacker News, and Exa MCP with fallbacks. No API key, no MCP tool pile in context.",
   "keywords": [
     "pi",
     "pi-package",

package/src/backends.ts

@@ -14,7 +14,7 @@ const COMMENT_CHARS = 300
 export const SOURCES = ['web', 'arxiv', 'hn'] as const
 export type Source = (typeof SOURCES)[number]
 
-export type Fetch = typeof fetch
+export type Fetch = (input: string | URL | Request, init?: RequestInit) => Promise<Response>
 
 export interface Hit {
     text: string
@@ -280,6 +280,8 @@ export async function search(
             return await special(query, numResults, signal, f)
         } catch (err) {
             if (signal?.aborted) throw err
+            const noResults = err instanceof Error && /^no (papers|posts)$/.test(err.message)
+            if (!noResults) throw err
         }
     }
     return firstOf<{ text: string; results: SearchResult[] }>(signal, [

package/src/x-post.ts

@@ -99,7 +99,7 @@ function xInline(text: string, block: XBlock, article: XArticle): string {
             const url = xEntity(article, range.key ?? -1)?.data?.url
             return {
                 ...range,
-                marker: url ? `[${text.slice(range.offset, range.offset + range.length)}](${url})]` : ''
+                marker: url ? `[${text.slice(range.offset, range.offset + range.length)}](${url})` : ''
             }
         })
     ]