@tduyng/pi-web 0.0.2 → 0.0.4

package/README.md

@@ -1,39 +1,62 @@
 # @tduyng/pi-web
 
-Web search and fetch for pi. The extension registers `web_search` and `web_fetch`, and registers the keyless Exa MCP server through pi's own MCP client: `pi.registerMcpServer('exa', ...)`, exposure `codemode`, listed in `/mcp`. The tools call that server's two tools, `web_search_exa` and `web_fetch_exa`, through `ctx.executeTool` and carry no MCP SDK of their own. The server takes a 60 second request timeout, which covers the connect handshake, and pi retries only HTTP 408, 429 and 5xx: a handshake that overruns marks the server failed for the session and `/mcp` shows why.
+Web search and fetch with no API key, no setup, and no MCP tool schemas sitting in the context. Two tools, 326 tokens a request.
 
-## Backends
+A model without web access answers from memory, and the reply does not say which parts are memory. Two ways out: paste pages by hand, or install a search MCP server whose tool schemas then ride in every request, including the ones that never search.
 
-`web_search` takes an array of up to six queries, one call each, and groups the results under the query that produced them. The primary backend is keyless Exa MCP, with DuckDuckGo HTML behind it. The Exa leg goes through the registered `exa` server, so an unconnected or `--no-mcp` session lands in the same DuckDuckGo fallback. x.com and twitter.com URLs go to Fixupx instead, which answers a page X refuses. It returns a post together with the whole long-form Article it links. An `x.com/i/article/` URL carries an article id rather than a post id, so the reply names the post URL to fetch.
+## Two tools, one hidden server
 
-`source: arxiv` searches the arXiv Atom API instead, sorted by submission date, and returns abstracts with authors and dates. A miss falls back to the open web, because arXiv indexes preprints and nothing else.
+pi registers Exa's MCP server with exposure `codemode`, so its tools never enter the prompt. A codemode script calls them through `ctx.executeTool`, and a model that never writes such a script never pays for them.
 
-`source: hn` searches Hacker News through Algolia's index, which needs no account, and returns each thread's points and comment count, the only measure of how much discussion a subject has. A comment hit is reported as the thread that holds it, with the comment's first 300 characters. A miss falls back to the open web.
+```text
+web_search({queries: ["typed decision model", "pi coding agent extensions"], numResults: 5})
+web_fetch({urls: ["https://example.com/post"], maxCharacters: 12000})
+```
 
-Search returns excerpts, so the result ends with the ranked URLs to read. Without that list the model answers from a snippet.
+`web_search` takes up to six queries and groups the results under the query that produced them. Each group carries `title`, `url`, `snippet`, plus `published` and `authors` where the backend has them, and `points` and `comments` on Hacker News. The results are a declared output schema, so a codemode script gets them as data in `structuredContent.groups` and can compare values without parsing text.
 
-## Typed results
+`web_fetch` reads a URL as Markdown. `x.com` and `twitter.com` links go to Fixupx, which answers a page X refuses, and it returns the post together with the long-form Article it links to.
 
-`web_search` declares an output schema, so a codemode script gets the answer as data in `structuredContent.groups`, one entry per query carrying `title`, `url` and `snippet`, plus `published` and `authors` where the backend has them, and `points` and `comments` on Hacker News. A script can compare values across results without parsing text. Exa's answer is prose: `Title:`, `URL:`, `Published:`, `Author:` and `Highlights:` blocks with a rule between them, which `parseExa` reads back into that shape, so the typed view holds for every backend. The model still receives the text, and the data is not sent to it a second time.
+## Search is a step, not an answer
 
-A backend that answers with nothing is a miss, and the next one runs. Before, an empty Exa answer ended the search and DuckDuckGo never got a turn.
+Results are excerpts, so the reply ends with the ranked URLs to read. Without that list the model answers from a snippet, and a snippet is where a wrong answer sounds most confident.
 
-## DuckDuckGo queue
+| Backend | When | What it returns |
+| --- | --- | --- |
+| Exa MCP | always, keyless | prose answer plus highlights |
+| DuckDuckGo | Exa unconnected, `--no-mcp`, or empty | result list |
+| arXiv Atom API | `source: "arxiv"` | abstracts, authors, dates, newest first |
+| Hacker News (Algolia) | `source: "hn"` | threads with points and comment count |
 
-DuckDuckGo answers a burst of requests from one address with a bot check, and keeps that address locked out for 10 to 15 minutes, with each retry extending it. Measured on 2026-10-26: six requests at once got three pages of results and three challenge pages at 14 KB apiece, and the challenge arrives as HTTP 202 as often as a 403. Searches therefore run one at a time, two seconds apart. A bot check pauses the provider for five minutes, doubling to an hour while it keeps answering that way, and a paused search fails at once instead of waiting, so the batch reports that query as failed rather than holding the tool open.
+A miss on arXiv or Hacker News falls back to the open web, because arXiv indexes preprints and nothing else.
 
-## Links in a fetched page
+## Measurements
 
-Exa's extraction returns plain text. Measured on 2026-09-27, `arxiv.org/abs/1706.03762` came back with no links at all, so a citation cannot be read out of the page that cites it. Jina Reader returns Markdown with the links intact, 295 of them on a Hacker News thread, but it blocks a domain that hammered it: `arxiv.org` answered 403 until 2026-09-27. Exa answers first for that reason, and the links survive only when Jina does. Follow a citation by searching for the paper's title.
+**DuckDuckGo queue.** DuckDuckGo answers a burst from one address with a bot check and keeps that address locked out for 10 to 15 minutes, with each retry extending it. Measured on 2026-10-26: six requests at once got three pages of results and three challenge pages at 14 KB apiece, and the challenge arrives as HTTP 202 as often as 403. So searches run one at a time, two seconds apart. A bot check pauses the provider for five minutes, doubling to an hour while it keeps answering that way, and a paused search fails at once instead of holding the tool open.
 
-## arXiv query form
+**arXiv query form.** arXiv ANDs nothing on its own. Measured on the live API: `typed decision model` returns 1,470,125 rows because the words are read as alternatives, and `all:"typed decision model"` returns 3 because the phrase is too strict. A plain query therefore has its words ANDed one at a time, which is the only form that ranks usefully. A query already written with arXiv field prefixes (`ti:`, `cat:`, `au:`) passes through untouched.
 
-arXiv ANDs nothing on its own. Measured on the live API: `typed decision model` returns 1,470,125 rows because the words are read as alternatives, and `all:"typed decision model"` returns 3 because the phrase is too strict. So a plain query has its words ANDed one at a time, which is the only form that ranks usefully. A query already written with arXiv field prefixes (`ti:`, `cat:`, `au:`, and the rest) is passed through untouched.
+**Links in a fetched page.** Exa's extraction returns plain text. Measured on 2026-09-27, `arxiv.org/abs/1706.03762` came back with no links at all, so a citation cannot be read out of the page that cites it. Jina Reader returns Markdown with links intact, 295 of them on a Hacker News thread, but it blocks a domain that hammered it. Exa answers first for that reason, and links survive only when Jina does. To follow a citation, search for the paper's title.
 
-## Reddit and X
+**Reddit and X.** Neither has a keyless search. Measured on 2026-09-27, `reddit.com/search.json` answers 403 on `www`, `old` and `api`, with a browser user agent and a plain one. DuckDuckGo's `site:` operator does reach both, so naming the forum in the query is the way in.
 
-Neither has a keyless search. Measured on 2026-09-27: `reddit.com/search.json` answers 403 on `www`, `old` and `api`, with a browser user agent and with a plain one, and X has no search without a key. DuckDuckGo's `site:` operator does return both, so a query that names the forum reaches those threads. The web backend finds them when a query names them: `Hacker News and Reddit discussion of typed decision models` returned a `news.ycombinator.com` host, while the bare query did not.
+## Cost
 
-## Quota
+`web_search` is 219 tokens a request and `web_fetch` is 107. The Exa tools never enter the prompt, so the cold start is 326 tokens on every session, with no key, account or handshake.
 
-The keyless Exa MCP endpoint needs no account. Set `EXA_API_KEY` to send your own key. The extension reads it at load and puts it on the registered server URL as the `exaApiKey` query parameter. Exa's free plan includes 1,000 requests per month. A batch of six queries costs six of them, so they run three at a time.
+## Install
+
+```bash
+pi install npm:@tduyng/pi-web
+```
+
+Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-web`
+
+Optionally set `EXA_API_KEY` to use an own key. The extension reads it at load and puts it on the registered server URL as the `exaApiKey` parameter. Exa's free plan includes 1,000 requests a month, so a batch of six queries costs six of them.
+
+## Limits
+
+- The keyless Exa server takes a 60 second request timeout covering the connect handshake. Pi retries only HTTP 408, 429 and 5xx, so a handshake that overruns marks the server failed for the session and `/mcp` shows why.
+- `numResults` is 1 to 10, default 5. `maxCharacters` defaults to 8,000.
+- Fetch has a 20 second timeout, and at most six queries run per call.
+- Exa's free tier, with an own key, is 1,000 requests a month.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@tduyng/pi-web",
-  "version": "0.0.2",
-  "description": "Keyless web search and fetch for pi: batched queries, arXiv preprints, Exa MCP with DuckDuckGo and Jina Reader fallbacks, zero dependencies.",
+  "version": "0.0.4",
+  "description": "Keyless web search and fetch: batched queries, arXiv, Hacker News, and Exa MCP with fallbacks. No API key, no MCP tool pile in context.",
   "keywords": [
     "pi",
     "pi-package",