pi-web-providers 1.0.0 → 2.0.0

package/example-config.json

@@ -0,0 +1,5 @@
+{
+  "tools": {
+    "search": "codex"
+  }
+}

package/examples/custom/README.md

@@ -0,0 +1,224 @@
+# Custom wrapper examples
+
+These examples keep the wrapper logic small. They are bash scripts that use
+`jq` for JSON handling. Each wrapper uses a different backend pattern:
+
+- `wrappers/codex-search.sh` — `codex --search exec`
+- `wrappers/gemini-contents.sh` — Gemini API via `curl`
+- `wrappers/claude-answer.sh` — `claude -p`
+- `wrappers/perplexity-research.sh` — Perplexity API via `curl`
+
+Each wrapper:
+
+- reads one JSON request from `stdin`
+- writes one JSON response to `stdout`
+- may write progress text to `stderr`
+
+## Requirements
+
+You need:
+
+- `bash`
+- `jq`
+- `curl`
+- `codex` on your `PATH` and authenticated locally
+- `claude` on your `PATH` and authenticated locally
+- `GOOGLE_API_KEY` for the Gemini example
+- `PERPLEXITY_API_KEY` for the Perplexity example
+
+## Copy the wrappers into your project
+
+```bash
+mkdir -p ./wrappers
+cp examples/custom/wrappers/codex-search.sh ./wrappers/
+cp examples/custom/wrappers/gemini-contents.sh ./wrappers/
+cp examples/custom/wrappers/claude-answer.sh ./wrappers/
+cp examples/custom/wrappers/perplexity-research.sh ./wrappers/
+chmod +x ./wrappers/*.sh
+```
+
+Then configure `custom` like this:
+
+```json
+{
+  "tools": {
+    "search": "custom",
+    "contents": "custom",
+    "answer": "custom",
+    "research": "custom"
+  },
+  "providers": {
+    "custom": {
+      "enabled": true,
+      "options": {
+        "search": {
+          "argv": ["bash", "./wrappers/codex-search.sh"]
+        },
+        "contents": {
+          "argv": ["bash", "./wrappers/gemini-contents.sh"]
+        },
+        "answer": {
+          "argv": ["bash", "./wrappers/claude-answer.sh"]
+        },
+        "research": {
+          "argv": ["bash", "./wrappers/perplexity-research.sh"]
+        }
+      }
+    }
+  }
+}
+```
+
+`web_research` uses the same async workflow as every other research provider:
+pi starts the wrapper in the background, tracks the job locally, and writes the
+final report to a file when it finishes.
+
+## Core command shapes
+
+### Search with Codex
+
+```bash
+codex --search exec \
+  --skip-git-repo-check \
+  --sandbox read-only \
+  --output-schema ./schema.json \
+  "Search the public web and return JSON only"
+```
+
+### Contents with Gemini and `curl`
+
+```bash
+curl -sS -X POST \
+  "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GOOGLE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "contents": [{"parts": [{"text": "Extract the main content from https://example.com and return JSON only"}]}],
+    "tools": [{"urlContext": {}}],
+    "generationConfig": {"responseMimeType": "application/json"}
+  }'
+```
+
+### Answers with Claude
+
+```bash
+claude -p \
+  --output-format json \
+  --json-schema "$schema" \
+  --permission-mode dontAsk \
+  --allowedTools "WebSearch,WebFetch" \
+  "Answer this question using current public web information"
+```
+
+### Research with Perplexity and `curl`
+
+```bash
+curl -sS https://api.perplexity.ai/chat/completions \
+  -H "Authorization: Bearer $PERPLEXITY_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "sonar-deep-research",
+    "stream": false,
+    "messages": [{"role": "user", "content": "Research this topic and return a long-form answer"}]
+  }'
+```
+
+## Try a wrapper directly
+
+### Search
+
+```bash
+printf '%s' '{
+  "capability": "search",
+  "query": "latest Codex CLI release notes",
+  "maxResults": 5,
+  "options": {},
+  "cwd": "'"$PWD"'"
+}' | bash examples/custom/wrappers/codex-search.sh
+```
+
+### Contents
+
+```bash
+printf '%s' '{
+  "capability": "contents",
+  "urls": ["https://example.com"],
+  "options": {},
+  "cwd": "'"$PWD"'"
+}' | bash examples/custom/wrappers/gemini-contents.sh
+```
+
+### Answer
+
+```bash
+printf '%s' '{
+  "capability": "answer",
+  "query": "What changed in the latest Claude Code release?",
+  "options": {},
+  "cwd": "'"$PWD"'"
+}' | bash examples/custom/wrappers/claude-answer.sh
+```
+
+### Research
+
+```bash
+printf '%s' '{
+  "capability": "research",
+  "input": "Compare current local agent CLIs for web-grounded tasks.",
+  "options": {},
+  "cwd": "'"$PWD"'"
+}' | bash examples/custom/wrappers/perplexity-research.sh
+```
+
+## Request and response contract
+
+### Search request
+
+```json
+{
+  "capability": "search",
+  "query": "latest Codex CLI release notes",
+  "maxResults": 5,
+  "options": {},
+  "cwd": "/path/to/project"
+}
+```
+
+### Search response
+
+```json
+{
+  "results": [
+    {
+      "title": "Codex CLI docs",
+      "url": "https://github.com/openai/codex",
+      "snippet": "CLI docs, examples, and release information."
+    }
+  ]
+}
+```
+
+### Contents response
+
+```json
+{
+  "answers": [
+    {
+      "url": "https://example.com",
+      "content": "# Example\n\nMain page content",
+      "summary": "Optional short summary",
+      "metadata": {}
+    }
+  ]
+}
+```
+
+### Answer and research response
+
+```json
+{
+  "text": "Rendered tool output",
+  "summary": "Optional short summary",
+  "itemCount": 1,
+  "metadata": {}
+}
+```

package/examples/custom/wrappers/claude-answer.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+request="$(cat)"
+cwd="$(jq -r '.cwd // "."' <<<"$request")"
+query="$(jq -r '.query' <<<"$request")"
+model="$(jq -r '.options.model // empty' <<<"$request")"
+
+schema='{"type":"object","properties":{"text":{"type":"string"},"summary":{"type":"string"},"itemCount":{"type":"integer"},"metadata":{"type":"object"}},"required":["text","summary","itemCount","metadata"],"additionalProperties":false}'
+prompt="$(
+  cat <<EOF
+Answer this question using current public web information:
+$query
+
+Return JSON only with these fields:
+- text: the full grounded answer
+- summary: a one-sentence summary
+- itemCount: use 1
+- metadata: include a short note such as the task type
+
+Use WebSearch and WebFetch when needed.
+EOF
+)"
+
+args=(
+  -p
+  --output-format json
+  --json-schema "$schema"
+  --permission-mode dontAsk
+  --allowedTools "WebSearch,WebFetch"
+  --no-session-persistence
+)
+
+if [[ -n "$model" ]]; then
+  args+=(--model "$model")
+fi
+
+echo "Answering with Claude..." >&2
+(
+  cd "$cwd"
+  claude "${args[@]}" "$prompt"
+)

package/examples/custom/wrappers/codex-search.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+request="$(cat)"
+cwd="$(jq -r '.cwd // "."' <<<"$request")"
+query="$(jq -r '.query' <<<"$request")"
+max_results="$(jq -r '.maxResults // 5' <<<"$request")"
+model="$(jq -r '.options.model // empty' <<<"$request")"
+
+schema_file="$(mktemp)"
+output_file="$(mktemp)"
+trap 'rm -f "$schema_file" "$output_file"' EXIT
+
+cat >"$schema_file" <<'JSON'
+{
+  "type": "object",
+  "properties": {
+    "results": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "title": { "type": "string" },
+          "url": { "type": "string" },
+          "snippet": { "type": "string" }
+        },
+        "required": ["title", "url", "snippet"],
+        "additionalProperties": false
+      }
+    }
+  },
+  "required": ["results"],
+  "additionalProperties": false
+}
+JSON
+
+prompt="$(
+  cat <<EOF
+Search the public web for: $query
+
+Return JSON only.
+Return at most $max_results results.
+Each result must include:
+- title
+- url
+- snippet
+
+Prefer primary or official sources when possible.
+EOF
+)"
+
+args=(
+  --search exec
+  --skip-git-repo-check
+  --sandbox read-only
+  --color never
+  --cd "$cwd"
+  --output-schema "$schema_file"
+  --output-last-message "$output_file"
+)
+
+if [[ -n "$model" ]]; then
+  args+=(--model "$model")
+fi
+
+echo "Searching with Codex..." >&2
+codex "${args[@]}" "$prompt" >/dev/null
+jq . "$output_file"

package/examples/custom/wrappers/gemini-contents.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+: "${GOOGLE_API_KEY:?GOOGLE_API_KEY is required}"
+
+request="$(cat)"
+model="$(jq -r '.options.model // "gemini-2.5-flash"' <<<"$request")"
+url_count="$(jq '.urls | length' <<<"$request")"
+urls="$(jq -r '.urls[]' <<<"$request")"
+
+prompt="$(
+  cat <<EOF
+Extract the main textual content from these URLs:
+$urls
+
+Return JSON only with this shape:
+- answers: an array with exactly $url_count items
+- each answer must include the input url, and either:
+  - content: the extracted text as a string
+  - or error: a short error string when extraction fails
+- optionally include summary and metadata
+EOF
+)"
+
+body="$(
+  jq -n \
+    --arg prompt "$prompt" \
+    '{
+    contents: [{parts: [{text: $prompt}]}],
+    tools: [{urlContext: {}}],
+    generationConfig: {responseMimeType: "application/json"}
+  }'
+)"
+
+echo "Fetching contents with Gemini..." >&2
+response="$(curl -sS -X POST \
+  "https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent?key=${GOOGLE_API_KEY}" \
+  -H "Content-Type: application/json" \
+  -d "$body")"
+
+error="$(jq -r '.error.message // empty' <<<"$response")"
+if [[ -n "$error" ]]; then
+  echo "$error" >&2
+  exit 1
+fi
+
+text="$(jq -r '[.candidates[]?.content.parts[]?.text // empty] | join("\n")' <<<"$response")"
+json_text="$(printf '%s\n' "$text" | sed -e '1s/^```json[[:space:]]*//' -e '1s/^```[[:space:]]*//' -e '$s/```$//')"
+jq . <<<"$json_text"

package/examples/custom/wrappers/perplexity-research.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+: "${PERPLEXITY_API_KEY:?PERPLEXITY_API_KEY is required}"
+
+request="$(cat)"
+input="$(jq -r '.input' <<<"$request")"
+model="$(jq -r '.options.model // "sonar-deep-research"' <<<"$request")"
+
+body="$(
+  jq -n \
+    --arg model "$model" \
+    --arg input "$input" \
+    '{
+    model: $model,
+    stream: false,
+    messages: [{role: "user", content: $input}]
+  }'
+)"
+
+echo "Researching with Perplexity..." >&2
+response="$(curl -sS https://api.perplexity.ai/chat/completions \
+  -H "Authorization: Bearer ${PERPLEXITY_API_KEY}" \
+  -H "Content-Type: application/json" \
+  -d "$body")"
+
+error="$(jq -r '.error.message // empty' <<<"$response")"
+if [[ -n "$error" ]]; then
+  echo "$error" >&2
+  exit 1
+fi
+
+citations="$(jq '.citations // []' <<<"$response")"
+count="$(jq '(.citations // []) | length' <<<"$response")"
+text="$(jq -r '
+  (.choices[0].message.content // "No research returned.") as $text
+  | (.citations // []) as $citations
+  | if ($citations | length) == 0 then
+      $text
+    else
+      $text + "\n\nSources:\n" + ($citations | to_entries | map("\(.key + 1). \(.value)") | join("\n"))
+    end
+' <<<"$response")"
+
+jq -n \
+  --arg text "$text" \
+  --arg summary "Research via Perplexity with $count source(s)" \
+  --argjson itemCount "$count" \
+  --argjson citations "$citations" \
+  '{
+    text: $text,
+    summary: $summary,
+    itemCount: $itemCount,
+    metadata: {citations: $citations}
+  }'

package/package.json

@@ -1,12 +1,14 @@
 {
   "name": "pi-web-providers",
-  "version": "1.0.0",
-  "description": "Configurable web access extension for pi with per-tool provider routing for search, contents, answers, and research across Claude, Codex, Exa, Gemini, Perplexity, Parallel, and Valyu.",
+  "version": "2.0.0",
+  "description": "Configurable web access extension for pi with per-tool provider routing for search, contents, answers, and research.",
   "type": "module",
   "files": [
     "dist",
     "README.md",
-    "LICENSE"
+    "LICENSE",
+    "example-config.json",
+    "examples"
   ],
   "keywords": [
     "pi-package",
@@ -14,11 +16,15 @@
     "coding-agent",
     "web-search",
     "claude",
+    "cloudflare",
     "codex",
+    "custom",
     "exa",
+    "firecrawl",
     "gemini",
     "perplexity",
     "parallel",
+    "tavily",
     "valyu"
   ],
   "author": "mavam",
@@ -39,20 +45,24 @@
     ]
   },
   "scripts": {
-    "build": "rm -rf dist && esbuild src/index.ts --bundle --format=esm --platform=node --outfile=dist/index.js --external:@mariozechner/pi-coding-agent --external:@mariozechner/pi-ai --external:@mariozechner/pi-tui --external:@sinclair/typebox --external:@anthropic-ai/claude-agent-sdk --external:@google/genai --external:@openai/codex-sdk --external:@perplexity-ai/perplexity_ai --external:exa-js --external:parallel-web --external:valyu-js",
+    "build": "rm -rf dist && esbuild src/index.ts --bundle --format=esm --platform=node --outfile=dist/index.js --external:@mariozechner/pi-coding-agent --external:@mariozechner/pi-ai --external:@mariozechner/pi-tui --external:@sinclair/typebox --external:@anthropic-ai/claude-agent-sdk --external:@google/genai --external:@mendable/firecrawl-js --external:@openai/codex-sdk --external:@perplexity-ai/perplexity_ai --external:@tavily/core --external:cloudflare --external:exa-js --external:parallel-web --external:valyu-js",
     "prepare": "npm run build",
     "prepack": "npm run build",
     "check": "tsc --noEmit",
     "format": "biome format --write .",
     "format:check": "biome format .",
+    "smoke:live": "npm run build && node scripts/live-smoke.mjs",
     "test": "vitest run",
     "test:watch": "vitest"
   },
   "dependencies": {
     "@anthropic-ai/claude-agent-sdk": "^0.2.71",
     "@google/genai": "^1.44.0",
+    "@mendable/firecrawl-js": "^4.18.1",
     "@openai/codex-sdk": "^0.111.0",
     "@perplexity-ai/perplexity_ai": "^0.26.1",
+    "@tavily/core": "^0.7.2",
+    "cloudflare": "^5.2.0",
     "exa-js": "^2.7.0",
     "parallel-web": "^0.3.1",
     "valyu-js": "^2.5.9",