webfox 0.0.0-test.0 → 4.0.0

package/docs/cli-experience.md

@@ -0,0 +1,177 @@
+# CLI reference
+
+Install the `webfox` package to use the `web` executable. Start with the
+[README](../README.md) for setup and everyday examples.
+
+## Commands and help
+
+```sh
+web --help
+web search --help
+web search --provider openai --help
+web providers
+web providers openai
+web config default search brave
+web config path
+web config show
+web config validate
+```
+
+Ordinary help shows all command controls. Explicit provider help adds that
+provider's exact supported flags. For example, OpenAI exposes `--model`,
+`--search-context-size`, and `--user-location-country`. Arrays use repeatable
+flags; booleans provide `--foo` and `--no-foo`.
+
+Root and capability help end with copyable examples. Help highlights headings,
+commands, flags, and arguments on a terminal. Piped help stays plain;
+`NO_COLOR` or `--no-color` disables styling.
+
+## Inputs
+
+Search and answer accept up to ten independent queries or questions. Quote each
+input; results keep input order. Research accepts exactly one brief. Contents
+accepts one or more URLs.
+
+```sh
+web search "Node.js cancellation" "Bun cancellation"
+web answer "What is MCP?" "What is A2A?" --provider openai
+web contents https://example.com/a https://example.com/b --provider tavily
+web research "Compare databases" --provider gemini --timeout 20m
+```
+
+Omit positional input to read piped or redirected stdin:
+
+```sh
+# Read from files
+web search < query.txt
+web answer --provider openai < question.txt
+web research --provider gemini < brief.md
+web contents --provider tavily < urls.txt
+
+# Pipe input
+echo "What is MCP?" | web answer --provider openai
+echo "https://example.com" | web contents --provider tavily
+```
+
+Search, answer, and research read one complete input, including newlines.
+Contents reads newline-separated URLs. Positional arguments take precedence over
+stdin. Without arguments, a terminal reports missing input instead of waiting;
+piped or redirected stdin is read through EOF. Empty or whitespace-only stdin is
+an input error. An explicit `-` still reads stdin, including from a terminal, but
+cannot be mixed with positional input. Help never reads stdin.
+
+None of these commands accepts repeated `--query`, `--output`, or `--raw` flags.
+
+## Common controls
+
+| Flag | Meaning |
+| --- | --- |
+| `--provider <id>` | Override the saved provider for this request. |
+| `--max-results <n>` | Limit results per query; search only. |
+| `--format text\|json` | Select the result format; text is the default, even when piped. |
+| `--timeout <duration>` | Set the overall deadline, including credential commands and retries. |
+| `--quiet` | Suppress progress and success notices on stderr, not errors. |
+| `--no-color` | Disable terminal colors. |
+| `--config <path>` | Select a configuration file. |
+| `--cwd <path>` | Set the working directory for custom providers and option files. |
+| `--options-json <json\|@file>` | Supply complex provider options; provider-specific flags take precedence. |
+
+Durations require a unit: `500ms`, `30s`, `20m`, or `1h`. Ctrl-C cancels waiting
+for stdin, credential commands, subprocesses, and provider operations.
+Cancellation stops the caller waiting even if an SDK cannot cancel its underlying
+request. Cancelling a remote research request does not necessarily cancel its
+billable job.
+
+Complex objects and options whose names collide with common flags remain
+available through `--options-json`. Retry tuning belongs in configuration, not
+CLI flags. See the [configuration reference](./reference.md#configuration) for
+option precedence, deadlines, and retries.
+
+## Provider inspection
+
+`web providers` shows providers as rows and capabilities as columns:
+
+- `☆` marks an unconfigured provider; `★` marks one with at least one configured
+  capability. Providers with separate credentials may still need configuration
+  for other capabilities.
+- `✔︎` means supported, `✘︎` means unsupported, and `◉` means selected as the
+  default for that capability. Selection is independent of configuration, so an
+  unconfigured provider can still show `◉`.
+
+The `custom` row appears only when a command is configured or a capability selects
+it as its default. Its support checkmarks reflect configured commands. Use
+`web providers custom` for inspection and setup guidance even before configuring
+commands.
+
+Configured means local settings or credential sources exist—not that credentials,
+executables, or connectivity have been verified. Inspection and help never run
+credential commands or initialize provider SDKs.
+
+## Output and progress
+
+Text remains the default when piped. Stdout contains results, not success banners
+or execution diagnostics. Status messages go to stderr:
+
+- `▶︎` for progress.
+- `✔︎` for success.
+- `✘︎` for errors.
+- `■` for cancellation.
+
+Errors include a stable error code and, for per-input failures, the affected input.
+Use `--quiet` to suppress progress and success notices. Status colors follow
+terminal detection, `NO_COLOR`, and `--no-color`; JSON never acquires terminal
+styling. The CLI and library do not truncate results.
+
+For remote research jobs, progress confirms submission and acceptance, then
+reports provider status changes or elapsed time roughly every 30 seconds while
+polling. Transient status-check failures show the retry delay and count; they do
+not start a new research job. Opaque job IDs and duplicate completion messages
+are omitted. One success line gives the total elapsed time; the report itself
+goes to stdout. These updates describe provider status, not estimated completion
+percentages or inferred research stages.
+
+In text mode, failed-input diagnostics go only to stderr. JSON retains structured
+errors in the result document as well.
+
+## JSON results
+
+Pipe JSON output into tools such as `jq`. For example, extract URLs from
+successful search results:
+
+```sh
+web search "TypeBox" --format json | jq -r '.results[] | select(.ok) | .value.results[].url'
+```
+
+JSON is one versioned document with results in input order. Status is `ok` when
+all inputs succeed and `partial` otherwise:
+
+```json
+{
+  "schemaVersion": 1,
+  "capability": "search",
+  "provider": "brave",
+  "status": "partial",
+  "results": [
+    { "input": "first query", "ok": true, "value": { "results": [] } },
+    {
+      "input": "second query",
+      "ok": false,
+      "error": { "code": "TIMEOUT", "message": "Operation exceeded its deadline." }
+    }
+  ]
+}
+```
+
+A result has either `ok: true` and `value`, or `ok: false` and a structured
+`error`. Partial batches retain completed successes, including when another input
+times out or is cancelled. A contents result's `input` is the requested URL;
+`value.url` is the final URL when the provider reports it.
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | Success or help. |
+| `1` | Provider failure, partial result, or timeout. |
+| `2` | Invalid input or configuration. |
+| `130` | Cancellation. |

package/docs/provider.md

@@ -0,0 +1,177 @@
+# Provider guide
+
+Choose a provider for each capability, then supply its credentials. Save your
+choices with `web config default`, or select a provider for one request with
+`--provider`.
+
+```sh
+web providers
+web providers brave
+web search --provider brave --help
+web config default search brave
+```
+
+`web providers <id>` shows setup guidance and supported capabilities. Configured
+means local settings or credential sources exist—not that credentials or
+connectivity have been verified. Inspection and help don't run credential commands
+or make provider requests.
+
+Use `web <capability> --provider <id> --help` for the full list of supported
+options. To save options or read credentials from a password manager, see the
+[configuration reference](./reference.md#configuration).
+
+## Brave
+
+Supports search, answers, and research. Set `BRAVE_SEARCH_API_KEY` for search and
+`BRAVE_ANSWERS_API_KEY` for answers and research. The keys are separate; configuring
+one doesn't configure the other capabilities.
+
+```sh
+web search "Node.js release notes" --provider brave
+```
+
+## Cloudflare
+
+Supports page extraction through Cloudflare Browser Rendering. Set both
+`CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID`.
+
+```sh
+web contents https://example.com --provider cloudflare
+```
+
+## Custom
+
+Connect your own command for any of the four capabilities. Configure
+`providers.custom.commands.<capability>.argv`, optionally with `cwd` and
+credential-source `env` entries. Only configured commands are available.
+
+The process receives one JSON request:
+
+```json
+{
+  "schemaVersion": 1,
+  "capability": "search",
+  "input": { "query": "example", "maxResults": 5 },
+  "options": {},
+  "cwd": "/working/directory"
+}
+```
+
+Write one normalized result object to stdout and newline-delimited progress to
+stderr. A nonzero exit signals failure. Contents answers must include a zero-based
+`inputIndex` relative to the process request; `url` separately represents the final
+URL. Errors are structured `{ "code": "PROVIDER_FAILURE", "message": "..." }`
+objects. See the [custom-provider example](../examples/custom/README.md) for a
+working command and the complete request/response contract.
+
+## Exa
+
+Supports search, page extraction, answers, and research. Set `EXA_API_KEY`.
+Research returns a synthesized report with source links.
+
+```sh
+web search "Node.js release notes" --provider exa
+web research "Compare databases for an analytics service" --provider exa --timeout 20m
+```
+
+## Firecrawl
+
+Supports search, page extraction, and page-scoped answers. Set `FIRECRAWL_API_KEY`
+for the hosted service. A self-hosted `providers.firecrawl.baseUrl` can work
+without an API key.
+
+Answers require a page URL, supplied through provider defaults or `--url`:
+
+```sh
+web contents https://example.com --provider firecrawl
+web answer "What does this page explain?" --provider firecrawl --url https://example.com
+```
+
+## Gemini
+
+Supports grounded answers and research, not standalone search. Set
+`GOOGLE_API_KEY`. Google's Search tool runs inside a model interaction; Webfox
+does not discard the generated answer and present its sources as ordinary search
+results.
+
+Answers default to `gemini-3.8-flash`. Override the model with
+`providers.gemini.options.answer.model` or the answer command's `--model` flag.
+Research uses `deep-research-preview-04-2026`, Google's standard
+[Deep Research agent](https://ai.google.dev/gemini-api/docs/deep-research).
+The research agent is fixed; answer model settings don't affect it.
+
+If you previously selected Gemini for search, choose a search-capable provider
+and remove `providers.gemini.options.search` from your configuration.
+
+## Linkup
+
+Supports search, page extraction, and research. Set `LINKUP_API_KEY`.
+
+```sh
+web search "Node.js release notes" --provider linkup
+```
+
+## Ollama
+
+Supports search and page extraction through **hosted web APIs** at
+`https://ollama.com`, not through the local Ollama model server. Set
+`OLLAMA_API_KEY` from your Ollama account; no model download or local daemon is
+required.
+
+```sh
+web search "Node.js streams documentation" --provider ollama --max-results 5
+web contents https://example.com --provider ollama
+```
+
+Search supports up to 10 results. The `contents` capability uses Ollama's web
+fetch endpoint. `providers.ollama.baseUrl` can point to a compatible proxy; it
+does not turn the local inference API into a web search service.
+
+## OpenAI
+
+Supports search, grounded answers, and research. Set `OPENAI_API_KEY`.
+
+```sh
+web answer "What is MCP?" --provider openai --model gpt-6-astra
+web search --provider openai --help
+```
+
+## Parallel
+
+Supports search and page extraction. Set `PARALLEL_API_KEY`.
+
+```sh
+web search "Node.js release notes" --provider parallel
+```
+
+## Perplexity
+
+Supports search, grounded answers, and research. Set `PERPLEXITY_API_KEY`.
+
+```sh
+web answer "What is MCP?" --provider perplexity
+```
+
+## Serper
+
+Supports search. Set `SERPER_API_KEY`.
+
+```sh
+web search "Node.js release notes" --provider serper
+```
+
+## Tavily
+
+Supports search and page extraction. Set `TAVILY_API_KEY`.
+
+```sh
+web contents https://example.com --provider tavily
+```
+
+## Valyu
+
+Supports search, page extraction, answers, and research. Set `VALYU_API_KEY`.
+
+```sh
+web search "Node.js release notes" --provider valyu
+```

package/docs/reference.md

@@ -0,0 +1,181 @@
+# Webfox reference
+
+This reference covers advanced configuration and the TypeScript API. For everyday
+commands, start with the [README](../README.md). See the
+[provider guide](./provider.md) for setup and provider-specific behavior, or the
+[CLI reference](./cli-experience.md) for scripting and output contracts.
+
+## Configuration
+
+### File selection and validation
+
+Configuration paths resolve in this order:
+
+1. `--config` or the library's `configPath`.
+2. `WEBFOX_CONFIG`.
+3. `%APPDATA%\webfox\config.yaml` on Windows, when `APPDATA` is set.
+4. `$XDG_CONFIG_HOME/webfox/config.yaml`.
+5. `~/.config/webfox/config.yaml`.
+
+A missing default file means no saved settings. An explicitly selected missing
+file is an error; `web config default` can create it. The setter preserves
+unrelated settings and comments and atomically replaces the file with owner-only
+permissions. `web config show` emits redacted YAML, hiding literal credentials
+and credential commands. `web config validate` checks configuration and provider
+option schemas without resolving credentials or making requests.
+
+Use YAML 1.2 with string mapping keys and finite numbers. Duplicate keys,
+multiple documents, aliases, explicit tags, and unknown configuration fields are
+rejected. Empty files are valid and mean no saved settings. The published JSON
+Schema supports validation and editor completion and is available as
+`webfox/config.schema.json`.
+
+JSON syntax is also valid YAML. To use a JSON configuration file, select it
+explicitly with `WEBFOX_CONFIG` or `--config`.
+
+See the [example configuration](../example-config.yaml).
+
+### Defaults and provider options
+
+Capability defaults contain only provider selection and portable settings
+(currently search's `maxResults`). Provider-specific options belong exclusively
+under that provider:
+
+```yaml
+defaults:
+  search:
+    provider: brave
+    maxResults: 5
+  answer:
+    provider: openai
+execution:
+  timeoutMs: 30000
+  researchTimeoutMs: 1800000
+  retries: 1
+  retryDelayMs: 2000
+  concurrency: 4
+providers:
+  brave:
+    options:
+      search:
+        mode: news
+  openai:
+    credentials:
+      api:
+        command: [op, read, "op://vault/openai/api-key"]
+    options:
+      answer:
+        model: gpt-6-astra
+```
+
+Options have one precedence order, from lowest to highest:
+
+1. Provider defaults.
+2. `providers.<id>.options.<capability>`.
+3. Request options, or CLI `--options-json`.
+4. Provider-specific CLI flags.
+
+Switching providers never carries the previous provider's options into a request.
+Defaults and overrides can be incomplete; required fields are checked after
+merging. For example, Firecrawl answers need a `url` supplied in defaults or
+with `--url`.
+
+### Credentials
+
+Each credential source has exactly one form:
+
+```yaml
+{ env: OPENAI_API_KEY }
+{ command: [program, argument] }
+{ value: "literal-secret" }
+```
+
+Standard environment references work without a provider section. `web providers
+<id>` lists credential names. See the [provider guide](./provider.md) for each
+provider's keys and setup requirements.
+
+Credential commands run asynchronously as argument arrays, never through a shell.
+They inherit the client's environment and working directory and obey its request
+signal. Successful outputs are cached per client, not globally. Overriding one
+credential preserves the provider's other standard references.
+
+Resolved credentials are redacted from results, progress, and public errors
+before they reach callers. Secret-bearing metadata fields are also redacted.
+This policy is not a sandbox for untrusted adapters or commands.
+
+### Timeouts, retries, and cancellation
+
+Defaults are 30 seconds for ordinary operations, 30 minutes for research, four
+concurrent inputs, and no retries. Retry backoff starts at two seconds and grows
+up to 30 seconds within the same overall deadline. The deadline includes
+credential commands, provider operations, and retries.
+
+Only transient failures on operations the adapter marks retry-safe are retried.
+Research job creation is never blindly retried; polling applies the configured
+retry count and backoff without creating another job. URL operations are
+scheduled separately so completed pages survive another page's failure or
+timeout.
+
+Cancellation stops the caller waiting even if an SDK cannot cancel its underlying
+request. Cancelling a remote research request does not necessarily cancel its
+billable job.
+
+## TypeScript library
+
+```ts
+import { createWebfox } from "webfox";
+
+const web = createWebfox();
+const document = await web.search({
+  provider: "openai",
+  queries: ["TypeBox validation", "Node.js AbortSignal"],
+  maxResults: 5,
+  options: { searchContextSize: "high" },
+  timeoutMs: 30_000,
+  onProgress: ({ message }) => console.error(message),
+});
+
+for (const result of document.results) {
+  if (result.ok) console.log(result.value.results);
+  else console.error(result.error.code, result.error.message);
+}
+```
+
+`createWebfox({ config?, configPath?, cwd?, env? })` snapshots configuration and
+environment for one client. It exposes:
+
+- `search({ queries, maxResults?, ...controls })`.
+- `contents({ urls, ...controls })`.
+- `answer({ queries, ...controls })`.
+- `research({ input, ...controls })`.
+- `inspectCapability(capability, provider?)`: selection, configuration status,
+  provider option-override schema, and effective non-secret defaults.
+- `listProviders()` and `getProvider(id)`: supported/configured capabilities and
+  saved selections.
+
+Request controls are `provider`, `options`, `timeoutMs`, `signal`, and
+`onProgress`. Planning errors throw `WebfoxError`; per-input execution failures
+remain in the document. Results use the same
+[document contract as CLI JSON](./cli-experience.md#json-results), without
+truncation. Provider registration is deliberately not public; use a
+[custom provider command](./provider.md#custom) to connect another backend.
+
+For all capabilities, `onProgress` includes per-input lifecycle events with
+`inputIndex`, `input`, and `state` (`queued`, `running`, `done`, `failed`, or
+`cancelled`). Indexes refer to the original input order, including duplicate
+inputs. Provider progress messages may omit these fields.
+
+## pi extension behavior
+
+Webfox exposes `web_search`, `web_contents`, `web_answer`, and `web_research`
+for capabilities with an explicitly configured default provider. Each tool's
+parameter schema is built dynamically from that provider: the model sees its
+supported options rather than parameters for every backend. Unsupported
+parameters are rejected with their exact paths and, when unambiguous, a suggested
+valid location.
+
+Restart pi after changing configuration to refresh the tool schemas. The
+extension forwards cancellation and progress, marks partial results as errors,
+and truncates tool output at 2,000 lines or 50 KiB, with full results saved to a
+temporary file. It needs no globally installed `web` command and starts no
+background research jobs.

package/example-config.yaml

@@ -0,0 +1,36 @@
+$schema: https://unpkg.com/webfox@3.5.1/dist/config.schema.json
+defaults:
+  search:
+    provider: brave
+    maxResults: 5
+  contents:
+    provider: tavily
+  answer:
+    provider: openai
+  research:
+    provider: gemini
+execution:
+  timeoutMs: 30000
+  retries: 1
+  retryDelayMs: 2000
+  researchTimeoutMs: 1800000
+providers:
+  brave:
+    credentials:
+      search:
+        env: BRAVE_SEARCH_API_KEY
+  tavily:
+    credentials:
+      api:
+        env: TAVILY_API_KEY
+  openai:
+    credentials:
+      api:
+        env: OPENAI_API_KEY
+    options:
+      answer:
+        model: gpt-6-astra
+  gemini:
+    credentials:
+      api:
+        env: GOOGLE_API_KEY

package/examples/custom/README.md

@@ -0,0 +1,57 @@
+# Custom provider example
+
+`provider.mjs` implements all four capabilities without network access. Run the
+example from the repository root using the included `webfox.json`:
+
+```sh
+web search "example query" --config examples/custom/webfox.json
+web contents https://example.com --config examples/custom/webfox.json --format json
+```
+
+The configuration defines explicit custom commands and saved defaults. In your
+own setup, use absolute command paths or set the command’s `cwd`.
+
+## Process contract
+
+Each invocation receives one JSON request on stdin, writes one JSON object to
+stdout, and sends newline-delimited progress to stderr. A nonzero exit signals
+failure. The runtime bounds process output and terminates the process group on
+cancellation or timeout (the direct child on Windows).
+
+```json
+{
+  "schemaVersion": 1,
+  "capability": "search",
+  "input": { "query": "example", "maxResults": 5 },
+  "options": {},
+  "cwd": "/working/directory"
+}
+```
+
+Payloads and responses:
+
+| Capability | Input | Response |
+| --- | --- | --- |
+| Search | `{ "query": "...", "maxResults": 5 }` | `{ "results": [{ "title": "...", "url": "https://...", "snippet": "..." }] }` |
+| Contents | `{ "urls": ["https://..."] }` | `{ "answers": [{ "inputIndex": 0, "url": "https://...", "content": "..." }] }` |
+| Answer | `{ "query": "..." }` | `{ "text": "..." }` |
+| Research | `{ "input": "..." }` | `{ "text": "..." }` |
+
+Contents answers must cover every request index exactly once. `inputIndex`
+associates a response with the requested URL; `url` can hold a redirected URL.
+A failed page uses a structured error instead of content:
+
+```json
+{
+  "inputIndex": 0,
+  "url": "https://example.com",
+  "error": { "code": "PROVIDER_FAILURE", "message": "Extraction failed." }
+}
+```
+
+The runtime currently schedules one URL per invocation. Treat indices as local
+to the supplied request, not the entire user batch.
+
+Replace the deterministic branches with your own integrations. Credentials for
+child processes go in the command’s `env` map as `{ "env": "NAME" }`,
+`{ "command": ["program", "arg"] }`, or `{ "value": "..." }` sources.

package/examples/custom/provider.mjs

@@ -0,0 +1,54 @@
+let body = "";
+for await (const chunk of process.stdin) body += chunk;
+
+const request = JSON.parse(body);
+if (request.schemaVersion !== 1) {
+  process.stderr.write("unsupported schemaVersion\n");
+  process.exit(2);
+}
+
+process.stderr.write(`running custom ${request.capability}\n`);
+
+switch (request.capability) {
+  case "search":
+    process.stdout.write(
+      JSON.stringify({
+        results: [
+          {
+            title: `Example result for ${request.input.query}`,
+            url: "https://example.com/",
+            snippet: "Replace this branch with your search integration.",
+          },
+        ],
+      }),
+    );
+    break;
+  case "contents":
+    process.stdout.write(
+      JSON.stringify({
+        answers: request.input.urls.map((url, inputIndex) => ({
+          inputIndex,
+          url,
+          content: `Example contents for ${url}`,
+        })),
+      }),
+    );
+    break;
+  case "answer":
+    process.stdout.write(
+      JSON.stringify({
+        text: `Example answer for ${request.input.query}`,
+      }),
+    );
+    break;
+  case "research":
+    process.stdout.write(
+      JSON.stringify({
+        text: `Example research report for ${request.input.input}`,
+      }),
+    );
+    break;
+  default:
+    process.stderr.write("unsupported capability\n");
+    process.exit(2);
+}