web-access-cli 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -27
- package/README_CN.md +39 -27
- package/THIRD_PARTY_NOTICES.md +11 -0
- package/dist/cli-output.js +16 -0
- package/dist/cli-output.js.map +1 -1
- package/dist/cli.js +48 -42
- package/dist/cli.js.map +1 -1
- package/dist/config/config.js +119 -53
- package/dist/config/config.js.map +1 -1
- package/dist/config/edit.js +88 -4
- package/dist/config/edit.js.map +1 -1
- package/dist/config/file.js +30 -0
- package/dist/config/file.js.map +1 -1
- package/dist/config/init.js +59 -0
- package/dist/config/init.js.map +1 -0
- package/dist/config/provider-order.js +14 -7
- package/dist/config/provider-order.js.map +1 -1
- package/dist/core/diagnostics.js +13 -7
- package/dist/core/diagnostics.js.map +1 -1
- package/dist/core/router.js +228 -94
- package/dist/core/router.js.map +1 -1
- package/dist/core/schema.js +33 -2
- package/dist/core/schema.js.map +1 -1
- package/dist/core/types.js +3 -0
- package/dist/core/types.js.map +1 -1
- package/dist/providers/extract.js +37 -0
- package/dist/providers/extract.js.map +1 -1
- package/dist/providers/registry.js +3 -0
- package/dist/providers/registry.js.map +1 -1
- package/dist/providers/xai.js +208 -0
- package/dist/providers/xai.js.map +1 -0
- package/dist/version.js +1 -1
- package/package.json +1 -1
- package/schemas/config.schema.json +87 -2
- package/schemas/output.schema.json +180 -0
package/README.md
CHANGED
|
@@ -13,10 +13,10 @@ The CLI is the primary product interface. It is not tied to Pi, Claude Code, Cod
|
|
|
13
13
|
|
|
14
14
|
| Capability | Provider types | Normalized output |
|
|
15
15
|
| --- | --- | --- |
|
|
16
|
-
| `search` | Tavily, Exa, Bocha, Brave, SearXNG, AnySearch, XCrawl, DeepSeek | `rank`, `title`, `url`, `snippet` |
|
|
16
|
+
| `search` | Tavily, Exa, Bocha, Brave, SearXNG, AnySearch, XCrawl, DeepSeek, xAI x_search, xAI web_search | `rank`, `title`, `url`, `snippet` |
|
|
17
17
|
| `extract` | Firecrawl v2, Jina Reader, Exa Contents, AnySearch, XCrawl, HTTP | Markdown `Document` |
|
|
18
18
|
|
|
19
|
-
A provider type describes an implementation, while a provider instance is a configurable instance of that type. One type can have multiple instances, such as `exa_team` and `exa_personal`.
|
|
19
|
+
A provider type describes an implementation, while a provider instance is a configurable instance of that type. One type can have multiple instances, such as `exa_team` and `exa_personal`. Each capability has a primary `providers` route and a `providers_fallback` route. Both arrays determine enabled instances and their current `auto` order; the CLI only reorders members within their existing group.
|
|
20
20
|
|
|
21
21
|
## Installation
|
|
22
22
|
|
|
@@ -84,22 +84,24 @@ web-access extract https://example.com/article
|
|
|
84
84
|
web-access extract https://example.com/article --provider http --timeout 30000
|
|
85
85
|
web-access extract https://example.com/article --json
|
|
86
86
|
|
|
87
|
-
web-access providers
|
|
88
|
-
web-access doctor
|
|
87
|
+
web-access providers
|
|
88
|
+
web-access doctor
|
|
89
89
|
|
|
90
90
|
web-access config edit
|
|
91
91
|
web-access --config "/path/to/config.json" config edit
|
|
92
|
+
web-access config init
|
|
93
|
+
web-access --config "/path/to/config.json" config init --json
|
|
92
94
|
```
|
|
93
95
|
|
|
94
|
-
The CLI provides the `search` and `extract` capability commands, the `providers` and `doctor` diagnostic commands, and the `config edit` configuration
|
|
96
|
+
The CLI provides the `search` and `extract` capability commands, the `providers` and `doctor` diagnostic commands, and the `config init` and `config edit` configuration commands. The current version does not provide batch/all, answer, a dedicated PDF path, a Node.js SDK, or a generic MCP integration. The repository includes an optional [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md) for source-checkout use; it is not included in the npm package.
|
|
95
97
|
|
|
96
98
|
### Global options
|
|
97
99
|
|
|
98
100
|
- `--config <path>`: Explicitly select a configuration file.
|
|
99
|
-
- `--
|
|
101
|
+
- `--json`: Output a compact schema v2 JSON envelope. Without it, successful commands write human-readable Markdown; all failures always write JSON.
|
|
100
102
|
- `--help`, `--version`: Print standard CLI help or version text.
|
|
101
103
|
|
|
102
|
-
|
|
104
|
+
Successful commands default to Markdown for human readers. `search` writes a YAML front matter block followed by a numbered result list; `extract` writes YAML front matter followed by the normalized Markdown body; `providers`, `doctor`, and `config edit` write a heading followed by pretty-printed JSON data; `config init` writes the created absolute path on one line. Add `--json` when a schema v2 envelope is required. Extract `partial` results and all failures, including invalid input and runtime errors, always remain JSON envelopes. Diagnostic messages are never mixed into stdout.
|
|
103
105
|
|
|
104
106
|
## Configuration
|
|
105
107
|
|
|
@@ -107,15 +109,15 @@ Configuration files use strict JSON; unknown fields are rejected. The CLI reads
|
|
|
107
109
|
|
|
108
110
|
The default path on every platform is `~/.config/web-access-cli/config.json`, where `~` is the current user's home directory. The `WEB_ACCESS_CONFIG` environment variable can select another path. The `--config` command-line option takes precedence.
|
|
109
111
|
|
|
110
|
-
`web-access config edit` creates parent directories and a complete default configuration when the file is missing, then opens it with the
|
|
112
|
+
`web-access config edit` creates parent directories and a complete default configuration when the file is missing, then opens it with the command in `VISUAL`, or `EDITOR` when `VISUAL` is empty. The value may contain a quoted executable path and arguments; it is parsed without shell evaluation and the configuration path is appended as a separate argument. An existing file is opened byte-for-byte as-is, even when it is temporarily invalid JSON; it is never overwritten or reformatted. The command waits only for the editor process to start, not for it to close. When neither variable is set, the command returns `open_failed`; a configuration file that was created successfully remains available for manual editing.
|
|
111
113
|
|
|
112
|
-
`config
|
|
114
|
+
`config init` creates a new configuration from standard credential environment variables and refuses to overwrite an existing file. It keeps all built-in instance definitions, writes detected Search providers to the primary route, leaves Search fallback empty, writes detected remote Extract providers to the primary route, and places `http` in the Extract fallback route. After an `auto` capability call succeeds or encounters fallback-eligible failures, the CLI atomically reorders the matching `providers` or `providers_fallback` array in place. Members never move between groups. The legacy `_providers` and `_providers_fallback` fields are rejected as unknown configuration fields.
|
|
113
115
|
|
|
114
116
|
The complete JSON Schema is available at [schemas/config.schema.json](schemas/config.schema.json). Example:
|
|
115
117
|
|
|
116
118
|
```json
|
|
117
119
|
{
|
|
118
|
-
"$schema": "https://unpkg.com/web-access-cli@0.
|
|
120
|
+
"$schema": "https://unpkg.com/web-access-cli@0.5.0/schemas/config.schema.json",
|
|
119
121
|
"providers": [
|
|
120
122
|
{
|
|
121
123
|
"id": "exa_team",
|
|
@@ -138,13 +140,15 @@ The complete JSON Schema is available at [schemas/config.schema.json](schemas/co
|
|
|
138
140
|
],
|
|
139
141
|
"search": {
|
|
140
142
|
"providers": ["searx_local", "exa_team", "brave"],
|
|
143
|
+
"providers_fallback": ["deepseek"],
|
|
141
144
|
"limit": 5,
|
|
142
|
-
"timeoutMs":
|
|
143
|
-
"attemptTimeoutMs":
|
|
145
|
+
"timeoutMs": 120000,
|
|
146
|
+
"attemptTimeoutMs": 60000,
|
|
144
147
|
"maxResponseBytes": 5242880
|
|
145
148
|
},
|
|
146
149
|
"extract": {
|
|
147
|
-
"providers": ["firecrawl_local", "jina", "exa_team"
|
|
150
|
+
"providers": ["firecrawl_local", "jina", "exa_team"],
|
|
151
|
+
"providers_fallback": ["http"],
|
|
148
152
|
"timeoutMs": 120000,
|
|
149
153
|
"attemptTimeoutMs": 45000,
|
|
150
154
|
"maxResponseBytes": 5242880,
|
|
@@ -153,16 +157,20 @@ The complete JSON Schema is available at [schemas/config.schema.json](schemas/co
|
|
|
153
157
|
}
|
|
154
158
|
```
|
|
155
159
|
|
|
156
|
-
The built-in instances are `tavily`, `exa`, `bocha`, `brave`, `searxng`, `firecrawl`, `jina`, `http`, `anysearch`, `xcrawl`, and `
|
|
160
|
+
The built-in instances are `tavily`, `exa`, `bocha`, `brave`, `searxng`, `firecrawl`, `jina`, `http`, `anysearch`, `xcrawl`, `deepseek`, `xai_x_search`, and `xai_web_search`. A configuration entry with the same ID overrides fields on the built-in instance. A custom ID creates another instance of the selected type. An instance is enabled when it appears in either corresponding route. Bocha defaults to `https://api.bocha.cn` and requires an API key; AnySearch defaults to `https://api.anysearch.com` and supports anonymous calls; XCrawl defaults to `https://run.xcrawl.com` and requires an API key; DeepSeek defaults to `https://api.deepseek.com/anthropic/v1` and requires an API key.
|
|
157
161
|
|
|
158
162
|
Default routes:
|
|
159
163
|
|
|
160
|
-
- Search: `tavily -> exa -> bocha -> brave -> searxng -> anysearch -> xcrawl
|
|
161
|
-
-
|
|
164
|
+
- Search primary: `tavily -> exa -> bocha -> brave -> searxng -> anysearch -> xcrawl`
|
|
165
|
+
- Search fallback: `deepseek -> xai_x_search -> xai_web_search`
|
|
166
|
+
- Extract primary: `firecrawl -> jina -> exa -> anysearch -> xcrawl`
|
|
167
|
+
- Extract fallback: `http`
|
|
168
|
+
|
|
169
|
+
The default routes split built-in providers into a primary group and a fallback group. Custom instance IDs are merged into the instance list but must still be added to a route explicitly. An omitted capability route uses the defaults above; an explicit Search primary empty array is invalid, while Extract may be explicitly disabled with an empty primary route. In `auto` mode, incompletely configured instances are skipped. AnySearch and XCrawl accept `searchFilterMode`: `strict` (default; freshness skips the provider) or `best_effort` (rewrites freshness into a query fragment). Domain constraints are rewritten into the query and strictly re-applied locally. XCrawl Extract always uses synchronous Scrape with Markdown output; Map, Crawl, and asynchronous jobs are outside the current CLI capabilities.
|
|
162
170
|
|
|
163
|
-
|
|
171
|
+
DeepSeek Search performs a full Anthropic-compatible Messages model turn with the native `web_search_20250305` server tool, so it can have higher latency and cost than a dedicated search endpoint. It is in the default Search fallback group and is reached only after every primary provider is unavailable, returns a final non-2xx HTTP response, or otherwise fails recoverably. The adapter accepts URLs only from structured `web_search_tool_result` blocks, joins citation excerpts by URL, and never extracts URLs from model prose. Domain constraints are rewritten into the query and strictly re-applied locally; `freshness` is unsupported and skips DeepSeek with a recoverable error. Redirects are rejected without contacting the `Location` target. Provider-private `encrypted_content` payloads are omitted from `raw` because they are opaque, cannot be displayed or decoded by the CLI, and add no diagnostic value.
|
|
164
172
|
|
|
165
|
-
|
|
173
|
+
XAI hosted Search reads an external OAuth JSON file from `XAI_AUTH_JSON` on every call and uses only its `access_token`; an external CLIProxyAPI process owns refresh and write-back. Instance fields `authJson`/`authJsonEnv` and `model`/`modelEnv` override the path and model. `xai_web_search` accepts at most five allow or exclude domains (mutually exclusive); `xai_x_search` rejects `freshness`. Model-backed search can take longer than ordinary API searches; Search defaults allow up to 60 seconds per attempt and 120 seconds total.
|
|
166
174
|
|
|
167
175
|
### Credentials and URLs
|
|
168
176
|
|
|
@@ -181,26 +189,29 @@ Environment variables take precedence over plaintext keys in JSON. Built-in inst
|
|
|
181
189
|
| AnySearch | `ANYSEARCH_API_KEY`, optional | `ANYSEARCH_BASE_URL`, default `https://api.anysearch.com` |
|
|
182
190
|
| XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`, default `https://run.xcrawl.com` |
|
|
183
191
|
| DeepSeek | `DEEPSEEK_API_KEY` | No standard environment variable; default `https://api.deepseek.com/anthropic/v1` |
|
|
192
|
+
| xAI Search | `XAI_AUTH_JSON` | No standard API-key variable; default `https://cli-chat-proxy.grok.com/v1`, model `grok-4.6` |
|
|
184
193
|
|
|
185
194
|
Custom instances use `apiKeyEnv` and `baseUrlEnv` to name their environment variables. Every remote provider can define a `baseUrl` and additional `headers`. The public `api.firecrawl.dev` service requires a key; a self-hosted Firecrawl v2 instance can run without one.
|
|
186
195
|
|
|
187
196
|
## Execution and fallback
|
|
188
197
|
|
|
189
|
-
`--provider auto` follows the
|
|
198
|
+
`--provider auto` follows the configured `providers` order first and enters `providers_fallback` only after the complete primary group fails with a fallback-eligible result:
|
|
190
199
|
|
|
191
200
|
1. An instance with incomplete configuration is recorded as a failed attempt and skipped.
|
|
192
201
|
2. A final provider HTTP response outside 2xx, including authentication, request, rate-limit, and server errors, causes the router to try the next instance.
|
|
193
202
|
3. Other recoverable errors, including network errors, timeouts, oversized responses, and missing usable content, also cause the router to continue.
|
|
194
203
|
4. Non-recoverable failures without a final non-2xx response, including invalid input and unsupported content detected in a successful response, stop execution immediately.
|
|
195
|
-
5. If every instance fails, the command returns `provider_exhausted`. The best short content produced during extraction is preserved in `partial`.
|
|
204
|
+
5. If every instance in both groups fails, the command returns `provider_exhausted`. The best short content produced during extraction is preserved in `partial`.
|
|
205
|
+
|
|
206
|
+
Before those routes run, `extract --provider auto` checks one Markdown variant when an HTTP instance is enabled in either group. It appends `.md` to file-like paths or uses `index.md` for directory paths, requests `text/markdown` and `text/plain`, and accepts only a sufficiently long non-HTML text response. A miss or request failure is transparent and falls through to the configured routes. A hit is reported as the enabled HTTP instance; it does not change provider order. Explicit non-HTTP selections are unchanged, while an explicit HTTP selection checks the Markdown variant before fetching the original URL.
|
|
196
207
|
|
|
197
|
-
After each `auto` call, a successful instance moves to the front, untried instances remain in the middle, and instances with the fallback-eligible failures above move stably to the back. Relative order within each group is preserved. The order stays unchanged when every instance fails. Search and Extract learn independently; explicit instances, non-fallback failures, and user cancellation never update the order. Writes use atomic replacement, and the last writer wins across concurrent processes.
|
|
208
|
+
After each `auto` call, a successful instance moves to the front of its group, untried instances remain in the middle, and instances with the fallback-eligible failures above move stably to the back. Relative order within each group is preserved; no provider crosses between primary and fallback. The order stays unchanged when every instance in a group fails. Search and Extract learn independently; explicit instances, non-fallback failures, and user cancellation never update the order. Writes use atomic replacement, and the last writer wins across concurrent processes.
|
|
198
209
|
|
|
199
210
|
An attempt keeps the provider's original error code, HTTP status, and `retryable` value. `retryable` describes whether the original operation can be retried against the same provider; it is not the sole condition for switching to the next provider in an automatic route.
|
|
200
211
|
|
|
201
212
|
Selecting an instance explicitly executes it strictly without fallback. If the instance is not in the route, the command returns `provider_disabled`.
|
|
202
213
|
|
|
203
|
-
Default limits are a
|
|
214
|
+
Default limits are a 120-second total timeout and a 60-second per-attempt timeout for Search, and 120 seconds and 45 seconds respectively for Extract. Each response has a hard limit of 5 MiB. `--timeout` overrides only the total timeout for the current command.
|
|
204
215
|
|
|
205
216
|
## Output contract
|
|
206
217
|
|
|
@@ -224,9 +235,9 @@ Capability commands use output schema version 2. Their default envelope is inten
|
|
|
224
235
|
}
|
|
225
236
|
```
|
|
226
237
|
|
|
227
|
-
|
|
238
|
+
With `--json`, capability output contains only `schemaVersion`, `ok`, the final Instance ID and normalized `data`. Failures contain a compact `error` and, when providers were attempted, `attempts` entries with only Instance ID, error code and optional HTTP status. Extract quality failures may include a `partial` document without raw provider data. If the learned order cannot be written, either a success or failure envelope additionally contains `warnings: [{ "code": "provider_order_update_failed", "message": "..." }]` without replacing the primary result. Search and extract input errors use the same compact failure shape.
|
|
228
239
|
|
|
229
|
-
For human-readable
|
|
240
|
+
For human-readable output, extraction starts with YAML front matter and then the provider-normalized Markdown body. Search results use a numbered Markdown list; diagnostics and config edit use a heading plus pretty-printed JSON data. Metadata values use JSON double-quoted scalar encoding and the body is not modified. Existing scripts and other machine consumers should append `--json` explicitly.
|
|
230
241
|
|
|
231
242
|
Example default extract output:
|
|
232
243
|
|
|
@@ -240,16 +251,16 @@ title: "Example title"
|
|
|
240
251
|
# Article content
|
|
241
252
|
```
|
|
242
253
|
|
|
243
|
-
|
|
254
|
+
All success and failure envelopes use schema version 2. Stable schemas are available in [schemas](schemas).
|
|
244
255
|
|
|
245
256
|
Exit codes:
|
|
246
257
|
|
|
247
258
|
- `0`: Success
|
|
248
259
|
- `2`: Invalid input or configuration, or an unknown or disabled provider
|
|
249
|
-
- `1`: Runtime, provider, doctor, or
|
|
260
|
+
- `1`: Runtime, provider, doctor, or editor launch failure
|
|
250
261
|
- `130`: User cancellation
|
|
251
262
|
|
|
252
|
-
`providers` lists each instance's type, capabilities, route status, credential source, and base URL source. Its `searchRoute` and `extractRoute` fields show the effective
|
|
263
|
+
`providers` lists each instance's type, capabilities, route status, credential source, and base URL source. Its `searchRoute`/`searchFallbackRoute` and `extractRoute`/`extractFallbackRoute` fields show the two effective orders for the next `auto` call. `doctor` performs local configuration checks only and does not call remote APIs. If an enabled route contains an unconfigured instance, the command returns `doctor_failed` with exit code `1`.
|
|
253
264
|
|
|
254
265
|
## HTTP and security boundaries
|
|
255
266
|
|
|
@@ -257,6 +268,7 @@ Exit codes:
|
|
|
257
268
|
- Follows up to five redirects. Cross-origin redirects remove authentication, Cookie, and common token headers.
|
|
258
269
|
- Enforces a streaming byte limit on every response instead of buffering an unbounded response first.
|
|
259
270
|
- HTTP Extract uses LinkeDOM, Mozilla Readability, and Turndown, with fallback parsing for Next.js RSC payloads.
|
|
271
|
+
- HTTP Extract prefers an enabled source Markdown variant (`.md` or directory `index.md`) before provider routes when that variant returns usable Markdown.
|
|
260
272
|
- Returns `unsupported_content` for PDF, image, audio, video, zip, and general binary content.
|
|
261
273
|
- By design, the CLI does not block localhost, private network addresses, or cloud metadata URLs. For untrusted input, callers must enforce their own URL allowlist, network isolation, or outbound proxy policy.
|
|
262
274
|
|
package/README_CN.md
CHANGED
|
@@ -13,10 +13,10 @@ CLI 是主要产品形态,不绑定 Pi、Claude Code、Codex、Cursor、OpenCo
|
|
|
13
13
|
|
|
14
14
|
| 能力 | Provider Type | 统一输出 |
|
|
15
15
|
| --- | --- | --- |
|
|
16
|
-
| `search` | Tavily、Exa、Bocha、Brave、SearXNG、AnySearch、XCrawl、DeepSeek | `rank`、`title`、`url`、`snippet` |
|
|
16
|
+
| `search` | Tavily、Exa、Bocha、Brave、SearXNG、AnySearch、XCrawl、DeepSeek、xAI x_search、xAI web_search | `rank`、`title`、`url`、`snippet` |
|
|
17
17
|
| `extract` | Firecrawl v2、Jina Reader、Exa Contents、AnySearch、XCrawl、HTTP | Markdown `Document` |
|
|
18
18
|
|
|
19
|
-
Provider Type 描述实现类型;Provider Instance 是一份可配置实例。一个 Type 可以有多个 Instance,例如 `exa_team` 和 `exa_personal
|
|
19
|
+
Provider Type 描述实现类型;Provider Instance 是一份可配置实例。一个 Type 可以有多个 Instance,例如 `exa_team` 和 `exa_personal`。每个能力都有主 `providers` Route 和兜底 `providers_fallback` Route;两组数组同时决定启用状态和 `auto` 顺序,CLI 只在原分组内部调整成员顺序。
|
|
20
20
|
|
|
21
21
|
## 安装
|
|
22
22
|
|
|
@@ -84,22 +84,24 @@ web-access extract https://example.com/article
|
|
|
84
84
|
web-access extract https://example.com/article --provider http --timeout 30000
|
|
85
85
|
web-access extract https://example.com/article --json
|
|
86
86
|
|
|
87
|
-
web-access providers
|
|
88
|
-
web-access doctor
|
|
87
|
+
web-access providers
|
|
88
|
+
web-access doctor
|
|
89
89
|
|
|
90
90
|
web-access config edit
|
|
91
91
|
web-access --config "/path/to/config.json" config edit
|
|
92
|
+
web-access config init
|
|
93
|
+
web-access --config "/path/to/config.json" config init --json
|
|
92
94
|
```
|
|
93
95
|
|
|
94
|
-
CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两个诊断命令,以及 `config edit` 配置维护命令。当前版本不提供 batch/all、answer、PDF 专线、Node SDK 或通用 MCP 集成。仓库提供可选的 [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md),仅供源码仓库使用,不包含在 npm 发布包中。
|
|
96
|
+
CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两个诊断命令,以及 `config init`、`config edit` 配置维护命令。当前版本不提供 batch/all、answer、PDF 专线、Node SDK 或通用 MCP 集成。仓库提供可选的 [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md),仅供源码仓库使用,不包含在 npm 发布包中。
|
|
95
97
|
|
|
96
98
|
### 通用选项
|
|
97
99
|
|
|
98
100
|
- `--config <path>`:显式指定配置文件。
|
|
99
|
-
- `--
|
|
101
|
+
- `--json`:输出紧凑 schema v2 JSON envelope;不指定时,成功命令输出面向人的 Markdown,失败始终输出 JSON。
|
|
100
102
|
- `--help`、`--version`:输出常规 CLI 帮助或版本文本。
|
|
101
103
|
|
|
102
|
-
`
|
|
104
|
+
成功命令默认输出面向人的 Markdown。`search` 输出 YAML front matter 和编号结果列表;`extract` 输出 YAML front matter 和规范化后的 Markdown 正文;`providers`、`doctor` 和 `config edit` 输出标题和格式化 JSON 数据;`config init` 输出新配置的绝对路径。需要 schema v2 envelope 时显式追加 `--json`。提取失败(包括 `partial`)以及所有失败路径始终输出 JSON envelope。诊断信息不会混入 stdout。
|
|
103
105
|
|
|
104
106
|
## 配置
|
|
105
107
|
|
|
@@ -107,15 +109,15 @@ CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两
|
|
|
107
109
|
|
|
108
110
|
所有平台的默认路径均为 `~/.config/web-access-cli/config.json`,其中 `~` 表示当前用户的主目录。环境变量 `WEB_ACCESS_CONFIG` 可以指定其他路径;命令行 `--config` 优先级更高。
|
|
109
111
|
|
|
110
|
-
`web-access config edit`
|
|
112
|
+
`web-access config edit` 会在配置文件缺失时创建父目录和完整默认配置,再优先使用 `VISUAL`、其次使用 `EDITOR` 打开;变量值可以包含带引号的可执行文件路径和参数,解析时不会执行 shell,并将配置路径作为独立的最后一个参数传入。已有文件会原样打开,即使内容暂时不是有效 JSON 也不会被覆盖或格式化。命令只等待编辑器进程启动,不等待编辑器关闭。两个变量都未设置、格式无效或启动失败时返回 `open_failed`;如果配置刚刚创建成功,文件仍会保留以便手动编辑。
|
|
111
113
|
|
|
112
|
-
`config
|
|
114
|
+
`config init` 按标准凭据环境变量创建新配置,已存在文件时拒绝覆盖。它保留全部内置 Instance 定义,将检测到的 Search provider 写入主 Route,Search fallback 留空,将检测到的远程 Extract provider 写入主 Route,并把 `http` 放入 Extract fallback。能力命令使用 `auto` 成功或发生可回退失败后,会原子重排对应的 `providers` 或 `providers_fallback` 数组;成员不会跨组移动。旧的 `_providers` 和 `_providers_fallback` 字段会被严格配置校验拒绝。
|
|
113
115
|
|
|
114
116
|
完整 JSON Schema 位于 [schemas/config.schema.json](schemas/config.schema.json)。示例:
|
|
115
117
|
|
|
116
118
|
```json
|
|
117
119
|
{
|
|
118
|
-
"$schema": "https://unpkg.com/web-access-cli@0.
|
|
120
|
+
"$schema": "https://unpkg.com/web-access-cli@0.5.0/schemas/config.schema.json",
|
|
119
121
|
"providers": [
|
|
120
122
|
{
|
|
121
123
|
"id": "exa_team",
|
|
@@ -138,13 +140,15 @@ CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两
|
|
|
138
140
|
],
|
|
139
141
|
"search": {
|
|
140
142
|
"providers": ["searx_local", "exa_team", "brave"],
|
|
143
|
+
"providers_fallback": ["deepseek"],
|
|
141
144
|
"limit": 5,
|
|
142
|
-
"timeoutMs":
|
|
143
|
-
"attemptTimeoutMs":
|
|
145
|
+
"timeoutMs": 120000,
|
|
146
|
+
"attemptTimeoutMs": 60000,
|
|
144
147
|
"maxResponseBytes": 5242880
|
|
145
148
|
},
|
|
146
149
|
"extract": {
|
|
147
|
-
"providers": ["firecrawl_local", "jina", "exa_team"
|
|
150
|
+
"providers": ["firecrawl_local", "jina", "exa_team"],
|
|
151
|
+
"providers_fallback": ["http"],
|
|
148
152
|
"timeoutMs": 120000,
|
|
149
153
|
"attemptTimeoutMs": 45000,
|
|
150
154
|
"maxResponseBytes": 5242880,
|
|
@@ -153,16 +157,20 @@ CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两
|
|
|
153
157
|
}
|
|
154
158
|
```
|
|
155
159
|
|
|
156
|
-
内置 Instance 为 `tavily`、`exa`、`bocha`、`brave`、`searxng`、`firecrawl`、`jina`、`http`、`anysearch`、`xcrawl`、`deepseek`。配置同 ID 时会覆盖内置实例的字段;自定义 ID 可以创建同 Type
|
|
160
|
+
内置 Instance 为 `tavily`、`exa`、`bocha`、`brave`、`searxng`、`firecrawl`、`jina`、`http`、`anysearch`、`xcrawl`、`deepseek`、`xai_x_search`、`xai_web_search`。配置同 ID 时会覆盖内置实例的字段;自定义 ID 可以创建同 Type 的额外实例。出现在对应主 Route 或 fallback Route 中的实例才启用。Bocha 默认 base URL 为 `https://api.bocha.cn`,必须配置 API key;AnySearch 默认 base URL 为 `https://api.anysearch.com`,支持匿名调用;XCrawl 默认 base URL 为 `https://run.xcrawl.com`,必须配置 API key;DeepSeek 默认 base URL 为 `https://api.deepseek.com/anthropic/v1`,必须配置 API key。
|
|
157
161
|
|
|
158
162
|
默认 Route:
|
|
159
163
|
|
|
160
|
-
- Search
|
|
161
|
-
-
|
|
164
|
+
- Search 主轮:`tavily -> exa -> bocha -> brave -> searxng -> anysearch -> xcrawl`
|
|
165
|
+
- Search fallback:`deepseek -> xai_x_search -> xai_web_search`
|
|
166
|
+
- Extract 主轮:`firecrawl -> jina -> exa -> anysearch -> xcrawl`
|
|
167
|
+
- Extract fallback:`http`
|
|
168
|
+
|
|
169
|
+
默认 Route 将内置 provider 分为主轮和 fallback。自定义 ID 会合并到 Instance 列表,但仍需显式加入 Route;省略能力 Route 时使用上述默认分组,Search 主 Route 显式空数组会报错,Extract 主 Route 可以显式为空以禁用 Extract。`auto` 会跳过未完成配置的 Instance。AnySearch 与 XCrawl 可设置 `searchFilterMode`:`strict`(默认,遇到 freshness 时跳过)或 `best_effort`(将日期改写为查询片段)。域名条件会改写查询并在本地再次严格过滤。XCrawl Extract 固定使用同步 Scrape 的 Markdown 输出;Map、Crawl 和异步任务不属于当前 CLI 能力。
|
|
162
170
|
|
|
163
|
-
|
|
171
|
+
DeepSeek Search 通过 Anthropic-compatible Messages API 调用原生 `web_search_20250305` server tool,一次搜索是完整模型轮次,因此延迟和成本可能高于专用搜索 endpoint。它位于默认 Search fallback,仅在全部主轮 Provider 未配置、返回最终非 2xx HTTP 响应或发生其他可恢复失败后触发。Adapter 只接受 `web_search_tool_result` 中的结构化 URL,按 URL 合并 citation 摘要,绝不从模型 prose 中猜测 URL。域名条件会改写查询并在本地再次严格过滤;DeepSeek 不支持 `freshness`,遇到该参数时以可恢复错误跳过;重定向会被严格拒绝,且不会访问 `Location` 目标。Provider 私有的 `encrypted_content` 是 CLI 无法展示或解码的 opaque payload,不具备诊断价值,因此会从 `raw` 中移除。
|
|
164
172
|
|
|
165
|
-
|
|
173
|
+
XAI hosted Search 使用 `XAI_AUTH_JSON` 指向外部 OAuth JSON 文件,每次调用只读取 `access_token`,由外部 CLIProxyAPI 负责刷新和回写。可用 `authJson`/`authJsonEnv` 与 `model`/`modelEnv` 为实例覆盖路径和模型。`xai_web_search` 支持最多 5 个 allow 或 exclude 域名(两者不能同时使用);`xai_x_search` 不接受 `freshness`。模型搜索可能需要更长时间;Search 默认单次超时 60 秒、总超时 120 秒。
|
|
166
174
|
|
|
167
175
|
### 凭据和 URL
|
|
168
176
|
|
|
@@ -181,26 +189,29 @@ DeepSeek Search 通过 Anthropic-compatible Messages API 调用原生 `web_searc
|
|
|
181
189
|
| AnySearch | `ANYSEARCH_API_KEY`,可选 | `ANYSEARCH_BASE_URL`,默认 `https://api.anysearch.com` |
|
|
182
190
|
| XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`,默认 `https://run.xcrawl.com` |
|
|
183
191
|
| DeepSeek | `DEEPSEEK_API_KEY` | 无标准环境变量;默认 `https://api.deepseek.com/anthropic/v1` |
|
|
192
|
+
| xAI Search | `XAI_AUTH_JSON` | 无标准环境变量;默认 `https://cli-chat-proxy.grok.com/v1`,模型 `grok-4.6` |
|
|
184
193
|
|
|
185
194
|
自定义 Instance 使用 `apiKeyEnv` 和 `baseUrlEnv` 指定自己的环境变量。所有远端 Provider 都可以设置 `baseUrl` 和附加 `headers`。公共 `api.firecrawl.dev` 需要 key;自托管 Firecrawl v2 可以不设置 key。
|
|
186
195
|
|
|
187
196
|
## 执行与回退
|
|
188
197
|
|
|
189
|
-
`--provider auto`
|
|
198
|
+
`--provider auto` 先按 `providers` 顺序执行;只有主轮全部失败且结果可回退时才进入 `providers_fallback`:
|
|
190
199
|
|
|
191
200
|
1. 未完成配置的实例会记录为失败 attempt 并跳过。
|
|
192
201
|
2. Provider 返回的最终 HTTP 响应不在 2xx 范围时,包括鉴权、请求、限流和服务器错误,Router 会尝试下一个实例。
|
|
193
202
|
3. 网络错误、超时、响应过大、无可用正文等其他可恢复错误也会继续 Route。
|
|
194
203
|
4. 没有最终非 2xx 响应的不可恢复错误,例如无效输入或在成功响应中检测到不支持的内容,会立即停止。
|
|
195
|
-
5.
|
|
204
|
+
5. 两组实例全部失败时返回 `provider_exhausted`。提取过程中产生的最佳短正文会保留在 `partial`。
|
|
205
|
+
|
|
206
|
+
执行这些 Route 前,`extract --provider auto` 会在任一组启用了 HTTP Instance 时先探测一个源站 Markdown 变体:文件型路径追加 `.md`,目录路径使用 `index.md`,请求 `text/markdown`/`text/plain`,只接受足够长且非 HTML 的文本响应。未命中或请求失败时不产生额外 attempt,并继续原有 Route;命中时结果归属已启用的 HTTP Instance,且不改变 Provider 顺序。显式非 HTTP 选择不变;显式 HTTP 会先探测 Markdown,未命中后再请求原 URL。
|
|
196
207
|
|
|
197
|
-
每次 `auto` 调用完成后,成功 Instance
|
|
208
|
+
每次 `auto` 调用完成后,成功 Instance 移到所在组队头,未尝试 Instance 保持在中间,发生上述可回退失败的 Instance 稳定移到所在组队尾;顺序学习不会跨组移动 provider。所有 Instance 都失败时顺序不变。Search 与 Extract 独立学习;显式 Instance、不可回退错误和用户取消不会更新顺序。写入采用原子替换,并发进程由最后写入者生效。
|
|
198
209
|
|
|
199
210
|
Attempt 会保留 Provider 的原始错误码、HTTP status 和 `retryable` 值。`retryable` 表示原始操作是否适合对同一 Provider 重试,不再是自动 Route 切换到下一 Provider 的唯一条件。
|
|
200
211
|
|
|
201
212
|
显式指定 Instance 时严格执行,不触发 fallback。若实例不在 Route 中,返回 `provider_disabled`。
|
|
202
213
|
|
|
203
|
-
默认上限:Search 总超时/单次超时为 60s
|
|
214
|
+
默认上限:Search 总超时/单次超时为 120s/60s,Extract 为 120s/45s;单个响应硬上限为 5 MiB。`--timeout` 只覆盖本次命令的总超时。
|
|
204
215
|
|
|
205
216
|
## 输出契约
|
|
206
217
|
|
|
@@ -224,9 +235,9 @@ Attempt 会保留 Provider 的原始错误码、HTTP status 和 `retryable` 值
|
|
|
224
235
|
}
|
|
225
236
|
```
|
|
226
237
|
|
|
227
|
-
|
|
238
|
+
使用 `--json` 时,能力命令成功只包含 `schemaVersion`、`ok`、最终 Instance ID 和规范化 `data`。失败时包含精简 `error`;实际尝试过 Provider 时,`attempts` 只保留 Instance ID、错误码和可选 HTTP status。提取质量失败可包含不含 raw 的 `partial` 文档。排序无法写回时,成功或失败 envelope 会额外包含 `warnings: [{ "code": "provider_order_update_failed", "message": "..." }]`,但不会覆盖主要结果。搜索和提取输入错误也使用相同的精简失败结构。
|
|
228
239
|
|
|
229
|
-
|
|
240
|
+
面向人的输出中,提取以 YAML front matter 开头、随后是 Provider 规范化的 Markdown 正文;搜索使用编号 Markdown 列表;诊断和配置编辑使用标题加格式化 JSON 数据。元数据值统一使用 JSON 双引号标量编码,正文内容不改写。已有脚本和其他机器消费者应显式追加 `--json`。
|
|
230
241
|
|
|
231
242
|
默认提取输出示例:
|
|
232
243
|
|
|
@@ -240,16 +251,16 @@ title: "Example title"
|
|
|
240
251
|
# Article content
|
|
241
252
|
```
|
|
242
253
|
|
|
243
|
-
|
|
254
|
+
所有成功和失败 envelope 使用 schema version 2。稳定 schema 位于 [schemas](schemas)。
|
|
244
255
|
|
|
245
256
|
退出码:
|
|
246
257
|
|
|
247
258
|
- `0`:成功
|
|
248
259
|
- `2`:输入、配置、未知或未启用 provider
|
|
249
|
-
- `1`:运行时/provider/doctor
|
|
260
|
+
- `1`:运行时/provider/doctor/编辑器启动失败
|
|
250
261
|
- `130`:用户取消
|
|
251
262
|
|
|
252
|
-
`providers` 会列出每个 Instance 的 Type、能力、Route 启用状态、凭据来源和 base URL 来源;`searchRoute` 与 `extractRoute` 是下一次 `auto`
|
|
263
|
+
`providers` 会列出每个 Instance 的 Type、能力、Route 启用状态、凭据来源和 base URL 来源;`searchRoute`/`searchFallbackRoute` 与 `extractRoute`/`extractFallbackRoute` 是下一次 `auto` 使用的两组有效顺序。`doctor` 只做本地配置检查,不主动调用远端 API;当已启用 Route 中存在未配置 Instance 时,命令返回 `doctor_failed` 和退出码 `1`。
|
|
253
264
|
|
|
254
265
|
## HTTP 与安全边界
|
|
255
266
|
|
|
@@ -257,6 +268,7 @@ title: "Example title"
|
|
|
257
268
|
- 最多跟随 5 次重定向;跨 origin 重定向会移除鉴权、Cookie 和常见 token headers。
|
|
258
269
|
- 所有响应都执行流式字节上限,避免先完整缓冲超大响应。
|
|
259
270
|
- HTTP Extract 使用 LinkeDOM、Mozilla Readability、Turndown,并对 Next.js RSC payload 做后备解析。
|
|
271
|
+
- HTTP Extract 在 route 已启用 HTTP Instance 时优先读取可用的源站 Markdown 变体(`.md` 或目录 `index.md`)。
|
|
260
272
|
- PDF、图片、音频、视频、zip 和通用二进制内容会返回 `unsupported_content`。
|
|
261
273
|
- 按项目设计,CLI 不阻止 localhost、私网地址或云元数据 URL。调用者必须在不可信输入场景中自行实施 URL allowlist、网络隔离或出站代理策略。
|
|
262
274
|
|
package/THIRD_PARTY_NOTICES.md
CHANGED
|
@@ -46,6 +46,17 @@ DeepSeek Search Provider 的 Anthropic-compatible Messages wire shape、结构
|
|
|
46
46
|
|
|
47
47
|
动态 Settings、Credential Service、Session event、LLM seam 和 Agent 集成代码没有进入本项目。
|
|
48
48
|
|
|
49
|
+
## Grok Build xAI Search
|
|
50
|
+
|
|
51
|
+
The xAI hosted search request and response mapping is derived from the `grok-build` project:
|
|
52
|
+
|
|
53
|
+
- Repository: `https://github.com/xai-org/grok-build`
|
|
54
|
+
- Source revision: `a549186d9d39311f2d3ee4208db62af8c65aa476`
|
|
55
|
+
- Copyright 2023-2026 SpaceXAI
|
|
56
|
+
- License: Apache-2.0
|
|
57
|
+
|
|
58
|
+
The adapted code is limited to the Responses tool wire shape and citation mapping. The full Apache-2.0 license is retained in the upstream repository.
|
|
59
|
+
|
|
49
60
|
## npm 依赖
|
|
50
61
|
|
|
51
62
|
发行包直接使用以下开源组件:
|
package/dist/cli-output.js
CHANGED
|
@@ -15,4 +15,20 @@ export function formatExtractMarkdown(envelope) {
|
|
|
15
15
|
const content = document.content;
|
|
16
16
|
return `${lines.join("\n")}${content.endsWith("\n") ? content : `${content}\n`}`;
|
|
17
17
|
}
|
|
18
|
+
export function formatMarkdown(envelope) {
|
|
19
|
+
if ("provider" in envelope && "document" in envelope.data)
|
|
20
|
+
return formatExtractMarkdown(envelope);
|
|
21
|
+
if ("provider" in envelope && "results" in envelope.data) {
|
|
22
|
+
const e = envelope;
|
|
23
|
+
const body = e.data.results
|
|
24
|
+
.map((result, index) => `${index + 1}. [${result.title}](${result.url})${result.snippet ? `\n ${result.snippet}` : ""}`)
|
|
25
|
+
.join("\n");
|
|
26
|
+
return `---\nprovider: ${JSON.stringify(e.provider)}\n---\n\n${body || "无搜索结果"}\n`;
|
|
27
|
+
}
|
|
28
|
+
if ("command" in envelope && envelope.ok) {
|
|
29
|
+
const e = envelope;
|
|
30
|
+
return `# ${e.command}\n\n${JSON.stringify(e.data, null, 2)}\n`;
|
|
31
|
+
}
|
|
32
|
+
return "";
|
|
33
|
+
}
|
|
18
34
|
//# sourceMappingURL=cli-output.js.map
|
package/dist/cli-output.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli-output.js","sourceRoot":"","sources":["../src/cli-output.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"cli-output.js","sourceRoot":"","sources":["../src/cli-output.ts"],"names":[],"mappings":"AAUA,SAAS,UAAU,CAAC,KAAa;IAChC,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;AAC9B,CAAC;AAED,MAAM,UAAU,qBAAqB,CACpC,QAAgC;IAEhC,MAAM,EAAE,QAAQ,EAAE,GAAG,QAAQ,CAAC,IAAI,CAAC;IACnC,MAAM,KAAK,GAAG;QACb,KAAK;QACL,aAAa,UAAU,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE;QAC5C,QAAQ,UAAU,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE;KACxC,CAAC;IACF,MAAM,KAAK,GAAG,QAAQ,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;IACpC,IAAI,KAAK;QAAE,KAAK,CAAC,IAAI,CAAC,UAAU,UAAU,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACrD,KAAK,CAAC,IAAI,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC;IAC1B,MAAM,OAAO,GAAG,QAAQ,CAAC,OAAO,CAAC;IACjC,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,IAAI,EAAE,CAAC;AAClF,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,QAAwB;IACtD,IAAI,UAAU,IAAI,QAAQ,IAAI,UAAU,IAAI,QAAQ,CAAC,IAAI;QACxD,OAAO,qBAAqB,CAAC,QAAkC,CAAC,CAAC;IAClE,IAAI,UAAU,IAAI,QAAQ,IAAI,SAAS,IAAI,QAAQ,CAAC,IAAI,EAAE,CAAC;QAC1D,MAAM,CAAC,GAAG,QAAiC,CAAC;QAC5C,MAAM,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,OAAO;aACzB,GAAG,CACH,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE,CACjB,GAAG,KAAK,GAAG,CAAC,MAAM,MAAM,CAAC,KAAK,KAAK,MAAM,CAAC,GAAG,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,QAAQ,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAClG;aACA,IAAI,CAAC,IAAI,CAAC,CAAC;QACb,OAAO,kBAAkB,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,CAAC,YAAY,IAAI,IAAI,OAAO,IAAI,CAAC;IACpF,CAAC;IACD,IAAI,SAAS,IAAI,QAAQ,IAAI,QAAQ,CAAC,EAAE,EAAE,CAAC;QAC1C,MAAM,CAAC,GAAG,QAAiE,CAAC;QAC5E,OAAO,KAAK,CAAC,CAAC,OAAO,OAAO,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC;IACjE,CAAC;IACD,OAAO,EAAE,CAAC;AACX,CAAC"}
|