web-access-cli 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +270 -0
  3. package/README_CN.md +268 -0
  4. package/THIRD_PARTY_NOTICES.md +48 -0
  5. package/dist/cli.js +197 -0
  6. package/dist/cli.js.map +1 -0
  7. package/dist/config/config.js +463 -0
  8. package/dist/config/config.js.map +1 -0
  9. package/dist/config/edit.js +126 -0
  10. package/dist/config/edit.js.map +1 -0
  11. package/dist/core/diagnostics.js +94 -0
  12. package/dist/core/diagnostics.js.map +1 -0
  13. package/dist/core/errors.js +47 -0
  14. package/dist/core/errors.js.map +1 -0
  15. package/dist/core/router.js +320 -0
  16. package/dist/core/router.js.map +1 -0
  17. package/dist/core/schema.js +120 -0
  18. package/dist/core/schema.js.map +1 -0
  19. package/dist/core/types.js +22 -0
  20. package/dist/core/types.js.map +1 -0
  21. package/dist/providers/anysearch.js +225 -0
  22. package/dist/providers/anysearch.js.map +1 -0
  23. package/dist/providers/common.js +210 -0
  24. package/dist/providers/common.js.map +1 -0
  25. package/dist/providers/extract.js +155 -0
  26. package/dist/providers/extract.js.map +1 -0
  27. package/dist/providers/registry.js +31 -0
  28. package/dist/providers/registry.js.map +1 -0
  29. package/dist/providers/rsc.js +109 -0
  30. package/dist/providers/rsc.js.map +1 -0
  31. package/dist/providers/search.js +178 -0
  32. package/dist/providers/search.js.map +1 -0
  33. package/dist/providers/xcrawl.js +170 -0
  34. package/dist/providers/xcrawl.js.map +1 -0
  35. package/dist/transport/http.js +175 -0
  36. package/dist/transport/http.js.map +1 -0
  37. package/dist/version.js +2 -0
  38. package/dist/version.js.map +1 -0
  39. package/package.json +61 -0
  40. package/schemas/config.schema.json +210 -0
  41. package/schemas/extractRequest.schema.json +21 -0
  42. package/schemas/output.schema.json +435 -0
  43. package/schemas/searchRequest.schema.json +57 -0
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 web-access-cli contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,270 @@
1
+ # web-access-cli
2
+
3
+ English | [简体中文](README_CN.md)
4
+
5
+ [![CI](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml)
6
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
7
+
8
+ An agent-neutral CLI for web capabilities. It defines search and web content extraction as stable capabilities while normalizing differences among Tavily, Exa, Brave, SearXNG, AnySearch, XCrawl, Firecrawl, Jina, and other providers behind a unified internal schema.
9
+
10
+ The CLI is the primary product interface. It is not tied to Pi, Claude Code, Codex, Cursor, OpenCode, or any other agent. Skills, MCP integrations, and agent plugins may only integrate as adapters on top of the CLI; they must not leak into the core capabilities or provider implementations.
11
+
12
+ ## Capabilities and providers
13
+
14
+ | Capability | Provider types | Normalized output |
15
+ | --- | --- | --- |
16
+ | `search` | Tavily, Exa, Brave, SearXNG, AnySearch, XCrawl | `rank`, `title`, `url`, `snippet` |
17
+ | `extract` | Firecrawl v2, Jina Reader, Exa Contents, AnySearch, XCrawl, HTTP | Markdown `Document` |
18
+
19
+ A provider type describes an implementation, while a provider instance is a configurable instance of that type. One type can have multiple instances, such as `exa_team` and `exa_personal`. A route is an ordered array of instance IDs that determines both which instances are enabled and the order in which `auto` tries them.
20
+
21
+ ## Installation
22
+
23
+ Node.js 22.19 or later is required.
24
+
25
+ The npm package has not been published yet. Build and run the CLI from source:
26
+
27
+ ```sh
28
+ git clone https://github.com/Juexe/web-access-cli.git
29
+ cd web-access-cli
30
+ pnpm install --frozen-lockfile
31
+ pnpm build
32
+ node dist/cli.js --help
33
+ ```
34
+
35
+ After building, an optional global installation can point to the absolute repository path:
36
+
37
+ ```sh
38
+ pnpm add --global <absolute-path-to-web-access-cli>
39
+ web-access --help
40
+ ```
41
+
42
+ For development in this repository:
43
+
44
+ ```sh
45
+ pnpm check
46
+ pnpm pack:check
47
+ ```
48
+
49
+ ## Project structure
50
+
51
+ ```text
52
+ src/cli.ts Commander CLI and exit codes
53
+ src/config/ Configuration files, environment variables, and route merging
54
+ src/core/ Public types, schemas, errors, routing, and diagnostics
55
+ src/providers/ Provider adapters, registry, and HTTP/RSC extraction
56
+ src/transport/ Proxies, timeouts, redirects, and response size limits
57
+ schemas/ Generated JSON Schemas
58
+ test/ node:test unit tests and local HTTP integration tests
59
+ ```
60
+
61
+ Provider adapters should handle protocol mapping only. Fallback, deadlines, attempts, envelopes, and exit codes are managed centrally by `src/core`. Do not add lifecycle or tool registration code for Pi, Claude Code, Codex, or any other agent to an adapter.
62
+
63
+ ## Development workflow
64
+
65
+ ```sh
66
+ pnpm install
67
+ pnpm lint
68
+ pnpm typecheck
69
+ pnpm test
70
+ pnpm build
71
+ pnpm pack:check
72
+ ```
73
+
74
+ `pnpm check` runs linting, type checking, tests, and the build in that order. After changing `src/core/schema.ts`, run `pnpm build` again and commit the updated `schemas/*.schema.json` files. Provider tests should use the mock transport or local HTTP server from `test/helpers.ts`; they must not depend on real API keys or external network reliability.
75
+
76
+ ## Commands
77
+
78
+ ```sh
79
+ web-access search "Agent-neutral web CLI"
80
+ web-access search "TypeScript release" --provider exa --limit 10 --freshness month
81
+ web-access search "browser automation" --include-domain example.com --exclude-domain blocked.example.com
82
+
83
+ web-access extract https://example.com/article
84
+ web-access extract https://example.com/article --provider http --timeout 30000
85
+
86
+ web-access providers --pretty
87
+ web-access doctor --pretty
88
+
89
+ web-access config edit
90
+ web-access --config "/path/to/config.json" config edit
91
+ ```
92
+
93
+ The CLI provides the `search` and `extract` capability commands, the `providers` and `doctor` diagnostic commands, and the `config edit` configuration command. The current version does not provide batch/all, answer, a dedicated PDF path, a Node.js SDK, or a generic MCP integration. The repository includes an optional [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md) for source-checkout use; it is not included in the npm package.
94
+
95
+ ### Global options
96
+
97
+ - `--config <path>`: Explicitly select a configuration file.
98
+ - `--pretty`: Pretty-print JSON. By default, JSON is written on a single line.
99
+ - `--help`, `--version`: Print standard CLI help or version text.
100
+
101
+ Except for help and version output, stdout always contains exactly one JSON envelope. Diagnostic messages are never mixed into stdout.
102
+
103
+ ## Configuration
104
+
105
+ Configuration files use strict JSON; unknown fields are rejected. The CLI reads only an explicit path or a user-level path and does not search parent project directories for configuration files.
106
+
107
+ The default path on every platform is `~/.config/web-access-cli/config.json`, where `~` is the current user's home directory. The `WEB_ACCESS_CONFIG` environment variable can select another path. The `--config` command-line option takes precedence.
108
+
109
+ `web-access config edit` creates parent directories and a complete default configuration when the file is missing, then opens it with the operating system's default application for JSON files. An existing file is opened byte-for-byte as-is, even when it is temporarily invalid JSON; it is never overwritten or reformatted. The command waits only for the operating system to accept the open request, not for the editor to close. On success, envelope `data` contains the absolute `path`, `created`, and `opened: true`. A default-application launch failure returns `open_failed`; a configuration file that was created successfully remains available for manual editing.
110
+
111
+ The complete JSON Schema is available at [schemas/config.schema.json](schemas/config.schema.json). Example:
112
+
113
+ ```json
114
+ {
115
+ "$schema": "https://unpkg.com/web-access-cli@0.1.0/schemas/config.schema.json",
116
+ "providers": [
117
+ {
118
+ "id": "exa_team",
119
+ "type": "exa",
120
+ "apiKeyEnv": "TEAM_EXA_API_KEY",
121
+ "headers": {
122
+ "X-Team": "docs"
123
+ }
124
+ },
125
+ {
126
+ "id": "searx_local",
127
+ "type": "searxng",
128
+ "baseUrl": "http://127.0.0.1:8080"
129
+ },
130
+ {
131
+ "id": "firecrawl_local",
132
+ "type": "firecrawl",
133
+ "baseUrl": "http://127.0.0.1:3002"
134
+ }
135
+ ],
136
+ "search": {
137
+ "providers": ["searx_local", "exa_team", "brave"],
138
+ "limit": 5,
139
+ "timeoutMs": 60000,
140
+ "attemptTimeoutMs": 20000,
141
+ "maxResponseBytes": 5242880
142
+ },
143
+ "extract": {
144
+ "providers": ["firecrawl_local", "jina", "exa_team", "http"],
145
+ "timeoutMs": 120000,
146
+ "attemptTimeoutMs": 45000,
147
+ "maxResponseBytes": 5242880,
148
+ "minContentCharacters": 500
149
+ }
150
+ }
151
+ ```
152
+
153
+ The built-in instances are `tavily`, `exa`, `brave`, `searxng`, `firecrawl`, `jina`, `http`, `anysearch`, and `xcrawl`. A configuration entry with the same ID overrides fields on the built-in instance. A custom ID creates another instance of the selected type. An instance is enabled only when it appears in the corresponding `providers` route. AnySearch defaults to `https://api.anysearch.com` and supports anonymous calls; XCrawl defaults to `https://run.xcrawl.com` and requires an API key.
154
+
155
+ Default routes:
156
+
157
+ - Search: `tavily -> exa -> brave -> searxng -> anysearch -> xcrawl`
158
+ - Extract: `firecrawl -> jina -> exa -> anysearch -> xcrawl -> http`
159
+
160
+ The default routes include every built-in instance that supports the corresponding capability. Custom instance IDs are merged into the instance list but must still be added to routes explicitly. An omitted route uses the defaults above, while an explicit empty array disables that capability. In `auto` mode, incompletely configured instances are skipped; AnySearch can be called anonymously with its default base URL, and XCrawl is skipped until an API key is configured. Both instance types accept `searchFilterMode`: `strict` (default; freshness skips the provider) or `best_effort` (rewrites freshness into a query fragment). Domain constraints are rewritten into the query and strictly re-applied locally. XCrawl Extract always uses synchronous Scrape with Markdown output; Map, Crawl, and asynchronous jobs are outside the current CLI capabilities.
161
+
162
+ ### Credentials and URLs
163
+
164
+ Environment variables take precedence over plaintext keys in JSON. Built-in instances support these standard variables:
165
+
166
+ | Type | API key | Base URL |
167
+ | --- | --- | --- |
168
+ | Tavily | `TAVILY_API_KEY` | `TAVILY_BASE_URL` |
169
+ | Exa | `EXA_API_KEY` | `EXA_BASE_URL` |
170
+ | Brave | `BRAVE_API_KEY` | `BRAVE_BASE_URL` |
171
+ | SearXNG | None | `SEARXNG_BASE_URL`, required |
172
+ | Firecrawl | `FIRECRAWL_API_KEY` | `FIRECRAWL_BASE_URL` |
173
+ | Jina | `JINA_API_KEY`, optional | `JINA_BASE_URL` |
174
+ | HTTP | None | None |
175
+ | AnySearch | `ANYSEARCH_API_KEY`, optional | `ANYSEARCH_BASE_URL`, default `https://api.anysearch.com` |
176
+ | XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`, default `https://run.xcrawl.com` |
177
+
178
+ Custom instances use `apiKeyEnv` and `baseUrlEnv` to name their environment variables. Every remote provider can define a `baseUrl` and additional `headers`. The public `api.firecrawl.dev` service requires a key; a self-hosted Firecrawl v2 instance can run without one.
179
+
180
+ ## Execution and fallback
181
+
182
+ `--provider auto` follows the configured route order:
183
+
184
+ 1. An instance with incomplete configuration is recorded as a failed attempt and skipped.
185
+ 2. Recoverable errors, including network errors, timeouts, rate limits, 5xx responses, oversized responses, and missing usable content, cause the router to try the next instance.
186
+ 3. Non-recoverable errors, including authentication errors, invalid input, and unsupported content, stop execution immediately.
187
+ 4. If every instance fails, the command returns `provider_exhausted`. The best short content produced during extraction is preserved in `partial`.
188
+
189
+ Selecting an instance explicitly executes it strictly without fallback. If the instance is not in the route, the command returns `provider_disabled`.
190
+
191
+ Default limits are a 60-second total timeout and a 20-second per-attempt timeout for Search, and 120 seconds and 45 seconds respectively for Extract. Each response has a hard limit of 5 MiB. `--timeout` overrides only the total timeout for the current command.
192
+
193
+ ## Output contract
194
+
195
+ Successful response example:
196
+
197
+ ```json
198
+ {
199
+ "schemaVersion": 1,
200
+ "ok": true,
201
+ "command": "search",
202
+ "durationMs": 183,
203
+ "request": {
204
+ "query": "Agent-neutral web CLI",
205
+ "provider": "auto",
206
+ "limit": 5,
207
+ "includeDomains": [],
208
+ "excludeDomains": []
209
+ },
210
+ "provider": { "id": "exa", "type": "exa" },
211
+ "attempts": [
212
+ {
213
+ "provider": { "id": "tavily", "type": "tavily" },
214
+ "status": "failed",
215
+ "durationMs": 0,
216
+ "error": {
217
+ "code": "provider_unavailable",
218
+ "message": "tavily 未完成 search 所需配置",
219
+ "retryable": true
220
+ }
221
+ },
222
+ {
223
+ "provider": { "id": "exa", "type": "exa" },
224
+ "status": "success",
225
+ "durationMs": 183
226
+ }
227
+ ],
228
+ "data": {
229
+ "results": [
230
+ {
231
+ "rank": 1,
232
+ "title": "Example",
233
+ "url": "https://example.com/",
234
+ "snippet": "Example snippet"
235
+ }
236
+ ]
237
+ },
238
+ "raw": {}
239
+ }
240
+ ```
241
+
242
+ The Chinese `message` value above matches the CLI's current error output contract.
243
+
244
+ `raw` contains only the response body from the final success or best failure. It never contains requests, response headers, or every historical response. API keys are redacted from serializable `raw` data and error messages. Stable schemas are available in [schemas](schemas).
245
+
246
+ Exit codes:
247
+
248
+ - `0`: Success
249
+ - `2`: Invalid input or configuration, or an unknown or disabled provider
250
+ - `1`: Runtime, provider, doctor, or default-application launch failure
251
+ - `130`: User cancellation
252
+
253
+ `providers` lists each instance's type, capabilities, route status, credential source, and base URL source. `doctor` performs local configuration checks only and does not call remote APIs. If an enabled route contains an unconfigured instance, the command returns `doctor_failed` with exit code `1`.
254
+
255
+ ## HTTP and security boundaries
256
+
257
+ - Honors `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY` by default.
258
+ - Follows up to five redirects. Cross-origin redirects remove authentication, Cookie, and common token headers.
259
+ - Enforces a streaming byte limit on every response instead of buffering an unbounded response first.
260
+ - HTTP Extract uses LinkeDOM, Mozilla Readability, and Turndown, with fallback parsing for Next.js RSC payloads.
261
+ - Returns `unsupported_content` for PDF, image, audio, video, zip, and general binary content.
262
+ - By design, the CLI does not block localhost, private network addresses, or cloud metadata URLs. For untrusted input, callers must enforce their own URL allowlist, network isolation, or outbound proxy policy.
263
+
264
+ ## Project scope
265
+
266
+ This project draws heavily from the provider and content extraction implementations in `pi-web-access`, but does not include Pi-specific registration, tool protocols, or UI code.
267
+
268
+ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) before opening an issue or pull request. Report security vulnerabilities privately according to [SECURITY.md](SECURITY.md).
269
+
270
+ Licensed under MIT. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for third-party and upstream attribution.
package/README_CN.md ADDED
@@ -0,0 +1,268 @@
1
+ # web-access-cli
2
+
3
+ [English](README.md) | 简体中文
4
+
5
+ [![CI](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml)
6
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
7
+
8
+ 一个 Agent-neutral 的网页能力 CLI。它把“搜索”和“网页正文提取”定义为稳定能力,把 Tavily、Exa、Brave、SearXNG、AnySearch、XCrawl、Firecrawl、Jina 等差异收敛到内部统一 schema。
9
+
10
+ CLI 是主要产品形态,不绑定 Pi、Claude Code、Codex、Cursor、OpenCode 或其他 Agent。Skill、MCP 和 Agent 插件只能作为 CLI 上层 adapter 接入,不能污染核心能力和 Provider 实现。
11
+
12
+ ## 能力与 provider
13
+
14
+ | 能力 | Provider Type | 统一输出 |
15
+ | --- | --- | --- |
16
+ | `search` | Tavily、Exa、Brave、SearXNG、AnySearch、XCrawl | `rank`、`title`、`url`、`snippet` |
17
+ | `extract` | Firecrawl v2、Jina Reader、Exa Contents、AnySearch、XCrawl、HTTP | Markdown `Document` |
18
+
19
+ Provider Type 描述实现类型;Provider Instance 是一份可配置实例。一个 Type 可以有多个 Instance,例如 `exa_team` 和 `exa_personal`。Route 是有序 Instance ID 数组,既决定启用状态,也决定 `auto` 的尝试顺序。
20
+
21
+ ## 安装
22
+
23
+ 要求 Node.js 22.19 或更高版本。
24
+
25
+ npm 包尚未发布。当前请从源码构建并运行:
26
+
27
+ ```sh
28
+ git clone https://github.com/Juexe/web-access-cli.git
29
+ cd web-access-cli
30
+ pnpm install --frozen-lockfile
31
+ pnpm build
32
+ node dist/cli.js --help
33
+ ```
34
+
35
+ 构建完成后,也可以使用仓库绝对路径安装为全局命令:
36
+
37
+ ```sh
38
+ pnpm add --global <web-access-cli 仓库绝对路径>
39
+ web-access --help
40
+ ```
41
+
42
+ 在仓库中开发:
43
+
44
+ ```sh
45
+ pnpm check
46
+ pnpm pack:check
47
+ ```
48
+
49
+ ## 项目结构
50
+
51
+ ```text
52
+ src/cli.ts Commander CLI 与退出码
53
+ src/config/ 配置文件、环境变量与 Route 合并
54
+ src/core/ 公共类型、schema、错误、路由和诊断
55
+ src/providers/ Provider adapter、registry、HTTP/RSC 提取
56
+ src/transport/ 代理、超时、重定向和响应大小限制
57
+ schemas/ 构建生成的 JSON Schema
58
+ test/ node:test 单元测试与本地 HTTP 集成测试
59
+ ```
60
+
61
+ Provider adapter 应只负责协议映射;fallback、deadline、attempts、envelope 和退出码由 `src/core` 统一处理。不要在 adapter 中引入 Pi、Claude Code、Codex 或其他 Agent 的生命周期和工具注册代码。
62
+
63
+ ## 开发工作流
64
+
65
+ ```sh
66
+ pnpm install
67
+ pnpm lint
68
+ pnpm typecheck
69
+ pnpm test
70
+ pnpm build
71
+ pnpm pack:check
72
+ ```
73
+
74
+ `pnpm check` 会按上述顺序执行 lint、类型检查、测试和构建。修改 `src/core/schema.ts` 后必须重新运行 `pnpm build`,并提交更新后的 `schemas/*.schema.json`。Provider 测试应使用 `test/helpers.ts` 的 mock transport 或本地 HTTP server,不依赖真实 API key 和外网稳定性。
75
+
76
+ ## 命令
77
+
78
+ ```sh
79
+ web-access search "Agent-neutral web CLI"
80
+ web-access search "TypeScript release" --provider exa --limit 10 --freshness month
81
+ web-access search "browser automation" --include-domain example.com --exclude-domain blocked.example.com
82
+
83
+ web-access extract https://example.com/article
84
+ web-access extract https://example.com/article --provider http --timeout 30000
85
+
86
+ web-access providers --pretty
87
+ web-access doctor --pretty
88
+
89
+ web-access config edit
90
+ web-access --config "/path/to/config.json" config edit
91
+ ```
92
+
93
+ CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两个诊断命令,以及 `config edit` 配置维护命令。当前版本不提供 batch/all、answer、PDF 专线、Node SDK 或通用 MCP 集成。仓库提供可选的 [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md),仅供源码仓库使用,不包含在 npm 发布包中。
94
+
95
+ ### 通用选项
96
+
97
+ - `--config <path>`:显式指定配置文件。
98
+ - `--pretty`:格式化 JSON;默认输出单行 JSON。
99
+ - `--help`、`--version`:输出常规 CLI 帮助或版本文本。
100
+
101
+ 除帮助与版本外,命令的 stdout 始终只有一个 JSON envelope。诊断信息不会混入 stdout。
102
+
103
+ ## 配置
104
+
105
+ 配置是严格 JSON;未知字段会报错。CLI 只读取显式路径或用户级路径,不会向上查找项目目录中的配置文件。
106
+
107
+ 所有平台的默认路径均为 `~/.config/web-access-cli/config.json`,其中 `~` 表示当前用户的主目录。环境变量 `WEB_ACCESS_CONFIG` 可以指定其他路径;命令行 `--config` 优先级更高。
108
+
109
+ `web-access config edit` 会在配置文件缺失时创建父目录和完整默认配置,再用系统为 JSON 文件关联的默认应用打开;已有文件会原样打开,即使内容暂时不是有效 JSON 也不会被覆盖或格式化。命令只等待系统接受打开请求,不等待编辑器关闭。成功 envelope 的 `data` 包含绝对 `path`、是否新建的 `created` 和 `opened: true`。无法启动默认应用时返回 `open_failed`;如果配置刚刚创建成功,文件仍会保留以便手动打开。
110
+
111
+ 完整 JSON Schema 位于 [schemas/config.schema.json](schemas/config.schema.json)。示例:
112
+
113
+ ```json
114
+ {
115
+ "$schema": "https://unpkg.com/web-access-cli@0.1.0/schemas/config.schema.json",
116
+ "providers": [
117
+ {
118
+ "id": "exa_team",
119
+ "type": "exa",
120
+ "apiKeyEnv": "TEAM_EXA_API_KEY",
121
+ "headers": {
122
+ "X-Team": "docs"
123
+ }
124
+ },
125
+ {
126
+ "id": "searx_local",
127
+ "type": "searxng",
128
+ "baseUrl": "http://127.0.0.1:8080"
129
+ },
130
+ {
131
+ "id": "firecrawl_local",
132
+ "type": "firecrawl",
133
+ "baseUrl": "http://127.0.0.1:3002"
134
+ }
135
+ ],
136
+ "search": {
137
+ "providers": ["searx_local", "exa_team", "brave"],
138
+ "limit": 5,
139
+ "timeoutMs": 60000,
140
+ "attemptTimeoutMs": 20000,
141
+ "maxResponseBytes": 5242880
142
+ },
143
+ "extract": {
144
+ "providers": ["firecrawl_local", "jina", "exa_team", "http"],
145
+ "timeoutMs": 120000,
146
+ "attemptTimeoutMs": 45000,
147
+ "maxResponseBytes": 5242880,
148
+ "minContentCharacters": 500
149
+ }
150
+ }
151
+ ```
152
+
153
+ 内置 Instance 为 `tavily`、`exa`、`brave`、`searxng`、`firecrawl`、`jina`、`http`、`anysearch`、`xcrawl`。配置同 ID 时会覆盖内置实例的字段;自定义 ID 可以创建同 Type 的额外实例。只有出现在对应 `providers` Route 中的实例才启用。AnySearch 默认 base URL 为 `https://api.anysearch.com`,支持匿名调用;XCrawl 默认 base URL 为 `https://run.xcrawl.com`,必须配置 API key。
154
+
155
+ 默认 Route:
156
+
157
+ - Search:`tavily -> exa -> brave -> searxng -> anysearch -> xcrawl`
158
+ - Extract:`firecrawl -> jina -> exa -> anysearch -> xcrawl -> http`
159
+
160
+ 默认 Route 包含支持对应 Capability 的全部内置 Instance。自定义 ID 会合并到 Instance 列表,但仍需显式加入 Route;省略 Route 时使用上述默认值,显式空数组则禁用对应能力。`auto` 会跳过未完成配置的 Instance;AnySearch 使用默认 base URL 时可匿名调用,XCrawl 则在配置 API key 前被跳过。两类 Instance 均可设置 `searchFilterMode`:`strict`(默认,遇到 freshness 时跳过)或 `best_effort`(将日期改写为查询片段)。域名条件会改写查询并在本地再次严格过滤。XCrawl Extract 固定使用同步 Scrape 的 Markdown 输出;Map、Crawl 和异步任务不属于当前 CLI 能力。
161
+
162
+ ### 凭据和 URL
163
+
164
+ 环境变量优先于 JSON 中的明文 key。内置 Instance 支持这些标准变量:
165
+
166
+ | Type | API key | Base URL |
167
+ | --- | --- | --- |
168
+ | Tavily | `TAVILY_API_KEY` | `TAVILY_BASE_URL` |
169
+ | Exa | `EXA_API_KEY` | `EXA_BASE_URL` |
170
+ | Brave | `BRAVE_API_KEY` | `BRAVE_BASE_URL` |
171
+ | SearXNG | 无 | `SEARXNG_BASE_URL`,必需 |
172
+ | Firecrawl | `FIRECRAWL_API_KEY` | `FIRECRAWL_BASE_URL` |
173
+ | Jina | `JINA_API_KEY`,可选 | `JINA_BASE_URL` |
174
+ | HTTP | 无 | 无 |
175
+ | AnySearch | `ANYSEARCH_API_KEY`,可选 | `ANYSEARCH_BASE_URL`,默认 `https://api.anysearch.com` |
176
+ | XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`,默认 `https://run.xcrawl.com` |
177
+
178
+ 自定义 Instance 使用 `apiKeyEnv` 和 `baseUrlEnv` 指定自己的环境变量。所有远端 Provider 都可以设置 `baseUrl` 和附加 `headers`。公共 `api.firecrawl.dev` 需要 key;自托管 Firecrawl v2 可以不设置 key。
179
+
180
+ ## 执行与回退
181
+
182
+ `--provider auto` 按 Route 顺序执行:
183
+
184
+ 1. 未完成配置的实例会记录为失败 attempt 并跳过。
185
+ 2. 网络错误、超时、限流、5xx、响应过大、无可用正文等可恢复错误会尝试下一个实例。
186
+ 3. 鉴权错误、无效输入、不支持的内容等不可恢复错误会立即停止。
187
+ 4. 所有实例失败时返回 `provider_exhausted`。提取过程中产生的最佳短正文会保留在 `partial`。
188
+
189
+ 显式指定 Instance 时严格执行,不触发 fallback。若实例不在 Route 中,返回 `provider_disabled`。
190
+
191
+ 默认上限:Search 总超时/单次超时为 60s/20s,Extract 为 120s/45s;单个响应硬上限为 5 MiB。`--timeout` 只覆盖本次命令的总超时。
192
+
193
+ ## 输出契约
194
+
195
+ 成功示例:
196
+
197
+ ```json
198
+ {
199
+ "schemaVersion": 1,
200
+ "ok": true,
201
+ "command": "search",
202
+ "durationMs": 183,
203
+ "request": {
204
+ "query": "Agent-neutral web CLI",
205
+ "provider": "auto",
206
+ "limit": 5,
207
+ "includeDomains": [],
208
+ "excludeDomains": []
209
+ },
210
+ "provider": { "id": "exa", "type": "exa" },
211
+ "attempts": [
212
+ {
213
+ "provider": { "id": "tavily", "type": "tavily" },
214
+ "status": "failed",
215
+ "durationMs": 0,
216
+ "error": {
217
+ "code": "provider_unavailable",
218
+ "message": "tavily 未完成 search 所需配置",
219
+ "retryable": true
220
+ }
221
+ },
222
+ {
223
+ "provider": { "id": "exa", "type": "exa" },
224
+ "status": "success",
225
+ "durationMs": 183
226
+ }
227
+ ],
228
+ "data": {
229
+ "results": [
230
+ {
231
+ "rank": 1,
232
+ "title": "Example",
233
+ "url": "https://example.com/",
234
+ "snippet": "Example snippet"
235
+ }
236
+ ]
237
+ },
238
+ "raw": {}
239
+ }
240
+ ```
241
+
242
+ `raw` 只包含最终成功或最佳失败的 response body,不包含请求、响应 headers 或所有历史响应。API key 会从可序列化的 `raw` 和错误消息中脱敏。稳定 schema 位于 [schemas](schemas)。
243
+
244
+ 退出码:
245
+
246
+ - `0`:成功
247
+ - `2`:输入、配置、未知或未启用 provider
248
+ - `1`:运行时/provider/doctor/默认应用启动失败
249
+ - `130`:用户取消
250
+
251
+ `providers` 会列出每个 Instance 的 Type、能力、Route 启用状态、凭据来源和 base URL 来源。`doctor` 只做本地配置检查,不主动调用远端 API;当已启用 Route 中存在未配置 Instance 时,命令返回 `doctor_failed` 和退出码 `1`。
252
+
253
+ ## HTTP 与安全边界
254
+
255
+ - 默认遵循 `HTTP_PROXY`、`HTTPS_PROXY`、`ALL_PROXY`、`NO_PROXY`。
256
+ - 最多跟随 5 次重定向;跨 origin 重定向会移除鉴权、Cookie 和常见 token headers。
257
+ - 所有响应都执行流式字节上限,避免先完整缓冲超大响应。
258
+ - HTTP Extract 使用 LinkeDOM、Mozilla Readability、Turndown,并对 Next.js RSC payload 做后备解析。
259
+ - PDF、图片、音频、视频、zip 和通用二进制内容会返回 `unsupported_content`。
260
+ - 按项目设计,CLI 不阻止 localhost、私网地址或云元数据 URL。调用者必须在不可信输入场景中自行实施 URL allowlist、网络隔离或出站代理策略。
261
+
262
+ ## 项目边界
263
+
264
+ 本项目大量参考 `pi-web-access` 的 provider 和内容提取实现,但不包含 Pi-specific 注册、工具协议或 UI 代码。
265
+
266
+ 欢迎贡献。提交 Issue 或 Pull Request 前请阅读 [CONTRIBUTING.md](CONTRIBUTING.md);安全漏洞请按照 [SECURITY.md](SECURITY.md) 私下报告。
267
+
268
+ 许可证为 MIT,第三方与上游归属见 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。
@@ -0,0 +1,48 @@
1
+ # 第三方与上游声明
2
+
3
+ ## pi-web-access
4
+
5
+ 本项目的 provider 映射、HTTP 正文提取和 Next.js RSC 后备解析大量参考并部分改写自:
6
+
7
+ - 项目:`pi-web-access`
8
+ - 仓库:`https://github.com/nicobailon/pi-web-access.git`
9
+ - 参考提交:`00b2271d0f1603ac780df3f324aed0fc92f3e849`
10
+ - 参考版本:`0.20.0`
11
+ - Copyright 2025 Nico Bailon
12
+ - 许可证:MIT
13
+
14
+ 上游 MIT 许可文本:
15
+
16
+ > Permission is hereby granted, free of charge, to any person obtaining a copy
17
+ > of this software and associated documentation files (the "Software"), to deal
18
+ > in the Software without restriction, including without limitation the rights
19
+ > to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
20
+ > copies of the Software, and to permit persons to whom the Software is
21
+ > furnished to do so, subject to the following conditions:
22
+ >
23
+ > The above copyright notice and this permission notice shall be included in all
24
+ > copies or substantial portions of the Software.
25
+ >
26
+ > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
27
+ > IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
28
+ > FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
29
+ > AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
30
+ > LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
31
+ > OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
32
+ > SOFTWARE.
33
+
34
+ Pi-specific 的工具注册、UI、Agent 生命周期与协议代码没有进入本项目。
35
+
36
+ ## npm 依赖
37
+
38
+ 发行包直接使用以下开源组件:
39
+
40
+ - `@mozilla/readability`:Apache-2.0
41
+ - `commander`:MIT
42
+ - `linkedom`:ISC
43
+ - `open`:MIT
44
+ - `turndown`:MIT
45
+ - `typebox`:MIT
46
+ - `undici`:MIT
47
+
48
+ 开发依赖包括 Biome、tsx、TypeScript 及对应类型包。各组件的完整许可证和版权声明以随 npm 包发布的许可证文件及其上游仓库为准。本文件不替代任何第三方许可证。