web-access-cli 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +270 -0
- package/README_CN.md +268 -0
- package/THIRD_PARTY_NOTICES.md +48 -0
- package/dist/cli.js +197 -0
- package/dist/cli.js.map +1 -0
- package/dist/config/config.js +463 -0
- package/dist/config/config.js.map +1 -0
- package/dist/config/edit.js +126 -0
- package/dist/config/edit.js.map +1 -0
- package/dist/core/diagnostics.js +94 -0
- package/dist/core/diagnostics.js.map +1 -0
- package/dist/core/errors.js +47 -0
- package/dist/core/errors.js.map +1 -0
- package/dist/core/router.js +320 -0
- package/dist/core/router.js.map +1 -0
- package/dist/core/schema.js +120 -0
- package/dist/core/schema.js.map +1 -0
- package/dist/core/types.js +22 -0
- package/dist/core/types.js.map +1 -0
- package/dist/providers/anysearch.js +225 -0
- package/dist/providers/anysearch.js.map +1 -0
- package/dist/providers/common.js +210 -0
- package/dist/providers/common.js.map +1 -0
- package/dist/providers/extract.js +155 -0
- package/dist/providers/extract.js.map +1 -0
- package/dist/providers/registry.js +31 -0
- package/dist/providers/registry.js.map +1 -0
- package/dist/providers/rsc.js +109 -0
- package/dist/providers/rsc.js.map +1 -0
- package/dist/providers/search.js +178 -0
- package/dist/providers/search.js.map +1 -0
- package/dist/providers/xcrawl.js +170 -0
- package/dist/providers/xcrawl.js.map +1 -0
- package/dist/transport/http.js +175 -0
- package/dist/transport/http.js.map +1 -0
- package/dist/version.js +2 -0
- package/dist/version.js.map +1 -0
- package/package.json +61 -0
- package/schemas/config.schema.json +210 -0
- package/schemas/extractRequest.schema.json +21 -0
- package/schemas/output.schema.json +435 -0
- package/schemas/searchRequest.schema.json +57 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 web-access-cli contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# web-access-cli
|
|
2
|
+
|
|
3
|
+
English | [简体中文](README_CN.md)
|
|
4
|
+
|
|
5
|
+
[](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
An agent-neutral CLI for web capabilities. It defines search and web content extraction as stable capabilities while normalizing differences among Tavily, Exa, Brave, SearXNG, AnySearch, XCrawl, Firecrawl, Jina, and other providers behind a unified internal schema.
|
|
9
|
+
|
|
10
|
+
The CLI is the primary product interface. It is not tied to Pi, Claude Code, Codex, Cursor, OpenCode, or any other agent. Skills, MCP integrations, and agent plugins may only integrate as adapters on top of the CLI; they must not leak into the core capabilities or provider implementations.
|
|
11
|
+
|
|
12
|
+
## Capabilities and providers
|
|
13
|
+
|
|
14
|
+
| Capability | Provider types | Normalized output |
|
|
15
|
+
| --- | --- | --- |
|
|
16
|
+
| `search` | Tavily, Exa, Brave, SearXNG, AnySearch, XCrawl | `rank`, `title`, `url`, `snippet` |
|
|
17
|
+
| `extract` | Firecrawl v2, Jina Reader, Exa Contents, AnySearch, XCrawl, HTTP | Markdown `Document` |
|
|
18
|
+
|
|
19
|
+
A provider type describes an implementation, while a provider instance is a configurable instance of that type. One type can have multiple instances, such as `exa_team` and `exa_personal`. A route is an ordered array of instance IDs that determines both which instances are enabled and the order in which `auto` tries them.
|
|
20
|
+
|
|
21
|
+
## Installation
|
|
22
|
+
|
|
23
|
+
Node.js 22.19 or later is required.
|
|
24
|
+
|
|
25
|
+
The npm package has not been published yet. Build and run the CLI from source:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
git clone https://github.com/Juexe/web-access-cli.git
|
|
29
|
+
cd web-access-cli
|
|
30
|
+
pnpm install --frozen-lockfile
|
|
31
|
+
pnpm build
|
|
32
|
+
node dist/cli.js --help
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
After building, an optional global installation can point to the absolute repository path:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
pnpm add --global <absolute-path-to-web-access-cli>
|
|
39
|
+
web-access --help
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
For development in this repository:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
pnpm check
|
|
46
|
+
pnpm pack:check
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Project structure
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
src/cli.ts Commander CLI and exit codes
|
|
53
|
+
src/config/ Configuration files, environment variables, and route merging
|
|
54
|
+
src/core/ Public types, schemas, errors, routing, and diagnostics
|
|
55
|
+
src/providers/ Provider adapters, registry, and HTTP/RSC extraction
|
|
56
|
+
src/transport/ Proxies, timeouts, redirects, and response size limits
|
|
57
|
+
schemas/ Generated JSON Schemas
|
|
58
|
+
test/ node:test unit tests and local HTTP integration tests
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Provider adapters should handle protocol mapping only. Fallback, deadlines, attempts, envelopes, and exit codes are managed centrally by `src/core`. Do not add lifecycle or tool registration code for Pi, Claude Code, Codex, or any other agent to an adapter.
|
|
62
|
+
|
|
63
|
+
## Development workflow
|
|
64
|
+
|
|
65
|
+
```sh
|
|
66
|
+
pnpm install
|
|
67
|
+
pnpm lint
|
|
68
|
+
pnpm typecheck
|
|
69
|
+
pnpm test
|
|
70
|
+
pnpm build
|
|
71
|
+
pnpm pack:check
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
`pnpm check` runs linting, type checking, tests, and the build in that order. After changing `src/core/schema.ts`, run `pnpm build` again and commit the updated `schemas/*.schema.json` files. Provider tests should use the mock transport or local HTTP server from `test/helpers.ts`; they must not depend on real API keys or external network reliability.
|
|
75
|
+
|
|
76
|
+
## Commands
|
|
77
|
+
|
|
78
|
+
```sh
|
|
79
|
+
web-access search "Agent-neutral web CLI"
|
|
80
|
+
web-access search "TypeScript release" --provider exa --limit 10 --freshness month
|
|
81
|
+
web-access search "browser automation" --include-domain example.com --exclude-domain blocked.example.com
|
|
82
|
+
|
|
83
|
+
web-access extract https://example.com/article
|
|
84
|
+
web-access extract https://example.com/article --provider http --timeout 30000
|
|
85
|
+
|
|
86
|
+
web-access providers --pretty
|
|
87
|
+
web-access doctor --pretty
|
|
88
|
+
|
|
89
|
+
web-access config edit
|
|
90
|
+
web-access --config "/path/to/config.json" config edit
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The CLI provides the `search` and `extract` capability commands, the `providers` and `doctor` diagnostic commands, and the `config edit` configuration command. The current version does not provide batch/all, answer, a dedicated PDF path, a Node.js SDK, or a generic MCP integration. The repository includes an optional [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md) for source-checkout use; it is not included in the npm package.
|
|
94
|
+
|
|
95
|
+
### Global options
|
|
96
|
+
|
|
97
|
+
- `--config <path>`: Explicitly select a configuration file.
|
|
98
|
+
- `--pretty`: Pretty-print JSON. By default, JSON is written on a single line.
|
|
99
|
+
- `--help`, `--version`: Print standard CLI help or version text.
|
|
100
|
+
|
|
101
|
+
Except for help and version output, stdout always contains exactly one JSON envelope. Diagnostic messages are never mixed into stdout.
|
|
102
|
+
|
|
103
|
+
## Configuration
|
|
104
|
+
|
|
105
|
+
Configuration files use strict JSON; unknown fields are rejected. The CLI reads only an explicit path or a user-level path and does not search parent project directories for configuration files.
|
|
106
|
+
|
|
107
|
+
The default path on every platform is `~/.config/web-access-cli/config.json`, where `~` is the current user's home directory. The `WEB_ACCESS_CONFIG` environment variable can select another path. The `--config` command-line option takes precedence.
|
|
108
|
+
|
|
109
|
+
`web-access config edit` creates parent directories and a complete default configuration when the file is missing, then opens it with the operating system's default application for JSON files. An existing file is opened byte-for-byte as-is, even when it is temporarily invalid JSON; it is never overwritten or reformatted. The command waits only for the operating system to accept the open request, not for the editor to close. On success, envelope `data` contains the absolute `path`, `created`, and `opened: true`. A default-application launch failure returns `open_failed`; a configuration file that was created successfully remains available for manual editing.
|
|
110
|
+
|
|
111
|
+
The complete JSON Schema is available at [schemas/config.schema.json](schemas/config.schema.json). Example:
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
{
|
|
115
|
+
"$schema": "https://unpkg.com/web-access-cli@0.1.0/schemas/config.schema.json",
|
|
116
|
+
"providers": [
|
|
117
|
+
{
|
|
118
|
+
"id": "exa_team",
|
|
119
|
+
"type": "exa",
|
|
120
|
+
"apiKeyEnv": "TEAM_EXA_API_KEY",
|
|
121
|
+
"headers": {
|
|
122
|
+
"X-Team": "docs"
|
|
123
|
+
}
|
|
124
|
+
},
|
|
125
|
+
{
|
|
126
|
+
"id": "searx_local",
|
|
127
|
+
"type": "searxng",
|
|
128
|
+
"baseUrl": "http://127.0.0.1:8080"
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"id": "firecrawl_local",
|
|
132
|
+
"type": "firecrawl",
|
|
133
|
+
"baseUrl": "http://127.0.0.1:3002"
|
|
134
|
+
}
|
|
135
|
+
],
|
|
136
|
+
"search": {
|
|
137
|
+
"providers": ["searx_local", "exa_team", "brave"],
|
|
138
|
+
"limit": 5,
|
|
139
|
+
"timeoutMs": 60000,
|
|
140
|
+
"attemptTimeoutMs": 20000,
|
|
141
|
+
"maxResponseBytes": 5242880
|
|
142
|
+
},
|
|
143
|
+
"extract": {
|
|
144
|
+
"providers": ["firecrawl_local", "jina", "exa_team", "http"],
|
|
145
|
+
"timeoutMs": 120000,
|
|
146
|
+
"attemptTimeoutMs": 45000,
|
|
147
|
+
"maxResponseBytes": 5242880,
|
|
148
|
+
"minContentCharacters": 500
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
The built-in instances are `tavily`, `exa`, `brave`, `searxng`, `firecrawl`, `jina`, `http`, `anysearch`, and `xcrawl`. A configuration entry with the same ID overrides fields on the built-in instance. A custom ID creates another instance of the selected type. An instance is enabled only when it appears in the corresponding `providers` route. AnySearch defaults to `https://api.anysearch.com` and supports anonymous calls; XCrawl defaults to `https://run.xcrawl.com` and requires an API key.
|
|
154
|
+
|
|
155
|
+
Default routes:
|
|
156
|
+
|
|
157
|
+
- Search: `tavily -> exa -> brave -> searxng -> anysearch -> xcrawl`
|
|
158
|
+
- Extract: `firecrawl -> jina -> exa -> anysearch -> xcrawl -> http`
|
|
159
|
+
|
|
160
|
+
The default routes include every built-in instance that supports the corresponding capability. Custom instance IDs are merged into the instance list but must still be added to routes explicitly. An omitted route uses the defaults above, while an explicit empty array disables that capability. In `auto` mode, incompletely configured instances are skipped; AnySearch can be called anonymously with its default base URL, and XCrawl is skipped until an API key is configured. Both instance types accept `searchFilterMode`: `strict` (default; freshness skips the provider) or `best_effort` (rewrites freshness into a query fragment). Domain constraints are rewritten into the query and strictly re-applied locally. XCrawl Extract always uses synchronous Scrape with Markdown output; Map, Crawl, and asynchronous jobs are outside the current CLI capabilities.
|
|
161
|
+
|
|
162
|
+
### Credentials and URLs
|
|
163
|
+
|
|
164
|
+
Environment variables take precedence over plaintext keys in JSON. Built-in instances support these standard variables:
|
|
165
|
+
|
|
166
|
+
| Type | API key | Base URL |
|
|
167
|
+
| --- | --- | --- |
|
|
168
|
+
| Tavily | `TAVILY_API_KEY` | `TAVILY_BASE_URL` |
|
|
169
|
+
| Exa | `EXA_API_KEY` | `EXA_BASE_URL` |
|
|
170
|
+
| Brave | `BRAVE_API_KEY` | `BRAVE_BASE_URL` |
|
|
171
|
+
| SearXNG | None | `SEARXNG_BASE_URL`, required |
|
|
172
|
+
| Firecrawl | `FIRECRAWL_API_KEY` | `FIRECRAWL_BASE_URL` |
|
|
173
|
+
| Jina | `JINA_API_KEY`, optional | `JINA_BASE_URL` |
|
|
174
|
+
| HTTP | None | None |
|
|
175
|
+
| AnySearch | `ANYSEARCH_API_KEY`, optional | `ANYSEARCH_BASE_URL`, default `https://api.anysearch.com` |
|
|
176
|
+
| XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`, default `https://run.xcrawl.com` |
|
|
177
|
+
|
|
178
|
+
Custom instances use `apiKeyEnv` and `baseUrlEnv` to name their environment variables. Every remote provider can define a `baseUrl` and additional `headers`. The public `api.firecrawl.dev` service requires a key; a self-hosted Firecrawl v2 instance can run without one.
|
|
179
|
+
|
|
180
|
+
## Execution and fallback
|
|
181
|
+
|
|
182
|
+
`--provider auto` follows the configured route order:
|
|
183
|
+
|
|
184
|
+
1. An instance with incomplete configuration is recorded as a failed attempt and skipped.
|
|
185
|
+
2. Recoverable errors, including network errors, timeouts, rate limits, 5xx responses, oversized responses, and missing usable content, cause the router to try the next instance.
|
|
186
|
+
3. Non-recoverable errors, including authentication errors, invalid input, and unsupported content, stop execution immediately.
|
|
187
|
+
4. If every instance fails, the command returns `provider_exhausted`. The best short content produced during extraction is preserved in `partial`.
|
|
188
|
+
|
|
189
|
+
Selecting an instance explicitly executes it strictly without fallback. If the instance is not in the route, the command returns `provider_disabled`.
|
|
190
|
+
|
|
191
|
+
Default limits are a 60-second total timeout and a 20-second per-attempt timeout for Search, and 120 seconds and 45 seconds respectively for Extract. Each response has a hard limit of 5 MiB. `--timeout` overrides only the total timeout for the current command.
|
|
192
|
+
|
|
193
|
+
## Output contract
|
|
194
|
+
|
|
195
|
+
Successful response example:
|
|
196
|
+
|
|
197
|
+
```json
|
|
198
|
+
{
|
|
199
|
+
"schemaVersion": 1,
|
|
200
|
+
"ok": true,
|
|
201
|
+
"command": "search",
|
|
202
|
+
"durationMs": 183,
|
|
203
|
+
"request": {
|
|
204
|
+
"query": "Agent-neutral web CLI",
|
|
205
|
+
"provider": "auto",
|
|
206
|
+
"limit": 5,
|
|
207
|
+
"includeDomains": [],
|
|
208
|
+
"excludeDomains": []
|
|
209
|
+
},
|
|
210
|
+
"provider": { "id": "exa", "type": "exa" },
|
|
211
|
+
"attempts": [
|
|
212
|
+
{
|
|
213
|
+
"provider": { "id": "tavily", "type": "tavily" },
|
|
214
|
+
"status": "failed",
|
|
215
|
+
"durationMs": 0,
|
|
216
|
+
"error": {
|
|
217
|
+
"code": "provider_unavailable",
|
|
218
|
+
"message": "tavily 未完成 search 所需配置",
|
|
219
|
+
"retryable": true
|
|
220
|
+
}
|
|
221
|
+
},
|
|
222
|
+
{
|
|
223
|
+
"provider": { "id": "exa", "type": "exa" },
|
|
224
|
+
"status": "success",
|
|
225
|
+
"durationMs": 183
|
|
226
|
+
}
|
|
227
|
+
],
|
|
228
|
+
"data": {
|
|
229
|
+
"results": [
|
|
230
|
+
{
|
|
231
|
+
"rank": 1,
|
|
232
|
+
"title": "Example",
|
|
233
|
+
"url": "https://example.com/",
|
|
234
|
+
"snippet": "Example snippet"
|
|
235
|
+
}
|
|
236
|
+
]
|
|
237
|
+
},
|
|
238
|
+
"raw": {}
|
|
239
|
+
}
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
The Chinese `message` value above matches the CLI's current error output contract.
|
|
243
|
+
|
|
244
|
+
`raw` contains only the response body from the final success or best failure. It never contains requests, response headers, or every historical response. API keys are redacted from serializable `raw` data and error messages. Stable schemas are available in [schemas](schemas).
|
|
245
|
+
|
|
246
|
+
Exit codes:
|
|
247
|
+
|
|
248
|
+
- `0`: Success
|
|
249
|
+
- `2`: Invalid input or configuration, or an unknown or disabled provider
|
|
250
|
+
- `1`: Runtime, provider, doctor, or default-application launch failure
|
|
251
|
+
- `130`: User cancellation
|
|
252
|
+
|
|
253
|
+
`providers` lists each instance's type, capabilities, route status, credential source, and base URL source. `doctor` performs local configuration checks only and does not call remote APIs. If an enabled route contains an unconfigured instance, the command returns `doctor_failed` with exit code `1`.
|
|
254
|
+
|
|
255
|
+
## HTTP and security boundaries
|
|
256
|
+
|
|
257
|
+
- Honors `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY` by default.
|
|
258
|
+
- Follows up to five redirects. Cross-origin redirects remove authentication, Cookie, and common token headers.
|
|
259
|
+
- Enforces a streaming byte limit on every response instead of buffering an unbounded response first.
|
|
260
|
+
- HTTP Extract uses LinkeDOM, Mozilla Readability, and Turndown, with fallback parsing for Next.js RSC payloads.
|
|
261
|
+
- Returns `unsupported_content` for PDF, image, audio, video, zip, and general binary content.
|
|
262
|
+
- By design, the CLI does not block localhost, private network addresses, or cloud metadata URLs. For untrusted input, callers must enforce their own URL allowlist, network isolation, or outbound proxy policy.
|
|
263
|
+
|
|
264
|
+
## Project scope
|
|
265
|
+
|
|
266
|
+
This project draws heavily from the provider and content extraction implementations in `pi-web-access`, but does not include Pi-specific registration, tool protocols, or UI code.
|
|
267
|
+
|
|
268
|
+
Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) before opening an issue or pull request. Report security vulnerabilities privately according to [SECURITY.md](SECURITY.md).
|
|
269
|
+
|
|
270
|
+
Licensed under MIT. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for third-party and upstream attribution.
|
package/README_CN.md
ADDED
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
# web-access-cli
|
|
2
|
+
|
|
3
|
+
[English](README.md) | 简体中文
|
|
4
|
+
|
|
5
|
+
[](https://github.com/Juexe/web-access-cli/actions/workflows/ci.yml)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
一个 Agent-neutral 的网页能力 CLI。它把“搜索”和“网页正文提取”定义为稳定能力,把 Tavily、Exa、Brave、SearXNG、AnySearch、XCrawl、Firecrawl、Jina 等差异收敛到内部统一 schema。
|
|
9
|
+
|
|
10
|
+
CLI 是主要产品形态,不绑定 Pi、Claude Code、Codex、Cursor、OpenCode 或其他 Agent。Skill、MCP 和 Agent 插件只能作为 CLI 上层 adapter 接入,不能污染核心能力和 Provider 实现。
|
|
11
|
+
|
|
12
|
+
## 能力与 provider
|
|
13
|
+
|
|
14
|
+
| 能力 | Provider Type | 统一输出 |
|
|
15
|
+
| --- | --- | --- |
|
|
16
|
+
| `search` | Tavily、Exa、Brave、SearXNG、AnySearch、XCrawl | `rank`、`title`、`url`、`snippet` |
|
|
17
|
+
| `extract` | Firecrawl v2、Jina Reader、Exa Contents、AnySearch、XCrawl、HTTP | Markdown `Document` |
|
|
18
|
+
|
|
19
|
+
Provider Type 描述实现类型;Provider Instance 是一份可配置实例。一个 Type 可以有多个 Instance,例如 `exa_team` 和 `exa_personal`。Route 是有序 Instance ID 数组,既决定启用状态,也决定 `auto` 的尝试顺序。
|
|
20
|
+
|
|
21
|
+
## 安装
|
|
22
|
+
|
|
23
|
+
要求 Node.js 22.19 或更高版本。
|
|
24
|
+
|
|
25
|
+
npm 包尚未发布。当前请从源码构建并运行:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
git clone https://github.com/Juexe/web-access-cli.git
|
|
29
|
+
cd web-access-cli
|
|
30
|
+
pnpm install --frozen-lockfile
|
|
31
|
+
pnpm build
|
|
32
|
+
node dist/cli.js --help
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
构建完成后,也可以使用仓库绝对路径安装为全局命令:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
pnpm add --global <web-access-cli 仓库绝对路径>
|
|
39
|
+
web-access --help
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
在仓库中开发:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
pnpm check
|
|
46
|
+
pnpm pack:check
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## 项目结构
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
src/cli.ts Commander CLI 与退出码
|
|
53
|
+
src/config/ 配置文件、环境变量与 Route 合并
|
|
54
|
+
src/core/ 公共类型、schema、错误、路由和诊断
|
|
55
|
+
src/providers/ Provider adapter、registry、HTTP/RSC 提取
|
|
56
|
+
src/transport/ 代理、超时、重定向和响应大小限制
|
|
57
|
+
schemas/ 构建生成的 JSON Schema
|
|
58
|
+
test/ node:test 单元测试与本地 HTTP 集成测试
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Provider adapter 应只负责协议映射;fallback、deadline、attempts、envelope 和退出码由 `src/core` 统一处理。不要在 adapter 中引入 Pi、Claude Code、Codex 或其他 Agent 的生命周期和工具注册代码。
|
|
62
|
+
|
|
63
|
+
## 开发工作流
|
|
64
|
+
|
|
65
|
+
```sh
|
|
66
|
+
pnpm install
|
|
67
|
+
pnpm lint
|
|
68
|
+
pnpm typecheck
|
|
69
|
+
pnpm test
|
|
70
|
+
pnpm build
|
|
71
|
+
pnpm pack:check
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
`pnpm check` 会按上述顺序执行 lint、类型检查、测试和构建。修改 `src/core/schema.ts` 后必须重新运行 `pnpm build`,并提交更新后的 `schemas/*.schema.json`。Provider 测试应使用 `test/helpers.ts` 的 mock transport 或本地 HTTP server,不依赖真实 API key 和外网稳定性。
|
|
75
|
+
|
|
76
|
+
## 命令
|
|
77
|
+
|
|
78
|
+
```sh
|
|
79
|
+
web-access search "Agent-neutral web CLI"
|
|
80
|
+
web-access search "TypeScript release" --provider exa --limit 10 --freshness month
|
|
81
|
+
web-access search "browser automation" --include-domain example.com --exclude-domain blocked.example.com
|
|
82
|
+
|
|
83
|
+
web-access extract https://example.com/article
|
|
84
|
+
web-access extract https://example.com/article --provider http --timeout 30000
|
|
85
|
+
|
|
86
|
+
web-access providers --pretty
|
|
87
|
+
web-access doctor --pretty
|
|
88
|
+
|
|
89
|
+
web-access config edit
|
|
90
|
+
web-access --config "/path/to/config.json" config edit
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
CLI 提供 `search`、`extract` 两个能力命令,`providers`、`doctor` 两个诊断命令,以及 `config edit` 配置维护命令。当前版本不提供 batch/all、answer、PDF 专线、Node SDK 或通用 MCP 集成。仓库提供可选的 [web-access-cli Agent Skill](skills/web-access-cli/SKILL.md),仅供源码仓库使用,不包含在 npm 发布包中。
|
|
94
|
+
|
|
95
|
+
### 通用选项
|
|
96
|
+
|
|
97
|
+
- `--config <path>`:显式指定配置文件。
|
|
98
|
+
- `--pretty`:格式化 JSON;默认输出单行 JSON。
|
|
99
|
+
- `--help`、`--version`:输出常规 CLI 帮助或版本文本。
|
|
100
|
+
|
|
101
|
+
除帮助与版本外,命令的 stdout 始终只有一个 JSON envelope。诊断信息不会混入 stdout。
|
|
102
|
+
|
|
103
|
+
## 配置
|
|
104
|
+
|
|
105
|
+
配置是严格 JSON;未知字段会报错。CLI 只读取显式路径或用户级路径,不会向上查找项目目录中的配置文件。
|
|
106
|
+
|
|
107
|
+
所有平台的默认路径均为 `~/.config/web-access-cli/config.json`,其中 `~` 表示当前用户的主目录。环境变量 `WEB_ACCESS_CONFIG` 可以指定其他路径;命令行 `--config` 优先级更高。
|
|
108
|
+
|
|
109
|
+
`web-access config edit` 会在配置文件缺失时创建父目录和完整默认配置,再用系统为 JSON 文件关联的默认应用打开;已有文件会原样打开,即使内容暂时不是有效 JSON 也不会被覆盖或格式化。命令只等待系统接受打开请求,不等待编辑器关闭。成功 envelope 的 `data` 包含绝对 `path`、是否新建的 `created` 和 `opened: true`。无法启动默认应用时返回 `open_failed`;如果配置刚刚创建成功,文件仍会保留以便手动打开。
|
|
110
|
+
|
|
111
|
+
完整 JSON Schema 位于 [schemas/config.schema.json](schemas/config.schema.json)。示例:
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
{
|
|
115
|
+
"$schema": "https://unpkg.com/web-access-cli@0.1.0/schemas/config.schema.json",
|
|
116
|
+
"providers": [
|
|
117
|
+
{
|
|
118
|
+
"id": "exa_team",
|
|
119
|
+
"type": "exa",
|
|
120
|
+
"apiKeyEnv": "TEAM_EXA_API_KEY",
|
|
121
|
+
"headers": {
|
|
122
|
+
"X-Team": "docs"
|
|
123
|
+
}
|
|
124
|
+
},
|
|
125
|
+
{
|
|
126
|
+
"id": "searx_local",
|
|
127
|
+
"type": "searxng",
|
|
128
|
+
"baseUrl": "http://127.0.0.1:8080"
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"id": "firecrawl_local",
|
|
132
|
+
"type": "firecrawl",
|
|
133
|
+
"baseUrl": "http://127.0.0.1:3002"
|
|
134
|
+
}
|
|
135
|
+
],
|
|
136
|
+
"search": {
|
|
137
|
+
"providers": ["searx_local", "exa_team", "brave"],
|
|
138
|
+
"limit": 5,
|
|
139
|
+
"timeoutMs": 60000,
|
|
140
|
+
"attemptTimeoutMs": 20000,
|
|
141
|
+
"maxResponseBytes": 5242880
|
|
142
|
+
},
|
|
143
|
+
"extract": {
|
|
144
|
+
"providers": ["firecrawl_local", "jina", "exa_team", "http"],
|
|
145
|
+
"timeoutMs": 120000,
|
|
146
|
+
"attemptTimeoutMs": 45000,
|
|
147
|
+
"maxResponseBytes": 5242880,
|
|
148
|
+
"minContentCharacters": 500
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
内置 Instance 为 `tavily`、`exa`、`brave`、`searxng`、`firecrawl`、`jina`、`http`、`anysearch`、`xcrawl`。配置同 ID 时会覆盖内置实例的字段;自定义 ID 可以创建同 Type 的额外实例。只有出现在对应 `providers` Route 中的实例才启用。AnySearch 默认 base URL 为 `https://api.anysearch.com`,支持匿名调用;XCrawl 默认 base URL 为 `https://run.xcrawl.com`,必须配置 API key。
|
|
154
|
+
|
|
155
|
+
默认 Route:
|
|
156
|
+
|
|
157
|
+
- Search:`tavily -> exa -> brave -> searxng -> anysearch -> xcrawl`
|
|
158
|
+
- Extract:`firecrawl -> jina -> exa -> anysearch -> xcrawl -> http`
|
|
159
|
+
|
|
160
|
+
默认 Route 包含支持对应 Capability 的全部内置 Instance。自定义 ID 会合并到 Instance 列表,但仍需显式加入 Route;省略 Route 时使用上述默认值,显式空数组则禁用对应能力。`auto` 会跳过未完成配置的 Instance;AnySearch 使用默认 base URL 时可匿名调用,XCrawl 则在配置 API key 前被跳过。两类 Instance 均可设置 `searchFilterMode`:`strict`(默认,遇到 freshness 时跳过)或 `best_effort`(将日期改写为查询片段)。域名条件会改写查询并在本地再次严格过滤。XCrawl Extract 固定使用同步 Scrape 的 Markdown 输出;Map、Crawl 和异步任务不属于当前 CLI 能力。
|
|
161
|
+
|
|
162
|
+
### 凭据和 URL
|
|
163
|
+
|
|
164
|
+
环境变量优先于 JSON 中的明文 key。内置 Instance 支持这些标准变量:
|
|
165
|
+
|
|
166
|
+
| Type | API key | Base URL |
|
|
167
|
+
| --- | --- | --- |
|
|
168
|
+
| Tavily | `TAVILY_API_KEY` | `TAVILY_BASE_URL` |
|
|
169
|
+
| Exa | `EXA_API_KEY` | `EXA_BASE_URL` |
|
|
170
|
+
| Brave | `BRAVE_API_KEY` | `BRAVE_BASE_URL` |
|
|
171
|
+
| SearXNG | 无 | `SEARXNG_BASE_URL`,必需 |
|
|
172
|
+
| Firecrawl | `FIRECRAWL_API_KEY` | `FIRECRAWL_BASE_URL` |
|
|
173
|
+
| Jina | `JINA_API_KEY`,可选 | `JINA_BASE_URL` |
|
|
174
|
+
| HTTP | 无 | 无 |
|
|
175
|
+
| AnySearch | `ANYSEARCH_API_KEY`,可选 | `ANYSEARCH_BASE_URL`,默认 `https://api.anysearch.com` |
|
|
176
|
+
| XCrawl | `XCRAWL_API_KEY` | `XCRAWL_BASE_URL`,默认 `https://run.xcrawl.com` |
|
|
177
|
+
|
|
178
|
+
自定义 Instance 使用 `apiKeyEnv` 和 `baseUrlEnv` 指定自己的环境变量。所有远端 Provider 都可以设置 `baseUrl` 和附加 `headers`。公共 `api.firecrawl.dev` 需要 key;自托管 Firecrawl v2 可以不设置 key。
|
|
179
|
+
|
|
180
|
+
## 执行与回退
|
|
181
|
+
|
|
182
|
+
`--provider auto` 按 Route 顺序执行:
|
|
183
|
+
|
|
184
|
+
1. 未完成配置的实例会记录为失败 attempt 并跳过。
|
|
185
|
+
2. 网络错误、超时、限流、5xx、响应过大、无可用正文等可恢复错误会尝试下一个实例。
|
|
186
|
+
3. 鉴权错误、无效输入、不支持的内容等不可恢复错误会立即停止。
|
|
187
|
+
4. 所有实例失败时返回 `provider_exhausted`。提取过程中产生的最佳短正文会保留在 `partial`。
|
|
188
|
+
|
|
189
|
+
显式指定 Instance 时严格执行,不触发 fallback。若实例不在 Route 中,返回 `provider_disabled`。
|
|
190
|
+
|
|
191
|
+
默认上限:Search 总超时/单次超时为 60s/20s,Extract 为 120s/45s;单个响应硬上限为 5 MiB。`--timeout` 只覆盖本次命令的总超时。
|
|
192
|
+
|
|
193
|
+
## 输出契约
|
|
194
|
+
|
|
195
|
+
成功示例:
|
|
196
|
+
|
|
197
|
+
```json
|
|
198
|
+
{
|
|
199
|
+
"schemaVersion": 1,
|
|
200
|
+
"ok": true,
|
|
201
|
+
"command": "search",
|
|
202
|
+
"durationMs": 183,
|
|
203
|
+
"request": {
|
|
204
|
+
"query": "Agent-neutral web CLI",
|
|
205
|
+
"provider": "auto",
|
|
206
|
+
"limit": 5,
|
|
207
|
+
"includeDomains": [],
|
|
208
|
+
"excludeDomains": []
|
|
209
|
+
},
|
|
210
|
+
"provider": { "id": "exa", "type": "exa" },
|
|
211
|
+
"attempts": [
|
|
212
|
+
{
|
|
213
|
+
"provider": { "id": "tavily", "type": "tavily" },
|
|
214
|
+
"status": "failed",
|
|
215
|
+
"durationMs": 0,
|
|
216
|
+
"error": {
|
|
217
|
+
"code": "provider_unavailable",
|
|
218
|
+
"message": "tavily 未完成 search 所需配置",
|
|
219
|
+
"retryable": true
|
|
220
|
+
}
|
|
221
|
+
},
|
|
222
|
+
{
|
|
223
|
+
"provider": { "id": "exa", "type": "exa" },
|
|
224
|
+
"status": "success",
|
|
225
|
+
"durationMs": 183
|
|
226
|
+
}
|
|
227
|
+
],
|
|
228
|
+
"data": {
|
|
229
|
+
"results": [
|
|
230
|
+
{
|
|
231
|
+
"rank": 1,
|
|
232
|
+
"title": "Example",
|
|
233
|
+
"url": "https://example.com/",
|
|
234
|
+
"snippet": "Example snippet"
|
|
235
|
+
}
|
|
236
|
+
]
|
|
237
|
+
},
|
|
238
|
+
"raw": {}
|
|
239
|
+
}
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
`raw` 只包含最终成功或最佳失败的 response body,不包含请求、响应 headers 或所有历史响应。API key 会从可序列化的 `raw` 和错误消息中脱敏。稳定 schema 位于 [schemas](schemas)。
|
|
243
|
+
|
|
244
|
+
退出码:
|
|
245
|
+
|
|
246
|
+
- `0`:成功
|
|
247
|
+
- `2`:输入、配置、未知或未启用 provider
|
|
248
|
+
- `1`:运行时/provider/doctor/默认应用启动失败
|
|
249
|
+
- `130`:用户取消
|
|
250
|
+
|
|
251
|
+
`providers` 会列出每个 Instance 的 Type、能力、Route 启用状态、凭据来源和 base URL 来源。`doctor` 只做本地配置检查,不主动调用远端 API;当已启用 Route 中存在未配置 Instance 时,命令返回 `doctor_failed` 和退出码 `1`。
|
|
252
|
+
|
|
253
|
+
## HTTP 与安全边界
|
|
254
|
+
|
|
255
|
+
- 默认遵循 `HTTP_PROXY`、`HTTPS_PROXY`、`ALL_PROXY`、`NO_PROXY`。
|
|
256
|
+
- 最多跟随 5 次重定向;跨 origin 重定向会移除鉴权、Cookie 和常见 token headers。
|
|
257
|
+
- 所有响应都执行流式字节上限,避免先完整缓冲超大响应。
|
|
258
|
+
- HTTP Extract 使用 LinkeDOM、Mozilla Readability、Turndown,并对 Next.js RSC payload 做后备解析。
|
|
259
|
+
- PDF、图片、音频、视频、zip 和通用二进制内容会返回 `unsupported_content`。
|
|
260
|
+
- 按项目设计,CLI 不阻止 localhost、私网地址或云元数据 URL。调用者必须在不可信输入场景中自行实施 URL allowlist、网络隔离或出站代理策略。
|
|
261
|
+
|
|
262
|
+
## 项目边界
|
|
263
|
+
|
|
264
|
+
本项目大量参考 `pi-web-access` 的 provider 和内容提取实现,但不包含 Pi-specific 注册、工具协议或 UI 代码。
|
|
265
|
+
|
|
266
|
+
欢迎贡献。提交 Issue 或 Pull Request 前请阅读 [CONTRIBUTING.md](CONTRIBUTING.md);安全漏洞请按照 [SECURITY.md](SECURITY.md) 私下报告。
|
|
267
|
+
|
|
268
|
+
许可证为 MIT,第三方与上游归属见 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# 第三方与上游声明
|
|
2
|
+
|
|
3
|
+
## pi-web-access
|
|
4
|
+
|
|
5
|
+
本项目的 provider 映射、HTTP 正文提取和 Next.js RSC 后备解析大量参考并部分改写自:
|
|
6
|
+
|
|
7
|
+
- 项目:`pi-web-access`
|
|
8
|
+
- 仓库:`https://github.com/nicobailon/pi-web-access.git`
|
|
9
|
+
- 参考提交:`00b2271d0f1603ac780df3f324aed0fc92f3e849`
|
|
10
|
+
- 参考版本:`0.20.0`
|
|
11
|
+
- Copyright 2025 Nico Bailon
|
|
12
|
+
- 许可证:MIT
|
|
13
|
+
|
|
14
|
+
上游 MIT 许可文本:
|
|
15
|
+
|
|
16
|
+
> Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
17
|
+
> of this software and associated documentation files (the "Software"), to deal
|
|
18
|
+
> in the Software without restriction, including without limitation the rights
|
|
19
|
+
> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
20
|
+
> copies of the Software, and to permit persons to whom the Software is
|
|
21
|
+
> furnished to do so, subject to the following conditions:
|
|
22
|
+
>
|
|
23
|
+
> The above copyright notice and this permission notice shall be included in all
|
|
24
|
+
> copies or substantial portions of the Software.
|
|
25
|
+
>
|
|
26
|
+
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
27
|
+
> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
28
|
+
> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
29
|
+
> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
30
|
+
> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
31
|
+
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
32
|
+
> SOFTWARE.
|
|
33
|
+
|
|
34
|
+
Pi-specific 的工具注册、UI、Agent 生命周期与协议代码没有进入本项目。
|
|
35
|
+
|
|
36
|
+
## npm 依赖
|
|
37
|
+
|
|
38
|
+
发行包直接使用以下开源组件:
|
|
39
|
+
|
|
40
|
+
- `@mozilla/readability`:Apache-2.0
|
|
41
|
+
- `commander`:MIT
|
|
42
|
+
- `linkedom`:ISC
|
|
43
|
+
- `open`:MIT
|
|
44
|
+
- `turndown`:MIT
|
|
45
|
+
- `typebox`:MIT
|
|
46
|
+
- `undici`:MIT
|
|
47
|
+
|
|
48
|
+
开发依赖包括 Biome、tsx、TypeScript 及对应类型包。各组件的完整许可证和版权声明以随 npm 包发布的许可证文件及其上游仓库为准。本文件不替代任何第三方许可证。
|