@jackwener/opencli 0.4.1 → 0.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLI-CREATOR.md +103 -142
- package/LICENSE +28 -0
- package/README.md +113 -63
- package/README.zh-CN.md +114 -63
- package/SKILL.md +21 -4
- package/dist/browser.d.ts +21 -2
- package/dist/browser.js +269 -15
- package/dist/browser.test.d.ts +1 -0
- package/dist/browser.test.js +43 -0
- package/dist/build-manifest.js +66 -2
- package/dist/cli-manifest.json +905 -109
- package/dist/clis/boss/search.js +186 -30
- package/dist/clis/twitter/delete.d.ts +1 -0
- package/dist/clis/twitter/delete.js +73 -0
- package/dist/clis/twitter/followers.d.ts +1 -0
- package/dist/clis/twitter/followers.js +104 -0
- package/dist/clis/twitter/following.d.ts +1 -0
- package/dist/clis/twitter/following.js +90 -0
- package/dist/clis/twitter/like.d.ts +1 -0
- package/dist/clis/twitter/like.js +69 -0
- package/dist/clis/twitter/notifications.d.ts +1 -0
- package/dist/clis/twitter/notifications.js +109 -0
- package/dist/clis/twitter/post.d.ts +1 -0
- package/dist/clis/twitter/post.js +63 -0
- package/dist/clis/twitter/reply.d.ts +1 -0
- package/dist/clis/twitter/reply.js +57 -0
- package/dist/clis/v2ex/daily.d.ts +1 -0
- package/dist/clis/v2ex/daily.js +98 -0
- package/dist/clis/v2ex/me.d.ts +1 -0
- package/dist/clis/v2ex/me.js +99 -0
- package/dist/clis/v2ex/notifications.d.ts +1 -0
- package/dist/clis/v2ex/notifications.js +72 -0
- package/dist/clis/xiaohongshu/search.d.ts +5 -2
- package/dist/clis/xiaohongshu/search.js +35 -41
- package/dist/doctor.d.ts +50 -0
- package/dist/doctor.js +372 -0
- package/dist/doctor.test.d.ts +1 -0
- package/dist/doctor.test.js +114 -0
- package/dist/main.js +47 -5
- package/dist/output.test.d.ts +1 -0
- package/dist/output.test.js +20 -0
- package/dist/registry.d.ts +4 -0
- package/dist/registry.js +1 -0
- package/dist/runtime.d.ts +3 -1
- package/dist/runtime.js +2 -2
- package/package.json +2 -2
- package/src/browser.test.ts +51 -0
- package/src/browser.ts +318 -22
- package/src/build-manifest.ts +67 -2
- package/src/clis/boss/search.ts +196 -29
- package/src/clis/twitter/delete.ts +78 -0
- package/src/clis/twitter/followers.ts +119 -0
- package/src/clis/twitter/following.ts +105 -0
- package/src/clis/twitter/like.ts +74 -0
- package/src/clis/twitter/notifications.ts +119 -0
- package/src/clis/twitter/post.ts +68 -0
- package/src/clis/twitter/reply.ts +62 -0
- package/src/clis/v2ex/daily.ts +105 -0
- package/src/clis/v2ex/me.ts +103 -0
- package/src/clis/v2ex/notifications.ts +77 -0
- package/src/clis/xiaohongshu/search.ts +41 -44
- package/src/doctor.test.ts +133 -0
- package/src/doctor.ts +424 -0
- package/src/main.ts +47 -4
- package/src/output.test.ts +27 -0
- package/src/registry.ts +5 -0
- package/src/runtime.ts +2 -1
package/CLI-CREATOR.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
##
|
|
8
|
+
## AI Agent 开发者必读:用 Playwright MCP Bridge 探索
|
|
9
9
|
|
|
10
10
|
> [!CAUTION]
|
|
11
11
|
> **你(AI Agent)必须通过 Playwright MCP Bridge 打开浏览器去访问目标网站!**
|
|
@@ -18,66 +18,15 @@
|
|
|
18
18
|
|
|
19
19
|
### AI Agent 探索工作流(必须遵循)
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
Step 4: 再次查看 Network,发现新触发的 API
|
|
31
|
-
↓
|
|
32
|
-
Step 5: 分析 API 的请求参数、响应结构、鉴权方式
|
|
33
|
-
↓
|
|
34
|
-
Step 6: 编写适配器代码
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
### 具体操作步骤
|
|
38
|
-
|
|
39
|
-
**Step 0: 打开浏览器**
|
|
40
|
-
```
|
|
41
|
-
工具: browser_navigate
|
|
42
|
-
URL: https://www.bilibili.com/video/BV1xxxxx
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
**Step 1: 获取页面快照,了解页面结构**
|
|
46
|
-
```
|
|
47
|
-
工具: browser_snapshot
|
|
48
|
-
→ 观察页面上有哪些可交互元素(按钮、标签、链接)
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
**Step 2: 查看已有的网络请求**
|
|
52
|
-
```
|
|
53
|
-
工具: browser_network_requests
|
|
54
|
-
→ 筛选出 JSON API 端点(忽略静态资源)
|
|
55
|
-
→ 记录 URL pattern、请求头、响应结构
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
**Step 3: 模拟用户交互发现深层 API**
|
|
59
|
-
```
|
|
60
|
-
工具: browser_click (点击"字幕"按钮、"评论"标签、"关注"链接等)
|
|
61
|
-
工具: browser_wait_for (等待数据加载)
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
**Step 4: 再次抓包,发现新 API**
|
|
65
|
-
```
|
|
66
|
-
工具: browser_network_requests
|
|
67
|
-
→ 对比 Step 2,找出新触发的 API 端点
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
**Step 5: 用 evaluate 测试 API 可行性**
|
|
71
|
-
```
|
|
72
|
-
工具: browser_evaluate
|
|
73
|
-
代码: async () => {
|
|
74
|
-
const res = await fetch('https://api.bilibili.com/x/player/wbi/v2?bvid=BV1xxx&cid=123',
|
|
75
|
-
{ credentials: 'include' });
|
|
76
|
-
return await res.json();
|
|
77
|
-
}
|
|
78
|
-
→ 验证返回的数据结构和字段
|
|
79
|
-
→ 如果返回空/403:检查是否需要签名(Wbi)或特殊 Header
|
|
80
|
-
```
|
|
21
|
+
| 步骤 | 工具 | 做什么 |
|
|
22
|
+
|------|------|--------|
|
|
23
|
+
| 0. 打开浏览器 | `browser_navigate` | 导航到目标页面 |
|
|
24
|
+
| 1. 观察页面 | `browser_snapshot` | 观察可交互元素(按钮/标签/链接) |
|
|
25
|
+
| 2. 首次抓包 | `browser_network_requests` | 筛选 JSON API 端点,记录 URL pattern |
|
|
26
|
+
| 3. 模拟交互 | `browser_click` + `browser_wait_for` | 点击"字幕""评论""关注"等按钮 |
|
|
27
|
+
| 4. 二次抓包 | `browser_network_requests` | 对比步骤 2,找出新触发的 API |
|
|
28
|
+
| 5. 验证 API | `browser_evaluate` | `fetch(url, {credentials:'include'})` 测试返回结构 |
|
|
29
|
+
| 6. 写代码 | — | 基于确认的 API 写适配器 |
|
|
81
30
|
|
|
82
31
|
### 常犯错误
|
|
83
32
|
|
|
@@ -89,6 +38,27 @@ URL: https://www.bilibili.com/video/BV1xxxxx
|
|
|
89
38
|
| 遇到 HTTP 200 但空数据就放弃 | 检查是否需要 Wbi 签名或 Cookie 鉴权 |
|
|
90
39
|
| 完全依赖 `__INITIAL_STATE__` 拿所有数据 | `__INITIAL_STATE__` 只有首屏数据,深层数据要调 API |
|
|
91
40
|
|
|
41
|
+
### 实战成功案例:5 分钟实现「关注列表」适配器
|
|
42
|
+
|
|
43
|
+
以下是用上述工作流实际发现 Bilibili 关注列表 API 的完整过程:
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
|
|
47
|
+
2. browser_network_requests → 发现:
|
|
48
|
+
GET /x/relation/followings?vmid={uid}&pn=1&ps=24 → [200]
|
|
49
|
+
GET /x/relation/stat?vmid={uid} → [200]
|
|
50
|
+
3. browser_evaluate → 验证 API:
|
|
51
|
+
fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
|
|
52
|
+
→ { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
|
|
53
|
+
4. 结论:标准 Cookie API,无需 Wbi 签名
|
|
54
|
+
5. 写 following.ts → 一次构建通过
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**关键决策点**:
|
|
58
|
+
- 直接访问 `fans/follow` 页面(不是首页),页面加载就会触发 following API
|
|
59
|
+
- 看到 URL 里没有 `/wbi/` → 不需要签名 → 直接用 `fetchJson` 而非 `apiGet`
|
|
60
|
+
- API 返回 `code: 0` + 非空 `list` → Tier 2 Cookie 策略确认
|
|
61
|
+
|
|
92
62
|
---
|
|
93
63
|
|
|
94
64
|
## 核心流程
|
|
@@ -202,6 +172,45 @@ opencli cascade https://api.example.com/hot
|
|
|
202
172
|
|
|
203
173
|
---
|
|
204
174
|
|
|
175
|
+
## Step 2.5: 准备工作(写代码之前)
|
|
176
|
+
|
|
177
|
+
### 先找模板:从最相似的现有适配器开始
|
|
178
|
+
|
|
179
|
+
**不要从零开始写**。先看看同站点已有哪些适配器:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
ls src/clis/<site>/ # 看看已有什么
|
|
183
|
+
cat src/clis/<site>/feed.ts # 读最相似的那个
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
最高效的方式是 **复制最相似的适配器,然后改 3 个地方**:
|
|
187
|
+
1. `name` → 新命令名
|
|
188
|
+
2. API URL → 你在 Step 1 发现的端点
|
|
189
|
+
3. 字段映射 → 对应新 API 的字段
|
|
190
|
+
|
|
191
|
+
### 平台 SDK 速查表
|
|
192
|
+
|
|
193
|
+
写 TS 适配器之前,先看看你的目标站点有没有**现成的 helper 函数**可以复用:
|
|
194
|
+
|
|
195
|
+
#### Bilibili (`src/bilibili.ts`)
|
|
196
|
+
|
|
197
|
+
| 函数 | 用途 | 何时使用 |
|
|
198
|
+
|------|------|----------|
|
|
199
|
+
| `fetchJson(page, url)` | 带 Cookie 的 fetch + JSON 解析 | 普通 Cookie-tier API |
|
|
200
|
+
| `apiGet(page, path, {signed, params})` | 带 Wbi 签名的 API 调用 | URL 含 `/wbi/` 的接口 |
|
|
201
|
+
| `getSelfUid(page)` | 获取当前登录用户的 UID | "我的xxx" 类命令 |
|
|
202
|
+
| `resolveUid(page, input)` | 解析用户输入的 UID(支持数字/URL) | `--uid` 参数处理 |
|
|
203
|
+
| `wbiSign(page, params)` | 底层 Wbi 签名生成 | 通常不直接用,`apiGet` 已封装 |
|
|
204
|
+
| `stripHtml(s)` | 去除 HTML 标签 | 清理富文本字段 |
|
|
205
|
+
|
|
206
|
+
**如何判断需不需要 `apiGet`**?看 Network 请求 URL:
|
|
207
|
+
- 含 `/wbi/` 或 `w_rid=` → 必须用 `apiGet(..., { signed: true })`
|
|
208
|
+
- 不含 → 直接用 `fetchJson`
|
|
209
|
+
|
|
210
|
+
> 其他站点(Twitter、小红书等)暂无专用 SDK,直接用 `page.evaluate` + `fetch` 即可。
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
205
214
|
## Step 3: 编写适配器
|
|
206
215
|
|
|
207
216
|
### YAML vs TS?先看决策树
|
|
@@ -224,6 +233,27 @@ opencli cascade https://api.example.com/hot
|
|
|
224
233
|
|
|
225
234
|
> **经验法则**:如果你发现 YAML 里嵌了超过 10 行 JS,改用 TS 更可维护。
|
|
226
235
|
|
|
236
|
+
### 通用模式:分页 API
|
|
237
|
+
|
|
238
|
+
很多 API 使用 `pn`(页码)+ `ps`(每页数量)分页。标准处理模式:
|
|
239
|
+
|
|
240
|
+
```typescript
|
|
241
|
+
args: [
|
|
242
|
+
{ name: 'page', type: 'int', required: false, default: 1, help: '页码' },
|
|
243
|
+
{ name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
|
|
244
|
+
],
|
|
245
|
+
func: async (page, kwargs) => {
|
|
246
|
+
const pn = kwargs.page ?? 1;
|
|
247
|
+
const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
|
|
248
|
+
const payload = await fetchJson(page,
|
|
249
|
+
`https://api.example.com/list?pn=${pn}&ps=${ps}`
|
|
250
|
+
);
|
|
251
|
+
return payload.data?.list || [];
|
|
252
|
+
},
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
> 大多数站点的 `ps` 上限是 20~50。超过会被静默截断或返回错误。
|
|
256
|
+
|
|
227
257
|
### 方式 A: YAML Pipeline(声明式,推荐)
|
|
228
258
|
|
|
229
259
|
文件路径: `src/clis/<site>/<name>.yaml`,放入即自动注册。
|
|
@@ -500,40 +530,13 @@ cli({
|
|
|
500
530
|
|
|
501
531
|
> **拦截核心思路**:不自己构造签名,而是利用 `installInterceptor` 劫持网站自己的 `XMLHttpRequest` 和 `fetch`,让网站发请求,我们直接在底层取出解析好的 `response.json()`。
|
|
502
532
|
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
部分 API 获取是非常复杂的连环请求(例如 B 站获取视频字幕:先需要 `bvid` 获取核心 `cid`,再通过 `cid` 获取包含签名/Wbi 的字幕列表拉取地址,最后 fetch 真实的 CDN 资源)。在此类场景中,你必须在一个 `evaluate` 块内部或者在 TypeScript Node 端编排整个请求链条:
|
|
506
|
-
|
|
507
|
-
```typescript
|
|
508
|
-
// 真实场景:B站获取视频字幕的级联获取思路
|
|
509
|
-
const subtitleUrls = await page.evaluate(async (bvid) => {
|
|
510
|
-
// Step 1: 拿 CID (通常可以通过页面全局状态极速提取)
|
|
511
|
-
const cid = window.__INITIAL_STATE__?.videoData?.cid;
|
|
512
|
-
|
|
513
|
-
// Step 2: 依据 BVID 和 CID 拿字幕配置 (可能需要携带 W_RID 签名或依赖浏览器当前登录状态 Cookie)
|
|
514
|
-
const res = await fetch(\`/x/player/wbi/v2?bvid=\${bvid}&cid=\${cid}\`, { credentials: 'include' });
|
|
515
|
-
const data = await res.json();
|
|
516
|
-
|
|
517
|
-
// Step 3: 风控拦截/未登录降级空值检测 (Anti-Bot Empty Value Detection) ⚠️ 极其重要
|
|
518
|
-
// 很多大厂 API 只要签名失败或无强登录 Cookie 依然会返回 HTTP 200,但把关键 URL 设为 ""
|
|
519
|
-
const firstSubUrl = data.data?.subtitle?.subtitles?.[0]?.subtitle_url;
|
|
520
|
-
if (!firstSubUrl) {
|
|
521
|
-
throw new Error('被风控降级或需登录:拿不到真实的 subtitle_url,请检查 Cookie 状态 (Tier 2/3)');
|
|
522
|
-
}
|
|
523
|
-
|
|
524
|
-
return firstSubUrl;
|
|
525
|
-
}, kwargs.bvid);
|
|
526
|
-
|
|
527
|
-
// Step 4: 拉取最终的 CDN 静态文件 (无鉴权)
|
|
528
|
-
const finalRes = await fetch(subtitleUrls.startsWith('//') ? 'https:' + subtitleUrls : subtitleUrls);
|
|
529
|
-
const subtitles = await finalRes.json();
|
|
530
|
-
```
|
|
533
|
+
> **级联请求**(如 BVID→CID→字幕)的完整模板和要点见下方[进阶模式: 级联请求](#进阶模式-级联请求-cascading-requests)章节。
|
|
531
534
|
|
|
532
535
|
---
|
|
533
536
|
|
|
534
537
|
## Step 4: 测试
|
|
535
538
|
|
|
536
|
-
>
|
|
539
|
+
> **构建通过 ≠ 功能正常**。`npm run build` 只验证 TypeScript / YAML 语法,不验证运行时行为。
|
|
537
540
|
> 每个新命令 **必须实际运行** 并确认输出正确后才算完成。
|
|
538
541
|
|
|
539
542
|
### 必做清单
|
|
@@ -552,7 +555,7 @@ opencli mysite hot --limit 3 -f json # JSON 输出确认字段完整
|
|
|
552
555
|
|
|
553
556
|
### tap 步骤调试(intercept 策略专用)
|
|
554
557
|
|
|
555
|
-
>
|
|
558
|
+
> **不要猜 store name / action name**。先用 evaluate 探索,再写 YAML。
|
|
556
559
|
|
|
557
560
|
#### Step 1: 列出所有 Pinia store
|
|
558
561
|
|
|
@@ -592,68 +595,26 @@ opencli evaluate "(() => {
|
|
|
592
595
|
└──────────────┘ └──────────────┘ └──────────────┘ └────────┘
|
|
593
596
|
```
|
|
594
597
|
|
|
595
|
-
### Verbose 模式
|
|
596
|
-
|
|
597
|
-
```bash
|
|
598
|
-
# 查看 pipeline 每步的输入输出
|
|
599
|
-
opencli bilibili hot --limit 1 -v
|
|
600
|
-
```
|
|
601
|
-
|
|
602
|
-
输出示例:
|
|
603
|
-
```
|
|
604
|
-
[1/4] navigate → https://www.bilibili.com
|
|
605
|
-
→ (no data)
|
|
606
|
-
[2/4] evaluate → (async () => { const res = await fetch(…
|
|
607
|
-
→ [{title: "…", author: "…", play: 230835}]
|
|
608
|
-
[3/4] map (rank, title, author, play, danmaku)
|
|
609
|
-
→ [{rank: 1, title: "…", author: "…"}]
|
|
610
|
-
[4/4] limit → 1
|
|
611
|
-
→ [{rank: 1, title: "…"}]
|
|
612
|
-
```
|
|
613
|
-
|
|
614
|
-
### 输出格式验证
|
|
598
|
+
### Verbose 模式 & 输出验证
|
|
615
599
|
|
|
616
600
|
```bash
|
|
617
|
-
#
|
|
618
|
-
opencli mysite hot -f
|
|
619
|
-
|
|
620
|
-
# 确认 JSON 可被 jq 解析
|
|
621
|
-
opencli mysite hot -f json | jq '.[0]'
|
|
622
|
-
|
|
623
|
-
# 确认 CSV 可被导入
|
|
624
|
-
opencli mysite hot -f csv > data.csv
|
|
601
|
+
opencli bilibili hot --limit 1 -v # 查看 pipeline 每步数据流
|
|
602
|
+
opencli mysite hot -f json | jq '.[0]' # 确认 JSON 可被解析
|
|
603
|
+
opencli mysite hot -f csv > data.csv # 确认 CSV 可导入
|
|
625
604
|
```
|
|
626
605
|
|
|
627
606
|
---
|
|
628
607
|
|
|
629
|
-
## Step 5:
|
|
630
|
-
|
|
631
|
-
### YAML 适配器
|
|
608
|
+
## Step 5: 提交发布
|
|
632
609
|
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
### TS 适配器
|
|
636
|
-
|
|
637
|
-
放入 `src/clis/<site>/<name>.ts` 即自动加载模块,无需在 `index.ts` 中写入 `import`。
|
|
638
|
-
|
|
639
|
-
### 验证注册
|
|
640
|
-
|
|
641
|
-
```bash
|
|
642
|
-
opencli list # 确认新命令出现
|
|
643
|
-
opencli validate mysite # 校验定义完整性
|
|
644
|
-
```
|
|
645
|
-
|
|
646
|
-
### 提交
|
|
610
|
+
文件放入 `src/clis/<site>/` 即自动注册(YAML 或 TS 无需手动 import),然后:
|
|
647
611
|
|
|
648
612
|
```bash
|
|
649
|
-
|
|
650
|
-
git commit -m "feat(mysite): add hot
|
|
651
|
-
git push
|
|
613
|
+
opencli list | grep mysite # 确认注册
|
|
614
|
+
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
|
|
652
615
|
```
|
|
653
616
|
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
> 💡 **架构理念升级**: OpenCLI 的原生机制本质上内建了一个 **Zero-Dependency jq 数据处理流**。使用时不需要依赖系统命令级别的 `jq` 包,而是将所有的解析拍平动作放在 `evaluate` 块内的原生 JavaScript 里,再由外层 YAML 通过 `select`、`map` 等命令提取。这将彻底消灭跨操作系统下产生的第三方二进制库依赖。
|
|
617
|
+
> **架构理念**:OpenCLI 内建 **Zero-Dependency jq** 数据流 — 所有解析在 `evaluate` 的原生 JS 内完成,外层 YAML 用 `select`/`map` 提取,无需依赖系统 `jq` 二进制。
|
|
657
618
|
|
|
658
619
|
---
|
|
659
620
|
|
package/LICENSE
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025, jackwener
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
16
|
+
contributors may be used to endorse or promote products derived from
|
|
17
|
+
this software without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
package/README.md
CHANGED
|
@@ -5,20 +5,84 @@
|
|
|
5
5
|
|
|
6
6
|
[中文文档](./README.zh-CN.md)
|
|
7
7
|
|
|
8
|
-
[](https://www.npmjs.com/package/@jackwener/opencli)
|
|
8
|
+
[](https://www.npmjs.com/package/@jackwener/opencli)
|
|
9
|
+
[](https://nodejs.org)
|
|
10
|
+
[](./LICENSE)
|
|
9
11
|
|
|
10
|
-
A CLI tool that turns **any website** into a command-line interface. **
|
|
12
|
+
A CLI tool that turns **any website** into a command-line interface. **57 commands** across **17 sites** — bilibili, zhihu, xiaohongshu, twitter, reddit, xueqiu, github, v2ex, hackernews, bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube — powered by browser session reuse and AI-native discovery.
|
|
11
13
|
|
|
12
|
-
|
|
14
|
+
---
|
|
13
15
|
|
|
14
|
-
|
|
15
|
-
- 🤖 **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies
|
|
16
|
-
- 🚀 **Dynamic Loader** — Simply drop `.ts` or `.yaml` adapters into the `clis/` folder for auto-registration
|
|
17
|
-
- 📝 **Dual-Engine Architecture**:
|
|
18
|
-
- **YAML Declarative Engine**: Most adapters are minimal ~30 lines of YAML pipeline
|
|
19
|
-
- **Native Browser Injection Engine**: Advanced TypeScript utilities (`installInterceptor`, `autoScroll`) for XHR hijacking, GraphQL unwrapping, and store mutation
|
|
16
|
+
## Table of Contents
|
|
20
17
|
|
|
21
|
-
|
|
18
|
+
- [Highlights](#highlights)
|
|
19
|
+
- [Prerequisites](#prerequisites)
|
|
20
|
+
- [Quick Start](#quick-start)
|
|
21
|
+
- [Built-in Commands](#built-in-commands)
|
|
22
|
+
- [Output Formats](#output-formats)
|
|
23
|
+
- [For AI Agents (Developer Guide)](#for-ai-agents-developer-guide)
|
|
24
|
+
- [Troubleshooting](#troubleshooting)
|
|
25
|
+
- [Releasing New Versions](#releasing-new-versions)
|
|
26
|
+
- [License](#license)
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## Highlights
|
|
31
|
+
|
|
32
|
+
- **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser.
|
|
33
|
+
- **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies.
|
|
34
|
+
- **Dynamic Loader** — Simply drop `.ts` or `.yaml` adapters into the `clis/` folder for auto-registration.
|
|
35
|
+
- **Dual-Engine Architecture** — Supports both YAML declarative data pipelines and robust browser runtime typescript injections.
|
|
36
|
+
|
|
37
|
+
## Prerequisites
|
|
38
|
+
|
|
39
|
+
- **Node.js**: >= 18.0.0
|
|
40
|
+
- **Chrome** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com).
|
|
41
|
+
|
|
42
|
+
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
|
|
43
|
+
|
|
44
|
+
OpenCLI needs a way to communicate with your browser. We highly recommend configuring **both** of the following methods for maximum reliability.
|
|
45
|
+
|
|
46
|
+
### Connection Method A: Playwright MCP Bridge Extension (Primary)
|
|
47
|
+
|
|
48
|
+
1. Install **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** extension in Chrome.
|
|
49
|
+
2. Obtain your token by clicking the extension icon in the browser toolbar or from the extension settings page.
|
|
50
|
+
|
|
51
|
+
**You must configure this token in BOTH your MCP configuration and system environment variables.**
|
|
52
|
+
|
|
53
|
+
First, add it to your MCP client config (e.g. Claude/Cursor):
|
|
54
|
+
|
|
55
|
+
```json
|
|
56
|
+
{
|
|
57
|
+
"mcpServers": {
|
|
58
|
+
"playwright": {
|
|
59
|
+
"command": "npx",
|
|
60
|
+
"args": ["-y", "@playwright/mcp@latest", "--extension"],
|
|
61
|
+
"env": {
|
|
62
|
+
"PLAYWRIGHT_MCP_EXTENSION_TOKEN": "<your-token-here>"
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
And, so that `opencli` commands can use it directly in the terminal, export it in your shell environment (e.g. `~/.zshrc`):
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
export PLAYWRIGHT_MCP_EXTENSION_TOKEN="<your-token-here>"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Connection Method B: Chrome 144+ Auto-Discovery (Fallback)
|
|
76
|
+
|
|
77
|
+
No extensions needed. Just enable Chrome's built-in remote debugging:
|
|
78
|
+
|
|
79
|
+
1. Open `chrome://inspect#remote-debugging` in Chrome
|
|
80
|
+
2. Check **"Allow remote debugging for this browser instance"**
|
|
81
|
+
3. Set `OPENCLI_USE_CDP=1` before running opencli
|
|
82
|
+
|
|
83
|
+
*You can also manually specify an endpoint via `OPENCLI_CDP_ENDPOINT` env var.*
|
|
84
|
+
|
|
85
|
+
## Quick Start
|
|
22
86
|
|
|
23
87
|
### Install via npm (recommended)
|
|
24
88
|
|
|
@@ -30,63 +94,39 @@ Then use directly:
|
|
|
30
94
|
|
|
31
95
|
```bash
|
|
32
96
|
opencli list # See all commands
|
|
97
|
+
opencli list -f yaml # List commands as YAML
|
|
33
98
|
opencli hackernews top --limit 5 # Public API, no browser
|
|
34
99
|
opencli bilibili hot --limit 5 # Browser command
|
|
35
100
|
opencli zhihu hot -f json # JSON output
|
|
101
|
+
opencli zhihu hot -f yaml # YAML output
|
|
36
102
|
```
|
|
37
103
|
|
|
38
|
-
### Install from source
|
|
104
|
+
### Install from source (for developers)
|
|
39
105
|
|
|
40
106
|
```bash
|
|
41
107
|
git clone git@github.com:jackwener/opencli.git
|
|
42
|
-
cd opencli
|
|
43
|
-
|
|
108
|
+
cd opencli
|
|
109
|
+
npm install
|
|
110
|
+
npm run build
|
|
111
|
+
npm link # Link binary globally
|
|
112
|
+
opencli list # Now you can use it anywhere!
|
|
44
113
|
```
|
|
45
114
|
|
|
46
115
|
### Update
|
|
47
116
|
|
|
48
117
|
```bash
|
|
49
|
-
# npm global
|
|
50
|
-
npm update -g @jackwener/opencli
|
|
51
|
-
|
|
52
|
-
# Or reinstall to latest
|
|
53
118
|
npm install -g @jackwener/opencli@latest
|
|
54
119
|
```
|
|
55
120
|
|
|
56
|
-
##
|
|
57
|
-
|
|
58
|
-
Browser commands need:
|
|
59
|
-
1. **Chrome** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com)
|
|
60
|
-
2. **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** extension installed
|
|
61
|
-
3. Configure `PLAYWRIGHT_MCP_EXTENSION_TOKEN` (from the extension settings page) in your MCP config:
|
|
62
|
-
|
|
63
|
-
```json
|
|
64
|
-
{
|
|
65
|
-
"mcpServers": {
|
|
66
|
-
"playwright": {
|
|
67
|
-
"command": "npx",
|
|
68
|
-
"args": ["@playwright/mcp@latest", "--extension"],
|
|
69
|
-
"env": {
|
|
70
|
-
"PLAYWRIGHT_MCP_EXTENSION_TOKEN": "<your-token>"
|
|
71
|
-
}
|
|
72
|
-
}
|
|
73
|
-
}
|
|
74
|
-
}
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
Public API commands (`hackernews`, `github search`, `v2ex`) need no browser at all.
|
|
78
|
-
|
|
79
|
-
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
|
|
80
|
-
|
|
81
|
-
## 📦 Built-in Commands
|
|
121
|
+
## Built-in Commands
|
|
82
122
|
|
|
83
123
|
| Site | Commands | Mode |
|
|
84
124
|
|------|----------|------|
|
|
85
|
-
| **bilibili** | `hot` `search` `me` `favorite`
|
|
125
|
+
| **bilibili** | `hot` `search` `me` `favorite` ... (11 commands) | 🔐 Browser |
|
|
86
126
|
| **zhihu** | `hot` `search` `question` | 🔐 Browser |
|
|
87
127
|
| **xiaohongshu** | `search` `notifications` `feed` `me` `user` | 🔐 Browser |
|
|
88
128
|
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` | 🔐 Browser |
|
|
89
|
-
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` | 🔐 Browser |
|
|
129
|
+
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `following` `followers` `notifications` `post` `reply` `delete` `like` | 🔐 Browser |
|
|
90
130
|
| **reddit** | `hot` `frontpage` `search` `subreddit` | 🔐 Browser |
|
|
91
131
|
| **weibo** | `hot` | 🔐 Browser |
|
|
92
132
|
| **boss** | `search` | 🔐 Browser |
|
|
@@ -100,17 +140,27 @@ Public API commands (`hackernews`, `github search`, `v2ex`) need no browser at a
|
|
|
100
140
|
| **hackernews** | `top` | 🌐 Public |
|
|
101
141
|
| **bbc** | `news` | 🌐 Public |
|
|
102
142
|
|
|
103
|
-
##
|
|
143
|
+
## Output Formats
|
|
144
|
+
|
|
145
|
+
All built-in commands support `--format` / `-f` with `table`, `json`, `yaml`, `md`, and `csv`.
|
|
146
|
+
The `list` command supports the same format options, and keeps `--json` for backward compatibility.
|
|
104
147
|
|
|
105
148
|
```bash
|
|
106
|
-
opencli
|
|
107
|
-
opencli bilibili hot -f
|
|
149
|
+
opencli list -f yaml # Command registry as YAML
|
|
150
|
+
opencli bilibili hot -f table # Default: rich terminal table
|
|
151
|
+
opencli bilibili hot -f json # JSON (pipe to jq or LLMs)
|
|
152
|
+
opencli bilibili hot -f yaml # YAML (human-readable structured output)
|
|
108
153
|
opencli bilibili hot -f md # Markdown
|
|
109
154
|
opencli bilibili hot -f csv # CSV
|
|
110
|
-
opencli bilibili hot -v # Verbose: show pipeline steps
|
|
155
|
+
opencli bilibili hot -v # Verbose: show pipeline debug steps
|
|
111
156
|
```
|
|
112
157
|
|
|
113
|
-
##
|
|
158
|
+
## For AI Agents (Developer Guide)
|
|
159
|
+
|
|
160
|
+
If you are an AI assistant tasked with creating a new command adapter for `opencli`, please follow the AI Agent workflow below:
|
|
161
|
+
|
|
162
|
+
> **Information for AI:**
|
|
163
|
+
> Before writing any adapter code, you **must** read [CLI-CREATOR.md](./CLI-CREATOR.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide. Skipping this will lead to preventable errors.
|
|
114
164
|
|
|
115
165
|
```bash
|
|
116
166
|
# 1. Deep Explore — discover APIs, infer capabilities, detect framework
|
|
@@ -126,30 +176,30 @@ opencli generate https://example.com --goal "hot"
|
|
|
126
176
|
opencli cascade https://api.example.com/data
|
|
127
177
|
```
|
|
128
178
|
|
|
129
|
-
Explore outputs to `.opencli/explore/<site
|
|
130
|
-
- `manifest.json` — site metadata, framework detection
|
|
131
|
-
- `endpoints.json` — scored API endpoints with response schemas
|
|
132
|
-
- `capabilities.json` — inferred capabilities with confidence scores
|
|
133
|
-
- `auth.json` — authentication strategy recommendations
|
|
179
|
+
Explore outputs to `.opencli/explore/<site>/` (manifest.json, endpoints.json, capabilities.json, auth.json).
|
|
134
180
|
|
|
135
|
-
##
|
|
181
|
+
## Troubleshooting
|
|
136
182
|
|
|
137
|
-
|
|
183
|
+
- **"Failed to connect to Playwright MCP Bridge"**
|
|
184
|
+
- Ensure the Playwright MCP extension is installed and **enabled** in your running Chrome.
|
|
185
|
+
- Restart the Chrome browser if you just installed the extension.
|
|
186
|
+
- **"CDP command failed" or "boss search blocked"**
|
|
187
|
+
- Some sites (like BOSS Zhipin) actively block Chrome DevTools Protocol connections. OpenCLI falls back to cookie extraction, but ensure you didn't force `--chrome-mode` unnecessarily.
|
|
188
|
+
- **Empty data returns or 'Unauthorized' error**
|
|
189
|
+
- Your login session in Chrome might have expired. Open a normal Chrome tab, navigate to the target site, and log in or refresh the page to prove you are human.
|
|
190
|
+
- **Node API errors**
|
|
191
|
+
- Make sure you are using Node.js >= 18. Some dependencies require modern Node APIs.
|
|
138
192
|
|
|
139
193
|
## Releasing New Versions
|
|
140
194
|
|
|
141
195
|
```bash
|
|
142
|
-
# Bump version
|
|
143
196
|
npm version patch # 0.1.0 → 0.1.1
|
|
144
197
|
npm version minor # 0.1.0 → 0.2.0
|
|
145
|
-
npm version major # 0.1.0 → 1.0.0
|
|
146
|
-
|
|
147
|
-
# Push tag to trigger GitHub Actions auto-release
|
|
148
198
|
git push --follow-tags
|
|
149
199
|
```
|
|
150
200
|
|
|
151
201
|
The CI will automatically build, create a GitHub release, and publish to npm.
|
|
152
202
|
|
|
153
|
-
##
|
|
203
|
+
## License
|
|
154
204
|
|
|
155
|
-
|
|
205
|
+
[BSD-3-Clause](./LICENSE)
|