@jackwener/opencli 1.6.1 → 1.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +1 -1
- package/README.md +27 -45
- package/README.zh-CN.md +32 -34
- package/autoresearch/browse-tasks.json +18 -20
- package/autoresearch/commands/debug.ts +163 -0
- package/autoresearch/commands/fix.ts +145 -0
- package/autoresearch/commands/plan.ts +88 -0
- package/autoresearch/commands/run.ts +138 -0
- package/autoresearch/config.ts +82 -0
- package/autoresearch/engine.ts +359 -0
- package/autoresearch/eval-all.ts +127 -0
- package/autoresearch/eval-browse.ts +1 -1
- package/autoresearch/eval-publish.ts +238 -0
- package/autoresearch/eval-save.ts +249 -0
- package/autoresearch/eval-skill.ts +14 -8
- package/autoresearch/eval-v2ex.ts +220 -0
- package/autoresearch/eval-zhihu.ts +230 -0
- package/autoresearch/logger.ts +69 -0
- package/autoresearch/presets/combined-reliability.ts +27 -0
- package/autoresearch/presets/index.ts +23 -0
- package/autoresearch/presets/operate-reliability.ts +24 -0
- package/autoresearch/presets/save-reliability.ts +26 -0
- package/autoresearch/presets/skill-quality.ts +20 -0
- package/autoresearch/presets/v2ex-reliability.ts +24 -0
- package/autoresearch/presets/zhihu-reliability.ts +25 -0
- package/autoresearch/publish-tasks.json +345 -0
- package/autoresearch/run-save.sh +11 -0
- package/autoresearch/save-adapters/xhs-explore-deep.ts +64 -0
- package/autoresearch/save-adapters/xhs-note-comments.ts +61 -0
- package/autoresearch/save-adapters/xhs-search-full.ts +62 -0
- package/autoresearch/save-adapters/zhihu-hot-detail.ts +52 -0
- package/autoresearch/save-adapters/zhihu-question-full.ts +57 -0
- package/autoresearch/save-adapters/zhihu-search-detail.ts +53 -0
- package/autoresearch/save-tasks.json +281 -0
- package/autoresearch/v2ex-tasks.json +899 -0
- package/autoresearch/zhihu-tasks.json +848 -0
- package/dist/browser/base-page.d.ts +4 -2
- package/dist/browser/base-page.js +37 -4
- package/dist/browser/bridge.js +10 -8
- package/dist/browser/cdp.js +2 -6
- package/dist/browser/daemon-client.d.ts +11 -1
- package/dist/browser/daemon-client.js +3 -0
- package/dist/browser/dom-helpers.d.ts +4 -2
- package/dist/browser/dom-helpers.js +42 -31
- package/dist/browser/dom-snapshot.js +23 -1
- package/dist/browser/page.d.ts +7 -2
- package/dist/browser/page.js +112 -30
- package/dist/browser.test.js +1 -1
- package/dist/build-manifest.d.ts +1 -0
- package/dist/build-manifest.js +1 -0
- package/dist/cli-manifest.json +1135 -184
- package/dist/cli.d.ts +2 -0
- package/dist/cli.js +48 -7
- package/dist/cli.test.d.ts +1 -0
- package/dist/cli.test.js +88 -0
- package/dist/clis/1688/item.d.ts +70 -0
- package/dist/clis/1688/item.js +187 -0
- package/dist/clis/1688/item.test.d.ts +1 -0
- package/dist/clis/1688/item.test.js +67 -0
- package/dist/clis/1688/search.d.ts +56 -0
- package/dist/clis/1688/search.js +309 -0
- package/dist/clis/1688/search.test.d.ts +1 -0
- package/dist/clis/1688/search.test.js +75 -0
- package/dist/clis/1688/shared.d.ts +112 -0
- package/dist/clis/1688/shared.js +514 -0
- package/dist/clis/1688/shared.test.d.ts +1 -0
- package/dist/clis/1688/shared.test.js +57 -0
- package/dist/clis/1688/store.d.ts +45 -0
- package/dist/clis/1688/store.js +226 -0
- package/dist/clis/1688/store.test.d.ts +1 -0
- package/dist/clis/1688/store.test.js +62 -0
- package/dist/clis/amazon/bestsellers.d.ts +0 -20
- package/dist/clis/amazon/bestsellers.js +6 -129
- package/dist/clis/amazon/bestsellers.test.js +12 -3
- package/dist/clis/amazon/movers-shakers.d.ts +1 -0
- package/dist/clis/amazon/movers-shakers.js +7 -0
- package/dist/clis/amazon/new-releases.d.ts +1 -0
- package/dist/clis/amazon/new-releases.js +7 -0
- package/dist/clis/amazon/rankings.d.ts +59 -0
- package/dist/clis/amazon/rankings.js +226 -0
- package/dist/clis/amazon/rankings.test.d.ts +1 -0
- package/dist/clis/amazon/rankings.test.js +41 -0
- package/dist/clis/amazon/shared.d.ts +11 -0
- package/dist/clis/amazon/shared.js +121 -11
- package/dist/clis/amazon/shared.test.js +11 -0
- package/dist/clis/bilibili/comments.js +2 -2
- package/dist/clis/bilibili/comments.test.js +3 -2
- package/dist/clis/bilibili/download.js +2 -1
- package/dist/clis/bilibili/subtitle.js +4 -3
- package/dist/clis/bilibili/subtitle.test.js +2 -1
- package/dist/clis/bilibili/utils.d.ts +5 -0
- package/dist/clis/bilibili/utils.js +30 -0
- package/dist/clis/bilibili/utils.test.d.ts +1 -0
- package/dist/clis/bilibili/utils.test.js +17 -0
- package/dist/clis/douban/marks.js +1 -1
- package/dist/clis/douban/subject.yaml +50 -19
- package/dist/clis/doubao/utils.js +32 -12
- package/dist/clis/douyin/_shared/browser-fetch.test.js +0 -1
- package/dist/clis/douyin/_shared/transcode.test.js +0 -2
- package/dist/clis/douyin/draft.test.js +0 -2
- package/dist/clis/facebook/search.test.js +0 -2
- package/dist/clis/gemini/ask.js +9 -3
- package/dist/clis/gemini/ask.test.d.ts +1 -0
- package/dist/clis/gemini/ask.test.js +100 -0
- package/dist/clis/gemini/reply-state.test.d.ts +1 -0
- package/dist/clis/gemini/reply-state.test.js +641 -0
- package/dist/clis/gemini/utils.d.ts +44 -1
- package/dist/clis/gemini/utils.js +528 -61
- package/dist/clis/gemini/utils.test.js +149 -2
- package/dist/clis/hupu/detail.d.ts +1 -0
- package/dist/clis/hupu/detail.js +72 -0
- package/dist/clis/hupu/hot.yaml +43 -0
- package/dist/clis/hupu/like.d.ts +1 -0
- package/dist/clis/hupu/like.js +75 -0
- package/dist/clis/hupu/reply.d.ts +1 -0
- package/dist/clis/hupu/reply.js +71 -0
- package/dist/clis/hupu/search.d.ts +1 -0
- package/dist/clis/hupu/search.js +59 -0
- package/dist/clis/hupu/unlike.d.ts +1 -0
- package/dist/clis/hupu/unlike.js +75 -0
- package/dist/clis/hupu/utils.d.ts +20 -0
- package/dist/clis/hupu/utils.js +319 -0
- package/dist/clis/instagram/_shared/private-publish.d.ts +138 -0
- package/dist/clis/instagram/_shared/private-publish.js +1030 -0
- package/dist/clis/instagram/_shared/private-publish.test.d.ts +1 -0
- package/dist/clis/instagram/_shared/private-publish.test.js +705 -0
- package/dist/clis/instagram/_shared/protocol-capture.d.ts +26 -0
- package/dist/clis/instagram/_shared/protocol-capture.js +282 -0
- package/dist/clis/instagram/_shared/protocol-capture.test.d.ts +1 -0
- package/dist/clis/instagram/_shared/protocol-capture.test.js +114 -0
- package/dist/clis/instagram/_shared/runtime-info.d.ts +9 -0
- package/dist/clis/instagram/_shared/runtime-info.js +81 -0
- package/dist/clis/instagram/note.d.ts +1 -0
- package/dist/clis/instagram/note.js +222 -0
- package/dist/clis/instagram/note.test.d.ts +1 -0
- package/dist/clis/instagram/note.test.js +81 -0
- package/dist/clis/instagram/post.d.ts +4 -0
- package/dist/clis/instagram/post.js +1496 -0
- package/dist/clis/instagram/post.test.d.ts +1 -0
- package/dist/clis/instagram/post.test.js +1647 -0
- package/dist/clis/instagram/reel.d.ts +1 -0
- package/dist/clis/instagram/reel.js +826 -0
- package/dist/clis/instagram/reel.test.d.ts +1 -0
- package/dist/clis/instagram/reel.test.js +167 -0
- package/dist/clis/instagram/story.d.ts +1 -0
- package/dist/clis/instagram/story.js +115 -0
- package/dist/clis/instagram/story.test.d.ts +1 -0
- package/dist/clis/instagram/story.test.js +167 -0
- package/dist/clis/sinafinance/stock-rank.d.ts +4 -0
- package/dist/clis/sinafinance/stock-rank.js +65 -0
- package/dist/clis/substack/utils.test.js +0 -2
- package/dist/clis/twitter/post.js +72 -45
- package/dist/clis/twitter/post.test.d.ts +1 -0
- package/dist/clis/twitter/post.test.js +116 -0
- package/dist/clis/twitter/reply.d.ts +12 -0
- package/dist/clis/twitter/reply.js +257 -35
- package/dist/clis/twitter/reply.test.d.ts +1 -0
- package/dist/clis/twitter/reply.test.js +151 -0
- package/dist/clis/xianyu/chat.d.ts +7 -0
- package/dist/clis/xianyu/chat.js +146 -0
- package/dist/clis/xianyu/chat.test.d.ts +1 -0
- package/dist/clis/xianyu/chat.test.js +15 -0
- package/dist/clis/xianyu/item.d.ts +7 -0
- package/dist/clis/xianyu/item.js +152 -0
- package/dist/clis/xianyu/item.test.d.ts +1 -0
- package/dist/clis/xianyu/item.test.js +56 -0
- package/dist/clis/xianyu/search.d.ts +10 -0
- package/dist/clis/xianyu/search.js +134 -0
- package/dist/clis/xianyu/search.test.d.ts +1 -0
- package/dist/clis/xianyu/search.test.js +17 -0
- package/dist/clis/xianyu/utils.d.ts +1 -0
- package/dist/clis/xianyu/utils.js +8 -0
- package/dist/clis/xiaoe/catalog.yaml +129 -0
- package/dist/clis/xiaoe/content.yaml +43 -0
- package/dist/clis/xiaoe/courses.yaml +73 -0
- package/dist/clis/xiaoe/detail.yaml +39 -0
- package/dist/clis/xiaoe/play-url.yaml +124 -0
- package/dist/clis/xiaohongshu/comments.test.js +0 -2
- package/dist/clis/xiaohongshu/creator-note-detail.test.js +0 -2
- package/dist/clis/xiaohongshu/creator-notes.test.js +0 -2
- package/dist/clis/xiaohongshu/download.test.js +0 -2
- package/dist/clis/xiaohongshu/note.test.js +0 -2
- package/dist/clis/xiaohongshu/publish.test.js +0 -2
- package/dist/clis/xiaohongshu/search.js +29 -20
- package/dist/clis/xiaohongshu/search.test.js +56 -48
- package/dist/clis/yuanbao/ask.d.ts +21 -0
- package/dist/clis/yuanbao/ask.js +427 -0
- package/dist/clis/yuanbao/ask.test.d.ts +1 -0
- package/dist/clis/yuanbao/ask.test.js +124 -0
- package/dist/clis/yuanbao/new.d.ts +1 -0
- package/dist/clis/yuanbao/new.js +70 -0
- package/dist/clis/yuanbao/new.test.d.ts +1 -0
- package/dist/clis/yuanbao/new.test.js +30 -0
- package/dist/clis/yuanbao/shared.d.ts +13 -0
- package/dist/clis/yuanbao/shared.js +49 -0
- package/dist/clis/zhihu/question.js +30 -19
- package/dist/clis/zhihu/question.test.js +34 -16
- package/dist/commanderAdapter.js +8 -4
- package/dist/commanderAdapter.test.js +42 -0
- package/dist/completion.js +3 -1
- package/dist/completion.test.d.ts +1 -0
- package/dist/completion.test.js +23 -0
- package/dist/doctor.js +1 -1
- package/dist/electron-apps.d.ts +2 -0
- package/dist/electron-apps.js +7 -1
- package/dist/errors.js +1 -1
- package/dist/execution.js +25 -35
- package/dist/explore.js +1 -1
- package/dist/launcher.d.ts +4 -0
- package/dist/launcher.js +64 -8
- package/dist/launcher.test.js +88 -7
- package/dist/output.d.ts +2 -0
- package/dist/output.js +10 -1
- package/dist/output.test.d.ts +0 -3
- package/dist/output.test.js +59 -92
- package/dist/pipeline/executor.test.js +0 -2
- package/dist/pipeline/steps/download.test.js +0 -2
- package/dist/registry.d.ts +2 -0
- package/dist/serialization.d.ts +1 -0
- package/dist/serialization.js +1 -0
- package/dist/types.d.ts +9 -2
- package/docs/.vitepress/config.mts +4 -0
- package/docs/adapters/browser/1688.md +52 -0
- package/docs/adapters/browser/36kr.md +2 -1
- package/docs/adapters/browser/doubao.md +5 -1
- package/docs/adapters/browser/hupu.md +53 -0
- package/docs/adapters/browser/sinafinance.md +32 -2
- package/docs/adapters/browser/weibo.md +6 -1
- package/docs/adapters/browser/wikipedia.md +2 -0
- package/docs/adapters/browser/xianyu.md +42 -0
- package/docs/adapters/browser/xiaoe.md +44 -0
- package/docs/adapters/browser/yuanbao.md +64 -0
- package/docs/adapters/index.md +14 -5
- package/docs/comparison.md +1 -1
- package/docs/developer/ai-workflow.md +2 -2
- package/docs/developer/contributing.md +1 -1
- package/docs/developer/testing.md +2 -0
- package/docs/guide/plugins.md +1 -0
- package/docs/guide/troubleshooting.md +11 -0
- package/docs/superpowers/specs/2026-04-03-v2ex-autoresearch-design.md +41 -0
- package/docs/zh/guide/plugins.md +1 -0
- package/extension/dist/background.js +1127 -0
- package/extension/src/background.test.ts +39 -0
- package/extension/src/background.ts +223 -34
- package/extension/src/cdp.ts +194 -4
- package/extension/src/protocol.ts +22 -1
- package/package.json +3 -2
- package/scripts/postinstall.js +1 -1
- package/skills/opencli-explorer/SKILL.md +1 -1
- package/skills/opencli-oneshot/SKILL.md +2 -2
- package/skills/opencli-operate/SKILL.md +120 -27
- package/skills/opencli-usage/SKILL.md +31 -20
- package/skills/opencli-usage/browser.md +114 -16
- package/skills/opencli-usage/public-api.md +32 -3
- package/skills/smart-search/SKILL.md +156 -0
- package/skills/smart-search/references/sources-ai.md +74 -0
- package/skills/smart-search/references/sources-info.md +43 -0
- package/skills/smart-search/references/sources-media.md +50 -0
- package/skills/smart-search/references/sources-other.md +42 -0
- package/skills/smart-search/references/sources-shopping.md +31 -0
- package/skills/smart-search/references/sources-social.md +51 -0
- package/skills/smart-search/references/sources-tech.md +42 -0
- package/skills/smart-search/references/sources-travel.md +20 -0
- package/src/browser/base-page.ts +41 -6
- package/src/browser/bridge.ts +11 -8
- package/src/browser/cdp.ts +1 -8
- package/src/browser/daemon-client.ts +11 -1
- package/src/browser/dom-helpers.ts +43 -31
- package/src/browser/dom-snapshot.ts +23 -1
- package/src/browser/page.ts +115 -31
- package/src/browser.test.ts +1 -1
- package/src/build-manifest.ts +2 -0
- package/src/cli.test.ts +133 -0
- package/src/cli.ts +73 -11
- package/src/clis/1688/item.test.ts +69 -0
- package/src/clis/1688/item.ts +282 -0
- package/src/clis/1688/search.test.ts +81 -0
- package/src/clis/1688/search.ts +402 -0
- package/src/clis/1688/shared.test.ts +75 -0
- package/src/clis/1688/shared.ts +623 -0
- package/src/clis/1688/store.test.ts +69 -0
- package/src/clis/1688/store.ts +300 -0
- package/src/clis/amazon/bestsellers.test.ts +12 -3
- package/src/clis/amazon/bestsellers.ts +6 -178
- package/src/clis/amazon/movers-shakers.ts +8 -0
- package/src/clis/amazon/new-releases.ts +8 -0
- package/src/clis/amazon/rankings.test.ts +47 -0
- package/src/clis/amazon/rankings.ts +312 -0
- package/src/clis/amazon/shared.test.ts +16 -0
- package/src/clis/amazon/shared.ts +134 -12
- package/src/clis/bilibili/comments.test.ts +4 -3
- package/src/clis/bilibili/comments.ts +2 -2
- package/src/clis/bilibili/download.ts +2 -1
- package/src/clis/bilibili/subtitle.test.ts +2 -1
- package/src/clis/bilibili/subtitle.ts +4 -3
- package/src/clis/bilibili/utils.test.ts +21 -0
- package/src/clis/bilibili/utils.ts +27 -0
- package/src/clis/douban/marks.ts +1 -1
- package/src/clis/douban/subject.yaml +50 -19
- package/src/clis/doubao/utils.ts +32 -12
- package/src/clis/douyin/_shared/browser-fetch.test.ts +0 -1
- package/src/clis/douyin/_shared/transcode.test.ts +0 -2
- package/src/clis/douyin/draft.test.ts +0 -2
- package/src/clis/facebook/search.test.ts +0 -2
- package/src/clis/gemini/ask.test.ts +116 -0
- package/src/clis/gemini/ask.ts +10 -3
- package/src/clis/gemini/reply-state.test.ts +708 -0
- package/src/clis/gemini/utils.test.ts +184 -2
- package/src/clis/gemini/utils.ts +588 -60
- package/src/clis/hupu/detail.ts +126 -0
- package/src/clis/hupu/hot.yaml +43 -0
- package/src/clis/hupu/like.ts +76 -0
- package/src/clis/hupu/reply.ts +76 -0
- package/src/clis/hupu/search.ts +95 -0
- package/src/clis/hupu/unlike.ts +76 -0
- package/src/clis/hupu/utils.ts +381 -0
- package/src/clis/instagram/_shared/private-publish.test.ts +827 -0
- package/src/clis/instagram/_shared/private-publish.ts +1303 -0
- package/src/clis/instagram/_shared/protocol-capture.test.ts +148 -0
- package/src/clis/instagram/_shared/protocol-capture.ts +321 -0
- package/src/clis/instagram/_shared/runtime-info.ts +91 -0
- package/src/clis/instagram/note.test.ts +96 -0
- package/src/clis/instagram/note.ts +254 -0
- package/src/clis/instagram/post.test.ts +1716 -0
- package/src/clis/instagram/post.ts +1620 -0
- package/src/clis/instagram/reel.test.ts +191 -0
- package/src/clis/instagram/reel.ts +886 -0
- package/src/clis/instagram/story.test.ts +191 -0
- package/src/clis/instagram/story.ts +151 -0
- package/src/clis/sinafinance/stock-rank.ts +68 -0
- package/src/clis/substack/utils.test.ts +0 -2
- package/src/clis/twitter/post.test.ts +157 -0
- package/src/clis/twitter/post.ts +82 -48
- package/src/clis/twitter/reply.test.ts +177 -0
- package/src/clis/twitter/reply.ts +285 -39
- package/src/clis/xianyu/chat.test.ts +20 -0
- package/src/clis/xianyu/chat.ts +175 -0
- package/src/clis/xianyu/item.test.ts +67 -0
- package/src/clis/xianyu/item.ts +172 -0
- package/src/clis/xianyu/search.test.ts +22 -0
- package/src/clis/xianyu/search.ts +151 -0
- package/src/clis/xianyu/utils.ts +9 -0
- package/src/clis/xiaoe/catalog.yaml +129 -0
- package/src/clis/xiaoe/content.yaml +43 -0
- package/src/clis/xiaoe/courses.yaml +73 -0
- package/src/clis/xiaoe/detail.yaml +39 -0
- package/src/clis/xiaoe/play-url.yaml +124 -0
- package/src/clis/xiaohongshu/comments.test.ts +0 -2
- package/src/clis/xiaohongshu/creator-note-detail.test.ts +0 -2
- package/src/clis/xiaohongshu/creator-notes.test.ts +0 -2
- package/src/clis/xiaohongshu/download.test.ts +0 -2
- package/src/clis/xiaohongshu/note.test.ts +0 -2
- package/src/clis/xiaohongshu/publish.test.ts +0 -2
- package/src/clis/xiaohongshu/search.test.ts +59 -48
- package/src/clis/xiaohongshu/search.ts +31 -21
- package/src/clis/yuanbao/ask.test.ts +156 -0
- package/src/clis/yuanbao/ask.ts +522 -0
- package/src/clis/yuanbao/new.test.ts +36 -0
- package/src/clis/yuanbao/new.ts +81 -0
- package/src/clis/yuanbao/shared.ts +57 -0
- package/src/clis/zhihu/question.test.ts +42 -17
- package/src/clis/zhihu/question.ts +31 -26
- package/src/commanderAdapter.test.ts +51 -0
- package/src/commanderAdapter.ts +8 -4
- package/src/completion.test.ts +30 -0
- package/src/completion.ts +3 -1
- package/src/doctor.ts +1 -1
- package/src/electron-apps.ts +9 -1
- package/src/errors.ts +1 -1
- package/src/execution.ts +26 -30
- package/src/explore.ts +1 -1
- package/src/launcher.test.ts +121 -7
- package/src/launcher.ts +87 -9
- package/src/output.test.ts +50 -90
- package/src/output.ts +10 -1
- package/src/pipeline/executor.test.ts +0 -2
- package/src/pipeline/steps/download.test.ts +0 -2
- package/src/registry.ts +2 -0
- package/src/serialization.ts +2 -0
- package/src/types.ts +9 -2
- package/tests/e2e/browser-auth.test.ts +9 -0
- package/CLI-EXPLORER.md +0 -724
- package/CLI-ONESHOT.md +0 -216
- package/SKILL.md +0 -59
package/CLI-EXPLORER.md
DELETED
|
@@ -1,724 +0,0 @@
|
|
|
1
|
-
# CLI-EXPLORER — 适配器探索式开发完全指南
|
|
2
|
-
|
|
3
|
-
> 本文档教你(或 AI Agent)如何为 OpenCLI 添加一个新网站的命令。
|
|
4
|
-
> 从零到发布,覆盖 API 发现、方案选择、适配器编写、测试验证全流程。
|
|
5
|
-
|
|
6
|
-
> [!TIP]
|
|
7
|
-
> **只想为一个具体页面快速生成一个命令?** 看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)(~150 行,4 步搞定)。
|
|
8
|
-
> 本文档适合从零探索一个新站点的完整流程。
|
|
9
|
-
|
|
10
|
-
---
|
|
11
|
-
|
|
12
|
-
## AI Agent 开发者必读:用浏览器探索
|
|
13
|
-
|
|
14
|
-
> [!CAUTION]
|
|
15
|
-
> **你(AI Agent)必须通过浏览器打开目标网站去探索!**
|
|
16
|
-
> 不要只靠 `opencli explore` 命令或静态分析来发现 API。
|
|
17
|
-
> 你拥有浏览器工具,必须主动用它们浏览网页、观察网络请求、模拟用户交互。
|
|
18
|
-
|
|
19
|
-
### 为什么?
|
|
20
|
-
|
|
21
|
-
很多 API 是**懒加载**的(用户必须点击某个按钮/标签才会触发网络请求)。字幕、评论、关注列表等深层数据不会在页面首次加载时出现在 Network 面板中。**如果你不主动去浏览和交互页面,你永远发现不了这些 API。**
|
|
22
|
-
|
|
23
|
-
### AI Agent 探索工作流(必须遵循)
|
|
24
|
-
|
|
25
|
-
| 步骤 | 工具 | 做什么 |
|
|
26
|
-
|------|------|--------|
|
|
27
|
-
| 0. 打开浏览器 | `browser_navigate` | 导航到目标页面 |
|
|
28
|
-
| 1. 观察页面 | `browser_snapshot` | 观察可交互元素(按钮/标签/链接) |
|
|
29
|
-
| 2. 首次抓包 | `browser_network_requests` | 筛选 JSON API 端点,记录 URL pattern |
|
|
30
|
-
| 3. 模拟交互 | `browser_click` + `browser_wait_for` | 点击"字幕""评论""关注"等按钮 |
|
|
31
|
-
| 4. 二次抓包 | `browser_network_requests` | 对比步骤 2,找出新触发的 API |
|
|
32
|
-
| 5. 验证 API | `browser_evaluate` | `fetch(url, {credentials:'include'})` 测试返回结构 |
|
|
33
|
-
| 6. 写代码 | — | 基于确认的 API 写适配器 |
|
|
34
|
-
|
|
35
|
-
### 常犯错误
|
|
36
|
-
|
|
37
|
-
| ❌ 错误做法 | ✅ 正确做法 |
|
|
38
|
-
|------------|------------|
|
|
39
|
-
| 只用 `opencli explore` 命令,等结果自动出来 | 用浏览器工具打开页面,主动浏览 |
|
|
40
|
-
| 直接在代码里 `fetch(url)`,不看浏览器实际请求 | 先在浏览器中确认 API 可用,再写代码 |
|
|
41
|
-
| 页面打开后直接抓包,期望所有 API 都出现 | 模拟点击交互(展开评论/切换标签/加载更多) |
|
|
42
|
-
| 遇到 HTTP 200 但空数据就放弃 | 检查是否需要 Wbi 签名或 Cookie 鉴权 |
|
|
43
|
-
| 完全依赖 `__INITIAL_STATE__` 拿所有数据 | `__INITIAL_STATE__` 只有首屏数据,深层数据要调 API |
|
|
44
|
-
|
|
45
|
-
### 实战成功案例:5 分钟实现「关注列表」适配器
|
|
46
|
-
|
|
47
|
-
以下是用上述工作流实际发现 Bilibili 关注列表 API 的完整过程:
|
|
48
|
-
|
|
49
|
-
```
|
|
50
|
-
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
|
|
51
|
-
2. browser_network_requests → 发现:
|
|
52
|
-
GET /x/relation/followings?vmid={uid}&pn=1&ps=24 → [200]
|
|
53
|
-
GET /x/relation/stat?vmid={uid} → [200]
|
|
54
|
-
3. browser_evaluate → 验证 API:
|
|
55
|
-
fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
|
|
56
|
-
→ { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
|
|
57
|
-
4. 结论:标准 Cookie API,无需 Wbi 签名
|
|
58
|
-
5. 写 following.ts → 一次构建通过
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
**关键决策点**:
|
|
62
|
-
- 直接访问 `fans/follow` 页面(不是首页),页面加载就会触发 following API
|
|
63
|
-
- 看到 URL 里没有 `/wbi/` → 不需要签名 → 直接用 `fetchJson` 而非 `apiGet`
|
|
64
|
-
- API 返回 `code: 0` + 非空 `list` → Tier 2 Cookie 策略确认
|
|
65
|
-
|
|
66
|
-
---
|
|
67
|
-
|
|
68
|
-
## 核心流程
|
|
69
|
-
|
|
70
|
-
```
|
|
71
|
-
┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌────────┐
|
|
72
|
-
│ 1. 发现 API │ ──▶ │ 2. 选择策略 │ ──▶ │ 3. 写适配器 │ ──▶ │ 4. 测试 │
|
|
73
|
-
└─────────────┘ └─────────────┘ └──────────────┘ └────────┘
|
|
74
|
-
explore cascade YAML / TS run + verify
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
---
|
|
78
|
-
|
|
79
|
-
## Step 1: 发现 API
|
|
80
|
-
|
|
81
|
-
### 1a. 自动化发现(推荐)
|
|
82
|
-
|
|
83
|
-
OpenCLI 内置 Deep Explore,自动分析网站网络请求:
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
opencli explore https://www.example.com --site mysite
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
输出到 `.opencli/explore/mysite/`:
|
|
90
|
-
|
|
91
|
-
| 文件 | 内容 |
|
|
92
|
-
|------|------|
|
|
93
|
-
| `manifest.json` | 站点元数据、框架检测(Vue2/3、React、Next.js、Pinia、Vuex) |
|
|
94
|
-
| `endpoints.json` | 已发现的 API 端点,按评分排序,含 URL pattern、方法、响应类型 |
|
|
95
|
-
| `capabilities.json` | 推理出的功能(`hot`、`search`、`feed`…),含置信度和推荐参数 |
|
|
96
|
-
| `auth.json` | 认证方式检测(Cookie/Header/无认证),策略候选列表 |
|
|
97
|
-
|
|
98
|
-
### 1b. 手动抓包验证
|
|
99
|
-
|
|
100
|
-
Explore 的自动分析可能不完美,用 verbose 模式手动确认:
|
|
101
|
-
|
|
102
|
-
```bash
|
|
103
|
-
# 在浏览器中打开目标页面,观察网络请求
|
|
104
|
-
opencli explore https://www.example.com --site mysite -v
|
|
105
|
-
|
|
106
|
-
# 或直接用 evaluate 测试 API
|
|
107
|
-
opencli bilibili hot -v # 查看已有命令的 pipeline 每步数据流
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
关注抓包结果中的关键信息:
|
|
111
|
-
- **URL pattern**: `/api/v2/hot?limit=20` → 这就是你要调用的端点
|
|
112
|
-
- **Method**: `GET` / `POST`
|
|
113
|
-
- **Request Headers**: Cookie? Bearer? 自定义签名头(X-s、X-t)?
|
|
114
|
-
- **Response Body**: JSON 结构,特别是数据在哪个路径(`data.items`、`data.list`)
|
|
115
|
-
|
|
116
|
-
### 1c. 高阶 API 发现捷径法则 (Heuristics)
|
|
117
|
-
|
|
118
|
-
在开始死磕复杂的抓包拦截之前,按照以下优先级进行尝试:
|
|
119
|
-
|
|
120
|
-
1. **后缀爆破法 (`.json`)**: 像 Reddit 这样复杂的网站,只要在其 URL 后加上 `.json`(例如 `/r/all.json`),就能在带 Cookie 的情况下直接利用 `fetch` 拿到极其干净的 REST 数据(Tier 2 Cookie 策略极速秒杀)。另外如功能完备的**雪球 (xueqiu)** 也可以走这种纯 API 的方式极简获取,成为你构建简单 YAML 的黄金标杆。
|
|
121
|
-
2. **全局状态查找法 (`__INITIAL_STATE__`)**: 许多服务端渲染 (SSR) 的网站(如小红书、Bilibili)会将首页或详情页的完整数据挂载到全局 window 对象上。与其去拦截网络请求,不如直接 `page.evaluate('() => window.__INITIAL_STATE__')` 获取整个数据树。
|
|
122
|
-
3. **主动交互触发法 (Active Interaction)**: 很多深层 API(如视频字幕、评论下的回复)是懒加载的。在静态抓包找不到数据时,尝试在 `evaluate` 步骤或手动打断点时,主动去**点击(Click)页面上的对应按钮**(如"CC"、"展开全部"),从而诱发隐藏的 Network Fetch。
|
|
123
|
-
4. **框架探测与 Store Action 截断**: 如果站点使用 Vue + Pinia,可以使用 `tap` 步骤调用 action,让前端框架代替你完成复杂的鉴权签名封装。
|
|
124
|
-
5. **底层 XHR/Fetch 拦截**: 最后手段,当上述都不行时,使用 TypeScript 适配器进行无侵入式的请求抓取。
|
|
125
|
-
|
|
126
|
-
### 1d. 框架检测
|
|
127
|
-
|
|
128
|
-
Explore 自动检测前端框架。如果需要手动确认:
|
|
129
|
-
|
|
130
|
-
```bash
|
|
131
|
-
# 在已打开目标网站的情况下
|
|
132
|
-
opencli evaluate "(()=>{
|
|
133
|
-
const vue3 = !!document.querySelector('#app')?.__vue_app__;
|
|
134
|
-
const vue2 = !!document.querySelector('#app')?.__vue__;
|
|
135
|
-
const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
|
|
136
|
-
const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
|
|
137
|
-
return JSON.stringify({vue3, vue2, react, pinia});
|
|
138
|
-
})()"
|
|
139
|
-
```
|
|
140
|
-
|
|
141
|
-
Vue + Pinia 的站点(如小红书)可以直接通过 Store Action 绕过签名。
|
|
142
|
-
|
|
143
|
-
---
|
|
144
|
-
|
|
145
|
-
## Step 2: 选择认证策略
|
|
146
|
-
|
|
147
|
-
OpenCLI 提供 5 级认证策略。使用 `cascade` 命令自动探测:
|
|
148
|
-
|
|
149
|
-
```bash
|
|
150
|
-
opencli cascade https://api.example.com/hot
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
### 策略决策树
|
|
154
|
-
|
|
155
|
-
```
|
|
156
|
-
直接 fetch(url) 能拿到数据?
|
|
157
|
-
→ ✅ Tier 1: public(公开 API,不需要浏览器)
|
|
158
|
-
→ ❌ fetch(url, {credentials:'include'}) 带 Cookie 能拿到?
|
|
159
|
-
→ ✅ Tier 2: cookie(最常见,evaluate 步骤内 fetch)
|
|
160
|
-
→ ❌ → 加上 Bearer / CSRF header 后能拿到?
|
|
161
|
-
→ ✅ Tier 3: header(如 Twitter ct0 + Bearer)
|
|
162
|
-
→ ❌ → 网站有 Pinia/Vuex Store?
|
|
163
|
-
→ ✅ Tier 4: intercept(Store Action + XHR 拦截)
|
|
164
|
-
→ ❌ Tier 5: ui(UI 自动化,最后手段)
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
### 各策略对比
|
|
168
|
-
|
|
169
|
-
| Tier | 策略 | 速度 | 复杂度 | 适用场景 | 实例 |
|
|
170
|
-
|------|------|------|--------|---------|------|
|
|
171
|
-
| 1 | `public` | ⚡ ~1s | 最简 | 公开 API,无需登录 | Hacker News, V2EX |
|
|
172
|
-
| 2 | `cookie` | 🔄 ~7s | 简单 | Cookie 认证即可 | Bilibili, Zhihu, Reddit |
|
|
173
|
-
| 3 | `header` | 🔄 ~7s | 中等 | 需要 CSRF token 或 Bearer | Twitter GraphQL |
|
|
174
|
-
| 4 | `intercept` | 🔄 ~10s | 较高 | 请求有复杂签名 | 小红书 (Pinia + XHR) |
|
|
175
|
-
| 5 | `ui` | 🐌 ~15s+ | 最高 | 无 API,纯 DOM 解析 | 遗留网站 |
|
|
176
|
-
|
|
177
|
-
---
|
|
178
|
-
|
|
179
|
-
## Step 2.5: 准备工作(写代码之前)
|
|
180
|
-
|
|
181
|
-
### 先找模板:从最相似的现有适配器开始
|
|
182
|
-
|
|
183
|
-
**不要从零开始写**。先看看同站点已有哪些适配器:
|
|
184
|
-
|
|
185
|
-
```bash
|
|
186
|
-
ls src/clis/<site>/ # 看看已有什么
|
|
187
|
-
cat src/clis/<site>/feed.ts # 读最相似的那个
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
最高效的方式是 **复制最相似的适配器,然后改 3 个地方**:
|
|
191
|
-
1. `name` → 新命令名
|
|
192
|
-
2. API URL → 你在 Step 1 发现的端点
|
|
193
|
-
3. 字段映射 → 对应新 API 的字段
|
|
194
|
-
|
|
195
|
-
### 平台 SDK 速查表
|
|
196
|
-
|
|
197
|
-
写 TS 适配器之前,先看看你的目标站点有没有**现成的 helper 函数**可以复用:
|
|
198
|
-
|
|
199
|
-
#### Bilibili (`src/clis/bilibili/utils.ts`)
|
|
200
|
-
|
|
201
|
-
| 函数 | 用途 | 何时使用 |
|
|
202
|
-
|------|------|----------|
|
|
203
|
-
| `fetchJson(page, url)` | 带 Cookie 的 fetch + JSON 解析 | 普通 Cookie-tier API |
|
|
204
|
-
| `apiGet(page, path, {signed, params})` | 带 Wbi 签名的 API 调用 | URL 含 `/wbi/` 的接口 |
|
|
205
|
-
| `getSelfUid(page)` | 获取当前登录用户的 UID | "我的xxx" 类命令 |
|
|
206
|
-
| `resolveUid(page, input)` | 解析用户输入的 UID(支持数字/URL) | `--uid` 参数处理 |
|
|
207
|
-
| `wbiSign(page, params)` | 底层 Wbi 签名生成 | 通常不直接用,`apiGet` 已封装 |
|
|
208
|
-
| `stripHtml(s)` | 去除 HTML 标签 | 清理富文本字段 |
|
|
209
|
-
|
|
210
|
-
**如何判断需不需要 `apiGet`**?看 Network 请求 URL:
|
|
211
|
-
- 含 `/wbi/` 或 `w_rid=` → 必须用 `apiGet(..., { signed: true })`
|
|
212
|
-
- 不含 → 直接用 `fetchJson`
|
|
213
|
-
|
|
214
|
-
> 其他站点(Twitter、小红书等)暂无专用 SDK,直接用 `page.evaluate` + `fetch` 即可。
|
|
215
|
-
|
|
216
|
-
---
|
|
217
|
-
|
|
218
|
-
## Step 3: 编写适配器
|
|
219
|
-
|
|
220
|
-
### YAML vs TS?先看决策树
|
|
221
|
-
|
|
222
|
-
```
|
|
223
|
-
你的 pipeline 里有 evaluate 步骤(内嵌 JS 代码)?
|
|
224
|
-
→ ✅ 用 TypeScript (src/clis/<site>/<name>.ts),保存即自动动态注册
|
|
225
|
-
→ ❌ 纯声明式(navigate + tap + map + limit)?
|
|
226
|
-
→ ✅ 用 YAML (src/clis/<site>/<name>.yaml),保存即自动注册
|
|
227
|
-
```
|
|
228
|
-
|
|
229
|
-
| 场景 | 选择 | 示例 |
|
|
230
|
-
|------|------|------|
|
|
231
|
-
| 纯 fetch/select/map/limit | YAML | `v2ex/hot.yaml`, `hackernews/top.yaml` |
|
|
232
|
-
| navigate + evaluate(fetch) + map | YAML(评估复杂度) | `zhihu/hot.yaml` |
|
|
233
|
-
| navigate + tap + map | YAML ✅ | `xiaohongshu/feed.yaml`, `xiaohongshu/notifications.yaml` |
|
|
234
|
-
| 有复杂 JS 逻辑(Pinia state 读取、条件分支) | TS | `xiaohongshu/me.ts`, `bilibili/me.ts` |
|
|
235
|
-
| XHR 拦截 + 签名 | TS | `xiaohongshu/search.ts` |
|
|
236
|
-
| GraphQL / 分页 / Wbi 签名 | TS | `bilibili/search.ts`, `twitter/search.ts` |
|
|
237
|
-
|
|
238
|
-
> **经验法则**:如果你发现 YAML 里嵌了超过 10 行 JS,改用 TS 更可维护。
|
|
239
|
-
|
|
240
|
-
### 通用模式:分页 API
|
|
241
|
-
|
|
242
|
-
很多 API 使用 `pn`(页码)+ `ps`(每页数量)分页。标准处理模式:
|
|
243
|
-
|
|
244
|
-
```typescript
|
|
245
|
-
args: [
|
|
246
|
-
{ name: 'page', type: 'int', required: false, default: 1, help: '页码' },
|
|
247
|
-
{ name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
|
|
248
|
-
],
|
|
249
|
-
func: async (page, kwargs) => {
|
|
250
|
-
const pn = kwargs.page ?? 1;
|
|
251
|
-
const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
|
|
252
|
-
const payload = await fetchJson(page,
|
|
253
|
-
`https://api.example.com/list?pn=${pn}&ps=${ps}`
|
|
254
|
-
);
|
|
255
|
-
return payload.data?.list || [];
|
|
256
|
-
},
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
> 大多数站点的 `ps` 上限是 20~50。超过会被静默截断或返回错误。
|
|
260
|
-
|
|
261
|
-
### 方式 A: YAML Pipeline(声明式,推荐)
|
|
262
|
-
|
|
263
|
-
文件路径: `src/clis/<site>/<name>.yaml`,放入即自动注册。
|
|
264
|
-
|
|
265
|
-
#### Tier 1 — 公开 API 模板
|
|
266
|
-
|
|
267
|
-
```yaml
|
|
268
|
-
# src/clis/v2ex/hot.yaml
|
|
269
|
-
site: v2ex
|
|
270
|
-
name: hot
|
|
271
|
-
description: V2EX 热门话题
|
|
272
|
-
domain: www.v2ex.com
|
|
273
|
-
strategy: public
|
|
274
|
-
browser: false
|
|
275
|
-
|
|
276
|
-
args:
|
|
277
|
-
limit:
|
|
278
|
-
type: int
|
|
279
|
-
default: 20
|
|
280
|
-
|
|
281
|
-
pipeline:
|
|
282
|
-
- fetch:
|
|
283
|
-
url: https://www.v2ex.com/api/topics/hot.json
|
|
284
|
-
|
|
285
|
-
- map:
|
|
286
|
-
rank: ${{ index + 1 }}
|
|
287
|
-
title: ${{ item.title }}
|
|
288
|
-
replies: ${{ item.replies }}
|
|
289
|
-
|
|
290
|
-
- limit: ${{ args.limit }}
|
|
291
|
-
|
|
292
|
-
columns: [rank, title, replies]
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
#### Tier 2 — Cookie 认证模板(最常用)
|
|
296
|
-
|
|
297
|
-
```yaml
|
|
298
|
-
# src/clis/zhihu/hot.yaml
|
|
299
|
-
site: zhihu
|
|
300
|
-
name: hot
|
|
301
|
-
description: 知乎热榜
|
|
302
|
-
domain: www.zhihu.com
|
|
303
|
-
|
|
304
|
-
pipeline:
|
|
305
|
-
- navigate: https://www.zhihu.com # 先加载页面建立 session
|
|
306
|
-
|
|
307
|
-
- evaluate: | # 在浏览器内发请求,自动带 Cookie
|
|
308
|
-
(async () => {
|
|
309
|
-
const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
|
|
310
|
-
credentials: 'include'
|
|
311
|
-
});
|
|
312
|
-
const d = await res.json();
|
|
313
|
-
return (d?.data || []).map(item => {
|
|
314
|
-
const t = item.target || {};
|
|
315
|
-
return {
|
|
316
|
-
title: t.title,
|
|
317
|
-
heat: item.detail_text || '',
|
|
318
|
-
answers: t.answer_count,
|
|
319
|
-
};
|
|
320
|
-
});
|
|
321
|
-
})()
|
|
322
|
-
|
|
323
|
-
- map:
|
|
324
|
-
rank: ${{ index + 1 }}
|
|
325
|
-
title: ${{ item.title }}
|
|
326
|
-
heat: ${{ item.heat }}
|
|
327
|
-
answers: ${{ item.answers }}
|
|
328
|
-
|
|
329
|
-
- limit: ${{ args.limit }}
|
|
330
|
-
|
|
331
|
-
columns: [rank, title, heat, answers]
|
|
332
|
-
```
|
|
333
|
-
|
|
334
|
-
> **关键**: `evaluate` 步骤内的 `fetch` 运行在浏览器页面内,自动携带 `credentials: 'include'`,无需手动处理 Cookie。
|
|
335
|
-
|
|
336
|
-
#### 进阶 — 带搜索参数
|
|
337
|
-
|
|
338
|
-
```yaml
|
|
339
|
-
# src/clis/zhihu/search.yaml
|
|
340
|
-
site: zhihu
|
|
341
|
-
name: search
|
|
342
|
-
description: 知乎搜索
|
|
343
|
-
|
|
344
|
-
args:
|
|
345
|
-
query:
|
|
346
|
-
type: str
|
|
347
|
-
required: true
|
|
348
|
-
positional: true
|
|
349
|
-
description: Search query
|
|
350
|
-
limit:
|
|
351
|
-
type: int
|
|
352
|
-
default: 10
|
|
353
|
-
|
|
354
|
-
pipeline:
|
|
355
|
-
- navigate: https://www.zhihu.com
|
|
356
|
-
|
|
357
|
-
- evaluate: |
|
|
358
|
-
(async () => {
|
|
359
|
-
const q = encodeURIComponent('${{ args.query }}');
|
|
360
|
-
const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
|
|
361
|
-
credentials: 'include'
|
|
362
|
-
});
|
|
363
|
-
const d = await res.json();
|
|
364
|
-
return (d?.data || [])
|
|
365
|
-
.filter(item => item.type === 'search_result')
|
|
366
|
-
.map(item => ({
|
|
367
|
-
title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
|
|
368
|
-
type: item.object?.type || '',
|
|
369
|
-
author: item.object?.author?.name || '',
|
|
370
|
-
votes: item.object?.voteup_count || 0,
|
|
371
|
-
}));
|
|
372
|
-
})()
|
|
373
|
-
|
|
374
|
-
- map:
|
|
375
|
-
rank: ${{ index + 1 }}
|
|
376
|
-
title: ${{ item.title }}
|
|
377
|
-
type: ${{ item.type }}
|
|
378
|
-
author: ${{ item.author }}
|
|
379
|
-
votes: ${{ item.votes }}
|
|
380
|
-
|
|
381
|
-
- limit: ${{ args.limit }}
|
|
382
|
-
|
|
383
|
-
columns: [rank, title, type, author, votes]
|
|
384
|
-
```
|
|
385
|
-
|
|
386
|
-
#### Tier 4 — Store Action Bridge(`tap` 步骤,intercept 策略推荐)
|
|
387
|
-
|
|
388
|
-
适用于 Vue + Pinia/Vuex 的网站(如小红书),无须手动写 XHR 拦截代码:
|
|
389
|
-
|
|
390
|
-
```yaml
|
|
391
|
-
# src/clis/xiaohongshu/notifications.yaml
|
|
392
|
-
site: xiaohongshu
|
|
393
|
-
name: notifications
|
|
394
|
-
description: "小红书通知"
|
|
395
|
-
domain: www.xiaohongshu.com
|
|
396
|
-
strategy: intercept
|
|
397
|
-
browser: true
|
|
398
|
-
|
|
399
|
-
args:
|
|
400
|
-
type:
|
|
401
|
-
type: str
|
|
402
|
-
default: mentions
|
|
403
|
-
description: "Notification type: mentions, likes, or connections"
|
|
404
|
-
limit:
|
|
405
|
-
type: int
|
|
406
|
-
default: 20
|
|
407
|
-
|
|
408
|
-
columns: [rank, user, action, content, note, time]
|
|
409
|
-
|
|
410
|
-
pipeline:
|
|
411
|
-
- navigate: https://www.xiaohongshu.com/notification
|
|
412
|
-
- wait: 3
|
|
413
|
-
- tap:
|
|
414
|
-
store: notification # Pinia store name
|
|
415
|
-
action: getNotification # Store action to call
|
|
416
|
-
args: # Action arguments
|
|
417
|
-
- ${{ args.type | default('mentions') }}
|
|
418
|
-
capture: /you/ # URL pattern to capture response
|
|
419
|
-
select: data.message_list # Extract sub-path from response
|
|
420
|
-
timeout: 8
|
|
421
|
-
- map:
|
|
422
|
-
rank: ${{ index + 1 }}
|
|
423
|
-
user: ${{ item.user_info.nickname }}
|
|
424
|
-
action: ${{ item.title }}
|
|
425
|
-
content: ${{ item.comment_info.content }}
|
|
426
|
-
- limit: ${{ args.limit | default(20) }}
|
|
427
|
-
```
|
|
428
|
-
|
|
429
|
-
> **`tap` 步骤自动完成**:注入 fetch+XHR 双拦截 → 查找 Pinia/Vuex store → 调用 action → 捕获匹配 URL 的响应 → 清理拦截。
|
|
430
|
-
> 如果 store 或 action 找不到,会返回 `hint` 列出所有可用的 store actions,方便调试。
|
|
431
|
-
|
|
432
|
-
| tap 参数 | 必填 | 说明 |
|
|
433
|
-
|---------|------|------|
|
|
434
|
-
| `store` | ✅ | Pinia store 名称(如 `feed`, `search`, `notification`) |
|
|
435
|
-
| `action` | ✅ | Store action 方法名 |
|
|
436
|
-
| `capture` | ✅ | URL 子串匹配(匹配网络请求 URL) |
|
|
437
|
-
| `args` | ❌ | 传给 action 的参数数组 |
|
|
438
|
-
| `select` | ❌ | 从 captured JSON 中提取的路径(如 `data.items`) |
|
|
439
|
-
| `timeout` | ❌ | 等待网络响应的超时秒数(默认 5s) |
|
|
440
|
-
| `framework` | ❌ | `pinia` 或 `vuex`(默认自动检测) |
|
|
441
|
-
|
|
442
|
-
### 方式 B: TypeScript 适配器(编程式)
|
|
443
|
-
|
|
444
|
-
适用于需要嵌入 JS 代码读取 Pinia state、XHR 拦截、GraphQL、分页、复杂数据转换等场景。
|
|
445
|
-
|
|
446
|
-
文件路径: `src/clis/<site>/<name>.ts`。文件将会在运行时被动态扫描并注册(切勿在 `index.ts` 中手动 `import`)。
|
|
447
|
-
|
|
448
|
-
#### Tier 3 — Header 认证(Twitter)
|
|
449
|
-
|
|
450
|
-
```typescript
|
|
451
|
-
// src/clis/twitter/search.ts
|
|
452
|
-
import { cli, Strategy } from '../../registry.js';
|
|
453
|
-
|
|
454
|
-
cli({
|
|
455
|
-
site: 'twitter',
|
|
456
|
-
name: 'search',
|
|
457
|
-
description: 'Search tweets',
|
|
458
|
-
strategy: Strategy.HEADER,
|
|
459
|
-
args: [{ name: 'query', required: true, positional: true }],
|
|
460
|
-
columns: ['rank', 'author', 'text', 'likes'],
|
|
461
|
-
func: async (page, kwargs) => {
|
|
462
|
-
await page.goto('https://x.com');
|
|
463
|
-
const data = await page.evaluate(`
|
|
464
|
-
(async () => {
|
|
465
|
-
// 从 Cookie 提取 CSRF token
|
|
466
|
-
const ct0 = document.cookie.split(';')
|
|
467
|
-
.map(c => c.trim())
|
|
468
|
-
.find(c => c.startsWith('ct0='))?.split('=')[1];
|
|
469
|
-
if (!ct0) return { error: 'Not logged in' };
|
|
470
|
-
|
|
471
|
-
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
|
|
472
|
-
const headers = {
|
|
473
|
-
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
|
|
474
|
-
'X-Csrf-Token': ct0,
|
|
475
|
-
'X-Twitter-Auth-Type': 'OAuth2Session',
|
|
476
|
-
};
|
|
477
|
-
|
|
478
|
-
const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
|
|
479
|
-
const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
|
|
480
|
-
const res = await fetch(url, { headers, credentials: 'include' });
|
|
481
|
-
return await res.json();
|
|
482
|
-
})()
|
|
483
|
-
`);
|
|
484
|
-
// ... 解析 data
|
|
485
|
-
},
|
|
486
|
-
});
|
|
487
|
-
```
|
|
488
|
-
|
|
489
|
-
#### Tier 4 — XHR/Fetch 双重拦截 (Twitter/小红书 通用模式)
|
|
490
|
-
|
|
491
|
-
```typescript
|
|
492
|
-
// src/clis/xiaohongshu/user.ts
|
|
493
|
-
import { cli, Strategy } from '../../registry.js';
|
|
494
|
-
|
|
495
|
-
cli({
|
|
496
|
-
site: 'xiaohongshu',
|
|
497
|
-
name: 'user',
|
|
498
|
-
description: '获取用户笔记',
|
|
499
|
-
strategy: Strategy.INTERCEPT,
|
|
500
|
-
args: [{ name: 'id', required: true }],
|
|
501
|
-
columns: ['rank', 'title', 'likes', 'url'],
|
|
502
|
-
func: async (page, kwargs) => {
|
|
503
|
-
await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
|
|
504
|
-
await page.wait(5);
|
|
505
|
-
|
|
506
|
-
// XHR/Fetch 底层拦截:捕获所有包含 'v1/user/posted' 的请求
|
|
507
|
-
await page.installInterceptor('v1/user/posted');
|
|
508
|
-
|
|
509
|
-
// 触发后端 API:模拟人类用户向底部滚动2次
|
|
510
|
-
await page.autoScroll({ times: 2, delayMs: 2000 });
|
|
511
|
-
|
|
512
|
-
// 提取所有被拦截捕获的 JSON 响应体
|
|
513
|
-
const requests = await page.getInterceptedRequests();
|
|
514
|
-
if (!requests || requests.length === 0) return [];
|
|
515
|
-
|
|
516
|
-
let results = [];
|
|
517
|
-
for (const req of requests) {
|
|
518
|
-
if (req.data?.data?.notes) {
|
|
519
|
-
for (const note of req.data.data.notes) {
|
|
520
|
-
results.push({
|
|
521
|
-
title: note.display_title || '',
|
|
522
|
-
likes: note.interact_info?.liked_count || '0',
|
|
523
|
-
url: `https://explore/${note.note_id || note.id}`
|
|
524
|
-
});
|
|
525
|
-
}
|
|
526
|
-
}
|
|
527
|
-
}
|
|
528
|
-
|
|
529
|
-
return results.slice(0, 20).map((item, i) => ({
|
|
530
|
-
rank: i + 1, ...item,
|
|
531
|
-
}));
|
|
532
|
-
},
|
|
533
|
-
});
|
|
534
|
-
```
|
|
535
|
-
|
|
536
|
-
> **拦截核心思路**:不自己构造签名,而是利用 `installInterceptor` 劫持网站自己的 `XMLHttpRequest` 和 `fetch`,让网站发请求,我们直接在底层取出解析好的 `response.json()`。
|
|
537
|
-
|
|
538
|
-
> **级联请求**(如 BVID→CID→字幕)的完整模板和要点见下方[进阶模式: 级联请求](#进阶模式-级联请求-cascading-requests)章节。
|
|
539
|
-
|
|
540
|
-
---
|
|
541
|
-
|
|
542
|
-
## Step 4: 测试
|
|
543
|
-
|
|
544
|
-
> **构建通过 ≠ 功能正常**。`npm run build` 只验证 TypeScript / YAML 语法,不验证运行时行为。
|
|
545
|
-
> 每个新命令 **必须实际运行** 并确认输出正确后才算完成。
|
|
546
|
-
|
|
547
|
-
### 必做清单
|
|
548
|
-
|
|
549
|
-
```bash
|
|
550
|
-
# 1. 构建(确认语法无误)
|
|
551
|
-
npm run build
|
|
552
|
-
|
|
553
|
-
# 2. 确认命令已注册
|
|
554
|
-
opencli list | grep mysite
|
|
555
|
-
|
|
556
|
-
# 3. 实际运行命令(最关键!)
|
|
557
|
-
opencli mysite hot --limit 3 -v # verbose 查看每步数据流
|
|
558
|
-
opencli mysite hot --limit 3 -f json # JSON 输出确认字段完整
|
|
559
|
-
```
|
|
560
|
-
|
|
561
|
-
### tap 步骤调试(intercept 策略专用)
|
|
562
|
-
|
|
563
|
-
> **不要猜 store name / action name**。先用 evaluate 探索,再写 YAML。
|
|
564
|
-
|
|
565
|
-
#### Step 1: 列出所有 Pinia store
|
|
566
|
-
|
|
567
|
-
在浏览器中打开目标网站后:
|
|
568
|
-
|
|
569
|
-
```bash
|
|
570
|
-
opencli evaluate "(() => {
|
|
571
|
-
const app = document.querySelector('#app')?.__vue_app__;
|
|
572
|
-
const pinia = app?.config?.globalProperties?.\$pinia;
|
|
573
|
-
return [...pinia._s.keys()];
|
|
574
|
-
})()"
|
|
575
|
-
# 输出: ["user", "feed", "search", "notification", ...]
|
|
576
|
-
```
|
|
577
|
-
|
|
578
|
-
#### Step 2: 查看 store 的 action 名称
|
|
579
|
-
|
|
580
|
-
故意写一个错误 action 名,tap 会返回所有可用 actions:
|
|
581
|
-
|
|
582
|
-
```
|
|
583
|
-
⚠ tap: Action not found: wrongName on store notification
|
|
584
|
-
💡 Available: getNotification, replyComment, getNotificationCount, reset
|
|
585
|
-
```
|
|
586
|
-
|
|
587
|
-
#### Step 3: 用 network requests 确认 capture 模式
|
|
588
|
-
|
|
589
|
-
```bash
|
|
590
|
-
# 在浏览器打开目标页面,查看网络请求
|
|
591
|
-
# 找到目标 API 的 URL 特征(如 "/you/mentions"、"homefeed")
|
|
592
|
-
```
|
|
593
|
-
|
|
594
|
-
#### 完整流程
|
|
595
|
-
|
|
596
|
-
```
|
|
597
|
-
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────┐
|
|
598
|
-
│ 1. navigate │ ──▶ │ 2. 探索 store │ ──▶ │ 3. 写 YAML │ ──▶ │ 4. 测试 │
|
|
599
|
-
│ 到目标页面 │ │ name/action │ │ tap 步骤 │ │ 运行验证 │
|
|
600
|
-
└──────────────┘ └──────────────┘ └──────────────┘ └────────┘
|
|
601
|
-
```
|
|
602
|
-
|
|
603
|
-
### Verbose 模式 & 输出验证
|
|
604
|
-
|
|
605
|
-
```bash
|
|
606
|
-
opencli bilibili hot --limit 1 -v # 查看 pipeline 每步数据流
|
|
607
|
-
opencli mysite hot -f json | jq '.[0]' # 确认 JSON 可被解析
|
|
608
|
-
opencli mysite hot -f csv > data.csv # 确认 CSV 可导入
|
|
609
|
-
```
|
|
610
|
-
|
|
611
|
-
---
|
|
612
|
-
|
|
613
|
-
## Step 5: 提交发布
|
|
614
|
-
|
|
615
|
-
文件放入 `src/clis/<site>/` 即自动注册(YAML 或 TS 无需手动 import),然后:
|
|
616
|
-
|
|
617
|
-
```bash
|
|
618
|
-
opencli list | grep mysite # 确认注册
|
|
619
|
-
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
|
|
620
|
-
```
|
|
621
|
-
|
|
622
|
-
> **架构理念**:OpenCLI 内建 **Zero-Dependency jq** 数据流 — 所有解析在 `evaluate` 的原生 JS 内完成,外层 YAML 用 `select`/`map` 提取,无需依赖系统 `jq` 二进制。
|
|
623
|
-
|
|
624
|
-
---
|
|
625
|
-
|
|
626
|
-
## 进阶模式: 级联请求 (Cascading Requests)
|
|
627
|
-
|
|
628
|
-
当目标数据需要多步 API 链式获取时(如 `BVID → CID → 字幕列表 → 字幕内容`),必须使用 **TS 适配器**。YAML 无法处理这种多步逻辑。
|
|
629
|
-
|
|
630
|
-
### 模板代码
|
|
631
|
-
|
|
632
|
-
```typescript
|
|
633
|
-
import { cli, Strategy } from '../../registry.js';
|
|
634
|
-
import type { IPage } from '../../types.js';
|
|
635
|
-
import { apiGet } from './utils.js'; // 复用平台 SDK
|
|
636
|
-
|
|
637
|
-
cli({
|
|
638
|
-
site: 'bilibili',
|
|
639
|
-
name: 'subtitle',
|
|
640
|
-
strategy: Strategy.COOKIE,
|
|
641
|
-
args: [{ name: 'bvid', required: true }],
|
|
642
|
-
columns: ['index', 'from', 'to', 'content'],
|
|
643
|
-
func: async (page: IPage | null, kwargs: any) => {
|
|
644
|
-
if (!page) throw new Error('Requires browser');
|
|
645
|
-
|
|
646
|
-
// Step 1: 建立 Session
|
|
647
|
-
await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);
|
|
648
|
-
|
|
649
|
-
// Step 2: 从页面提取中间 ID (__INITIAL_STATE__)
|
|
650
|
-
const cid = await page.evaluate(`(async () => {
|
|
651
|
-
return window.__INITIAL_STATE__?.videoData?.cid;
|
|
652
|
-
})()`);
|
|
653
|
-
if (!cid) throw new Error('无法提取 CID');
|
|
654
|
-
|
|
655
|
-
// Step 3: 用中间 ID 调用下一级 API (自动 Wbi 签名)
|
|
656
|
-
const payload = await apiGet(page, '/x/player/wbi/v2', {
|
|
657
|
-
params: { bvid: kwargs.bvid, cid },
|
|
658
|
-
signed: true, // ← 自动生成 w_rid
|
|
659
|
-
});
|
|
660
|
-
|
|
661
|
-
// Step 4: 检测风控降级 (空值断言)
|
|
662
|
-
const subtitles = payload.data?.subtitle?.subtitles || [];
|
|
663
|
-
const url = subtitles[0]?.subtitle_url;
|
|
664
|
-
if (!url) throw new Error('subtitle_url 为空,疑似风控降级');
|
|
665
|
-
|
|
666
|
-
// Step 5: 拉取最终数据 (CDN JSON)
|
|
667
|
-
const items = await page.evaluate(`(async () => {
|
|
668
|
-
const res = await fetch(${JSON.stringify('https:' + url)});
|
|
669
|
-
const json = await res.json();
|
|
670
|
-
return { data: json.body || json };
|
|
671
|
-
})()`);
|
|
672
|
-
|
|
673
|
-
return items.data.map((item, idx) => ({ ... }));
|
|
674
|
-
},
|
|
675
|
-
});
|
|
676
|
-
```
|
|
677
|
-
|
|
678
|
-
### 关键要点
|
|
679
|
-
|
|
680
|
-
| 步骤 | 注意事项 |
|
|
681
|
-
|------|----------|
|
|
682
|
-
| 提取中间 ID | 优先从 `__INITIAL_STATE__` 拿,避免额外 API 调用 |
|
|
683
|
-
| Wbi 签名 | B 站 `/wbi/` 接口**强制校验** `w_rid`,纯 `fetch` 会被 403 |
|
|
684
|
-
| 空值断言 | 即使 HTTP 200,核心字段可能为空串(风控降级) |
|
|
685
|
-
| CDN URL | 常以 `//` 开头,记得补 `https:` |
|
|
686
|
-
| `JSON.stringify` | 拼接 URL 到 evaluate 时必须用它转义,避免注入 |
|
|
687
|
-
|
|
688
|
-
---
|
|
689
|
-
|
|
690
|
-
## 常见陷阱
|
|
691
|
-
|
|
692
|
-
| 陷阱 | 表现 | 解决方案 |
|
|
693
|
-
|------|------|---------|
|
|
694
|
-
| 缺少 `navigate` | evaluate 报 `Target page context` 错误 | 在 evaluate 前加 `navigate:` 步骤 |
|
|
695
|
-
| 嵌套字段访问 | `${{ item.node?.title }}` 不工作 | 在 evaluate 中 flatten 数据,不在模板中用 optional chaining |
|
|
696
|
-
| 缺少 `strategy: public` | 公开 API 也启动浏览器,7s → 1s | 公开 API 加上 `strategy: public` + `browser: false` |
|
|
697
|
-
| evaluate 返回字符串 | map 步骤收到 `""` 而非数组 | pipeline 有 auto-parse,但建议在 evaluate 内 `.map()` 整形 |
|
|
698
|
-
| 搜索参数被 URL 编码 | `${{ args.query }}` 被浏览器二次编码 | 在 evaluate 内用 `encodeURIComponent()` 手动编码 |
|
|
699
|
-
| Cookie 过期 | 返回 401 / 空数据 | 在浏览器里重新登录目标站点 |
|
|
700
|
-
| Extension tab 残留 | Chrome 多出 `chrome-extension://` tab | 已自动清理;若残留,手动关闭即可 |
|
|
701
|
-
| TS evaluate 格式 | `() => {}` 报 `result is not a function` | TS 中 `page.evaluate()` 必须用 IIFE:`(async () => { ... })()` |
|
|
702
|
-
| 页面异步加载 | evaluate 拿到空数据(store state 还没更新) | 在 evaluate 内用 polling 等待数据出现,或增加 `wait` 时间 |
|
|
703
|
-
| YAML 内嵌大段 JS | 调试困难,字符串转义问题 | 超过 10 行 JS 的命令改用 TS adapter |
|
|
704
|
-
| **风控被拦截(伪200)** | 获取到的 JSON 里核心数据是 `""` (空串) | 极易被误判。必须添加断言!无核心数据立刻要求升级鉴权 Tier 并重新配置 Cookie |
|
|
705
|
-
| **API 没找见** | `explore` 工具打分出来的都拿不到深层数据 | 点击页面按钮诱发懒加载数据,再结合 `getInterceptedRequests` 获取 |
|
|
706
|
-
|
|
707
|
-
---
|
|
708
|
-
|
|
709
|
-
## 用 AI Agent 自动生成适配器
|
|
710
|
-
|
|
711
|
-
最快的方式是让 AI Agent 完成全流程:
|
|
712
|
-
|
|
713
|
-
```bash
|
|
714
|
-
# 一键:探索 → 分析 → 合成 → 注册
|
|
715
|
-
opencli generate https://www.example.com --goal "hot"
|
|
716
|
-
|
|
717
|
-
# 或分步执行:
|
|
718
|
-
opencli explore https://www.example.com --site mysite # 发现 API
|
|
719
|
-
opencli explore https://www.example.com --auto --click "字幕,CC" # 模拟点击触发懒加载 API
|
|
720
|
-
opencli synthesize mysite # 生成候选 YAML
|
|
721
|
-
opencli verify mysite/hot --smoke # 冒烟测试
|
|
722
|
-
```
|
|
723
|
-
|
|
724
|
-
生成的候选 YAML 保存在 `.opencli/explore/mysite/candidates/`,可直接复制到 `src/clis/mysite/` 并微调。
|