@jackwener/opencli 1.3.3 โ 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/pull_request_template.md +3 -1
- package/.github/workflows/build-extension.yml +7 -1
- package/.github/workflows/ci.yml +29 -3
- package/.github/workflows/docs.yml +1 -1
- package/.github/workflows/e2e-headed.yml +20 -0
- package/.github/workflows/release.yml +1 -1
- package/.github/workflows/security.yml +0 -3
- package/CHANGELOG.md +55 -0
- package/CONTRIBUTING.md +6 -3
- package/README.md +30 -3
- package/README.zh-CN.md +30 -3
- package/SKILL.md +7 -1
- package/TESTING.md +1 -0
- package/chatwise-opencli.ps1 +82 -0
- package/dist/analysis.d.ts +38 -0
- package/dist/analysis.js +166 -0
- package/dist/browser/cdp.d.ts +0 -4
- package/dist/browser/cdp.js +53 -41
- package/dist/browser/cdp.test.d.ts +1 -0
- package/dist/browser/cdp.test.js +52 -0
- package/dist/browser/dom-snapshot.d.ts +2 -2
- package/dist/browser/dom-snapshot.js +54 -1
- package/dist/browser/dom-snapshot.test.js +36 -0
- package/dist/browser/index.d.ts +2 -2
- package/dist/browser/index.js +1 -1
- package/dist/browser/mcp.d.ts +0 -2
- package/dist/browser/mcp.js +2 -3
- package/dist/browser/page.d.ts +4 -3
- package/dist/browser/page.js +34 -37
- package/dist/browser/stealth.d.ts +0 -2
- package/dist/browser/stealth.js +24 -9
- package/dist/browser.test.js +2 -2
- package/dist/build-manifest.js +15 -9
- package/dist/build-manifest.test.js +12 -0
- package/dist/cascade.js +4 -2
- package/dist/cli-manifest.json +639 -258
- package/dist/cli.js +57 -29
- package/dist/clis/_shared/desktop-commands.d.ts +22 -0
- package/dist/clis/_shared/desktop-commands.js +108 -0
- package/dist/clis/antigravity/serve.js +5 -2
- package/dist/clis/arxiv/search.js +1 -1
- package/dist/clis/bilibili/dynamic.test.d.ts +1 -0
- package/dist/clis/bilibili/dynamic.test.js +68 -0
- package/dist/clis/bilibili/favorite.js +4 -2
- package/dist/clis/bilibili/following.js +3 -2
- package/dist/clis/bilibili/subtitle.js +8 -7
- package/dist/clis/bilibili/utils.js +2 -2
- package/dist/clis/boss/batchgreet.js +1 -1
- package/dist/clis/boss/chatlist.js +1 -1
- package/dist/clis/boss/chatmsg.js +1 -1
- package/dist/clis/boss/detail.js +1 -1
- package/dist/clis/boss/exchange.js +1 -1
- package/dist/clis/boss/greet.js +1 -1
- package/dist/clis/boss/invite.js +1 -1
- package/dist/clis/boss/joblist.js +1 -1
- package/dist/clis/boss/mark.js +4 -3
- package/dist/clis/boss/recommend.js +1 -1
- package/dist/clis/boss/resume.js +1 -1
- package/dist/clis/boss/search.js +1 -1
- package/dist/clis/boss/send.js +5 -4
- package/dist/clis/boss/stats.js +1 -1
- package/dist/clis/chatgpt/ask.js +4 -0
- package/dist/clis/chatgpt/new.js +5 -1
- package/dist/clis/chatgpt/read.js +5 -1
- package/dist/clis/chatgpt/send.js +2 -1
- package/dist/clis/chatgpt/status.js +5 -1
- package/dist/clis/chatwise/ask.js +8 -2
- package/dist/clis/chatwise/export.js +2 -0
- package/dist/clis/chatwise/history.js +2 -0
- package/dist/clis/chatwise/model.js +8 -3
- package/dist/clis/chatwise/new.js +3 -18
- package/dist/clis/chatwise/read.js +2 -0
- package/dist/clis/chatwise/screenshot.js +3 -27
- package/dist/clis/chatwise/send.js +8 -2
- package/dist/clis/chatwise/shared.d.ts +2 -0
- package/dist/clis/chatwise/shared.js +6 -0
- package/dist/clis/chatwise/status.js +3 -22
- package/dist/clis/codex/ask.js +6 -2
- package/dist/clis/codex/dump.js +2 -25
- package/dist/clis/codex/new.js +2 -25
- package/dist/clis/codex/screenshot.js +2 -27
- package/dist/clis/codex/send.js +6 -4
- package/dist/clis/codex/status.js +2 -22
- package/dist/clis/cursor/ask.js +2 -1
- package/dist/clis/cursor/composer.js +2 -1
- package/dist/clis/cursor/dump.js +2 -25
- package/dist/clis/cursor/new.js +2 -18
- package/dist/clis/cursor/read.js +2 -1
- package/dist/clis/cursor/screenshot.js +1 -30
- package/dist/clis/cursor/send.js +2 -1
- package/dist/clis/cursor/status.js +2 -21
- package/dist/clis/dictionary/examples.yaml +25 -0
- package/dist/clis/dictionary/search.yaml +27 -0
- package/dist/clis/dictionary/synonyms.yaml +25 -0
- package/dist/clis/douban/book-hot.js +1 -1
- package/dist/clis/douban/movie-hot.js +1 -1
- package/dist/clis/douban/search.js +1 -1
- package/dist/clis/douban/utils.d.ts +4 -1
- package/dist/clis/douban/utils.js +156 -1
- package/dist/clis/doubao/ask.js +1 -1
- package/dist/clis/doubao/new.js +1 -1
- package/dist/clis/doubao/read.js +1 -1
- package/dist/clis/doubao/send.js +1 -1
- package/dist/clis/doubao/status.js +1 -1
- package/dist/clis/doubao-app/ask.js +1 -1
- package/dist/clis/doubao-app/new.js +1 -1
- package/dist/clis/doubao-app/read.js +1 -1
- package/dist/clis/doubao-app/send.js +1 -1
- package/dist/clis/grok/ask.d.ts +4 -0
- package/dist/clis/grok/ask.js +28 -10
- package/dist/clis/grok/ask.test.js +18 -0
- package/dist/clis/jd/item.d.ts +1 -0
- package/dist/clis/jd/item.js +96 -0
- package/dist/clis/jd/item.test.d.ts +1 -0
- package/dist/clis/jd/item.test.js +28 -0
- package/dist/clis/jike/feed.js +1 -1
- package/dist/clis/jike/search.js +1 -1
- package/dist/clis/linkedin/search.js +5 -4
- package/dist/clis/linkedin/timeline.d.ts +21 -0
- package/dist/clis/linkedin/timeline.js +503 -0
- package/dist/clis/linkedin/timeline.test.d.ts +1 -0
- package/dist/clis/linkedin/timeline.test.js +81 -0
- package/dist/clis/medium/feed.js +1 -1
- package/dist/clis/medium/search.js +1 -1
- package/dist/clis/medium/user.js +1 -1
- package/dist/clis/medium/{shared.js โ utils.js} +2 -1
- package/dist/clis/pixiv/detail.yaml +49 -0
- package/dist/clis/pixiv/download.d.ts +7 -0
- package/dist/clis/pixiv/download.js +78 -0
- package/dist/clis/pixiv/download.test.d.ts +1 -0
- package/dist/clis/pixiv/download.test.js +87 -0
- package/dist/clis/pixiv/illusts.d.ts +8 -0
- package/dist/clis/pixiv/illusts.js +65 -0
- package/dist/clis/pixiv/illusts.test.d.ts +1 -0
- package/dist/clis/pixiv/illusts.test.js +99 -0
- package/dist/clis/pixiv/ranking.yaml +53 -0
- package/dist/clis/pixiv/search.d.ts +6 -0
- package/dist/clis/pixiv/search.js +43 -0
- package/dist/clis/pixiv/search.test.d.ts +1 -0
- package/dist/clis/pixiv/search.test.js +83 -0
- package/dist/clis/pixiv/test-utils.d.ts +12 -0
- package/dist/clis/pixiv/test-utils.js +23 -0
- package/dist/clis/pixiv/user.yaml +46 -0
- package/dist/clis/pixiv/utils.d.ts +27 -0
- package/dist/clis/pixiv/utils.js +49 -0
- package/dist/clis/reddit/comment.js +2 -1
- package/dist/clis/reddit/read.js +4 -3
- package/dist/clis/reddit/read.test.d.ts +1 -0
- package/dist/clis/reddit/read.test.js +28 -0
- package/dist/clis/reddit/save.js +2 -1
- package/dist/clis/reddit/saved.js +7 -3
- package/dist/clis/reddit/subscribe.js +2 -1
- package/dist/clis/reddit/upvote.js +2 -1
- package/dist/clis/reddit/upvoted.js +7 -3
- package/dist/clis/sinablog/article.js +1 -1
- package/dist/clis/sinablog/hot.js +1 -1
- package/dist/clis/sinablog/user.js +1 -1
- package/dist/clis/substack/feed.js +1 -1
- package/dist/clis/substack/publication.js +1 -1
- package/dist/clis/substack/search.js +3 -2
- package/dist/clis/substack/{shared.js โ utils.js} +3 -2
- package/dist/clis/tiktok/search.yaml +2 -1
- package/dist/clis/twitter/accept.js +2 -1
- package/dist/clis/twitter/article.js +4 -1
- package/dist/clis/twitter/block.js +2 -1
- package/dist/clis/twitter/bookmark.js +2 -1
- package/dist/clis/twitter/bookmarks.js +3 -2
- package/dist/clis/twitter/delete.js +2 -1
- package/dist/clis/twitter/follow.js +2 -1
- package/dist/clis/twitter/followers.js +3 -2
- package/dist/clis/twitter/following.js +3 -2
- package/dist/clis/twitter/hide-reply.js +2 -1
- package/dist/clis/twitter/like.js +2 -1
- package/dist/clis/twitter/notifications.js +2 -1
- package/dist/clis/twitter/post.js +2 -1
- package/dist/clis/twitter/profile.js +5 -2
- package/dist/clis/twitter/reply-dm.js +2 -1
- package/dist/clis/twitter/reply.js +2 -1
- package/dist/clis/twitter/search.js +30 -13
- package/dist/clis/twitter/search.test.d.ts +1 -0
- package/dist/clis/twitter/search.test.js +104 -0
- package/dist/clis/twitter/thread.js +2 -2
- package/dist/clis/twitter/timeline.js +3 -2
- package/dist/clis/twitter/trending.js +3 -2
- package/dist/clis/twitter/unblock.js +2 -1
- package/dist/clis/twitter/unbookmark.js +2 -1
- package/dist/clis/twitter/unfollow.js +2 -1
- package/dist/clis/v2ex/daily.js +3 -2
- package/dist/clis/v2ex/me.js +3 -2
- package/dist/clis/v2ex/notifications.js +4 -4
- package/dist/clis/web/read.d.ts +16 -0
- package/dist/clis/web/read.js +202 -0
- package/dist/clis/xueqiu/danjuan-utils.d.ts +55 -0
- package/dist/clis/xueqiu/danjuan-utils.js +126 -0
- package/dist/clis/xueqiu/danjuan-utils.test.d.ts +1 -0
- package/dist/clis/xueqiu/danjuan-utils.test.js +41 -0
- package/dist/clis/xueqiu/fund-holdings.d.ts +1 -0
- package/dist/clis/xueqiu/fund-holdings.js +28 -0
- package/dist/clis/xueqiu/fund-snapshot.d.ts +1 -0
- package/dist/clis/xueqiu/fund-snapshot.js +25 -0
- package/dist/clis/youtube/transcript.js +5 -4
- package/dist/clis/youtube/video.js +3 -2
- package/dist/daemon.js +7 -3
- package/dist/discovery.js +11 -10
- package/dist/doctor.js +2 -1
- package/dist/download/index.d.ts +4 -12
- package/dist/download/index.js +33 -12
- package/dist/download/index.test.js +79 -2
- package/dist/download/media-download.js +4 -2
- package/dist/engine.test.js +76 -4
- package/dist/execution.d.ts +1 -9
- package/dist/execution.js +56 -46
- package/dist/explore.js +12 -111
- package/dist/external-clis.yaml +0 -8
- package/dist/external.js +7 -5
- package/dist/external.test.js +4 -0
- package/dist/generate.d.ts +0 -9
- package/dist/generate.js +4 -20
- package/dist/hooks.d.ts +46 -0
- package/dist/hooks.js +56 -0
- package/dist/hooks.test.d.ts +4 -0
- package/dist/hooks.test.js +92 -0
- package/dist/interceptor.js +70 -23
- package/dist/main.js +2 -0
- package/dist/output.js +12 -6
- package/dist/pipeline/executor.js +1 -1
- package/dist/pipeline/steps/browser.js +1 -3
- package/dist/pipeline/steps/download.js +42 -26
- package/dist/pipeline/steps/download.test.d.ts +1 -0
- package/dist/pipeline/steps/download.test.js +101 -0
- package/dist/pipeline/steps/fetch.js +40 -22
- package/dist/pipeline/steps/fetch.test.d.ts +1 -0
- package/dist/pipeline/steps/fetch.test.js +123 -0
- package/dist/pipeline/steps/transform.js +2 -6
- package/dist/pipeline/template.js +66 -52
- package/dist/pipeline/template.test.js +28 -0
- package/dist/pipeline/transform.test.js +18 -0
- package/dist/plugin.d.ts +40 -1
- package/dist/plugin.js +214 -17
- package/dist/plugin.test.d.ts +1 -1
- package/dist/plugin.test.js +219 -3
- package/dist/record.js +6 -98
- package/dist/registry-api.d.ts +2 -0
- package/dist/registry-api.js +1 -0
- package/dist/registry.d.ts +5 -2
- package/dist/registry.js +1 -2
- package/dist/runtime.d.ts +0 -1
- package/dist/runtime.js +14 -4
- package/dist/snapshotFormatter.d.ts +7 -14
- package/dist/snapshotFormatter.js +38 -78
- package/dist/utils.d.ts +9 -0
- package/dist/utils.js +29 -0
- package/dist/validate.js +3 -5
- package/dist/yaml-schema.d.ts +26 -0
- package/dist/yaml-schema.js +5 -0
- package/docs/.vitepress/config.mts +3 -0
- package/docs/adapters/browser/dictionary.md +27 -0
- package/docs/adapters/browser/jd.md +27 -0
- package/docs/adapters/browser/linkedin.md +6 -0
- package/docs/adapters/browser/pixiv.md +92 -0
- package/docs/adapters/browser/web.md +30 -0
- package/docs/adapters/browser/xueqiu.md +27 -9
- package/docs/adapters/index.md +3 -1
- package/docs/comparison.md +125 -0
- package/docs/developer/contributing.md +21 -2
- package/docs/developer/testing.md +14 -8
- package/docs/developer/ts-adapter.md +18 -0
- package/docs/developer/yaml-adapter.md +16 -0
- package/docs/guide/plugins.md +10 -0
- package/docs/zh/guide/plugins.md +10 -0
- package/extension/dist/background.js +519 -444
- package/extension/manifest.json +1 -1
- package/extension/package.json +1 -1
- package/extension/src/background.test.ts +46 -1
- package/extension/src/background.ts +108 -33
- package/extension/src/cdp.ts +9 -9
- package/package.json +3 -2
- package/scripts/check-doc-coverage.sh +2 -0
- package/src/analysis.ts +170 -0
- package/src/browser/cdp.test.ts +66 -0
- package/src/browser/cdp.ts +59 -44
- package/src/browser/dom-snapshot.test.ts +42 -0
- package/src/browser/dom-snapshot.ts +56 -3
- package/src/browser/index.ts +2 -2
- package/src/browser/mcp.ts +2 -4
- package/src/browser/page.ts +34 -37
- package/src/browser/stealth.ts +24 -10
- package/src/browser.test.ts +2 -2
- package/src/build-manifest.test.ts +14 -0
- package/src/build-manifest.ts +13 -31
- package/src/cascade.ts +5 -3
- package/src/cli.ts +66 -34
- package/src/clis/_shared/desktop-commands.ts +121 -0
- package/src/clis/antigravity/serve.ts +6 -3
- package/src/clis/arxiv/search.ts +1 -1
- package/src/clis/bilibili/dynamic.test.ts +79 -0
- package/src/clis/bilibili/favorite.ts +5 -2
- package/src/clis/bilibili/following.ts +3 -2
- package/src/clis/bilibili/subtitle.ts +8 -7
- package/src/clis/bilibili/utils.ts +2 -2
- package/src/clis/boss/batchgreet.ts +1 -1
- package/src/clis/boss/chatlist.ts +1 -1
- package/src/clis/boss/chatmsg.ts +1 -1
- package/src/clis/boss/detail.ts +1 -1
- package/src/clis/boss/exchange.ts +1 -1
- package/src/clis/boss/greet.ts +1 -1
- package/src/clis/boss/invite.ts +1 -1
- package/src/clis/boss/joblist.ts +1 -1
- package/src/clis/boss/mark.ts +4 -3
- package/src/clis/boss/recommend.ts +1 -1
- package/src/clis/boss/resume.ts +1 -1
- package/src/clis/boss/search.ts +1 -1
- package/src/clis/boss/send.ts +5 -4
- package/src/clis/boss/stats.ts +1 -1
- package/src/clis/chatgpt/ask.ts +5 -0
- package/src/clis/chatgpt/new.ts +7 -2
- package/src/clis/chatgpt/read.ts +7 -2
- package/src/clis/chatgpt/send.ts +3 -2
- package/src/clis/chatgpt/status.ts +6 -1
- package/src/clis/chatwise/ask.ts +7 -2
- package/src/clis/chatwise/export.ts +2 -0
- package/src/clis/chatwise/history.ts +2 -0
- package/src/clis/chatwise/model.ts +7 -3
- package/src/clis/chatwise/new.ts +3 -20
- package/src/clis/chatwise/read.ts +2 -0
- package/src/clis/chatwise/screenshot.ts +3 -32
- package/src/clis/chatwise/send.ts +7 -2
- package/src/clis/chatwise/shared.ts +8 -0
- package/src/clis/chatwise/status.ts +3 -24
- package/src/clis/codex/ask.ts +5 -2
- package/src/clis/codex/dump.ts +2 -27
- package/src/clis/codex/new.ts +2 -28
- package/src/clis/codex/screenshot.ts +2 -32
- package/src/clis/codex/send.ts +5 -4
- package/src/clis/codex/status.ts +2 -24
- package/src/clis/cursor/ask.ts +2 -1
- package/src/clis/cursor/composer.ts +2 -1
- package/src/clis/cursor/dump.ts +2 -27
- package/src/clis/cursor/new.ts +2 -20
- package/src/clis/cursor/read.ts +2 -1
- package/src/clis/cursor/screenshot.ts +1 -36
- package/src/clis/cursor/send.ts +2 -1
- package/src/clis/cursor/status.ts +2 -22
- package/src/clis/dictionary/examples.yaml +25 -0
- package/src/clis/dictionary/search.yaml +27 -0
- package/src/clis/dictionary/synonyms.yaml +25 -0
- package/src/clis/douban/book-hot.ts +1 -1
- package/src/clis/douban/movie-hot.ts +1 -1
- package/src/clis/douban/search.ts +1 -1
- package/src/clis/douban/utils.ts +165 -1
- package/src/clis/doubao/ask.ts +1 -1
- package/src/clis/doubao/new.ts +1 -1
- package/src/clis/doubao/read.ts +1 -1
- package/src/clis/doubao/send.ts +1 -1
- package/src/clis/doubao/status.ts +1 -1
- package/src/clis/doubao-app/ask.ts +1 -1
- package/src/clis/doubao-app/new.ts +1 -1
- package/src/clis/doubao-app/read.ts +1 -1
- package/src/clis/doubao-app/send.ts +1 -1
- package/src/clis/grok/ask.test.ts +25 -0
- package/src/clis/grok/ask.ts +25 -12
- package/src/clis/jd/item.test.ts +35 -0
- package/src/clis/jd/item.ts +101 -0
- package/src/clis/jike/feed.ts +1 -1
- package/src/clis/jike/search.ts +1 -1
- package/src/clis/linkedin/search.ts +5 -4
- package/src/clis/linkedin/timeline.test.ts +99 -0
- package/src/clis/linkedin/timeline.ts +532 -0
- package/src/clis/medium/feed.ts +1 -1
- package/src/clis/medium/search.ts +1 -1
- package/src/clis/medium/user.ts +1 -1
- package/src/clis/medium/{shared.ts โ utils.ts} +2 -1
- package/src/clis/pixiv/detail.yaml +49 -0
- package/src/clis/pixiv/download.test.ts +114 -0
- package/src/clis/pixiv/download.ts +91 -0
- package/src/clis/pixiv/illusts.test.ts +115 -0
- package/src/clis/pixiv/illusts.ts +78 -0
- package/src/clis/pixiv/ranking.yaml +53 -0
- package/src/clis/pixiv/search.test.ts +97 -0
- package/src/clis/pixiv/search.ts +53 -0
- package/src/clis/pixiv/test-utils.ts +29 -0
- package/src/clis/pixiv/user.yaml +46 -0
- package/src/clis/pixiv/utils.ts +62 -0
- package/src/clis/reddit/comment.ts +2 -1
- package/src/clis/reddit/read.test.ts +34 -0
- package/src/clis/reddit/read.ts +4 -3
- package/src/clis/reddit/save.ts +2 -1
- package/src/clis/reddit/saved.ts +6 -2
- package/src/clis/reddit/subscribe.ts +2 -1
- package/src/clis/reddit/upvote.ts +2 -1
- package/src/clis/reddit/upvoted.ts +6 -2
- package/src/clis/sinablog/article.ts +1 -1
- package/src/clis/sinablog/hot.ts +1 -1
- package/src/clis/sinablog/user.ts +1 -1
- package/src/clis/substack/feed.ts +1 -1
- package/src/clis/substack/publication.ts +1 -1
- package/src/clis/substack/search.ts +3 -2
- package/src/clis/substack/{shared.ts โ utils.ts} +3 -2
- package/src/clis/tiktok/search.yaml +2 -1
- package/src/clis/twitter/accept.ts +2 -1
- package/src/clis/twitter/article.ts +3 -1
- package/src/clis/twitter/block.ts +2 -1
- package/src/clis/twitter/bookmark.ts +2 -1
- package/src/clis/twitter/bookmarks.ts +3 -2
- package/src/clis/twitter/delete.ts +2 -1
- package/src/clis/twitter/follow.ts +2 -1
- package/src/clis/twitter/followers.ts +3 -2
- package/src/clis/twitter/following.ts +3 -2
- package/src/clis/twitter/hide-reply.ts +2 -1
- package/src/clis/twitter/like.ts +2 -1
- package/src/clis/twitter/notifications.ts +2 -1
- package/src/clis/twitter/post.ts +2 -1
- package/src/clis/twitter/profile.ts +4 -2
- package/src/clis/twitter/reply-dm.ts +2 -1
- package/src/clis/twitter/reply.ts +2 -1
- package/src/clis/twitter/search.test.ts +113 -0
- package/src/clis/twitter/search.ts +38 -14
- package/src/clis/twitter/thread.ts +2 -2
- package/src/clis/twitter/timeline.ts +3 -2
- package/src/clis/twitter/trending.ts +3 -2
- package/src/clis/twitter/unblock.ts +2 -1
- package/src/clis/twitter/unbookmark.ts +2 -1
- package/src/clis/twitter/unfollow.ts +2 -1
- package/src/clis/v2ex/daily.ts +3 -2
- package/src/clis/v2ex/me.ts +3 -2
- package/src/clis/v2ex/notifications.ts +3 -4
- package/src/clis/web/read.ts +210 -0
- package/src/clis/xueqiu/danjuan-utils.test.ts +49 -0
- package/src/clis/xueqiu/danjuan-utils.ts +176 -0
- package/src/clis/xueqiu/fund-holdings.ts +32 -0
- package/src/clis/xueqiu/fund-snapshot.ts +27 -0
- package/src/clis/youtube/transcript.ts +5 -4
- package/src/clis/youtube/video.ts +3 -2
- package/src/daemon.ts +5 -4
- package/src/discovery.ts +12 -34
- package/src/doctor.ts +3 -2
- package/src/download/index.test.ts +93 -2
- package/src/download/index.ts +44 -23
- package/src/download/media-download.ts +5 -3
- package/src/engine.test.ts +84 -3
- package/src/execution.ts +62 -46
- package/src/explore.ts +21 -90
- package/src/external-clis.yaml +0 -8
- package/src/external.test.ts +9 -0
- package/src/external.ts +12 -10
- package/src/generate.ts +4 -41
- package/src/hooks.test.ts +126 -0
- package/src/hooks.ts +90 -0
- package/src/interceptor.ts +73 -23
- package/src/main.ts +2 -0
- package/src/output.ts +14 -6
- package/src/pipeline/executor.ts +1 -1
- package/src/pipeline/steps/browser.ts +1 -3
- package/src/pipeline/steps/download.test.ts +136 -0
- package/src/pipeline/steps/download.ts +47 -34
- package/src/pipeline/steps/fetch.test.ts +179 -0
- package/src/pipeline/steps/fetch.ts +39 -23
- package/src/pipeline/steps/transform.ts +2 -6
- package/src/pipeline/template.test.ts +28 -0
- package/src/pipeline/template.ts +67 -79
- package/src/pipeline/transform.test.ts +20 -0
- package/src/plugin.test.ts +251 -3
- package/src/plugin.ts +265 -21
- package/src/record.ts +12 -84
- package/src/registry-api.ts +2 -0
- package/src/registry.ts +7 -4
- package/src/runtime.ts +14 -4
- package/src/snapshotFormatter.ts +43 -121
- package/src/utils.ts +39 -0
- package/src/validate.ts +3 -5
- package/src/yaml-schema.ts +28 -0
- package/tests/e2e/browser-auth.test.ts +25 -0
- package/tests/e2e/plugin-management.test.ts +137 -0
- package/tests/e2e/public-commands.test.ts +34 -1
- package/vitest.config.ts +19 -1
- package/.github/workflows/pkg-pr-new.yml +0 -30
- package/dist/clis/douban/shared.d.ts +0 -4
- package/dist/clis/douban/shared.js +0 -155
- package/src/clis/douban/shared.ts +0 -165
- /package/dist/clis/boss/{common.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/boss/{common.js โ utils.js} +0 -0
- /package/dist/clis/doubao/{common.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/doubao/{common.js โ utils.js} +0 -0
- /package/dist/clis/doubao-app/{common.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/doubao-app/{common.js โ utils.js} +0 -0
- /package/dist/clis/jike/{shared.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/jike/{shared.js โ utils.js} +0 -0
- /package/dist/clis/medium/{shared.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/sinablog/{shared.d.ts โ utils.d.ts} +0 -0
- /package/dist/clis/sinablog/{shared.js โ utils.js} +0 -0
- /package/dist/clis/substack/{shared.d.ts โ utils.d.ts} +0 -0
- /package/src/clis/boss/{common.ts โ utils.ts} +0 -0
- /package/src/clis/doubao/{common.ts โ utils.ts} +0 -0
- /package/src/clis/doubao-app/{common.ts โ utils.ts} +0 -0
- /package/src/clis/jike/{shared.ts โ utils.ts} +0 -0
- /package/src/clis/sinablog/{shared.ts โ utils.ts} +0 -0
package/dist/record.js
CHANGED
|
@@ -17,7 +17,8 @@ import * as readline from 'node:readline';
|
|
|
17
17
|
import chalk from 'chalk';
|
|
18
18
|
import yaml from 'js-yaml';
|
|
19
19
|
import { sendCommand } from './browser/daemon-client.js';
|
|
20
|
-
import {
|
|
20
|
+
import { SEARCH_PARAMS, PAGINATION_PARAMS, FIELD_ROLES } from './constants.js';
|
|
21
|
+
import { urlToPattern, findArrayPath, inferCapabilityName, inferStrategy, detectAuthFromContent, classifyQueryParams, } from './analysis.js';
|
|
21
22
|
// โโ Interceptor JS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
22
23
|
/**
|
|
23
24
|
* Generates a full-capture interceptor that stores {url, method, status, body}
|
|
@@ -110,92 +111,6 @@ function generateReadRecordedJs() {
|
|
|
110
111
|
`;
|
|
111
112
|
}
|
|
112
113
|
// โโ Analysis helpers โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
113
|
-
function urlToPattern(url) {
|
|
114
|
-
try {
|
|
115
|
-
const p = new URL(url);
|
|
116
|
-
const pathNorm = p.pathname
|
|
117
|
-
.replace(/\/\d+/g, '/{id}')
|
|
118
|
-
.replace(/\/[0-9a-fA-F]{8,}/g, '/{hex}')
|
|
119
|
-
.replace(/\/BV[a-zA-Z0-9]{10}/g, '/{bvid}');
|
|
120
|
-
const params = [];
|
|
121
|
-
p.searchParams.forEach((_v, k) => { if (!VOLATILE_PARAMS.has(k))
|
|
122
|
-
params.push(k); });
|
|
123
|
-
return `${p.host}${pathNorm}${params.length ? '?' + params.sort().map(k => `${k}={}`).join('&') : ''}`;
|
|
124
|
-
}
|
|
125
|
-
catch {
|
|
126
|
-
return url;
|
|
127
|
-
}
|
|
128
|
-
}
|
|
129
|
-
function detectAuthIndicators(url, body) {
|
|
130
|
-
const indicators = [];
|
|
131
|
-
// Heuristic: if body contains sign/w_rid fields, it's likely signed
|
|
132
|
-
if (body && typeof body === 'object') {
|
|
133
|
-
const keys = Object.keys(body).map(k => k.toLowerCase());
|
|
134
|
-
if (keys.some(k => k.includes('sign') || k === 'w_rid' || k.includes('token'))) {
|
|
135
|
-
indicators.push('signature');
|
|
136
|
-
}
|
|
137
|
-
}
|
|
138
|
-
// Check URL for common auth patterns
|
|
139
|
-
if (url.includes('/wbi/') || url.includes('w_rid='))
|
|
140
|
-
indicators.push('signature');
|
|
141
|
-
if (url.includes('bearer') || url.includes('access_token'))
|
|
142
|
-
indicators.push('bearer');
|
|
143
|
-
return indicators;
|
|
144
|
-
}
|
|
145
|
-
function findArrayPath(obj, depth = 0) {
|
|
146
|
-
if (depth > 5 || !obj || typeof obj !== 'object')
|
|
147
|
-
return null;
|
|
148
|
-
if (Array.isArray(obj)) {
|
|
149
|
-
if (obj.length >= 2 && obj.some(i => i && typeof i === 'object' && !Array.isArray(i))) {
|
|
150
|
-
return { path: '', items: obj };
|
|
151
|
-
}
|
|
152
|
-
return null;
|
|
153
|
-
}
|
|
154
|
-
let best = null;
|
|
155
|
-
for (const [key, val] of Object.entries(obj)) {
|
|
156
|
-
const found = findArrayPath(val, depth + 1);
|
|
157
|
-
if (found) {
|
|
158
|
-
const fullPath = found.path ? `${key}.${found.path}` : key;
|
|
159
|
-
const candidate = { path: fullPath, items: found.items };
|
|
160
|
-
if (!best || candidate.items.length > best.items.length)
|
|
161
|
-
best = candidate;
|
|
162
|
-
}
|
|
163
|
-
}
|
|
164
|
-
return best;
|
|
165
|
-
}
|
|
166
|
-
function inferCapabilityName(url) {
|
|
167
|
-
const u = url.toLowerCase();
|
|
168
|
-
if (u.includes('hot') || u.includes('popular') || u.includes('ranking') || u.includes('trending'))
|
|
169
|
-
return 'hot';
|
|
170
|
-
if (u.includes('search'))
|
|
171
|
-
return 'search';
|
|
172
|
-
if (u.includes('feed') || u.includes('timeline') || u.includes('dynamic'))
|
|
173
|
-
return 'feed';
|
|
174
|
-
if (u.includes('comment') || u.includes('reply'))
|
|
175
|
-
return 'comments';
|
|
176
|
-
if (u.includes('history'))
|
|
177
|
-
return 'history';
|
|
178
|
-
if (u.includes('profile') || u.includes('me'))
|
|
179
|
-
return 'me';
|
|
180
|
-
if (u.includes('favorite') || u.includes('collect') || u.includes('bookmark'))
|
|
181
|
-
return 'favorite';
|
|
182
|
-
try {
|
|
183
|
-
const segs = new URL(url).pathname
|
|
184
|
-
.split('/')
|
|
185
|
-
.filter(s => s && !s.match(/^\d+$/) && !s.match(/^[0-9a-f]{8,}$/i) && !s.match(/^v\d+$/));
|
|
186
|
-
if (segs.length)
|
|
187
|
-
return segs[segs.length - 1].replace(/[^a-z0-9]/gi, '_').toLowerCase();
|
|
188
|
-
}
|
|
189
|
-
catch { }
|
|
190
|
-
return 'data';
|
|
191
|
-
}
|
|
192
|
-
function inferStrategy(authIndicators) {
|
|
193
|
-
if (authIndicators.includes('signature'))
|
|
194
|
-
return 'intercept';
|
|
195
|
-
if (authIndicators.includes('bearer') || authIndicators.includes('csrf'))
|
|
196
|
-
return 'header';
|
|
197
|
-
return 'cookie';
|
|
198
|
-
}
|
|
199
114
|
function scoreRequest(req, arrayResult) {
|
|
200
115
|
let s = 0;
|
|
201
116
|
if (arrayResult) {
|
|
@@ -247,14 +162,7 @@ function buildRecordedYaml(site, pageUrl, req, capName, arrayResult, authIndicat
|
|
|
247
162
|
? ''
|
|
248
163
|
: itemPath.split('.').map(p => `?.${p}`).join('');
|
|
249
164
|
// Detect search/limit/page params (must be before fetch URL building to use hasSearch/hasPage)
|
|
250
|
-
const
|
|
251
|
-
try {
|
|
252
|
-
new URL(req.url).searchParams.forEach((_v, k) => { if (!VOLATILE_PARAMS.has(k))
|
|
253
|
-
qp.push(k); });
|
|
254
|
-
}
|
|
255
|
-
catch { }
|
|
256
|
-
const hasSearch = qp.some(p => SEARCH_PARAMS.has(p));
|
|
257
|
-
const hasPage = qp.some(p => PAGINATION_PARAMS.has(p));
|
|
165
|
+
const { hasSearch, hasPagination: hasPage } = classifyQueryParams(req.url);
|
|
258
166
|
// Build evaluate script
|
|
259
167
|
const mapLines = Object.entries(detectedFields)
|
|
260
168
|
.map(([role, field]) => ` ${role}: item?.${field}`)
|
|
@@ -363,10 +271,9 @@ export async function recordSession(opts) {
|
|
|
363
271
|
let stopped = false;
|
|
364
272
|
const stop = () => { stopped = true; };
|
|
365
273
|
const { promise: enterPromise, cleanup: cleanupEnter } = waitForEnter();
|
|
366
|
-
|
|
274
|
+
enterPromise.then(stop);
|
|
367
275
|
const timeoutPromise = new Promise(r => setTimeout(() => {
|
|
368
276
|
stop();
|
|
369
|
-
cleanupEnter(); // close readline to prevent process from hanging
|
|
370
277
|
r();
|
|
371
278
|
}, timeoutMs));
|
|
372
279
|
// Poll loop: drain captured data + inject interceptor into any new tabs
|
|
@@ -394,6 +301,7 @@ export async function recordSession(opts) {
|
|
|
394
301
|
}
|
|
395
302
|
}, pollMs);
|
|
396
303
|
await Promise.race([enterPromise, timeoutPromise]);
|
|
304
|
+
cleanupEnter(); // Always clean up readline to prevent process from hanging
|
|
397
305
|
clearInterval(pollInterval);
|
|
398
306
|
// Final drain from all known tabs
|
|
399
307
|
for (const tabId of injectedTabs) {
|
|
@@ -476,7 +384,7 @@ function analyzeAndWrite(site, pageUrl, requests, outDir) {
|
|
|
476
384
|
const scored = [];
|
|
477
385
|
for (const [pattern, req] of seen) {
|
|
478
386
|
const arrayResult = findArrayPath(req.body);
|
|
479
|
-
const authIndicators =
|
|
387
|
+
const authIndicators = detectAuthFromContent(req.url, req.body);
|
|
480
388
|
const score = scoreRequest(req, arrayResult);
|
|
481
389
|
if (score > 0) {
|
|
482
390
|
scored.push({ req, pattern, arrayResult, authIndicators, score });
|
package/dist/registry-api.d.ts
CHANGED
|
@@ -9,3 +9,5 @@
|
|
|
9
9
|
export { cli, Strategy, getRegistry, fullName, registerCommand } from './registry.js';
|
|
10
10
|
export type { CliCommand, Arg, CliOptions } from './registry.js';
|
|
11
11
|
export type { IPage } from './types.js';
|
|
12
|
+
export { onStartup, onBeforeExecute, onAfterExecute } from './hooks.js';
|
|
13
|
+
export type { HookFn, HookContext, HookName } from './hooks.js';
|
package/dist/registry-api.js
CHANGED
package/dist/registry.d.ts
CHANGED
|
@@ -18,6 +18,10 @@ export interface Arg {
|
|
|
18
18
|
help?: string;
|
|
19
19
|
choices?: string[];
|
|
20
20
|
}
|
|
21
|
+
export interface RequiredEnv {
|
|
22
|
+
name: string;
|
|
23
|
+
help?: string;
|
|
24
|
+
}
|
|
21
25
|
export type CommandArgs = Record<string, any>;
|
|
22
26
|
export interface CliCommand {
|
|
23
27
|
site: string;
|
|
@@ -34,6 +38,7 @@ export interface CliCommand {
|
|
|
34
38
|
/** Origin of this command: 'yaml', 'ts', or plugin name. */
|
|
35
39
|
source?: string;
|
|
36
40
|
footerExtra?: (kwargs: CommandArgs) => string | undefined;
|
|
41
|
+
requiredEnv?: RequiredEnv[];
|
|
37
42
|
/**
|
|
38
43
|
* Control pre-navigation for cookie/header context before command execution.
|
|
39
44
|
*
|
|
@@ -65,5 +70,3 @@ export declare function getRegistry(): Map<string, CliCommand>;
|
|
|
65
70
|
export declare function fullName(cmd: CliCommand): string;
|
|
66
71
|
export declare function strategyLabel(cmd: CliCommand): string;
|
|
67
72
|
export declare function registerCommand(cmd: CliCommand): void;
|
|
68
|
-
export { serializeArg, serializeCommand, formatArgSummary, formatRegistryHelpText } from './serialization.js';
|
|
69
|
-
export type { SerializedArg } from './serialization.js';
|
package/dist/registry.js
CHANGED
|
@@ -26,6 +26,7 @@ export function cli(opts) {
|
|
|
26
26
|
pipeline: opts.pipeline,
|
|
27
27
|
timeoutSeconds: opts.timeoutSeconds,
|
|
28
28
|
footerExtra: opts.footerExtra,
|
|
29
|
+
requiredEnv: opts.requiredEnv,
|
|
29
30
|
navigateBefore: opts.navigateBefore,
|
|
30
31
|
};
|
|
31
32
|
const key = fullName(cmd);
|
|
@@ -44,5 +45,3 @@ export function strategyLabel(cmd) {
|
|
|
44
45
|
export function registerCommand(cmd) {
|
|
45
46
|
_registry.set(fullName(cmd), cmd);
|
|
46
47
|
}
|
|
47
|
-
// Re-export serialization helpers from their dedicated module
|
|
48
|
-
export { serializeArg, serializeCommand, formatArgSummary, formatRegistryHelpText } from './serialization.js';
|
package/dist/runtime.d.ts
CHANGED
|
@@ -7,7 +7,6 @@ export declare function getBrowserFactory(): new () => IBrowserFactory;
|
|
|
7
7
|
export declare const DEFAULT_BROWSER_CONNECT_TIMEOUT: number;
|
|
8
8
|
export declare const DEFAULT_BROWSER_COMMAND_TIMEOUT: number;
|
|
9
9
|
export declare const DEFAULT_BROWSER_EXPLORE_TIMEOUT: number;
|
|
10
|
-
export declare const DEFAULT_BROWSER_SMOKE_TIMEOUT: number;
|
|
11
10
|
/**
|
|
12
11
|
* Timeout with seconds unit. Used for high-level command timeouts.
|
|
13
12
|
*/
|
package/dist/runtime.js
CHANGED
|
@@ -7,10 +7,20 @@ import { TimeoutError } from './errors.js';
|
|
|
7
7
|
export function getBrowserFactory() {
|
|
8
8
|
return (process.env.OPENCLI_CDP_ENDPOINT ? CDPBridge : BrowserBridge);
|
|
9
9
|
}
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
10
|
+
function parseEnvTimeout(envVar, fallback) {
|
|
11
|
+
const raw = process.env[envVar];
|
|
12
|
+
if (raw === undefined)
|
|
13
|
+
return fallback;
|
|
14
|
+
const parsed = parseInt(raw, 10);
|
|
15
|
+
if (Number.isNaN(parsed) || parsed <= 0) {
|
|
16
|
+
console.error(`[runtime] Invalid ${envVar}="${raw}", using default ${fallback}s`);
|
|
17
|
+
return fallback;
|
|
18
|
+
}
|
|
19
|
+
return parsed;
|
|
20
|
+
}
|
|
21
|
+
export const DEFAULT_BROWSER_CONNECT_TIMEOUT = parseEnvTimeout('OPENCLI_BROWSER_CONNECT_TIMEOUT', 30);
|
|
22
|
+
export const DEFAULT_BROWSER_COMMAND_TIMEOUT = parseEnvTimeout('OPENCLI_BROWSER_COMMAND_TIMEOUT', 60);
|
|
23
|
+
export const DEFAULT_BROWSER_EXPLORE_TIMEOUT = parseEnvTimeout('OPENCLI_BROWSER_EXPLORE_TIMEOUT', 120);
|
|
14
24
|
/**
|
|
15
25
|
* Timeout with seconds unit. Used for high-level command timeouts.
|
|
16
26
|
*/
|
|
@@ -1,18 +1,11 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Aria snapshot formatter: parses Playwright MCP snapshot text into clean format.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
* 1. Parse & filter: strip annotations, metadata, noise
|
|
6
|
-
* 2. Deduplicate: generic/text
|
|
7
|
-
* 3.
|
|
8
|
-
* 4.
|
|
9
|
-
* 5. Prune: empty containers (iterative bottom-up)
|
|
10
|
-
* 6. Collapse: single-child containers
|
|
4
|
+
* 4-pass pipeline:
|
|
5
|
+
* 1. Parse & filter: strip annotations, metadata, noise, ads, boilerplate subtrees
|
|
6
|
+
* 2. Deduplicate: generic/text parent match, heading+link, nested identical links
|
|
7
|
+
* 3. Prune: empty containers (iterative bottom-up)
|
|
8
|
+
* 4. Collapse: single-child containers
|
|
11
9
|
*/
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
compact?: boolean;
|
|
15
|
-
maxDepth?: number;
|
|
16
|
-
maxTextLength?: number;
|
|
17
|
-
}
|
|
18
|
-
export declare function formatSnapshot(raw: string, opts?: FormatOptions): string;
|
|
10
|
+
import type { SnapshotOptions } from './types.js';
|
|
11
|
+
export declare function formatSnapshot(raw: string, opts?: SnapshotOptions): string;
|
|
@@ -1,13 +1,11 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Aria snapshot formatter: parses Playwright MCP snapshot text into clean format.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
* 1. Parse & filter: strip annotations, metadata, noise
|
|
6
|
-
* 2. Deduplicate: generic/text
|
|
7
|
-
* 3.
|
|
8
|
-
* 4.
|
|
9
|
-
* 5. Prune: empty containers (iterative bottom-up)
|
|
10
|
-
* 6. Collapse: single-child containers
|
|
4
|
+
* 4-pass pipeline:
|
|
5
|
+
* 1. Parse & filter: strip annotations, metadata, noise, ads, boilerplate subtrees
|
|
6
|
+
* 2. Deduplicate: generic/text parent match, heading+link, nested identical links
|
|
7
|
+
* 3. Prune: empty containers (iterative bottom-up)
|
|
8
|
+
* 4. Collapse: single-child containers
|
|
11
9
|
*/
|
|
12
10
|
const DEFAULT_MAX_TEXT_LENGTH = 200;
|
|
13
11
|
// Roles that are pure noise and should always be filtered
|
|
@@ -174,88 +172,64 @@ export function formatSnapshot(raw, opts = {}) {
|
|
|
174
172
|
return '';
|
|
175
173
|
const maxTextLen = opts.maxTextLength ?? DEFAULT_MAX_TEXT_LENGTH;
|
|
176
174
|
const lines = raw.split('\n');
|
|
177
|
-
// === Pass 1: Parse, filter, and collect entries ===
|
|
178
|
-
const
|
|
175
|
+
// === Pass 1: Parse, filter, and collect entries (merged with ad/boilerplate subtree skip) ===
|
|
176
|
+
const parsed = [];
|
|
179
177
|
let refCounter = 0;
|
|
180
|
-
let skipUntilDepth = -1;
|
|
178
|
+
let skipUntilDepth = -1;
|
|
181
179
|
for (let i = 0; i < lines.length; i++) {
|
|
182
180
|
const line = lines[i];
|
|
183
181
|
if (!line.trim())
|
|
184
182
|
continue;
|
|
185
183
|
const indent = line.length - line.trimStart().length;
|
|
186
184
|
const depth = Math.floor(indent / 2);
|
|
187
|
-
//
|
|
185
|
+
// Subtree skip zone (noise roles, ads, boilerplate)
|
|
188
186
|
if (skipUntilDepth >= 0) {
|
|
189
187
|
if (depth > skipUntilDepth)
|
|
190
|
-
continue;
|
|
191
|
-
skipUntilDepth = -1;
|
|
188
|
+
continue;
|
|
189
|
+
skipUntilDepth = -1;
|
|
192
190
|
}
|
|
193
191
|
let content = line.trimStart();
|
|
194
|
-
|
|
195
|
-
if (content.startsWith('- ')) {
|
|
192
|
+
if (content.startsWith('- '))
|
|
196
193
|
content = content.slice(2);
|
|
197
|
-
}
|
|
198
|
-
// Skip metadata lines
|
|
199
194
|
if (isMetadataLine(content))
|
|
200
195
|
continue;
|
|
201
|
-
// Apply maxDepth filter
|
|
202
196
|
if (opts.maxDepth !== undefined && depth > opts.maxDepth)
|
|
203
197
|
continue;
|
|
204
198
|
const { role, text, hasText, trailingText } = parseLine(content);
|
|
205
|
-
// Skip noise nodes
|
|
206
199
|
if (isNoiseNode(role, hasText, text, trailingText))
|
|
207
200
|
continue;
|
|
208
|
-
//
|
|
201
|
+
// Subtree noise roles (contentinfo footer, etc.)
|
|
209
202
|
if (SUBTREE_NOISE_ROLES.has(role)) {
|
|
210
203
|
skipUntilDepth = depth;
|
|
211
204
|
continue;
|
|
212
205
|
}
|
|
213
|
-
//
|
|
206
|
+
// Ads and boilerplate โ skip entire subtree (merged from old Pass 2)
|
|
207
|
+
if (isAdNode(text, trailingText) || isBoilerplateNode(text)) {
|
|
208
|
+
skipUntilDepth = depth;
|
|
209
|
+
continue;
|
|
210
|
+
}
|
|
214
211
|
content = stripAnnotations(content);
|
|
215
|
-
// Check if node should trigger subtree skip (ads, boilerplate)
|
|
216
|
-
const isSubtreeSkip = isAdNode(text, trailingText) || isBoilerplateNode(text);
|
|
217
|
-
// Interactive mode filter
|
|
218
212
|
const isInteractive = INTERACTIVE_ROLES.has(role);
|
|
219
213
|
const isLandmark = LANDMARK_ROLES.has(role);
|
|
220
214
|
if (opts.interactive && !isInteractive && !isLandmark && !hasText)
|
|
221
215
|
continue;
|
|
222
|
-
// Compact mode
|
|
223
216
|
if (opts.compact) {
|
|
224
|
-
content = content
|
|
225
|
-
.replace(/\s*\[.*?\]\s*/g, ' ')
|
|
226
|
-
.replace(/\s+/g, ' ')
|
|
227
|
-
.trim();
|
|
217
|
+
content = content.replace(/\s*\[.*?\]\s*/g, ' ').replace(/\s+/g, ' ').trim();
|
|
228
218
|
}
|
|
229
|
-
// Text truncation
|
|
230
219
|
if (maxTextLen > 0 && content.length > maxTextLen) {
|
|
231
220
|
content = content.slice(0, maxTextLen) + 'โฆ';
|
|
232
221
|
}
|
|
233
|
-
// Assign refs to interactive elements
|
|
234
222
|
if (isInteractive) {
|
|
235
223
|
refCounter++;
|
|
236
224
|
content = `[@${refCounter}] ${content}`;
|
|
237
225
|
}
|
|
238
|
-
|
|
226
|
+
parsed.push({ depth, content, role, text, trailingText, isInteractive, isLandmark });
|
|
239
227
|
}
|
|
240
|
-
// === Pass 2:
|
|
241
|
-
|
|
242
|
-
for (let i = 0; i <
|
|
243
|
-
const entry =
|
|
244
|
-
|
|
245
|
-
const skipDepth = entry.depth;
|
|
246
|
-
i++;
|
|
247
|
-
while (i < entries.length && entries[i].depth > skipDepth) {
|
|
248
|
-
i++;
|
|
249
|
-
}
|
|
250
|
-
i--;
|
|
251
|
-
continue;
|
|
252
|
-
}
|
|
253
|
-
noAds.push(entry);
|
|
254
|
-
}
|
|
255
|
-
// === Pass 3: Deduplicate child generic/text matching parent label ===
|
|
256
|
-
let deduped = [];
|
|
257
|
-
for (let i = 0; i < noAds.length; i++) {
|
|
258
|
-
const entry = noAds[i];
|
|
228
|
+
// === Pass 2: Deduplicate (merged: generic/text parent match + heading+link + nested links) ===
|
|
229
|
+
const deduped = [];
|
|
230
|
+
for (let i = 0; i < parsed.length; i++) {
|
|
231
|
+
const entry = parsed[i];
|
|
232
|
+
// Dedup: generic/text child matching parent label
|
|
259
233
|
if (entry.role === 'generic' || entry.role === 'text') {
|
|
260
234
|
let parent;
|
|
261
235
|
for (let j = deduped.length - 1; j >= 0; j--) {
|
|
@@ -268,43 +242,30 @@ export function formatSnapshot(raw, opts = {}) {
|
|
|
268
242
|
}
|
|
269
243
|
if (parent) {
|
|
270
244
|
const childText = entry.trailingText || entry.text;
|
|
271
|
-
if (childText && parent.text && childText === parent.text)
|
|
245
|
+
if (childText && parent.text && childText === parent.text)
|
|
272
246
|
continue;
|
|
273
|
-
}
|
|
274
247
|
}
|
|
275
248
|
}
|
|
276
|
-
|
|
277
|
-
}
|
|
278
|
-
// === Pass 4: Deduplicate heading + child link with identical label ===
|
|
279
|
-
// Pattern: heading "Title": โ link "Title": (same text) โ skip the link
|
|
280
|
-
const deduped2 = [];
|
|
281
|
-
for (let i = 0; i < deduped.length; i++) {
|
|
282
|
-
const entry = deduped[i];
|
|
249
|
+
// Dedup: heading + child link with identical label
|
|
283
250
|
if (entry.role === 'heading' && entry.text) {
|
|
284
|
-
const next =
|
|
251
|
+
const next = parsed[i + 1];
|
|
285
252
|
if (next && next.role === 'link' && next.text === entry.text && next.depth === entry.depth + 1) {
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
i++; // skip the link
|
|
253
|
+
deduped.push(entry);
|
|
254
|
+
i++; // skip the link, preserve its children
|
|
289
255
|
continue;
|
|
290
256
|
}
|
|
291
257
|
}
|
|
292
|
-
|
|
293
|
-
}
|
|
294
|
-
// === Pass 5: Deduplicate nested identical links ===
|
|
295
|
-
const deduped3 = [];
|
|
296
|
-
for (let i = 0; i < deduped2.length; i++) {
|
|
297
|
-
const entry = deduped2[i];
|
|
258
|
+
// Dedup: nested identical links (skip parent, keep child)
|
|
298
259
|
if (entry.role === 'link' && entry.text) {
|
|
299
|
-
const next =
|
|
260
|
+
const next = parsed[i + 1];
|
|
300
261
|
if (next && next.role === 'link' && next.text === entry.text && next.depth === entry.depth + 1) {
|
|
301
|
-
continue;
|
|
262
|
+
continue;
|
|
302
263
|
}
|
|
303
264
|
}
|
|
304
|
-
|
|
265
|
+
deduped.push(entry);
|
|
305
266
|
}
|
|
306
|
-
// === Pass
|
|
307
|
-
let current =
|
|
267
|
+
// === Pass 3: Iteratively prune empty containers (bottom-up) ===
|
|
268
|
+
let current = deduped;
|
|
308
269
|
let changed = true;
|
|
309
270
|
while (changed) {
|
|
310
271
|
changed = false;
|
|
@@ -330,7 +291,7 @@ export function formatSnapshot(raw, opts = {}) {
|
|
|
330
291
|
}
|
|
331
292
|
current = next;
|
|
332
293
|
}
|
|
333
|
-
// === Pass
|
|
294
|
+
// === Pass 4: Collapse single-child containers ===
|
|
334
295
|
const collapsed = [];
|
|
335
296
|
for (let i = 0; i < current.length; i++) {
|
|
336
297
|
const entry = current[i];
|
|
@@ -358,10 +319,9 @@ export function formatSnapshot(raw, opts = {}) {
|
|
|
358
319
|
}
|
|
359
320
|
}
|
|
360
321
|
if (!hasGrandchildren) {
|
|
361
|
-
const mergedContent = entry.content.replace(/:$/, '') + ' > ' + child.content;
|
|
362
322
|
collapsed.push({
|
|
363
323
|
...entry,
|
|
364
|
-
content:
|
|
324
|
+
content: entry.content.replace(/:$/, '') + ' > ' + child.content,
|
|
365
325
|
role: child.role,
|
|
366
326
|
text: child.text,
|
|
367
327
|
trailingText: child.trailingText,
|
package/dist/utils.d.ts
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared utility functions used across the codebase.
|
|
3
|
+
*/
|
|
4
|
+
/** Type guard: checks if a value is a non-null, non-array object. */
|
|
5
|
+
export declare function isRecord(value: unknown): value is Record<string, unknown>;
|
|
6
|
+
/** Simple async concurrency limiter. */
|
|
7
|
+
export declare function mapConcurrent<T, R>(items: T[], limit: number, fn: (item: T, index: number) => Promise<R>): Promise<R[]>;
|
|
8
|
+
/** Save a base64-encoded string to a file, creating parent directories as needed. */
|
|
9
|
+
export declare function saveBase64ToFile(base64: string, filePath: string): Promise<void>;
|
package/dist/utils.js
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared utility functions used across the codebase.
|
|
3
|
+
*/
|
|
4
|
+
import * as fs from 'node:fs';
|
|
5
|
+
import * as path from 'node:path';
|
|
6
|
+
/** Type guard: checks if a value is a non-null, non-array object. */
|
|
7
|
+
export function isRecord(value) {
|
|
8
|
+
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
|
9
|
+
}
|
|
10
|
+
/** Simple async concurrency limiter. */
|
|
11
|
+
export async function mapConcurrent(items, limit, fn) {
|
|
12
|
+
const results = new Array(items.length);
|
|
13
|
+
let index = 0;
|
|
14
|
+
async function worker() {
|
|
15
|
+
while (index < items.length) {
|
|
16
|
+
const i = index++;
|
|
17
|
+
results[i] = await fn(items[i], i);
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
const workers = Array.from({ length: Math.min(limit, items.length) }, () => worker());
|
|
21
|
+
await Promise.all(workers);
|
|
22
|
+
return results;
|
|
23
|
+
}
|
|
24
|
+
/** Save a base64-encoded string to a file, creating parent directories as needed. */
|
|
25
|
+
export async function saveBase64ToFile(base64, filePath) {
|
|
26
|
+
const dir = path.dirname(filePath);
|
|
27
|
+
await fs.promises.mkdir(dir, { recursive: true });
|
|
28
|
+
await fs.promises.writeFile(filePath, Buffer.from(base64, 'base64'));
|
|
29
|
+
}
|
package/dist/validate.js
CHANGED
|
@@ -5,14 +5,12 @@ import yaml from 'js-yaml';
|
|
|
5
5
|
import { getErrorMessage } from './errors.js';
|
|
6
6
|
/** All recognized pipeline step names */
|
|
7
7
|
const KNOWN_STEP_NAMES = new Set([
|
|
8
|
-
'navigate', 'click', 'type', 'wait', 'press', 'snapshot',
|
|
8
|
+
'navigate', 'click', 'type', 'wait', 'press', 'snapshot',
|
|
9
9
|
'fetch', 'evaluate',
|
|
10
10
|
'select', 'map', 'filter', 'sort', 'limit',
|
|
11
|
-
'intercept', 'tap',
|
|
11
|
+
'intercept', 'tap', 'download',
|
|
12
12
|
]);
|
|
13
|
-
|
|
14
|
-
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
|
15
|
-
}
|
|
13
|
+
import { isRecord } from './utils.js';
|
|
16
14
|
export function validateClisWithTarget(dirs, target) {
|
|
17
15
|
const results = [];
|
|
18
16
|
let errors = 0;
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared YAML CLI definition types.
|
|
3
|
+
* Used by both discovery.ts (runtime) and build-manifest.ts (build-time).
|
|
4
|
+
*/
|
|
5
|
+
export interface YamlArgDefinition {
|
|
6
|
+
type?: string;
|
|
7
|
+
default?: unknown;
|
|
8
|
+
required?: boolean;
|
|
9
|
+
positional?: boolean;
|
|
10
|
+
description?: string;
|
|
11
|
+
help?: string;
|
|
12
|
+
choices?: string[];
|
|
13
|
+
}
|
|
14
|
+
export interface YamlCliDefinition {
|
|
15
|
+
site?: string;
|
|
16
|
+
name?: string;
|
|
17
|
+
description?: string;
|
|
18
|
+
domain?: string;
|
|
19
|
+
strategy?: string;
|
|
20
|
+
browser?: boolean;
|
|
21
|
+
args?: Record<string, YamlArgDefinition>;
|
|
22
|
+
columns?: string[];
|
|
23
|
+
pipeline?: Record<string, unknown>[];
|
|
24
|
+
timeout?: number;
|
|
25
|
+
navigateBefore?: boolean | string;
|
|
26
|
+
}
|
|
@@ -29,6 +29,7 @@ export default defineConfig({
|
|
|
29
29
|
items: [
|
|
30
30
|
{ text: 'Getting Started', link: '/guide/getting-started' },
|
|
31
31
|
{ text: 'Installation', link: '/guide/installation' },
|
|
32
|
+
{ text: 'Comparison', link: '/comparison' },
|
|
32
33
|
{ text: 'Browser Bridge', link: '/guide/browser-bridge' },
|
|
33
34
|
{ text: 'Troubleshooting', link: '/guide/troubleshooting' },
|
|
34
35
|
{ text: 'Plugins', link: '/guide/plugins' },
|
|
@@ -72,6 +73,7 @@ export default defineConfig({
|
|
|
72
73
|
{ text: 'Douban', link: '/adapters/browser/douban' },
|
|
73
74
|
{ text: 'Sina Blog', link: '/adapters/browser/sinablog' },
|
|
74
75
|
{ text: 'Substack', link: '/adapters/browser/substack' },
|
|
76
|
+
{ text: 'Pixiv', link: '/adapters/browser/pixiv' },
|
|
75
77
|
],
|
|
76
78
|
},
|
|
77
79
|
{
|
|
@@ -80,6 +82,7 @@ export default defineConfig({
|
|
|
80
82
|
items: [
|
|
81
83
|
{ text: 'HackerNews', link: '/adapters/browser/hackernews' },
|
|
82
84
|
{ text: 'Dev.to', link: '/adapters/browser/devto' },
|
|
85
|
+
{ text: 'Dictionary', link: '/adapters/browser/dictionary' },
|
|
83
86
|
{ text: 'BBC', link: '/adapters/browser/bbc' },
|
|
84
87
|
{ text: 'Apple Podcasts', link: '/adapters/browser/apple-podcasts' },
|
|
85
88
|
{ text: 'Xiaoyuzhou', link: '/adapters/browser/xiaoyuzhou' },
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Dictionary
|
|
2
|
+
|
|
3
|
+
**Mode**: ๐ Public ยท **Domain**: `api.dictionaryapi.dev`
|
|
4
|
+
|
|
5
|
+
Search the open dictionary to quickly fetch native definitions, part of speech contexts, and phonetic pronunciations directly in your IDE terminal.
|
|
6
|
+
|
|
7
|
+
## Commands
|
|
8
|
+
|
|
9
|
+
| Command | Description |
|
|
10
|
+
|---------|-------------|
|
|
11
|
+
| `opencli dictionary search` | Fetch the exact definition of a word |
|
|
12
|
+
| `opencli dictionary synonyms` | Find related synonyms for a word |
|
|
13
|
+
| `opencli dictionary examples` | Read real-world sentence usage examples |
|
|
14
|
+
|
|
15
|
+
## Usage Examples
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
# Look up a complex term
|
|
19
|
+
opencli dictionary search serendipity
|
|
20
|
+
|
|
21
|
+
# Discover phonetics
|
|
22
|
+
opencli dictionary search ephemeral
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Prerequisites
|
|
26
|
+
|
|
27
|
+
- No browser required โ utilizes the fast, open JSON definitions API.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# JD.com
|
|
2
|
+
|
|
3
|
+
**Mode**: ๐ Browser ยท **Domain**: `item.jd.com`
|
|
4
|
+
|
|
5
|
+
## Commands
|
|
6
|
+
|
|
7
|
+
| Command | Description |
|
|
8
|
+
|---------|-------------|
|
|
9
|
+
| `opencli jd item <sku>` | Fetch product details (price, images, specs) |
|
|
10
|
+
|
|
11
|
+
## Usage Examples
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
# Get product details by SKU
|
|
15
|
+
opencli jd item 100291143898
|
|
16
|
+
|
|
17
|
+
# Limit detail images
|
|
18
|
+
opencli jd item 100291143898 --images 5
|
|
19
|
+
|
|
20
|
+
# JSON output
|
|
21
|
+
opencli jd item 100291143898 -f json
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Prerequisites
|
|
25
|
+
|
|
26
|
+
- Chrome running and **logged into** jd.com
|
|
27
|
+
- [Browser Bridge extension](/guide/browser-bridge) installed
|