@jackwener/opencli 1.0.3 → 1.0.5

package/docs/developer/contributing.md

@@ -0,0 +1,136 @@
+# Contributing
+
+Thanks for your interest in contributing to OpenCLI.
+
+## Quick Start
+
+```bash
+# 1. Fork & clone
+git clone git@github.com:<your-username>/opencli.git
+cd opencli
+
+# 2. Install dependencies
+npm install
+
+# 3. Build
+npm run build
+
+# 4. Run a few checks
+npx tsc --noEmit
+npx vitest run src/
+
+# 5. Link globally (optional, for testing `opencli` command)
+npm link
+```
+
+## Adding a New Site Adapter
+
+This is the most common type of contribution. Start with YAML when possible, and use TypeScript only when you need browser-side logic or multi-step flows.
+
+### YAML Adapter (Recommended for data-fetching commands)
+
+Create a file like `src/clis/<site>/<command>.yaml`:
+
+::: v-pre
+```yaml
+site: mysite
+name: trending
+description: Trending posts on MySite
+domain: www.mysite.com
+strategy: public      # public | cookie | header
+browser: false        # true if browser session is needed
+
+args:
+  limit:
+    type: int
+    default: 20
+    description: Number of items
+
+pipeline:
+  - fetch:
+      url: https://api.mysite.com/trending
+
+  - map:
+      rank: ${{ index + 1 }}
+      title: ${{ item.title }}
+      score: ${{ item.score }}
+      url: ${{ item.url }}
+
+  - limit: ${{ args.limit }}
+
+columns: [rank, title, score, url]
+```
+:::
+
+See [`hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml) for a real example.
+
+### TypeScript Adapter (For complex browser interactions)
+
+Create a file like `src/clis/<site>/<command>.ts`:
+
+```typescript
+import { cli, Strategy } from '../../registry.js';
+
+cli({
+  site: 'mysite',
+  name: 'search',
+  description: 'Search MySite',
+  domain: 'www.mysite.com',
+  strategy: Strategy.COOKIE,
+  args: [
+    { name: 'query', required: true, help: 'Search query' },
+    { name: 'limit', type: 'int', default: 10, help: 'Max results' },
+  ],
+  columns: ['title', 'url', 'date'],
+
+  func: async (page, kwargs) => {
+    const { query, limit = 10 } = kwargs;
+    // ... browser automation logic
+    return data.slice(0, Number(limit)).map((item: any) => ({
+      title: item.title,
+      url: item.url,
+      date: item.created_at,
+    }));
+  },
+});
+```
+
+### Validate Your Adapter
+
+```bash
+opencli validate               # Validate YAML syntax and schema
+opencli <site> <command> --limit 3 -f json   # Test your command
+opencli <site> <command> -v    # Verbose mode for debugging
+```
+
+## Code Style
+
+- **TypeScript strict mode** — avoid `any` where possible.
+- **ES Modules** — use `.js` extensions in imports (TypeScript output).
+- **Naming**: `kebab-case` for files, `camelCase` for variables/functions, `PascalCase` for types/classes.
+- **No default exports** — use named exports.
+
+## Commit Convention
+
+We use [Conventional Commits](https://www.conventionalcommits.org/):
+
+```
+feat(twitter): add thread command
+fix(browser): handle CDP timeout gracefully
+docs: update CONTRIBUTING.md
+test(reddit): add e2e test for save command
+chore: bump vitest to v4
+```
+
+## Submitting a Pull Request
+
+1. Create a feature branch: `git checkout -b feat/mysite-trending`
+2. Make your changes and add tests when relevant
+3. Run the checks:
+   ```bash
+   npx tsc --noEmit           # Type check
+   npx vitest run src/        # Unit tests
+   opencli validate           # YAML validation (if applicable)
+   ```
+4. Commit using conventional commit format
+5. Push and open a PR

package/docs/developer/testing.md

@@ -0,0 +1,237 @@
+# Testing Guide
+
+> 面向开发者和 AI Agent 的测试参考手册。
+
+## 目录
+
+- [测试架构](#测试架构)
+- [当前覆盖范围](#当前覆盖范围)
+- [本地运行测试](#本地运行测试)
+- [如何添加新测试](#如何添加新测试)
+- [CI/CD 流水线](#cicd-流水线)
+- [浏览器模式](#浏览器模式)
+- [站点兼容性](#站点兼容性)
+
+---
+
+## 测试架构
+
+测试分为三层，全部使用 **vitest** 运行：
+
+```
+tests/
+├── e2e/                           # E2E 集成测试（子进程运行真实 CLI）
+│   ├── helpers.ts                 # runCli() 共享工具
+│   ├── public-commands.test.ts    # 公开 API 命令（无需浏览器）
+│   ├── browser-public.test.ts     # 浏览器命令（公开数据）
+│   ├── browser-auth.test.ts       # 需登录命令（graceful failure 测试）
+│   ├── management.test.ts         # 管理命令（list, validate, verify, help）
+│   └── output-formats.test.ts     # 输出格式（json/yaml/csv/md）
+├── smoke/                         # 烟雾测试（仅定时 / 手动触发）
+│   └── api-health.test.ts         # 外部 API 可用性检测
+src/
+├── *.test.ts                      # 单元测试（已有 8 个）
+```
+
+| 层 | 位置 | 运行方式 | 用途 |
+|---|---|---|---|
+| 单元测试 | `src/**/*.test.ts` | `npx vitest run src/` | 内部模块逻辑 |
+| E2E 测试 | `tests/e2e/*.test.ts` | `npx vitest run tests/e2e/` | 真实 CLI 命令执行 |
+| 烟雾测试 | `tests/smoke/*.test.ts` | `npx vitest run tests/smoke/` | 外部 API 健康 |
+
+---
+
+## 当前覆盖范围
+
+### 单元测试（8 个文件）
+
+| 文件 | 覆盖内容 |
+|---|---|
+| `browser.test.ts` | JSON-RPC、tab 管理、extension/standalone 模式切换 |
+| `engine.test.ts` | 命令发现与执行 |
+| `registry.test.ts` | 命令注册与策略分配 |
+| `output.test.ts` | 输出格式渲染 |
+| `doctor.test.ts` | Token 诊断 |
+| `coupang.test.ts` | 数据归一化 |
+| `pipeline/template.test.ts` | 模板表达式求值 |
+| `pipeline/transform.test.ts` | 数据变换步骤 |
+
+### E2E 测试（~52 个用例）
+
+| 文件 | 覆盖站点/功能 | 测试数 |
+|---|---|---|
+| `public-commands.test.ts` | hackernews/top, v2ex/hot, v2ex/latest, v2ex/topic | 5 |
+| `browser-public.test.ts` | bbc, bilibili×3, weibo, zhihu×2, reddit×2, twitter, xueqiu×2, reuters, youtube, smzdm, boss, ctrip, coupang, xiaohongshu, yahoo-finance, v2ex/daily | 21 |
+| `browser-auth.test.ts` | bilibili/me,dynamic,favorite,history,following + twitter/bookmarks,timeline,notifications + v2ex/me,notifications + xueqiu/feed,watchlist + xiaohongshu/feed,notifications | 14 |
+| `management.test.ts` | list×5 格式, validate×3 级别, verify, --version, --help, unknown cmd | 12 |
+| `output-formats.test.ts` | json, yaml, csv, md 格式验证 | 5 |
+
+### 烟雾测试
+
+公开 API 可用性（hackernews, v2ex×2, v2ex/topic）+ 全站点注册完整性检查。
+
+---
+
+## 本地运行测试
+
+### 前置条件
+
+```bash
+npm ci                # 安装依赖
+npm run build         # 编译（E2E 测试需要 dist/main.js）
+```
+
+### 运行命令
+
+```bash
+# 全部单元测试
+npx vitest run src/
+
+# 全部 E2E 测试（会真实调用外部 API）
+npx vitest run tests/e2e/
+
+# 单个测试文件
+npx vitest run tests/e2e/management.test.ts
+
+# 全部测试（单元 + E2E）
+npx vitest run
+
+# 烟雾测试
+npx vitest run tests/smoke/
+
+# watch 模式（开发时推荐）
+npx vitest src/
+```
+
+### 浏览器命令本地测试须知
+
+- opencli 通过 Browser Bridge 扩展连接已运行的 Chrome 浏览器
+- `browser-public.test.ts` 使用 `tryBrowserCommand()`，站点反爬导致空数据时 warn + pass
+- `browser-auth.test.ts` 验证 **graceful failure**（不 crash 不 hang 即通过）
+- 如需测试完整登录态，保持 Chrome 登录态并安装 Browser Bridge 扩展，手动跑对应测试
+
+---
+
+## 如何添加新测试
+
+### 新增 YAML Adapter（如 `src/clis/producthunt/trending.yaml`）
+
+1. **无需额外操作**：`validate` 测试会自动覆盖 YAML 结构验证
+2. 根据 adapter 类型，在对应文件加一个 `it()` block：
+
+```typescript
+// 如果 browser: false（公开 API）→ tests/e2e/public-commands.test.ts
+it('producthunt trending returns data', async () => {
+  const { stdout, code } = await runCli(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
+  expect(code).toBe(0);
+  const data = parseJsonOutput(stdout);
+  expect(Array.isArray(data)).toBe(true);
+  expect(data.length).toBeGreaterThanOrEqual(1);
+  expect(data[0]).toHaveProperty('title');
+}, 30_000);
+```
+
+```typescript
+// 如果 browser: true 但可公开访问 → tests/e2e/browser-public.test.ts
+it('producthunt trending returns data', async () => {
+  const data = await tryBrowserCommand(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
+  expectDataOrSkip(data, 'producthunt trending');
+}, 60_000);
+```
+
+```typescript
+// 如果 browser: true 且需登录 → tests/e2e/browser-auth.test.ts
+it('producthunt me fails gracefully without login', async () => {
+  await expectGracefulAuthFailure(['producthunt', 'me', '-f', 'json'], 'producthunt me');
+}, 60_000);
+```
+
+### 新增管理命令（如 `opencli export`）
+
+在 `tests/e2e/management.test.ts` 添加测试。
+
+### 新增内部模块
+
+在 `src/` 下对应位置创建 `*.test.ts`。
+
+### 决策流程图
+
+```
+新增功能 → 是内部模块？ → 是 → src/ 下加 *.test.ts
+                ↓ 否
+         是 CLI 命令？ → browser: false? → tests/e2e/public-commands.test.ts
+                              ↓ true
+                        公开数据？ → tests/e2e/browser-public.test.ts
+                              ↓ 需登录
+                        tests/e2e/browser-auth.test.ts
+```
+
+---
+
+## CI/CD 流水线
+
+### ci.yml（主流水线）
+
+| Job | 触发条件 | 内容 |
+|---|---|---|
+| **build** | push/PR to main,dev | typecheck + build |
+| **unit-test** | push/PR to main,dev | 单元测试，2 shard 并行 |
+| **smoke-test** | 每周一 08:00 UTC / 手动 | xvfb + real Chrome，外部 API 健康检查 |
+
+### e2e-headed.yml（E2E 测试）
+
+| Job | 触发条件 | 内容 |
+|---|---|---|
+| **e2e-headed** | push/PR to main,dev | xvfb + real Chrome，全部 E2E 测试 |
+
+E2E 使用 `browser-actions/setup-chrome` 安装真实 Chrome，配合 `xvfb-run` 提供虚拟显示器，以 headed 模式运行浏览器。
+
+### Sharding
+
+单元测试使用 vitest 内置 shard：
+
+::: v-pre
+```yaml
+strategy:
+  matrix:
+    shard: [1, 2]
+steps:
+  - run: npx vitest run src/ --shard=${{ matrix.shard }}/2
+```
+:::
+
+---
+
+## 浏览器模式
+
+opencli 通过 Browser Bridge 扩展连接浏览器：
+
+| 条件 | 模式 | 使用场景 |
+|---|---|---|
+| 扩展已安装 | Extension 模式 | 本地用户，连接已登录的 Chrome |
+| 扩展未安装 | CLI 报错提示安装 | 需要安装 Browser Bridge 扩展 |
+
+CI 中使用 `OPENCLI_BROWSER_EXECUTABLE_PATH` 指定真实 Chrome 路径：
+
+::: v-pre
+```yaml
+env:
+  OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
+```
+:::
+
+---
+
+## 站点兼容性
+
+在 GitHub Actions 美国 runner 上，部分站点因地域限制或登录要求返回空数据。E2E 测试对这些站点使用 warn + pass 策略，不影响 CI 绿灯。
+
+| 站点 | CI 状态 | 限制原因 |
+|---|---|---|
+| hackernews, bbc, v2ex | ✅ 返回数据 | 无限制 |
+| yahoo-finance | ✅ 返回数据 | 无限制 |
+| bilibili, zhihu, weibo, xiaohongshu | ⚠️ 空数据 | 地域限制（中国站点） |
+| reddit, twitter, youtube | ⚠️ 空数据 | 需登录或 cookie |
+| smzdm, boss, ctrip, coupang, xueqiu | ⚠️ 空数据 | 地域限制 / 需登录 |
+
+> 使用 self-hosted runner（国内服务器）可解决地域限制问题。

package/docs/developer/ts-adapter.md

@@ -0,0 +1,87 @@
+# TypeScript Adapter Guide
+
+Use TypeScript adapters when you need browser-side logic, multi-step flows, DOM manipulation, or complex data extraction that goes beyond simple API fetching.
+
+## Basic Structure
+
+```typescript
+import { cli, Strategy } from '../../registry.js';
+
+cli({
+  site: 'mysite',
+  name: 'search',
+  description: 'Search MySite',
+  domain: 'www.mysite.com',
+  strategy: Strategy.COOKIE,      // PUBLIC | COOKIE | HEADER
+  args: [
+    { name: 'query', required: true, help: 'Search query' },
+    { name: 'limit', type: 'int', default: 10, help: 'Max results' },
+  ],
+  columns: ['title', 'url', 'date'],
+
+  func: async (page, kwargs) => {
+    const { query, limit = 10 } = kwargs;
+
+    // Navigate and extract data
+    await page.goto('https://www.mysite.com');
+
+    const data = await page.evaluate(`
+      (async () => {
+        const res = await fetch('/api/search?q=${encodeURIComponent(String(query))}', {
+          credentials: 'include'
+        });
+        return (await res.json()).results;
+      })()
+    `);
+
+    return data.slice(0, Number(limit)).map((item: any) => ({
+      title: item.title,
+      url: item.url,
+      date: item.created_at,
+    }));
+  },
+});
+```
+
+## Strategy Types
+
+| Strategy | Constant | Use Case |
+|----------|----------|----------|
+| Public | `Strategy.PUBLIC` | No auth needed |
+| Cookie | `Strategy.COOKIE` | Browser session cookies |
+| Header | `Strategy.HEADER` | Custom headers/tokens |
+
+## The `page` Object
+
+The `page` parameter provides browser interaction methods:
+
+- `page.goto(url)` — Navigate to a URL
+- `page.evaluate(script)` — Execute JavaScript in the page context
+- `page.waitForSelector(selector)` — Wait for an element
+- `page.click(selector)` — Click an element
+- `page.type(selector, text)` — Type text into an input
+
+## The `kwargs` Object
+
+Contains parsed CLI arguments as key-value pairs. Always destructure with defaults:
+
+```typescript
+const { query, limit = 10, format = 'json' } = kwargs;
+```
+
+## AI-Assisted Development
+
+Use the AI workflow tools to accelerate adapter creation:
+
+```bash
+# Discover APIs and page structure
+opencli explore https://example.com --site mysite
+
+# Auto-generate adapter from explore artifacts
+opencli synthesize mysite
+
+# One-shot: explore → synthesize → register
+opencli generate https://example.com --goal "trending"
+```
+
+See [AI Workflow](/developer/ai-workflow) for the complete guide.

package/docs/developer/yaml-adapter.md

@@ -0,0 +1,108 @@
+# YAML Adapter Guide
+
+YAML adapters are the recommended way to add new commands when the site offers a straightforward API. They use a declarative pipeline approach — no TypeScript required.
+
+## Basic Structure
+
+::: v-pre
+```yaml
+site: mysite          # Site identifier
+name: trending        # Command name (opencli mysite trending)
+description: ...      # Help text
+domain: www.mysite.com
+strategy: public      # public | cookie | header
+browser: false        # true if browser session is needed
+
+args:                 # CLI arguments
+  limit:
+    type: int
+    default: 20
+    description: Number of items
+
+pipeline:             # Data processing steps
+  - fetch:
+      url: https://api.mysite.com/trending
+
+  - map:
+      rank: ${{ index + 1 }}
+      title: ${{ item.title }}
+
+  - limit: ${{ args.limit }}
+
+columns: [rank, title, score, url]
+```
+:::
+
+## Pipeline Steps
+
+### `fetch`
+Fetch data from a URL. Supports template expressions for dynamic URLs.
+
+::: v-pre
+```yaml
+- fetch:
+    url: https://api.example.com/search?q=${{ args.query }}
+    headers:
+      Accept: application/json
+```
+:::
+
+### `map`
+
+::: v-pre
+Transform each item in the result array. Use `${{ item.xxx }}` for field access and `${{ index }}` for position.
+
+```yaml
+- map:
+    rank: ${{ index + 1 }}
+    title: ${{ item.title }}
+    url: https://example.com${{ item.path }}
+```
+:::
+
+### `limit`
+Truncate results to N items.
+
+::: v-pre
+```yaml
+- limit: ${{ args.limit }}
+```
+:::
+
+### `filter`
+Filter items by condition.
+
+::: v-pre
+```yaml
+- filter: ${{ item.score > 100 }}
+```
+:::
+
+### `download`
+Download media files.
+
+::: v-pre
+```yaml
+- download:
+    url: ${{ item.imageUrl }}
+    dir: ./downloads
+    filename: ${{ item.title | sanitize }}.jpg
+```
+:::
+
+## Template Expressions
+
+::: v-pre
+Use `${{ ... }}` for dynamic values:
+
+| Expression | Description |
+|-----------|-------------|
+| `${{ args.limit }}` | CLI argument |
+| `${{ item.title }}` | Current item field |
+| `${{ index }}` | Current index (0-based) |
+| `${{ item.x \| sanitize }}` | Pipe filters |
+:::
+
+## Real Example
+
+See [`src/clis/hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml).

package/docs/guide/browser-bridge.md

@@ -0,0 +1,38 @@
+# Browser Bridge Setup
+
+> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands.
+
+OpenCLI connects to your browser through a lightweight **Browser Bridge** Chrome Extension + micro-daemon (zero config, auto-start).
+
+## Extension Installation
+
+### Method 1: Download Pre-built Release (Recommended)
+
+1. Go to the GitHub [Releases page](https://github.com/jackwener/opencli/releases) and download the latest `opencli-extension.zip` or `opencli-extension.crx`.
+2. Open `chrome://extensions` and enable **Developer mode** (top-right toggle).
+3. Drag and drop the `.crx` file or the unzipped folder into the extensions page.
+
+### Method 2: Load Unpacked Source (For Developers)
+
+1. Open `chrome://extensions` and enable **Developer mode**.
+2. Click **Load unpacked** and select the `extension/` directory from the repository.
+
+## Verification
+
+That's it! The daemon auto-starts when you run any browser command. No tokens, no manual configuration.
+
+```bash
+opencli doctor            # Check extension + daemon connectivity
+opencli doctor --live     # Also test live browser commands
+```
+
+## How It Works
+
+```
+┌─────────────┐     WebSocket      ┌──────────────┐     Chrome API     ┌─────────┐
+│  opencli    │ ◄──────────────► │  micro-daemon │ ◄──────────────► │  Chrome  │
+│  (Node.js)  │    localhost:19825  │  (auto-start) │    Extension       │ Browser  │
+└─────────────┘                    └──────────────┘                    └─────────┘
+```
+
+The daemon manages the WebSocket connection between your CLI commands and the Chrome extension. The extension executes JavaScript in the context of web pages, with access to the logged-in session.

package/docs/guide/getting-started.md

@@ -0,0 +1,56 @@
+# Getting Started
+
+> **Make any website or Electron App your CLI.**
+> Zero risk · Reuse Chrome login · AI-powered discovery · Browser + Desktop automation
+
+[![npm](https://img.shields.io/npm/v/@jackwener/opencli?style=flat-square)](https://www.npmjs.com/package/@jackwener/opencli)
+[![Node.js Version](https://img.shields.io/node/v/@jackwener/opencli?style=flat-square)](https://nodejs.org)
+[![License](https://img.shields.io/npm/l/@jackwener/opencli?style=flat-square)](https://github.com/jackwener/opencli/blob/main/LICENSE)
+
+OpenCLI turns **any website** or **Electron app** into a command-line interface — Bilibili, Zhihu, 小红书, Twitter/X, Reddit, YouTube, Antigravity, and [many more](/adapters/) — powered by browser session reuse and AI-native discovery.
+
+## Highlights
+
+- **CLI All Electron** — CLI-ify apps like Antigravity Ultra! Now AI can control itself natively.
+- **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser.
+- **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies.
+- **Self-healing setup** — `opencli setup` verifies Browser Bridge connectivity; `opencli doctor` diagnoses daemon, extension, and live browser connectivity.
+- **Dynamic Loader** — Simply drop `.ts` or `.yaml` adapters into the `clis/` folder for auto-registration.
+- **Dual-Engine Architecture** — Supports both YAML declarative data pipelines and robust browser runtime TypeScript injections.
+
+## Quick Start
+
+### Install via npm
+
+```bash
+npm install -g @jackwener/opencli
+```
+
+### Basic Usage
+
+```bash
+opencli list                              # See all commands
+opencli hackernews top --limit 5          # Public API, no browser
+opencli bilibili hot --limit 5            # Browser command
+opencli zhihu hot -f json                 # JSON output
+```
+
+### Output Formats
+
+All built-in commands support `--format` / `-f`:
+
+```bash
+opencli bilibili hot -f table   # Default: rich terminal table
+opencli bilibili hot -f json    # JSON (pipe to jq or LLMs)
+opencli bilibili hot -f yaml    # YAML (human-readable)
+opencli bilibili hot -f md      # Markdown
+opencli bilibili hot -f csv     # CSV
+opencli bilibili hot -v         # Verbose: show pipeline debug
+```
+
+## Next Steps
+
+- [Installation details](/guide/installation)
+- [Browser Bridge setup](/guide/browser-bridge)
+- [All available adapters](/adapters/)
+- [For developers / AI agents](/developer/contributing)

package/docs/guide/installation.md

@@ -0,0 +1,37 @@
+# Installation
+
+## Requirements
+
+- **Node.js**: >= 20.0.0
+- **Chrome** running and logged into the target site (for browser commands)
+
+## Install via npm (Recommended)
+
+```bash
+npm install -g @jackwener/opencli
+```
+
+## Install from Source
+
+```bash
+git clone git@github.com:jackwener/opencli.git
+cd opencli
+npm install
+npm run build
+npm link      # Link binary globally
+opencli list  # Now you can use it anywhere!
+```
+
+## Update
+
+```bash
+npm install -g @jackwener/opencli@latest
+```
+
+## Verify Installation
+
+```bash
+opencli --version   # Check version
+opencli list        # List all commands
+opencli doctor      # Diagnose connectivity
+```