dp-cli 0.1.0__tar.gz

dp_cli-0.1.0/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: dp-cli
+Version: 0.1.0
+Summary: A powerful CLI for DrissionPage — browser automation, structured data extraction, network listening and more.
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/mofanx/dp-cli
+Project-URL: Repository, https://github.com/mofanx/dp-cli
+Keywords: drissionpage,browser,automation,cli,web-scraping
+Classifier: Programming Language :: Python :: 3
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Topic :: Utilities
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: DrissionPage>=4.0
+Requires-Dist: click>=8.0
+
+# dp-cli
+
+A powerful CLI for [DrissionPage](https://github.com/g1879/DrissionPage) — browser automation, structured data extraction, network listening and more.
+
+## Features
+
+- **Anti-detection by default** — not based on webdriver, `navigator.webdriver` is `false`
+- **Reuse your own browser** — connect to a running Chrome via `--port`, keeping login state and cookies
+- **Powerful locator syntax** — descriptive strings stable across navigation (no ephemeral refs)
+- **Structured data extraction** — `extract` + `query` + `snapshot --mode content` for scraping list pages
+- **Network listening** — capture XHR/Fetch requests and response bodies
+- **Dual mode** — browser control + pure HTTP requests
+- **Shadow-root / iframe** — traverse directly without switching context
+- **JSON output** — all commands output JSON, AI-friendly
+
+## Installation
+
+```bash
+pip install dp-cli
+dp --help
+```
+
+## Quick Start
+
+```bash
+# Auto-managed browser
+dp open https://example.com
+dp snapshot
+dp click "text:Login"
+dp fill "@name=username" admin
+dp press Enter
+dp close
+
+# Connect to your own logged-in browser
+google-chrome --remote-debugging-port=9222
+dp open https://example.com --port 9222
+dp snapshot
+```
+
+## Data Extraction (3-step workflow)
+
+```bash
+# 1. Discover CSS class names via noise-filtered content tree
+dp snapshot --mode content --max-text 40
+
+# 2. Verify field selectors
+dp query "css:.item-title" --fields "text,loc"
+
+# 3. Batch extract to CSV
+dp extract "css:.item-card" \
+  '{"title":"css:.item-title",
+    "price":"css:.item-price",
+    "tags":{"selector":"css:.tag","multi":true},
+    "url":{"selector":"css:a","attr":"href"}}' \
+  --limit 100 --output csv --filename result.csv
+```
+
+## Project Structure
+
+```
+dp_cli/
+├── main.py              # CLI entry point (~47 lines)
+├── session.py           # Browser session management
+├── snapshot.py          # Page snapshot & data extraction engine
+├── output.py            # JSON output helpers
+└── commands/
+    ├── _utils.py        # Shared decorators & helpers
+    ├── browser.py       # open / goto / reload / close / list
+    ├── snapshot_cmd.py  # snapshot / extract / query / find / inspect
+    ├── element.py       # click / fill / select / hover / drag / check / upload
+    ├── keyboard.py      # press / type / scroll / scroll-to
+    ├── page.py          # screenshot / pdf / eval / wait / dialog
+    ├── tab.py           # tab-list / tab-new / tab-select / tab-close
+    ├── storage.py       # cookie-* / localstorage-* / sessionstorage-*
+    ├── network.py       # listen / listen-stop / http-get / http-post
+    └── misc.py          # resize / maximize / state-save / state-load / config-set
+```
+
+## Documentation
+
+See [`skills/SKILL.md`](skills/SKILL.md) for full workflow guide and [`skills/references/commands.md`](skills/references/commands.md) for complete command reference.
+
+## License
+
+BSD-3-Clause

dp_cli-0.1.0/README.md

@@ -0,0 +1,85 @@
+# dp-cli
+
+A powerful CLI for [DrissionPage](https://github.com/g1879/DrissionPage) — browser automation, structured data extraction, network listening and more.
+
+## Features
+
+- **Anti-detection by default** — not based on webdriver, `navigator.webdriver` is `false`
+- **Reuse your own browser** — connect to a running Chrome via `--port`, keeping login state and cookies
+- **Powerful locator syntax** — descriptive strings stable across navigation (no ephemeral refs)
+- **Structured data extraction** — `extract` + `query` + `snapshot --mode content` for scraping list pages
+- **Network listening** — capture XHR/Fetch requests and response bodies
+- **Dual mode** — browser control + pure HTTP requests
+- **Shadow-root / iframe** — traverse directly without switching context
+- **JSON output** — all commands output JSON, AI-friendly
+
+## Installation
+
+```bash
+pip install dp-cli
+dp --help
+```
+
+## Quick Start
+
+```bash
+# Auto-managed browser
+dp open https://example.com
+dp snapshot
+dp click "text:Login"
+dp fill "@name=username" admin
+dp press Enter
+dp close
+
+# Connect to your own logged-in browser
+google-chrome --remote-debugging-port=9222
+dp open https://example.com --port 9222
+dp snapshot
+```
+
+## Data Extraction (3-step workflow)
+
+```bash
+# 1. Discover CSS class names via noise-filtered content tree
+dp snapshot --mode content --max-text 40
+
+# 2. Verify field selectors
+dp query "css:.item-title" --fields "text,loc"
+
+# 3. Batch extract to CSV
+dp extract "css:.item-card" \
+  '{"title":"css:.item-title",
+    "price":"css:.item-price",
+    "tags":{"selector":"css:.tag","multi":true},
+    "url":{"selector":"css:a","attr":"href"}}' \
+  --limit 100 --output csv --filename result.csv
+```
+
+## Project Structure
+
+```
+dp_cli/
+├── main.py              # CLI entry point (~47 lines)
+├── session.py           # Browser session management
+├── snapshot.py          # Page snapshot & data extraction engine
+├── output.py            # JSON output helpers
+└── commands/
+    ├── _utils.py        # Shared decorators & helpers
+    ├── browser.py       # open / goto / reload / close / list
+    ├── snapshot_cmd.py  # snapshot / extract / query / find / inspect
+    ├── element.py       # click / fill / select / hover / drag / check / upload
+    ├── keyboard.py      # press / type / scroll / scroll-to
+    ├── page.py          # screenshot / pdf / eval / wait / dialog
+    ├── tab.py           # tab-list / tab-new / tab-select / tab-close
+    ├── storage.py       # cookie-* / localstorage-* / sessionstorage-*
+    ├── network.py       # listen / listen-stop / http-get / http-post
+    └── misc.py          # resize / maximize / state-save / state-load / config-set
+```
+
+## Documentation
+
+See [`skills/SKILL.md`](skills/SKILL.md) for full workflow guide and [`skills/references/commands.md`](skills/references/commands.md) for complete command reference.
+
+## License
+
+BSD-3-Clause

dp_cli-0.1.0/dp_cli/__init__.py

@@ -0,0 +1 @@
+# -*- coding:utf-8 -*-

dp_cli-0.1.0/dp_cli/commands/__init__.py

@@ -0,0 +1,12 @@
+# -*- coding:utf-8 -*-
+from dp_cli.commands import (
+    browser, snapshot_cmd, element, keyboard,
+    page, tab, storage, network, misc,
+)
+
+_MODULES = [browser, snapshot_cmd, element, keyboard, page, tab, storage, network, misc]
+
+
+def register_all(cli):
+    for mod in _MODULES:
+        mod.register(cli)

dp_cli-0.1.0/dp_cli/commands/_utils.py

@@ -0,0 +1,107 @@
+# -*- coding:utf-8 -*-
+"""所有命令模块共享的工具函数和装饰器"""
+import io
+import csv
+import click
+
+from dp_cli.session import get_browser, load_refs, load_session, save_session
+from dp_cli.output import error
+
+
+def normalize_url(url: str) -> str:
+    """补全 URL scheme，支持省略 http:// / https://"""
+    if not url:
+        return url
+    if not url.startswith(('http://', 'https://', 'file://')):
+        return 'https://' + url
+    return url
+
+
+def session_option(f):
+    return click.option('-s', '--session', default='default',
+                        help='会话名称，默认 default', show_default=True)(f)
+
+
+def _get_page(session: str, raw: bool = False):
+    """获取页面对象，失败则 error 退出。
+
+    :param raw: True 时始终返回 ChromiumPage（用于浏览器级操作如标签页管理）。
+                False 时返回绑定的标签页 ChromiumTab（如有），否则返回 ChromiumPage。
+    """
+    try:
+        page = get_browser(session)
+    except Exception as e:
+        error(f'无法连接浏览器会话 [{session}]，请先执行 dp open',
+              code='SESSION_NOT_FOUND', detail=str(e))
+        return
+
+    if raw:
+        return page
+
+    # 检查是否有绑定的标签页
+    sess = load_session(session)
+    tab_id = sess.get('active_tab')
+    if tab_id:
+        try:
+            tab = page.get_tab(tab_id)
+            return tab
+        except Exception:
+            # 标签页可能已关闭，清除绑定
+            sess.pop('active_tab', None)
+            save_session(session, sess)
+
+    return page
+
+
+def resolve_locator(locator: str, session: str = 'default') -> str:
+    """解析定位器，支持 ref:N 语法。
+
+    如果 locator 以 'ref:' 开头，从 session 的 refs 映射中查找真实定位器。
+    否则原样返回。
+    """
+    if not locator.startswith('ref:'):
+        return locator
+
+    ref_id = locator[4:]
+    refs = load_refs(session)
+    if not refs:
+        error(f'没有可用的 ref 映射，请先执行 dp snapshot',
+              code='NO_REFS')
+        raise SystemExit(1)
+
+    ref_data = refs.get(ref_id)
+    if not ref_data:
+        available = sorted(refs.keys(), key=lambda x: int(x) if x.isdigit() else 0)
+        hint = f"可用范围: ref:1 ~ ref:{available[-1]}" if available else ""
+        error(f'ref:{ref_id} 不存在。{hint}',
+              code='REF_NOT_FOUND')
+        raise SystemExit(1)
+
+    real_loc = ref_data.get('locator')
+    if real_loc and not real_loc.startswith('t:'):
+        return real_loc
+
+    # locator 不可用时（如 t:p），尝试用 name 作为 text 定位器
+    name = ref_data.get('name', '')
+    if name and len(name) <= 50:
+        return f'text:{name}'
+
+    error(f'ref:{ref_id} 无法解析为有效定位器 (role={ref_data.get("role")})',
+          code='REF_UNRESOLVABLE')
+    raise SystemExit(1)
+
+
+def records_to_csv(records: list) -> str:
+    """将记录列表转为 CSV 字符串（含 BOM，Excel 直接打开不乱码）"""
+    if not records:
+        return ''
+    fields = list(records[0].keys())
+    buf = io.StringIO()
+    writer = csv.DictWriter(buf, fieldnames=fields, extrasaction='ignore',
+                            lineterminator='\n')
+    writer.writeheader()
+    for row in records:
+        clean = {k: ('|'.join(str(i) for i in v) if isinstance(v, list) else v)
+                 for k, v in row.items()}
+        writer.writerow(clean)
+    return buf.getvalue()

dp_cli-0.1.0/dp_cli/commands/browser.py

@@ -0,0 +1,159 @@
+# -*- coding:utf-8 -*-
+"""浏览器生命周期命令: open / goto / reload / go-back / go-forward / close / close-all / list / delete-data"""
+import click
+
+from dp_cli.session import (get_browser, close_browser, list_sessions,
+                            delete_session, load_session, save_session)
+from dp_cli.output import ok, error, format_page_info
+from dp_cli.commands._utils import session_option, _get_page, normalize_url
+
+
+def register(cli):
+
+    @cli.command('open')
+    @click.argument('url', required=False)
+    @session_option
+    @click.option('--headless', is_flag=True, help='无头模式')
+    @click.option('--browser', 'browser_path', default=None, help='浏览器可执行文件路径')
+    @click.option('--profile', 'user_data_dir', default=None, help='用户数据目录')
+    @click.option('--proxy', default=None, help='代理服务器，如 http://127.0.0.1:7890')
+    @click.option('--port', type=int, default=None, help='连接指定端口的已有浏览器实例')
+    @click.option('--new', is_flag=True, help='强制创建新实例（不复用已有会话）')
+    def cmd_open(url, session, headless, browser_path, user_data_dir, proxy, port, new):
+        """打开浏览器并可选导航到 URL。
+
+        \b
+        【复用用户自己的浏览器】(最常见场景，保留登录状态/Cookie/历史)
+        第一步：用调试模式启动你自己的 Chrome/Chromium：
+          google-chrome --remote-debugging-port=9222
+        第二步：用 dp 接管：
+          dp open --port 9222
+          dp open https://example.com --port 9222
+        第三步：后续命令无需再指定 --port（会话自动记住端口）：
+          dp snapshot
+          dp click "text:登录"
+
+        \b
+        【dp 自动管理浏览器】
+          dp open
+          dp open https://example.com
+          dp open https://example.com --headless
+          dp -s work open https://github.com
+        """
+        if new:
+            delete_session(session)
+        try:
+            page = get_browser(session, headless=headless, browser_path=browser_path,
+                               user_data_dir=user_data_dir, proxy=proxy, port=port)
+        except Exception as e:
+            error(f'启动浏览器失败: {e}', code='BROWSER_START_FAILED', detail=str(e))
+            return
+        if url:
+            try:
+                page.get(normalize_url(url))
+            except Exception as e:
+                error(f'导航失败: {e}', code='NAVIGATE_FAILED', detail=str(e))
+                return
+        ok(format_page_info(page), msg='浏览器已就绪')
+
+    @cli.command()
+    @click.argument('url')
+    @session_option
+    @click.option('--timeout', default=30, help='超时秒数', show_default=True)
+    @click.option('--retry', default=3, help='重试次数', show_default=True)
+    def goto(url, session, timeout, retry):
+        """导航到指定 URL。
+
+        \b
+        示例:
+          dp goto https://example.com
+          dp goto example.com
+          dp goto example.com --timeout 60
+        """
+        page = _get_page(session)
+        try:
+            page.get(normalize_url(url), timeout=timeout, retry=retry)
+            ok(format_page_info(page))
+        except Exception as e:
+            error(f'导航到 {url} 失败', code='NAVIGATE_FAILED', detail=str(e))
+
+    @cli.command()
+    @session_option
+    def reload(session):
+        """刷新当前页面。"""
+        page = _get_page(session)
+        try:
+            page.get(page.url)
+            ok(format_page_info(page))
+        except Exception as e:
+            error(f'刷新失败', code='RELOAD_FAILED', detail=str(e))
+
+    @cli.command('go-back')
+    @session_option
+    def go_back(session):
+        """浏览器后退。"""
+        page = _get_page(session)
+        try:
+            page.back()
+            ok(format_page_info(page))
+        except Exception as e:
+            error('后退失败', code='NAVIGATE_FAILED', detail=str(e))
+
+    @cli.command('go-forward')
+    @session_option
+    def go_forward(session):
+        """浏览器前进。"""
+        page = _get_page(session)
+        try:
+            page.forward()
+            ok(format_page_info(page))
+        except Exception as e:
+            error('前进失败', code='NAVIGATE_FAILED', detail=str(e))
+
+    @cli.command('close')
+    @session_option
+    @click.option('--del-data', is_flag=True, help='同时删除用户数据目录')
+    @click.option('--force', is_flag=True, help='强制关闭浏览器（user_connected 模式下默认只断开连接）')
+    def cmd_close(session, del_data, force):
+        """关闭浏览器会话。
+
+        如果是通过 --port 连接的用户自己的浏览器，默认只断开连接不关闭浏览器。
+        用 --force 才会真正关闭浏览器进程。
+        """
+        sess = load_session(session)
+        if not sess:
+            error(f'会话 [{session}] 不存在', code='SESSION_NOT_FOUND')
+            return
+        user_connected = sess.get('user_connected', False)
+        if user_connected and not force:
+            delete_session(session)
+            ok(msg=f'已断开与 [{session}] 的连接（浏览器仍运行）。用 --force 关闭浏览器。')
+        else:
+            result = close_browser(session, del_data=del_data)
+            if result:
+                ok(msg=f'会话 [{session}] 已关闭')
+            else:
+                error(f'关闭失败', code='CLOSE_FAILED')
+
+    @cli.command('close-all')
+    def close_all():
+        """关闭所有会话。"""
+        sessions = list_sessions()
+        closed = []
+        for s in sessions:
+            close_browser(s['name'])
+            closed.append(s['name'])
+        ok({'closed': closed}, msg=f'已关闭 {len(closed)} 个会话')
+
+    @cli.command('list')
+    def cmd_list():
+        """列出所有活跃会话。"""
+        sessions = list_sessions()
+        ok({'sessions': sessions, 'count': len(sessions)})
+
+    @cli.command('delete-data')
+    @session_option
+    def delete_data(session):
+        """删除会话的用户数据目录。"""
+        close_browser(session, del_data=True)
+        ok(msg=f'会话 [{session}] 数据已删除')