skysearch 0.2.0__tar.gz

skysearch-0.2.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: skysearch
+Version: 0.2.0
+Summary: Search the web, rank results, fetch any page content.
+Author: zimvir
+License: MIT
+Project-URL: Homepage, https://github.com/zimvir/skysearch
+Project-URL: Documentation, https://github.com/zimvir/skysearch#readme
+Keywords: search,fetch,bing,bm25,web-scraper,crawler
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: DrissionPage
+Requires-Dist: beautifulsoup4
+Requires-Dist: lxml
+Requires-Dist: readability-lxml
+Requires-Dist: jieba
+Requires-Dist: rank-bm25
+
+# SkySearch
+
+基于 Bing 搜索 + BM25 排序的轻量级搜索引擎，支持动态页面抓取。
+
+## 特性
+
+- **Bing 搜索**：SessionPage 底层 TLS 指纹伪装，绕过反爬
+- **动态页面**：Chromium 多标签页并行抓取
+- **BM25 排序**：jieba 中文分词 + rank-bm25 相关性排序
+- **多模式输出**：`text` / `info` / `raw` 三种输出模式
+- **CLI 工具**：支持命令行参数，也可用作 Python 库
+
+## 安装
+
+```bash
+pip install skysearch
+```
+
+## 命令行使用
+
+### 搜索模式
+
+```bash
+# 交互式输入
+skysearch
+
+# 指定关键词
+skysearch "深度学习框架"
+
+# 指定结果数量
+skysearch -n 20 "Python教程"
+
+# 保持浏览器打开
+skysearch -n 20 "关键词" --keep
+```
+
+### URL 抓取模式
+
+```bash
+# 默认：输出纯文本
+skysearch --url https://example.com
+
+# 指定输出模式
+skysearch --url https://example.com --mode text   # 纯文本（默认）
+skysearch --url https://example.com --mode info    # 结构化信息
+skysearch --url https://example.com --mode raw     # 原始 HTML
+
+# 保持浏览器打开
+skysearch --url https://example.com --keep
+```
+
+## 作为库使用
+
+```python
+import skysearch
+
+# 搜索
+results = skysearch.search("深度学习", num=10)
+# [{'title': ..., 'url': ..., 'score': 12.5, 'snippet': ...}, ...]
+
+# URL 抓取
+text = skysearch.fetch("https://example.com")
+info = skysearch.fetch("https://example.com", mode='info')
+raw = skysearch.fetch("https://example.com", mode='raw')
+
+# 单独函数
+links = skysearch.fetch_links("https://example.com")
+info_dict = skysearch.fetch_info("https://example.com")
+raw_dict = skysearch.fetch_raw("https://example.com")
+
+# 搜索 + 抓取一体化
+results = skysearch.search_and_fetch("关键词", mode='info')
+```
+
+## API 参数说明
+
+### search(query, num=10, verbose=False, keep=False, tuple_format=False)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| query | 搜索关键词 | - |
+| num | 结果数量 | 10 |
+| verbose | 打印详细过程 | False |
+| keep | 保持浏览器打开 | False |
+| tuple_format | 返回元组格式 | False |
+
+### fetch(url, mode='text', keep=False, timeout=10, retry=2)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| url | 页面 URL | - |
+| mode | 输出模式：`text` `info` `raw` | text |
+| keep | 保持浏览器打开 | False |
+| timeout | 请求超时秒数 | 10 |
+| retry | 重试次数 | 2 |
+
+### search_and_fetch(query, num=10, mode='text', verbose=False, keep=False, tuple_format=False)
+
+一体化搜索 + 抓取，返回列表包含结果信息和抓取内容。
+
+## 输出模式说明
+
+| 模式 | 说明 | 适用场景 |
+|------|------|----------|
+| `text` | 纯文本正文 | 人类阅读 |
+| `info` | 结构化 JSON（url, title, text, links, meta） | 数据分析 / agent |
+| `raw` | 原始 HTML | 深度解析 |
+
+## 技术栈
+
+| 模块 | 技术 |
+|------|------|
+| HTTP 请求 | DrissionPage (SessionPage) |
+| 动态渲染 | DrissionPage (ChromiumPage) |
+| HTML 解析 | BeautifulSoup4 + lxml |
+| 正文提取 | readability-lxml |
+| 中文分词 | jieba |
+| 排序算法 | rank-bm25 (BM25Okapi) |
+
+## 项目结构
+
+```
+src/skysearch/
+├── __init__.py       # 库入口，导出所有 API
+├── cli.py            # 命令行入口
+├── search.py         # Bing 搜索
+├── ranker.py         # BM25 排序
+├── api.py            # 简洁 API 接口
+└── fetcher/          # 页面抓取包
+    ├── __init__.py
+    ├── core.py       # 核心函数
+    ├── session.py    # 会话管理
+    └── parser.py     # HTML 解析
+```
+
+## License
+
+MIT

skysearch-0.2.0/README.md

@@ -0,0 +1,138 @@
+# SkySearch
+
+基于 Bing 搜索 + BM25 排序的轻量级搜索引擎，支持动态页面抓取。
+
+## 特性
+
+- **Bing 搜索**：SessionPage 底层 TLS 指纹伪装，绕过反爬
+- **动态页面**：Chromium 多标签页并行抓取
+- **BM25 排序**：jieba 中文分词 + rank-bm25 相关性排序
+- **多模式输出**：`text` / `info` / `raw` 三种输出模式
+- **CLI 工具**：支持命令行参数，也可用作 Python 库
+
+## 安装
+
+```bash
+pip install skysearch
+```
+
+## 命令行使用
+
+### 搜索模式
+
+```bash
+# 交互式输入
+skysearch
+
+# 指定关键词
+skysearch "深度学习框架"
+
+# 指定结果数量
+skysearch -n 20 "Python教程"
+
+# 保持浏览器打开
+skysearch -n 20 "关键词" --keep
+```
+
+### URL 抓取模式
+
+```bash
+# 默认：输出纯文本
+skysearch --url https://example.com
+
+# 指定输出模式
+skysearch --url https://example.com --mode text   # 纯文本（默认）
+skysearch --url https://example.com --mode info    # 结构化信息
+skysearch --url https://example.com --mode raw     # 原始 HTML
+
+# 保持浏览器打开
+skysearch --url https://example.com --keep
+```
+
+## 作为库使用
+
+```python
+import skysearch
+
+# 搜索
+results = skysearch.search("深度学习", num=10)
+# [{'title': ..., 'url': ..., 'score': 12.5, 'snippet': ...}, ...]
+
+# URL 抓取
+text = skysearch.fetch("https://example.com")
+info = skysearch.fetch("https://example.com", mode='info')
+raw = skysearch.fetch("https://example.com", mode='raw')
+
+# 单独函数
+links = skysearch.fetch_links("https://example.com")
+info_dict = skysearch.fetch_info("https://example.com")
+raw_dict = skysearch.fetch_raw("https://example.com")
+
+# 搜索 + 抓取一体化
+results = skysearch.search_and_fetch("关键词", mode='info')
+```
+
+## API 参数说明
+
+### search(query, num=10, verbose=False, keep=False, tuple_format=False)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| query | 搜索关键词 | - |
+| num | 结果数量 | 10 |
+| verbose | 打印详细过程 | False |
+| keep | 保持浏览器打开 | False |
+| tuple_format | 返回元组格式 | False |
+
+### fetch(url, mode='text', keep=False, timeout=10, retry=2)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| url | 页面 URL | - |
+| mode | 输出模式：`text` `info` `raw` | text |
+| keep | 保持浏览器打开 | False |
+| timeout | 请求超时秒数 | 10 |
+| retry | 重试次数 | 2 |
+
+### search_and_fetch(query, num=10, mode='text', verbose=False, keep=False, tuple_format=False)
+
+一体化搜索 + 抓取，返回列表包含结果信息和抓取内容。
+
+## 输出模式说明
+
+| 模式 | 说明 | 适用场景 |
+|------|------|----------|
+| `text` | 纯文本正文 | 人类阅读 |
+| `info` | 结构化 JSON（url, title, text, links, meta） | 数据分析 / agent |
+| `raw` | 原始 HTML | 深度解析 |
+
+## 技术栈
+
+| 模块 | 技术 |
+|------|------|
+| HTTP 请求 | DrissionPage (SessionPage) |
+| 动态渲染 | DrissionPage (ChromiumPage) |
+| HTML 解析 | BeautifulSoup4 + lxml |
+| 正文提取 | readability-lxml |
+| 中文分词 | jieba |
+| 排序算法 | rank-bm25 (BM25Okapi) |
+
+## 项目结构
+
+```
+src/skysearch/
+├── __init__.py       # 库入口，导出所有 API
+├── cli.py            # 命令行入口
+├── search.py         # Bing 搜索
+├── ranker.py         # BM25 排序
+├── api.py            # 简洁 API 接口
+└── fetcher/          # 页面抓取包
+    ├── __init__.py
+    ├── core.py       # 核心函数
+    ├── session.py    # 会话管理
+    └── parser.py     # HTML 解析
+```
+
+## License
+
+MIT

skysearch-0.2.0/pyproject.toml

@@ -0,0 +1,42 @@
+[project]
+name = "skysearch"
+version = "0.2.0"
+description = "Search the web, rank results, fetch any page content."
+readme = "README.md"
+requires-python = ">=3.11"
+license = {text = "MIT"}
+authors = [
+    {name = "zimvir"}
+]
+keywords = ["search", "fetch", "bing", "bm25", "web-scraper", "crawler"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+
+dependencies = [
+    "DrissionPage",
+    "beautifulsoup4",
+    "lxml",
+    "readability-lxml",
+    "jieba",
+    "rank-bm25",
+]
+
+[project.scripts]
+skysearch = "skysearch.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/zimvir/skysearch"
+Documentation = "https://github.com/zimvir/skysearch#readme"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]

skysearch-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

skysearch-0.2.0/src/skysearch/__init__.py

@@ -0,0 +1,40 @@
+"""SkySearch - 基于 Bing 搜索 + BM25 排序的轻量级搜索引擎。"""
+
+__version__ = "0.1.0"
+
+from .search import bing_search
+from .fetcher import (
+    fetch_page_text,
+    fetch_all_pages,
+    is_dynamic_page,
+    parse_html,
+    parse_info,
+)
+from .ranker import rank_documents
+from . import api
+
+__all__ = [
+    'bing_search',
+    'fetch_page_text',
+    'fetch_all_pages',
+    'is_dynamic_page',
+    'parse_html',
+    'parse_info',
+    'rank_documents',
+    'api',
+    # API 函数
+    'search',
+    'fetch',
+    'fetch_info',
+    'fetch_raw',
+    'fetch_links',
+    'search_and_fetch',
+]
+
+# 快捷引用
+search = api.search
+fetch = api.fetch
+fetch_info = api.fetch_info
+fetch_raw = api.fetch_raw
+fetch_links = api.fetch_links
+search_and_fetch = api.search_and_fetch

skysearch-0.2.0/src/skysearch/api.py

@@ -0,0 +1,279 @@
+"""简洁 API 接口，模拟 CLI 行为。"""
+
+import json
+from .search import bing_search as _bing_search
+from .fetcher import (
+    fetch_page_text as _fetch_page_text,
+    fetch_all_pages as _fetch_all_pages,
+    parse_html as _parse_html,
+    is_dynamic_page as _is_dynamic_page,
+    parse_info as _parse_info,
+)
+from .ranker import rank_documents as _rank_documents
+from DrissionPage import SessionPage, ChromiumOptions, ChromiumPage
+from bs4 import BeautifulSoup
+import time
+
+
+def search(query, num=10, verbose=False, keep=False, tuple_format=False):
+    """
+    搜索模式（类似 CLI: skysearch "关键词"）。
+
+    Args:
+        query: 搜索关键词
+        num: 结果数量（默认 10）
+        verbose: 是否打印详细过程（默认 False）
+        keep: 是否保持浏览器打开（默认 False）
+        tuple_format: 是否返回元组格式（默认 False）
+            False: [{"title": ..., "url": ..., "score": 12.5}, ...]
+            True: [({"title": ..., "url": ...}, 12.5), ...]
+
+    Returns:
+        排序后的结果列表
+    """
+    if verbose:
+        print(f"Searching Bing for: {query}\n")
+
+    results = _bing_search(query, num_results=num)
+    if not results:
+        if verbose:
+            print("No results found.")
+        return []
+
+    if verbose:
+        print(f"Found {len(results)} results, fetching content...\n")
+
+    urls = [r['url'] for r in results]
+    raw_results = _fetch_all_pages(urls, verbose=verbose)
+    raw_results.sort(key=lambda x: x[0])
+
+    texts = []
+    valid_results = []
+
+    for i, (idx, text, is_dynamic) in enumerate(raw_results):
+        if text:
+            texts.append(text)
+            valid_results.append(results[i])
+            if verbose:
+                status = "[D]" if is_dynamic else "[S]"
+                print(f"[{i+1}] {status} {results[i]['title'][:40]} | length: {len(text)}")
+
+    if not texts:
+        if verbose:
+            print("\nNo page content fetched.")
+        return []
+
+    if verbose:
+        print(f"\nRanking... (valid pages: {len(texts)})\n")
+
+    scores = _rank_documents(texts, query)
+
+    if tuple_format:
+        ranked = list(zip(valid_results, scores))
+        ranked.sort(key=lambda x: x[1], reverse=True)
+        return ranked
+
+    # 默认返回扁平列表
+    output = []
+    for r, score in zip(valid_results, scores):
+        item = dict(r)
+        item['score'] = score
+        output.append(item)
+
+    output.sort(key=lambda x: x['score'], reverse=True)
+    return output
+
+
+def fetch(url, mode='text', keep=False, timeout=10, retry=2):
+    """
+    URL 抓取模式（类似 CLI: skysearch --url xxx）。
+
+    Args:
+        url: 页面 URL
+        mode: 输出模式，'text' | 'info' | 'raw'（默认 'text'）
+        keep: 是否保持浏览器打开（默认 False）
+        timeout: 请求超时秒数（默认 10）
+        retry: 重试次数（默认 2）
+
+    Returns:
+        根据模式返回不同内容
+    """
+    page = SessionPage()
+    page.get(url, retry=retry, timeout=timeout)
+    html = page.html
+
+    # 动态页面用浏览器渲染
+    if _is_dynamic_page(html):
+        options = ChromiumOptions()
+        options.headless = True
+        browser = ChromiumPage(options)
+        try:
+            browser.get(url)
+            browser.wait.load_start()
+            time.sleep(2)
+            html = browser.html
+        finally:
+            if not keep:
+                browser.quit()
+
+    if mode == 'raw':
+        return json.dumps({"url": url, "html": html}, ensure_ascii=False)
+
+    if mode == 'info':
+        result = {"url": url}
+        result.update(_parse_html(html, mode='info'))
+        return json.dumps(result, ensure_ascii=False, indent=2)
+
+    if mode == 'text':
+        return _fetch_page_text(url, close_browser=not keep)
+
+    return None
+
+
+def fetch_info(url, keep=False, timeout=10, retry=2):
+    """
+    获取结构化信息（url, title, text, links, meta）。
+
+    Args:
+        url: 页面 URL
+        keep: 是否保持浏览器打开（默认 False）
+        timeout: 请求超时秒数（默认 10）
+        retry: 重试次数（默认 2）
+
+    Returns:
+        dict: {"url": ..., "title": ..., "text": ..., "links": [...], "meta": {...}}
+    """
+    page = SessionPage()
+    page.get(url, retry=retry, timeout=timeout)
+    html = page.html
+
+    if _is_dynamic_page(html):
+        options = ChromiumOptions()
+        options.headless = True
+        browser = ChromiumPage(options)
+        try:
+            browser.get(url)
+            browser.wait.load_start()
+            time.sleep(2)
+            html = browser.html
+        finally:
+            if not keep:
+                browser.quit()
+
+    return {"url": url, **_parse_info(html)}
+
+
+def fetch_raw(url, keep=False, timeout=10, retry=2):
+    """
+    获取原始 HTML。
+
+    Args:
+        url: 页面 URL
+        keep: 是否保持浏览器打开（默认 False）
+        timeout: 请求超时秒数（默认 10）
+        retry: 重试次数（默认 2）
+
+    Returns:
+        dict: {"url": ..., "html": ...}
+    """
+    page = SessionPage()
+    page.get(url, retry=retry, timeout=timeout)
+    html = page.html
+
+    if _is_dynamic_page(html):
+        options = ChromiumOptions()
+        options.headless = True
+        browser = ChromiumPage(options)
+        try:
+            browser.get(url)
+            browser.wait.load_start()
+            time.sleep(2)
+            html = browser.html
+        finally:
+            if not keep:
+                browser.quit()
+
+    return {"url": url, "html": html}
+
+
+def fetch_links(url, timeout=10, retry=2):
+    """
+    提取页面所有链接。
+
+    Args:
+        url: 页面 URL
+        timeout: 请求超时秒数（默认 10）
+        retry: 重试次数（默认 2）
+
+    Returns:
+        list: [{"text": "链接文字", "href": "https://..."}, ...]
+    """
+    page = SessionPage()
+    page.get(url, retry=retry, timeout=timeout)
+    html = page.html
+
+    if _is_dynamic_page(html):
+        options = ChromiumOptions()
+        options.headless = True
+        browser = ChromiumPage(options)
+        try:
+            browser.get(url)
+            browser.wait.load_start()
+            time.sleep(2)
+            html = browser.html
+        finally:
+            browser.quit()
+
+    soup = BeautifulSoup(html, 'lxml')
+    links = []
+    for a in soup.find_all('a', href=True):
+        links.append({
+            "text": a.get_text(strip=True),
+            "href": a['href']
+        })
+
+    return links
+
+
+def search_and_fetch(query, num=10, mode='text', verbose=False, keep=False, tuple_format=False):
+    """
+    搜索 + 抓取一体化。
+
+    Args:
+        query: 搜索关键词
+        num: 结果数量（默认 10）
+        mode: 输出模式，'text' | 'info' | 'raw'（默认 'text'）
+        verbose: 是否打印详细过程（默认 False）
+        keep: 是否保持浏览器打开（默认 False）
+        tuple_format: 是否返回元组格式（默认 False）
+
+    Returns:
+        list: 搜索结果列表，每个元素包含结果信息和抓取内容
+    """
+    results = search(query, num=num, verbose=verbose, keep=keep, tuple_format=tuple_format)
+
+    output = []
+    for item in results:
+        if tuple_format:
+            url = item[0]['url']
+            score = item[1]
+            result_dict = item[0]
+        else:
+            url = item['url']
+            score = item.get('score', 0)
+            result_dict = item
+
+        if mode == 'raw':
+            content = fetch_raw(url, keep=keep)
+        elif mode == 'info':
+            content = fetch_info(url, keep=keep)
+        else:
+            content = fetch(url, mode='text', keep=keep)
+
+        output.append({
+            **result_dict,
+            "score": score,
+            "content": content
+        })
+
+    return output

skysearch-0.2.0/src/skysearch/cli.py

@@ -0,0 +1,129 @@
+"""命令行入口。"""
+
+import argparse
+import time
+import json
+from .search import bing_search
+from .fetcher import fetch_all_pages, fetch_page_text, is_dynamic_page, parse_html
+from .ranker import rank_documents
+from DrissionPage import SessionPage, ChromiumOptions, ChromiumPage
+
+
+def handle_url(args):
+    """处理 --url 模式。"""
+    print(f"Fetching: {args.url}\n")
+
+    page = SessionPage()
+    page.get(args.url, retry=2, timeout=10)
+    html = page.html
+
+    # 动态页面用浏览器渲染
+    if is_dynamic_page(html):
+        options = ChromiumOptions()
+        options.headless = True
+        browser = ChromiumPage(options)
+        try:
+            browser.get(args.url)
+            browser.wait.load_start()
+            time.sleep(2)
+            html = browser.html
+        finally:
+            if not args.keep:
+                browser.quit()
+
+    # 根据模式解析输出
+    if args.mode == 'raw':
+        result = {"url": args.url, "html": html}
+        print(json.dumps(result, ensure_ascii=False))
+    elif args.mode == 'info':
+        result = {"url": args.url}
+        result.update(parse_html(html, mode='info'))
+        print(json.dumps(result, ensure_ascii=False, indent=2))
+    else:
+        text = fetch_page_text(args.url, close_browser=not args.keep)
+        if args.keep:
+            print("Browser kept open (use Ctrl+C to exit)")
+        elif text:
+            print(f"Content ({len(text)} chars):\n")
+            print(text)
+        else:
+            print("Failed to fetch page content.")
+
+
+def handle_search(args):
+    """处理搜索模式。"""
+    query = args.query or args.query2
+    if not query:
+        query = input("Enter search query: ")
+
+    print("Searching Bing...\n")
+
+    results = bing_search(query, num_results=args.num)
+
+    if not results:
+        print("No results found.")
+        return
+
+    print(f"Found {len(results)} results, fetching content...\n")
+
+    urls = [r['url'] for r in results]
+    raw_results = fetch_all_pages(urls, verbose=True)
+    raw_results.sort(key=lambda x: x[0])
+
+    texts = []
+    valid_results = []
+
+    print("="*50)
+    print("Fetch summary:")
+    print("="*50)
+
+    for i, (idx, text, is_dynamic) in enumerate(raw_results):
+        if text:
+            texts.append(text)
+            valid_results.append(results[i])
+            status = "[D]" if is_dynamic else "[S]"
+        else:
+            status = "[X]"
+
+        print(f"[{i+1}] {status} {results[i]['title'][:40]} | length: {len(text) if text else 0}")
+
+    if not texts:
+        print("\nNo page content fetched.")
+        return
+
+    print(f"\nRanking... (valid pages: {len(texts)})\n")
+
+    scores = rank_documents(texts, query)
+    ranked = list(zip(valid_results, scores))
+    ranked.sort(key=lambda x: x[1], reverse=True)
+
+    print("="*50)
+    print("Ranked results (higher score = more relevant):")
+    print("="*50)
+
+    for i, (r, score) in enumerate(ranked[:10], 1):
+        print(f"\n{i}. Score: {score:.3f}")
+        print(f"   Title: {r['title']}")
+        print(f"   URL:   {r['url']}")
+
+
+def main():
+    """主入口函数。"""
+    parser = argparse.ArgumentParser(
+        description='SkySearch - A lightweight search engine based on Bing + BM25',
+        usage='%(prog)s [-n NUM] [-q QUERY] [QUERY]\n'
+              '       %(prog)s --url <url>'
+    )
+    parser.add_argument('query', nargs='?', help='search query')
+    parser.add_argument('-n', '--num', type=int, default=10, help='number of results (default: 10)')
+    parser.add_argument('-q', '--query', dest='query2', help='search query (long form)')
+    parser.add_argument('--url', help='fetch a single URL directly')
+    parser.add_argument('--mode', choices=['text', 'info', 'raw'], default='text', help='output mode (default: text)')
+    parser.add_argument('--keep', action='store_true', help='keep browser open after fetching')
+
+    args = parser.parse_args()
+
+    if args.url:
+        handle_url(args)
+    else:
+        handle_search(args)

skysearch-0.2.0/src/skysearch/fetcher/__init__.py

@@ -0,0 +1,17 @@
+"""页面抓取模块。"""
+
+from .core import fetch_page_text, is_dynamic_page, extract_text
+from .session import fetch_page_session, fetch_all_pages
+from .parser import parse_html, parse_info, parse_raw, parse_text
+
+__all__ = [
+    'fetch_page_text',
+    'is_dynamic_page',
+    'extract_text',
+    'fetch_page_session',
+    'fetch_all_pages',
+    'parse_html',
+    'parse_info',
+    'parse_raw',
+    'parse_text',
+]

skysearch-0.2.0/src/skysearch/fetcher/core.py

@@ -0,0 +1,77 @@
+"""页面抓取核心模块。"""
+
+import time
+import os
+from DrissionPage import SessionPage, ChromiumPage, ChromiumOptions
+from bs4 import BeautifulSoup
+from readability.readability import Document
+
+DEBUG = os.getenv('SKYSEARCH_DEBUG', 'false').lower() in ('1', 'true', 'yes')
+
+
+def is_dynamic_page(html: str) -> bool:
+    """判断页面是否需要浏览器渲染。"""
+    dynamic_markers = ['id="root"', 'id="app"', 'id="__next"', '__NEXT_DATA__']
+
+    for marker in dynamic_markers:
+        if marker in html:
+            return True
+
+    soup = BeautifulSoup(html, 'lxml')
+    text = soup.get_text()
+    if len(text) < 500:
+        return True
+
+    return False
+
+
+def extract_text(html: str) -> str:
+    """使用 readability-lxml 提取正文文本。"""
+    if not html or len(html.strip()) == 0:
+        return ''
+
+    try:
+        doc = Document(html)
+        summary_html = doc.summary()
+        soup = BeautifulSoup(summary_html, 'lxml')
+        text = soup.get_text(separator=' ', strip=True)
+        return text if len(text) >= 50 else ''
+    except Exception:
+        return ''
+
+
+def fetch_page_text(url: str, close_browser: bool = True) -> str:
+    """抓取单个页面并提取正文文本。"""
+    browser = None
+    try:
+        page = SessionPage()
+        page.get(url, retry=2, timeout=10)
+        html = page.html
+
+        if is_dynamic_page(html):
+            if DEBUG:
+                print(f"  [DEBUG] Dynamic page, using browser | original length: {len(html)}")
+            try:
+                options = ChromiumOptions()
+                options.headless = True
+                browser = ChromiumPage(options)
+                browser.get(url)
+                browser.wait.load_start()
+                time.sleep(2)
+                html = browser.html
+            except Exception as e:
+                if DEBUG:
+                    print(f"  [DEBUG] Browser failed, using original: {e}")
+                if browser:
+                    browser.quit()
+                    browser = None
+
+        return extract_text(html)
+
+    except Exception as e:
+        if DEBUG:
+            print(f"  [DEBUG] Fetch failed: {e}")
+        return ''
+    finally:
+        if close_browser and browser:
+            browser.quit()

skysearch-0.2.0/src/skysearch/fetcher/parser.py

@@ -0,0 +1,64 @@
+"""HTML 解析模块。"""
+
+from bs4 import BeautifulSoup
+from readability.readability import Document
+
+
+def parse_info(html):
+    """解析结构化信息：url, title, text, links, meta。"""
+    soup = BeautifulSoup(html, 'lxml')
+
+    # 提取标题
+    title = soup.title.string if soup.title else ""
+
+    # 提取 meta 信息
+    meta = {}
+    for tag in soup.find_all('meta'):
+        name = tag.get('property') or tag.get('name')
+        content = tag.get('content', '')
+        if name and content:
+            meta[name] = content
+
+    # 提取链接
+    links = []
+    for a in soup.find_all('a', href=True):
+        links.append({
+            "text": a.get_text(strip=True),
+            "href": a['href']
+        })
+
+    # 提取正文
+    doc = Document(html)
+    summary_html = doc.summary()
+    text_soup = BeautifulSoup(summary_html, 'lxml')
+    text = text_soup.get_text(separator=' ', strip=True)
+
+    return {
+        "title": title,
+        "text": text,
+        "links": links,
+        "meta": meta
+    }
+
+
+def parse_raw(html):
+    """返回原始 HTML。"""
+    return {"html": html}
+
+
+def parse_text(html):
+    """提取纯文本。"""
+    doc = Document(html)
+    summary_html = doc.summary()
+    soup = BeautifulSoup(summary_html, 'lxml')
+    return soup.get_text(separator=' ', strip=True)
+
+
+def parse_html(html, mode='text'):
+    """根据模式解析 HTML。"""
+    if mode == 'info':
+        return parse_info(html)
+    elif mode == 'raw':
+        return parse_raw(html)
+    else:
+        return parse_text(html)

skysearch-0.2.0/src/skysearch/fetcher/session.py

@@ -0,0 +1,95 @@
+"""页面抓取会话模块。"""
+
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from DrissionPage import SessionPage, ChromiumPage, ChromiumOptions
+from .core import extract_text, is_dynamic_page
+
+
+def fetch_page_session(url: str):
+    """使用 SessionPage 抓取页面。"""
+    try:
+        page = SessionPage()
+        page.get(url, retry=2, timeout=10)
+        html = page.html
+        text = extract_text(html)
+        return text, is_dynamic_page(html)
+    except Exception:
+        return '', True
+
+
+def fetch_all_pages(urls, verbose=False):
+    """两轮抓取：SessionPage 并行 + Chromium 多标签重试。
+
+    Args:
+        urls: URL 列表
+        verbose: 是否打印详细过程（默认 False）
+    """
+    if verbose:
+        print("[1/2] Fetching pages with SessionPage...")
+    results = [None] * len(urls)
+    failed_indices = []
+
+    with ThreadPoolExecutor(max_workers=5) as executor:
+        futures = {executor.submit(fetch_page_session, url): i for i, url in enumerate(urls)}
+
+        for future in as_completed(futures):
+            i = futures[future]
+            text, maybe_dynamic = future.result()
+            results[i] = (i, text, maybe_dynamic)
+
+            if verbose:
+                if text:
+                    print(f"  OK [{i+1}/{len(urls)}] length: {len(text)}")
+                else:
+                    failed_indices.append(i)
+                    print(f"  RETRY [{i+1}/{len(urls)}] will retry with browser")
+            elif not text:
+                failed_indices.append(i)
+
+    # 第二轮：Chromium 多标签重试
+    if failed_indices:
+        if verbose:
+            print(f"\n[2/2] Retrying {len(failed_indices)} pages with Chromium (multi-tab)...")
+
+        retry_urls = [urls[i] for i in failed_indices]
+
+        try:
+            options = ChromiumOptions()
+            options.headless = True
+            browser = ChromiumPage(options)
+
+            try:
+                browser.get(retry_urls[0])
+                browser.wait.load_start()
+                time.sleep(2)
+
+                for url in retry_urls[1:]:
+                    browser.new_tab(url)
+
+                time.sleep(3)
+
+                tab_ids = browser.tab_ids
+
+                for tab_idx, tab_id in enumerate(tab_ids):
+                    if tab_idx < len(retry_urls):
+                        try:
+                            browser.activate_tab(tab_id)
+                            time.sleep(1)
+                            html = browser.html
+                            text = extract_text(html)
+                            original_idx = failed_indices[tab_idx]
+                            results[original_idx] = (original_idx, text, True)
+                            if verbose:
+                                print(f"  OK [retry {tab_idx+1}/{len(retry_urls)}] length: {len(text)}")
+                        except Exception as e:
+                            if verbose:
+                                print(f"  FAIL [retry {tab_idx+1}/{len(retry_urls)}] error: {e}")
+            finally:
+                browser.quit()
+
+        except Exception as e:
+            if verbose:
+                print(f"Chromium retry failed: {e}")
+
+    return results

skysearch-0.2.0/src/skysearch/ranker.py

@@ -0,0 +1,29 @@
+"""BM25 排序模块。"""
+
+import jieba
+from rank_bm25 import BM25Okapi
+from typing import List
+
+
+def rank_documents(documents: List[str], query: str) -> List[float]:
+    """
+    使用 BM25 算法对文档进行相关性排序。
+
+    Args:
+        documents: 文档文本列表。
+        query: 搜索关键词。
+
+    Returns:
+        与输入文档顺序对应的分数列表。
+    """
+    if not documents:
+        return []
+
+    # jieba 中文分词
+    tokenized_corpus = [list(jieba.cut(doc)) for doc in documents]
+    bm25 = BM25Okapi(tokenized_corpus)
+
+    tokenized_query = list(jieba.cut(query))
+    scores = bm25.get_scores(tokenized_query)
+
+    return scores.tolist()

skysearch-0.2.0/src/skysearch/search.py

@@ -0,0 +1,67 @@
+"""Bing 搜索模块。"""
+
+from typing import List, Dict
+from DrissionPage import SessionPage
+from bs4 import BeautifulSoup
+from urllib.parse import quote, urlparse, parse_qs
+
+
+def bing_search(query: str, num_results: int = 10) -> List[Dict[str, str]]:
+    """
+    从 Bing 获取搜索结果。
+
+    Args:
+        query: 搜索关键词。
+        num_results: 返回结果数量。
+
+    Returns:
+        列表，每个元素为 dict，包含 title、url、snippet。
+    """
+    url = f"https://www.bing.com/search?q={quote(query)}"
+    page = SessionPage()
+    page.get(url, retry=2)
+
+    soup = BeautifulSoup(page.html, 'lxml')
+    results = []
+
+    # Bing 结果在 <li class="b_algo"> 中
+    for li in soup.select('li.b_algo')[:num_results]:
+        a_tag = li.select_one('h2 a')
+        p_tag = li.select_one('p')
+
+        if not a_tag:
+            continue
+
+        title = a_tag.text.strip()
+        href = a_tag.get('href', '')
+        url = parse_real_url(href)
+        snippet = p_tag.text.strip() if p_tag else ''
+
+        results.append({
+            'title': title,
+            'url': url,
+            'snippet': snippet,
+        })
+
+    return results
+
+
+def parse_real_url(redirect_url: str) -> str:
+    """
+    从 Bing 重定向 URL 中提取真实 URL。
+
+    Args:
+        redirect_url: Bing 重定向 URL，格式如 /url?q=实际URL&...
+
+    Returns:
+        真实 URL 字符串。
+    """
+    if not redirect_url:
+        return ''
+
+    if redirect_url.startswith('/url?'):
+        parsed = urlparse(redirect_url)
+        qs = parse_qs(parsed.query)
+        return qs.get('q', [''])[0]
+
+    return redirect_url

skysearch-0.2.0/src/skysearch.egg-info/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: skysearch
+Version: 0.2.0
+Summary: Search the web, rank results, fetch any page content.
+Author: zimvir
+License: MIT
+Project-URL: Homepage, https://github.com/zimvir/skysearch
+Project-URL: Documentation, https://github.com/zimvir/skysearch#readme
+Keywords: search,fetch,bing,bm25,web-scraper,crawler
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: DrissionPage
+Requires-Dist: beautifulsoup4
+Requires-Dist: lxml
+Requires-Dist: readability-lxml
+Requires-Dist: jieba
+Requires-Dist: rank-bm25
+
+# SkySearch
+
+基于 Bing 搜索 + BM25 排序的轻量级搜索引擎，支持动态页面抓取。
+
+## 特性
+
+- **Bing 搜索**：SessionPage 底层 TLS 指纹伪装，绕过反爬
+- **动态页面**：Chromium 多标签页并行抓取
+- **BM25 排序**：jieba 中文分词 + rank-bm25 相关性排序
+- **多模式输出**：`text` / `info` / `raw` 三种输出模式
+- **CLI 工具**：支持命令行参数，也可用作 Python 库
+
+## 安装
+
+```bash
+pip install skysearch
+```
+
+## 命令行使用
+
+### 搜索模式
+
+```bash
+# 交互式输入
+skysearch
+
+# 指定关键词
+skysearch "深度学习框架"
+
+# 指定结果数量
+skysearch -n 20 "Python教程"
+
+# 保持浏览器打开
+skysearch -n 20 "关键词" --keep
+```
+
+### URL 抓取模式
+
+```bash
+# 默认：输出纯文本
+skysearch --url https://example.com
+
+# 指定输出模式
+skysearch --url https://example.com --mode text   # 纯文本（默认）
+skysearch --url https://example.com --mode info    # 结构化信息
+skysearch --url https://example.com --mode raw     # 原始 HTML
+
+# 保持浏览器打开
+skysearch --url https://example.com --keep
+```
+
+## 作为库使用
+
+```python
+import skysearch
+
+# 搜索
+results = skysearch.search("深度学习", num=10)
+# [{'title': ..., 'url': ..., 'score': 12.5, 'snippet': ...}, ...]
+
+# URL 抓取
+text = skysearch.fetch("https://example.com")
+info = skysearch.fetch("https://example.com", mode='info')
+raw = skysearch.fetch("https://example.com", mode='raw')
+
+# 单独函数
+links = skysearch.fetch_links("https://example.com")
+info_dict = skysearch.fetch_info("https://example.com")
+raw_dict = skysearch.fetch_raw("https://example.com")
+
+# 搜索 + 抓取一体化
+results = skysearch.search_and_fetch("关键词", mode='info')
+```
+
+## API 参数说明
+
+### search(query, num=10, verbose=False, keep=False, tuple_format=False)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| query | 搜索关键词 | - |
+| num | 结果数量 | 10 |
+| verbose | 打印详细过程 | False |
+| keep | 保持浏览器打开 | False |
+| tuple_format | 返回元组格式 | False |
+
+### fetch(url, mode='text', keep=False, timeout=10, retry=2)
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| url | 页面 URL | - |
+| mode | 输出模式：`text` `info` `raw` | text |
+| keep | 保持浏览器打开 | False |
+| timeout | 请求超时秒数 | 10 |
+| retry | 重试次数 | 2 |
+
+### search_and_fetch(query, num=10, mode='text', verbose=False, keep=False, tuple_format=False)
+
+一体化搜索 + 抓取，返回列表包含结果信息和抓取内容。
+
+## 输出模式说明
+
+| 模式 | 说明 | 适用场景 |
+|------|------|----------|
+| `text` | 纯文本正文 | 人类阅读 |
+| `info` | 结构化 JSON（url, title, text, links, meta） | 数据分析 / agent |
+| `raw` | 原始 HTML | 深度解析 |
+
+## 技术栈
+
+| 模块 | 技术 |
+|------|------|
+| HTTP 请求 | DrissionPage (SessionPage) |
+| 动态渲染 | DrissionPage (ChromiumPage) |
+| HTML 解析 | BeautifulSoup4 + lxml |
+| 正文提取 | readability-lxml |
+| 中文分词 | jieba |
+| 排序算法 | rank-bm25 (BM25Okapi) |
+
+## 项目结构
+
+```
+src/skysearch/
+├── __init__.py       # 库入口，导出所有 API
+├── cli.py            # 命令行入口
+├── search.py         # Bing 搜索
+├── ranker.py         # BM25 排序
+├── api.py            # 简洁 API 接口
+└── fetcher/          # 页面抓取包
+    ├── __init__.py
+    ├── core.py       # 核心函数
+    ├── session.py    # 会话管理
+    └── parser.py     # HTML 解析
+```
+
+## License
+
+MIT

skysearch-0.2.0/src/skysearch.egg-info/SOURCES.txt

@@ -0,0 +1,17 @@
+README.md
+pyproject.toml
+src/skysearch/__init__.py
+src/skysearch/api.py
+src/skysearch/cli.py
+src/skysearch/ranker.py
+src/skysearch/search.py
+src/skysearch.egg-info/PKG-INFO
+src/skysearch.egg-info/SOURCES.txt
+src/skysearch.egg-info/dependency_links.txt
+src/skysearch.egg-info/entry_points.txt
+src/skysearch.egg-info/requires.txt
+src/skysearch.egg-info/top_level.txt
+src/skysearch/fetcher/__init__.py
+src/skysearch/fetcher/core.py
+src/skysearch/fetcher/parser.py
+src/skysearch/fetcher/session.py

skysearch-0.2.0/src/skysearch.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

skysearch-0.2.0/src/skysearch.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+skysearch = skysearch.cli:main

skysearch-0.2.0/src/skysearch.egg-info/requires.txt

@@ -0,0 +1,6 @@
+DrissionPage
+beautifulsoup4
+lxml
+readability-lxml
+jieba
+rank-bm25

skysearch-0.2.0/src/skysearch.egg-info/top_level.txt

@@ -0,0 +1 @@
+skysearch