sophhub 0.1.0

package/skills/builtin/web-scraper/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: web-scraper
+description: Scrape web pages on the server side. Supports static HTTP scraping (curl) and dynamic JavaScript rendering (Chrome print-to-pdf + PyMuPDF). Zero extra pip dependencies. Use when web_fetch returns empty or minimal content.
+metadata: { "openclaw": { "emoji": "🕸️", "requires": { "bins": ["python3", "curl"] } } }
+---
+
+# Web Scraper
+
+Server-side web page scraping with two modes. **No extra pip packages needed** -- uses only curl, Chrome, and PyMuPDF (already in the Docker image).
+
+## When to Use
+
+- `web_fetch` returned empty or very short content (common with SPA/JS-heavy sites)
+- The user explicitly asks to scrape or crawl a URL
+
+## Quick Start
+
+```bash
+# Static scrape (fast, curl + text extraction, works for most pages)
+python3 {baseDir}/scripts/scrape.py "https://example.com"
+
+# Dynamic scrape (Chrome renders JS, saves PDF, PyMuPDF extracts text)
+python3 {baseDir}/scripts/scrape.py "https://www.bitmain.com/" --mode dynamic
+```
+
+## Options
+
+| Option                   | Default  | Description                                                                        |
+| ------------------------ | -------- | ---------------------------------------------------------------------------------- |
+| `--mode static\|dynamic` | `static` | `static`: curl fetch + regex text extract. `dynamic`: Chrome PDF render + PyMuPDF. |
+| `--timeout <seconds>`    | `15`     | Request/render timeout                                                             |
+| `--max-chars <n>`        | `50000`  | Truncate output beyond this length                                                 |
+
+## Decision Logic
+
+1. **Try static mode first** (fast, < 2 seconds).
+2. If the result `text` is very short (< 200 characters) or empty, the page likely needs JavaScript.
+3. **Retry with `--mode dynamic`** to render JavaScript via headless Chrome.
+4. If dynamic mode fails (no Chrome), report the error to the user.
+
+## Output Format
+
+JSON to stdout:
+
+```json
+{
+  "status": "ok",
+  "mode": "dynamic",
+  "url": "https://www.bitmain.com/",
+  "title": "BITMAIN",
+  "text": "BITMAIN\nProducts\nBitcoin Miner S23 Hyd...",
+  "length": 1779
+}
+```
+
+## Modes Explained
+
+### Static Mode (`--mode static`)
+
+- Uses `curl` to fetch raw HTML
+- Regex-based tag stripping for text extraction
+- Fast (< 2 seconds), no Python dependencies needed
+- Does NOT execute JavaScript
+- Good for: documentation, blogs, news, static content
+
+### Dynamic Mode (`--mode dynamic`)
+
+- Chrome `--headless=new --print-to-pdf` renders the page including all JavaScript
+- PyMuPDF (`fitz`) extracts text from the resulting PDF
+- Slower (15-40 seconds) but captures dynamically generated content
+- Requires `google-chrome` or `chromium` on PATH
+- Good for: SPA (React/Vue/Angular), JS-rendered dashboards
+
+## Examples
+
+```bash
+# Basic static scrape
+python3 {baseDir}/scripts/scrape.py "https://docs.python.org/3/tutorial/index.html"
+
+# SPA page (needs JavaScript rendering)
+python3 {baseDir}/scripts/scrape.py "https://www.bitmain.com/" --mode dynamic
+
+# Longer timeout for slow pages
+python3 {baseDir}/scripts/scrape.py "https://slow-site.example.com" --mode dynamic --timeout 30
+```
+
+## Dependencies
+
+| Tool                         | Required For             | Already Installed      |
+| ---------------------------- | ------------------------ | ---------------------- |
+| `python3`                    | Script runtime           | Yes (Docker image)     |
+| `curl`                       | Static mode              | Yes (Docker image)     |
+| `google-chrome` / `chromium` | Dynamic mode only        | Depends on image       |
+| `PyMuPDF` (fitz)             | Dynamic mode PDF reading | Yes (requirements.txt) |
+| `pdfminer`                   | Fallback PDF reading     | Yes (requirements.txt) |
+
+## Security
+
+- Only `http://` and `https://` URLs are allowed
+- Requests to `localhost`, private IPs, and reserved addresses are blocked (SSRF protection)
+- Output is truncated at `--max-chars` to prevent excessive memory usage

package/skills/builtin/web-scraper/scripts/scrape.py

@@ -0,0 +1,270 @@
+#!/usr/bin/env python3
+"""
+Web scraper via headless Chrome print-to-pdf + PyMuPDF text extraction.
+No requests/beautifulsoup4 needed -- only uses packages already in the Docker image.
+Outputs JSON to stdout for agent consumption.
+"""
+
+import argparse
+import ipaddress
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+from typing import Dict, Optional
+from urllib.parse import urlparse
+
+# ---------------------------------------------------------------------------
+# SSRF guard
+# ---------------------------------------------------------------------------
+
+BLOCKED_HOSTS = {"localhost", "127.0.0.1", "0.0.0.0", "[::1]", "metadata.google.internal"}
+
+
+def is_private_ip(hostname):
+    # type: (str) -> bool
+    try:
+        addr = ipaddress.ip_address(hostname)
+        return addr.is_private or addr.is_loopback or addr.is_link_local or addr.is_reserved
+    except ValueError:
+        return False
+
+
+def validate_url(url):
+    # type: (str) -> Optional[str]
+    parsed = urlparse(url)
+    if parsed.scheme not in ("http", "https"):
+        return "Unsupported scheme: %s. Only http/https allowed." % parsed.scheme
+    hostname = parsed.hostname or ""
+    if hostname in BLOCKED_HOSTS:
+        return "Blocked host: %s" % hostname
+    if is_private_ip(hostname):
+        return "Private/reserved IP blocked: %s" % hostname
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Chrome binary detection
+# ---------------------------------------------------------------------------
+
+CHROME_CANDIDATES = ["google-chrome", "google-chrome-stable", "chromium", "chromium-browser"]
+
+
+def find_chrome():
+    # type: () -> Optional[str]
+    for name in CHROME_CANDIDATES:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Static scrape via curl (no extra deps)
+# ---------------------------------------------------------------------------
+
+def scrape_static(url, timeout):
+    # type: (str, int) -> Dict
+    """Lightweight fetch using curl + basic text extraction."""
+    curl = shutil.which("curl")
+    if not curl:
+        return {"status": "error", "mode": "static", "url": url,
+                "error": "curl not found on PATH"}
+
+    try:
+        result = subprocess.run(
+            [curl, "-sL", "--max-time", str(timeout),
+             "-H", "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/137.0",
+             "-H", "Accept: text/html,*/*;q=0.8",
+             url],
+            capture_output=True, text=True, timeout=timeout + 5,
+        )
+    except subprocess.TimeoutExpired:
+        return {"status": "error", "mode": "static", "url": url,
+                "error": "curl timed out after %ds" % timeout}
+
+    if result.returncode != 0:
+        return {"status": "error", "mode": "static", "url": url,
+                "error": "curl failed (code %d): %s" % (result.returncode, result.stderr[:300])}
+
+    html = result.stdout
+    text = _extract_text_from_html(html)
+    title = _extract_title_from_html(html)
+
+    return {
+        "status": "ok",
+        "mode": "static",
+        "url": url,
+        "title": title,
+        "text": text,
+        "length": len(text),
+    }
+
+
+def _extract_title_from_html(html):
+    # type: (str) -> str
+    """Extract <title> content without bs4."""
+    import re
+    m = re.search(r"<title[^>]*>(.*?)</title>", html, re.IGNORECASE | re.DOTALL)
+    return m.group(1).strip() if m else ""
+
+
+def _extract_text_from_html(html):
+    # type: (str) -> str
+    """Rough HTML-to-text without bs4: strip tags, collapse whitespace."""
+    import re
+    text = re.sub(r"<script[^>]*>.*?</script>", "", html, flags=re.DOTALL | re.IGNORECASE)
+    text = re.sub(r"<style[^>]*>.*?</style>", "", text, flags=re.DOTALL | re.IGNORECASE)
+    text = re.sub(r"<[^>]+>", "\n", text)
+    text = re.sub(r"&nbsp;", " ", text)
+    text = re.sub(r"&amp;", "&", text)
+    text = re.sub(r"&lt;", "<", text)
+    text = re.sub(r"&gt;", ">", text)
+    text = re.sub(r"&#\d+;", "", text)
+    text = re.sub(r"&\w+;", "", text)
+    lines = [line.strip() for line in text.split("\n") if line.strip()]
+    return "\n".join(lines)
+
+
+# ---------------------------------------------------------------------------
+# Dynamic scrape: Chrome --print-to-pdf + PyMuPDF
+# ---------------------------------------------------------------------------
+
+def scrape_dynamic(url, timeout):
+    # type: (str, int) -> Dict
+    chrome = find_chrome()
+    if not chrome:
+        return {
+            "status": "error", "mode": "dynamic", "url": url,
+            "error": "No Chrome/Chromium found. Searched: %s. Use --mode static instead."
+                     % ", ".join(CHROME_CANDIDATES),
+        }
+
+    tmpdir = tempfile.mkdtemp(prefix="web-scraper-")
+    pdf_path = os.path.join(tmpdir, "page.pdf")
+
+    try:
+        js_budget_ms = max(5000, timeout * 1000)
+        cmd = [
+            chrome,
+            "--headless=new",
+            "--print-to-pdf=" + pdf_path,
+            "--no-sandbox",
+            "--disable-gpu",
+            "--disable-software-rasterizer",
+            "--disable-dev-shm-usage",
+            "--no-pdf-header-footer",
+            "--virtual-time-budget=%d" % js_budget_ms,
+            url,
+        ]
+
+        try:
+            subprocess.run(
+                cmd, capture_output=True, text=True,
+                timeout=timeout + 30,
+            )
+        except subprocess.TimeoutExpired:
+            return {"status": "error", "mode": "dynamic", "url": url,
+                    "error": "Chrome timed out after %ds" % timeout}
+
+        if not os.path.exists(pdf_path) or os.path.getsize(pdf_path) == 0:
+            return {"status": "error", "mode": "dynamic", "url": url,
+                    "error": "Chrome did not produce a PDF file"}
+
+        text = _extract_text_from_pdf(pdf_path)
+        title = _guess_title(text)
+
+        return {
+            "status": "ok",
+            "mode": "dynamic",
+            "url": url,
+            "title": title,
+            "text": text,
+            "length": len(text),
+        }
+    finally:
+        try:
+            shutil.rmtree(tmpdir, ignore_errors=True)
+        except Exception:
+            pass
+
+
+def _extract_text_from_pdf(pdf_path):
+    # type: (str) -> str
+    """Extract text from PDF using PyMuPDF (already in requirements.txt)."""
+    try:
+        import fitz  # PyMuPDF
+    except ImportError:
+        return _extract_text_from_pdf_fallback(pdf_path)
+
+    doc = fitz.open(pdf_path)
+    pages = []
+    for page in doc:
+        pages.append(page.get_text())
+    doc.close()
+    text = "\n".join(pages).strip()
+    lines = [line.strip() for line in text.split("\n") if line.strip()]
+    return "\n".join(lines)
+
+
+def _extract_text_from_pdf_fallback(pdf_path):
+    # type: (str) -> str
+    """Fallback: try pdfminer if PyMuPDF is not available."""
+    try:
+        from pdfminer.high_level import extract_text
+        return extract_text(pdf_path).strip()
+    except ImportError:
+        return "[error] Neither PyMuPDF nor pdfminer available. Install with: pip3 install --break-system-packages PyMuPDF"
+
+
+def _guess_title(text):
+    # type: (str) -> str
+    """Use the first non-empty line as title."""
+    for line in text.split("\n"):
+        stripped = line.strip()
+        if stripped and len(stripped) < 200:
+            return stripped
+    return ""
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main():
+    parser = argparse.ArgumentParser(description="Web scraper for OpenClaw agents")
+    parser.add_argument("url", help="URL to scrape (http/https only)")
+    parser.add_argument("--mode", choices=["static", "dynamic"], default="static",
+                        help="static: curl + text extract; dynamic: Chrome PDF + PyMuPDF (default: static)")
+    parser.add_argument("--timeout", type=int, default=15,
+                        help="Request timeout in seconds (default: 15)")
+    parser.add_argument("--max-chars", type=int, default=50000,
+                        help="Max output characters (default: 50000)")
+    args = parser.parse_args()
+
+    err = validate_url(args.url)
+    if err:
+        print(json.dumps({"status": "error", "url": args.url, "error": err}, ensure_ascii=False))
+        sys.exit(1)
+
+    try:
+        if args.mode == "static":
+            result = scrape_static(args.url, args.timeout)
+        else:
+            result = scrape_dynamic(args.url, args.timeout)
+    except Exception as e:
+        result = {"status": "error", "url": args.url, "mode": args.mode,
+                  "error": "%s: %s" % (type(e).__name__, e)}
+
+    if result.get("status") == "ok" and len(result.get("text", "")) > args.max_chars:
+        result["text"] = result["text"][:args.max_chars] + "\n\n... [truncated]"
+        result["truncated"] = True
+        result["length"] = len(result["text"])
+
+    print(json.dumps(result, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

package/skills/builtin/website-builder/SKILL.md

@@ -0,0 +1,266 @@
+---
+name: website-builder
+description: >
+  一键建站助手：根据用户需求生成完整网站（HTML/CSS/JS），自动部署到 moltbot 内置静态托管，
+  立即给出可公开访问的链接（http://<host>:18789/canvas/<site-name>/）。
+  支持：上传截图/UI图进行像素级复刻、多轮自然语言修改、响应式设计、多页面网站。
+  触发场景：用户说"帮我做一个网站"、"做一个landing page"、"做一个作品集"、"复刻这个界面"、
+  "帮我建一个xxx网站"等。
+metadata: { "openclaw": { "emoji": "🌐", "requires": { "bins": ["node"] } } }
+---
+
+## 加载时说明
+
+当用户启用此 skill 时，发送以下欢迎语：
+
+> 🌐 **建站助手已就绪！**
+>
+> 我可以帮你：
+>
+> - 用一句话描述需求，自动生成并部署网站
+> - 上传截图 / UI 图，像素级复刻
+> - 多轮对话实时修改，所见即所得
+>
+> 生成后你会收到一个可直接访问的链接，无需额外配置。
+>
+> 你想做什么样的网站？
+
+---
+
+# 建站助手 (Website Builder)
+
+根据需求生成完整网站，部署到 moltbot gateway 的 `/canvas/` 静态路由，立即可访问。
+
+---
+
+## 一、访问路径说明
+
+网站文件存放在：
+
+```
+<workspace>/canvas/<site-name>/index.html
+```
+
+访问地址：
+
+```
+http://<moltbot-host>:18789/canvas/<site-name>/
+```
+
+其中 `<moltbot-host>` 是运行 moltbot 的服务器 IP 或域名（如 `192.168.1.100`）。
+
+---
+
+## 二、工作流程
+
+### Step 1：理解需求，输出确认卡片
+
+收到建站需求后，先展示理解摘要让用户确认，**不要直接开始写代码**：
+
+```
+🌐 建站方案确认
+
+- 网站类型：个人作品集
+- 风格：极简黑白
+- 页面：首页、作品、关于、联系
+- 技术栈：纯 HTML + CSS + JS（无需构建）
+- 站点名称（URL路径）：portfolio
+- 访问地址：http://<your-ip>:18789/canvas/portfolio/
+
+确认开始生成？
+```
+
+### Step 2：生成网站文件
+
+用户确认后，调用部署脚本：
+
+```bash
+bash /app/skills/website-builder/scripts/deploy_site.sh \
+  --name "<site-name>" \
+  --workspace "<workspace-dir>"
+```
+
+脚本负责：
+
+1. 创建 `<workspace>/canvas/<site-name>/` 目录
+2. Agent 将生成的 HTML/CSS/JS 文件写入该目录
+3. 输出访问链接
+
+#### 生成代码的要求
+
+- **单文件优先**：能写在一个 `index.html` 里就不拆分（内联 CSS + JS），更易修改
+- **多页面**：每个页面一个 HTML 文件（`about.html`、`works.html` 等），共享一个 `style.css`
+- **响应式**：必须适配手机和桌面
+- **无外部依赖**：不引用 CDN，所有样式和脚本内联或本地，确保离线可用
+- **现代设计**：使用 CSS Grid / Flexbox，避免过时的表格布局
+
+### Step 3：写入文件
+
+使用 exec 工具直接写文件到 canvas 目录：
+
+```python
+# 写入 index.html
+with open('<workspace>/canvas/<site-name>/index.html', 'w') as f:
+    f.write('''...生成的HTML内容...''')
+```
+
+### Step 4：输出访问链接
+
+```
+✅ 网站已部署！
+
+🔗 访问链接：http://<your-ip>:18789/canvas/<site-name>/
+
+📁 文件位置：<workspace>/canvas/<site-name>/
+   ├── index.html
+   ├── about.html（如有）
+   └── style.css（如有）
+
+需要修改？直接告诉我（如"把导航栏改成深色"、"加一个联系表单"）。
+```
+
+---
+
+## 三、获取 moltbot 访问地址
+
+Agent 需要告知用户正确的访问地址。通过以下方式获取 host：
+
+```bash
+# 获取容器内可用的网络地址
+hostname -I 2>/dev/null | awk '{print $1}'
+```
+
+或直接提示用户：
+
+> 请将 `<your-ip>` 替换为你访问 moltbot 的 IP 地址（就是你打开 moltbot 界面时用的那个 IP）。
+
+---
+
+## 四、部署脚本
+
+```bash
+bash /app/skills/website-builder/scripts/deploy_site.sh --name <site-name> --workspace <workspace-dir>
+```
+
+脚本仅负责创建目录，文件由 agent 用 exec 写入。
+
+---
+
+## 五、多轮修改流程
+
+用户发出修改指令后：
+
+1. 读取现有文件：`cat <workspace>/canvas/<site-name>/index.html`
+2. 按需修改代码
+3. 重新写入文件
+4. 告知用户刷新浏览器即可看到效果（支持 live reload 如已开启）
+
+**常见修改示例：**
+
+| 用户说                 | Agent 动作                         |
+| ---------------------- | ---------------------------------- |
+| "把导航栏改成深色背景" | 修改 CSS `.navbar` 背景色          |
+| "按钮换成蓝色渐变"     | 修改 button 样式                   |
+| "加一个联系我表单"     | 新增 form 区块 + 样式              |
+| "首页加个轮播图"       | 添加 JS 轮播逻辑                   |
+| "字体换成更现代的"     | 引入 Google Fonts 或换用系统字体栈 |
+| "把这个部分删掉"       | 删除对应 HTML 区块                 |
+
+---
+
+## 六、上传截图复刻流程
+
+用户上传网页截图或 UI 图时：
+
+1. **读取图片描述**：系统会在消息上下文中提供图片的文字描述（`[Image] Description: ...`）
+2. **提取设计元素**：从描述中理解布局、配色、字体、组件结构
+3. **生成复刻代码**：按理解的设计还原 HTML/CSS
+4. **展示确认**：告知用户已识别的设计要素，确认后部署
+
+若需要更精确的 OCR 识别，可配合 `sophnet-image-ocr` skill 提取图片中的文字内容。
+
+---
+
+## 七、站点命名规则
+
+站点名称（`<site-name>`）用于 URL 路径，规则：
+
+- 只含小写字母、数字、连字符
+- 不含空格和特殊字符
+- 示例：`portfolio`、`my-blog`、`landing-2026`
+
+若用户未指定，Agent 根据需求自动生成一个合适的名称并告知用户。
+
+---
+
+## 八、管理已部署站点
+
+```bash
+# 查看所有已部署站点
+ls <workspace>/canvas/
+
+# 删除某个站点
+rm -rf <workspace>/canvas/<site-name>/
+
+# 更新站点（直接覆盖文件即可）
+```
+
+---
+
+## 九、快速建站示例
+
+### 示例 A：个人作品集
+
+**用户**：帮我做一个极简风格的个人作品集网站，包含首页、作品、关于我三个页面
+
+**Bot**：
+
+```
+🌐 建站方案确认
+
+- 类型：个人作品集
+- 风格：极简（白底、细线、大量留白）
+- 页面：首页(index.html)、作品(works.html)、关于(about.html)
+- 站点名称：portfolio
+- 访问地址：http://<your-ip>:18789/canvas/portfolio/
+
+确认开始？
+```
+
+### 示例 B：产品 Landing Page
+
+**用户**：帮我做一个卖运动鞋的 landing page，苹果官网风格
+
+**Bot**：
+
+```
+🌐 建站方案确认
+
+- 类型：产品 Landing Page
+- 风格：极简（深色背景、全屏大图、苹果式排版）
+- 内容：Hero区、产品特性、图片展示、购买CTA
+- 站点名称：sneaker-landing
+- 访问地址：http://<your-ip>:18789/canvas/sneaker-landing/
+
+确认开始？
+```
+
+### 示例 C：复刻截图
+
+**用户**：（上传网页截图）帮我复刻这个界面
+
+**Bot**：
+
+```
+📸 识别到以下设计要素：
+
+- 布局：顶部导航 + Hero大图 + 三栏特性介绍 + 底部
+- 配色：深蓝主色 (#1a237e)，白色文字，橙色强调色
+- 字体：无衬线，标题大号加粗
+- 组件：固定导航栏、全屏背景图、卡片组件
+
+站点名称：cloned-site
+访问地址：http://<your-ip>:18789/canvas/cloned-site/
+
+开始复刻？
+```

package/skills/builtin/website-builder/scripts/deploy_site.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+#
+# deploy_site.sh - Create canvas directory for a website
+#
+# Usage:
+#   bash deploy_site.sh --name <site-name> --workspace <workspace-dir>
+#
+# Outputs:
+#   SITE_DIR=<absolute path to site directory>
+#   SITE_URL_PATH=/canvas/<site-name>/
+#
+
+set -euo pipefail
+
+SITE_NAME=""
+WORKSPACE_DIR=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --name) SITE_NAME="$2"; shift 2 ;;
+    --workspace) WORKSPACE_DIR="$2"; shift 2 ;;
+    *) echo "Unknown option: $1" >&2; exit 1 ;;
+  esac
+done
+
+if [[ -z "$SITE_NAME" ]]; then
+  echo "Error: --name is required" >&2
+  exit 1
+fi
+
+# Sanitize site name
+SITE_NAME="$(echo "$SITE_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//')"
+
+if [[ -z "$WORKSPACE_DIR" ]]; then
+  # Try to infer from environment
+  STATE_DIR="${HOME}/.openclaw"
+  WORKSPACE_DIR="${STATE_DIR}/workspace"
+fi
+
+CANVAS_ROOT="${WORKSPACE_DIR}/canvas"
+SITE_DIR="${CANVAS_ROOT}/${SITE_NAME}"
+
+mkdir -p "$SITE_DIR"
+
+echo "SITE_DIR=${SITE_DIR}"
+echo "SITE_URL_PATH=/canvas/${SITE_NAME}/"