@seanyao/roll 0.5.0

package/tools/roll-fetch/SKILL.md

@@ -0,0 +1,182 @@
+---
+hidden: true
+name: roll-fetch
+description: Web page fetching and crawling for AI agents. Extract content from URLs for research, documentation, and competitive analysis.
+---
+
+# Roll Fetch - Web Content Extraction
+
+Extract content from web pages for research and analysis.
+
+## When to Use
+
+- Product research (competitor analysis)
+- Technical documentation gathering
+- Code examples and best practices
+- Full site crawling for backup/analysis
+
+## Environment Setup
+
+Configure API keys per machine:
+
+```bash
+# Required for Tavily
+export TAVILY_API_KEY=tvly-dev-...
+
+# Optional for cloud browser fallback
+export BROWSER_USE_API_KEY=bu-...
+```
+
+Or create `.env` file in project root:
+```
+TAVILY_API_KEY=tvly-dev-...
+BROWSER_USE_API_KEY=bu-...
+```
+
+## Methods
+
+### 1. Tavily API (Recommended)
+
+Best quality extraction, requires `TAVILY_API_KEY`.
+
+```bash
+# Using Tavily CLI or API
+curl -X POST https://api.tavily.com/extract \
+  -H "Content-Type: application/json" \
+  -d '{
+    "urls": ["https://example.com"],
+    "api_key": "your_tavily_api_key"
+  }'
+```
+
+**Pros**: AI-optimized extraction, handles complex layouts
+**Cons**: Requires API key, rate limited
+
+### 2. LLM Native Fetch (Default)
+
+Use your built-in URL fetching capability directly.
+
+**When to use**: When Tavily is unavailable or for quick checks.
+
+**Note**: Most modern AI agents (Kimi, Codex, Claude) have native URL fetching. Use `FetchURL` tool or equivalent.
+
+### 3. Browser Automation (Fallback)
+
+Local browser automation for stubborn pages using **[browser-use](https://github.com/browser-use/browser-use)**.
+
+**How to Choose:**
+
+| If | Then Use | Why |
+|----|---------|-----|
+| `BROWSER_USE_API_KEY` in env | **Cloud** | Managed browsers, less setup |
+| No API key, but `browser-use` installed | **Local** | Free, no external dependency |
+| Neither | Skip to manual extraction | Tell user "Need browser automation setup" |
+
+**Option A: Local (Free, No API Key)**
+```python
+from browser_use import Agent, Browser, BrowserConfig
+import asyncio
+
+async def fetch_page(url):
+    # Pure local, no API key needed
+    browser = Browser(config=BrowserConfig(headless=True))
+    await browser.start()
+    page = await browser.get_current_page()
+    await page.goto(url)
+    content = await page.content()
+    await browser.stop()
+    return content
+
+# Run
+content = asyncio.run(fetch_page("https://example.com"))
+```
+
+**Option B: Cloud API**
+```python
+from browser_use import Agent
+
+agent = Agent(
+    task=f"Extract the main content from {url} and return as markdown",
+    llm="moonshot"  # or openai, anthropic
+)
+result = await agent.run()
+```
+
+**Setup** (Local):
+```bash
+pip install browser-use
+playwright install chromium
+```
+
+## Usage
+
+### CLI Usage (via smart-web-fetch.js)
+
+```bash
+# Auto mode (Tavily → Native → Browser)
+node smart-web-fetch.js fetch https://example.com
+
+# Explicit method
+node smart-web-fetch.js fetch https://example.com tavily
+node smart-web-fetch.js fetch https://example.com native
+node smart-web-fetch.js fetch https://example.com browser
+
+# Search
+node smart-web-fetch.js search "Python async" 5
+```
+
+### Programmatic Usage
+
+```javascript
+const { smartFetch, smartSearch } = require('./smart-web-fetch.js');
+
+// Fetch a page
+const result = await smartFetch('https://example.com');
+console.log(result.content);
+
+// Search
+const searchResult = await smartSearch('OpenAI GPT-5', 5);
+console.log(searchResult.results);
+```
+
+### Single Page Fetch
+
+```
+User: "Fetch https://docs.example.com/api"
+→ Use smart-web-fetch.js with auto mode
+→ Return clean markdown content
+```
+
+### Full Site Crawl
+
+```
+User: "Crawl https://docs.example.com"
+→ Use smart-web-fetch.js recursively
+→ Extract all internal links
+→ Recursively fetch up to max depth (default: 2)
+→ Save each page as separate markdown file
+```
+
+## Output Format
+
+Always return clean Markdown:
+- Extract main content only (remove nav, ads, footers)
+- Preserve code blocks and tables
+- Include source URL as header
+
+## Quality Check
+
+Validate extracted content:
+- Min length: 500 chars (reject if shorter)
+- Check for captcha/error messages
+- Verify main content structure (headings, paragraphs)
+
+## Examples
+
+| Task | Method | Command |
+|------|--------|---------|
+| Quick article | Auto | `node smart-web-fetch.js fetch https://blog.example.com` |
+| API docs | Tavily | `node smart-web-fetch.js fetch https://docs.example.com tavily` |
+| SPA site | Browser | `node smart-web-fetch.js fetch https://spa.example.com browser` |
+| Search | Tavily | `node smart-web-fetch.js search "Python async" 5` |
+| Fallback test | Native | `node smart-web-fetch.js fetch https://example.com native` |

package/tools/roll-fetch/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "smart-web-fetch",
+  "version": "1.0.0",
+  "description": "Intelligent web fetching with automatic Tavily → Scrapling fallback",
+  "main": "smart-web-fetch.js",
+  "bin": {
+    "smart-web-fetch": "./smart-web-fetch.js"
+  },
+  "scripts": {
+    "test": "node smart-web-fetch.js fetch https://example.com"
+  },
+  "keywords": ["web-scraping", "tavily", "scrapling", "fallback"],
+  "author": "R0_lobster",
+  "license": "MIT"
+}