aidevops 2.52.1 → 2.53.2

package/.agent/tools/browser/crawl4ai-usage.md

@@ -1,416 +0,0 @@
----
-description: Crawl4AI usage patterns and best practices
-mode: subagent
-tools:
-  read: true
-  write: false
-  edit: false
-  bash: true
-  glob: true
-  grep: true
-  webfetch: true
-  task: true
----
-
-# Crawl4AI Usage Guide for AI Assistants
-
-<!-- AI-CONTEXT-START -->
-
-## Quick Reference
-
-- **Helper**: `.agent/scripts/crawl4ai-helper.sh`
-- **API Port**: `localhost:11235`
-- **Commands**: `install | docker-setup | docker-start | status | crawl | extract | mcp-setup`
-- **Crawl**: `./crawl4ai-helper.sh crawl URL markdown output.json`
-- **Extract**: `./crawl4ai-helper.sh extract URL '{"title":"h1"}' data.json`
-- **MCP Tools**: `crawl_url | crawl_multiple | extract_structured | take_screenshot | generate_pdf`
-- **Dashboard**: `http://localhost:11235/dashboard`
-- **Playground**: `http://localhost:11235/playground`
-- **Output**: JSON with markdown, html, extracted_content, links, media, metadata
-- **Process results**: `jq -r '.results[0].markdown' output.json`
-<!-- AI-CONTEXT-END -->
-
-## Purpose
-
-This guide provides AI assistants with comprehensive instructions for using Crawl4AI within the AI DevOps Framework for web crawling, data extraction, and content processing tasks.
-
-## Quick Start Commands
-
-### Basic Setup
-
-```bash
-# Install Crawl4AI
-./.agent/scripts/crawl4ai-helper.sh install
-
-# Setup Docker deployment
-./.agent/scripts/crawl4ai-helper.sh docker-setup
-
-# Start services
-./.agent/scripts/crawl4ai-helper.sh docker-start
-
-# Check status
-./.agent/scripts/crawl4ai-helper.sh status
-```
-
-### MCP Integration
-
-```bash
-# Setup MCP server for AI assistants
-./.agent/scripts/crawl4ai-helper.sh mcp-setup
-```
-
-## Core Operations
-
-### 1. Web Crawling
-
-```bash
-# Basic crawling - extract markdown
-./.agent/scripts/crawl4ai-helper.sh crawl https://example.com markdown output.json
-
-# Crawl with specific format
-./.agent/scripts/crawl4ai-helper.sh crawl https://news.com html news.json
-
-# Save to file
-./.agent/scripts/crawl4ai-helper.sh crawl https://docs.com markdown ~/Downloads/docs.json
-```
-
-### 2. Structured Data Extraction
-
-```bash
-# Extract with CSS selectors
-./.agent/scripts/crawl4ai-helper.sh extract https://example.com '{"title":"h1","content":".article"}' data.json
-
-# Complex schema extraction
-./.agent/scripts/crawl4ai-helper.sh extract https://ecommerce.com '{
-  "products": {
-    "selector": ".product",
-    "fields": [
-      {"name": "title", "selector": "h2", "type": "text"},
-      {"name": "price", "selector": ".price", "type": "text"},
-      {"name": "image", "selector": "img", "type": "attribute", "attribute": "src"}
-    ]
-  }
-}' products.json
-```
-
-## AI Assistant Integration Patterns
-
-### For Claude Desktop
-
-1. **Setup MCP Configuration**:
-
-   ```json
-   {
-     "mcpServers": {
-       "crawl4ai": {
-         "command": "npx",
-         "args": ["crawl4ai-mcp-server@latest"]
-       }
-     }
-   }
-   ```
-
-2. **Available Tools**:
-   - `crawl_url`: Single URL crawling
-   - `crawl_multiple`: Batch URL processing
-   - `extract_structured`: Data extraction
-   - `take_screenshot`: Page screenshots
-   - `generate_pdf`: PDF conversion
-
-### For Other AI Assistants
-
-Use the REST API directly:
-
-```python
-import requests
-
-# Basic crawl
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://example.com"],
-    "crawler_config": {
-        "type": "CrawlerRunConfig",
-        "params": {"cache_mode": "bypass"}
-    }
-})
-
-# Extract structured data
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://example.com"],
-    "crawler_config": {
-        "type": "CrawlerRunConfig",
-        "params": {
-            "extraction_strategy": {
-                "type": "JsonCssExtractionStrategy",
-                "params": {
-                    "schema": {
-                        "type": "dict",
-                        "value": {"title": "h1", "content": ".article"}
-                    }
-                }
-            }
-        }
-    }
-})
-```
-
-## Common Use Cases
-
-### 1. Content Research
-
-```bash
-# Research articles
-./.agent/scripts/crawl4ai-helper.sh crawl https://research-site.com markdown research.json
-
-# Extract key information
-./.agent/scripts/crawl4ai-helper.sh extract https://paper.com '{
-  "title": "h1",
-  "authors": ".authors",
-  "abstract": ".abstract",
-  "keywords": ".keywords"
-}' paper-data.json
-```
-
-### 2. News Aggregation
-
-```bash
-# Multiple news sources
-for url in "https://news1.com" "https://news2.com" "https://news3.com"; do
-    ./.agent/scripts/crawl4ai-helper.sh crawl "$url" markdown "news-$(basename $url).json"
-done
-```
-
-### 3. E-commerce Data
-
-```bash
-# Product information
-./.agent/scripts/crawl4ai-helper.sh extract https://shop.com/product '{
-  "name": "h1.product-title",
-  "price": ".price-current",
-  "description": ".product-description",
-  "specs": {
-    "selector": ".specs tr",
-    "fields": [
-      {"name": "feature", "selector": "td:first-child", "type": "text"},
-      {"name": "value", "selector": "td:last-child", "type": "text"}
-    ]
-  }
-}' product.json
-```
-
-### 4. Documentation Processing
-
-```bash
-# API documentation
-./.agent/scripts/crawl4ai-helper.sh extract https://api-docs.com '{
-  "endpoints": {
-    "selector": ".endpoint",
-    "fields": [
-      {"name": "method", "selector": ".method", "type": "text"},
-      {"name": "path", "selector": ".path", "type": "text"},
-      {"name": "description", "selector": ".description", "type": "text"},
-      {"name": "parameters", "selector": ".params", "type": "html"}
-    ]
-  }
-}' api-docs.json
-```
-
-## Advanced Workflows
-
-### Batch Processing
-
-```bash
-#!/bin/bash
-# Process multiple URLs with different strategies
-
-urls=(
-    "https://news.com"
-    "https://blog.com" 
-    "https://docs.com"
-)
-
-for url in "${urls[@]}"; do
-    echo "Processing: $url"
-    ./.agent/scripts/crawl4ai-helper.sh crawl "$url" markdown "output-$(date +%s).json"
-    sleep 2  # Rate limiting
-done
-```
-
-### Content Analysis Pipeline
-
-```bash
-#!/bin/bash
-# Complete content analysis workflow
-
-URL="https://example.com"
-TIMESTAMP=$(date +%Y%m%d_%H%M%S)
-
-# 1. Basic crawl
-./.agent/scripts/crawl4ai-helper.sh crawl "$URL" markdown "raw-$TIMESTAMP.json"
-
-# 2. Extract structured data
-./.agent/scripts/crawl4ai-helper.sh extract "$URL" '{
-  "title": "h1",
-  "headings": "h2, h3",
-  "links": {"selector": "a", "type": "attribute", "attribute": "href"},
-  "images": {"selector": "img", "type": "attribute", "attribute": "src"}
-}' "structured-$TIMESTAMP.json"
-
-echo "Analysis complete: raw-$TIMESTAMP.json and structured-$TIMESTAMP.json"
-```
-
-## Configuration Best Practices
-
-### Environment Setup
-
-```bash
-# Create dedicated environment file
-cat > ~/.aidevops/.agent-workspace/tmp/crawl4ai.env << EOF
-OPENAI_API_KEY=your-key-here
-LLM_PROVIDER=openai/gpt-4o-mini
-LLM_TEMPERATURE=0.7
-CRAWL4AI_MAX_PAGES=50
-CRAWL4AI_TIMEOUT=60
-EOF
-```
-
-### Performance Optimization
-
-```bash
-# For high-volume crawling
-export CRAWL4AI_CONCURRENT_REQUESTS=5
-export CRAWL4AI_BROWSER_POOL_SIZE=3
-export CRAWL4AI_MEMORY_THRESHOLD=90
-```
-
-## Monitoring & Debugging
-
-### Status Checks
-
-```bash
-# Comprehensive status
-./.agent/scripts/crawl4ai-helper.sh status
-
-# Docker container status
-docker ps | grep crawl4ai
-
-# API health
-curl -s http://localhost:11235/health | jq '.'
-
-# Metrics
-curl -s http://localhost:11235/metrics
-```
-
-### Dashboard Access
-
-- **Monitoring Dashboard**: http://localhost:11235/dashboard
-- **Interactive Playground**: http://localhost:11235/playground
-- **API Documentation**: http://localhost:11235/schema
-
-### Troubleshooting
-
-```bash
-# Container logs
-docker logs crawl4ai --tail 50
-
-# Restart services
-./.agent/scripts/crawl4ai-helper.sh docker-stop
-./.agent/scripts/crawl4ai-helper.sh docker-start
-
-# Test basic functionality
-curl -X POST http://localhost:11235/crawl \
-  -H "Content-Type: application/json" \
-  -d '{"urls": ["https://httpbin.org/html"]}'
-```
-
-## Output Processing
-
-### JSON Response Structure
-
-```json
-{
-  "success": true,
-  "results": [
-    {
-      "url": "https://example.com",
-      "success": true,
-      "markdown": "# Page Title\n\nContent...",
-      "html": "<html>...</html>",
-      "extracted_content": {...},
-      "links": {...},
-      "media": {...},
-      "metadata": {...}
-    }
-  ]
-}
-```
-
-### Processing Results
-
-```bash
-# Extract just the markdown
-jq -r '.results[0].markdown' output.json > content.md
-
-# Get extracted data
-jq '.results[0].extracted_content' output.json > data.json
-
-# List all links
-jq -r '.results[0].links.internal[]' output.json
-```
-
-## Security Considerations
-
-### Safe Crawling Practices
-
-1. **Respect robots.txt**: Always enabled by default
-2. **Rate limiting**: Built-in delays between requests
-3. **User agent**: Identifies as Crawl4AI
-4. **Timeout protection**: Prevents hanging requests
-
-### Data Privacy
-
-```bash
-# Use cache mode for repeated requests
-./.agent/scripts/crawl4ai-helper.sh crawl https://example.com markdown output.json
-
-# Clear cache when needed
-docker exec crawl4ai redis-cli FLUSHALL
-```
-
-## Integration Tips
-
-### With Other Framework Tools
-
-```bash
-# Combine with quality tools
-./.agent/scripts/crawl4ai-helper.sh crawl https://docs.com markdown docs.json
-cat docs.json | jq -r '.results[0].markdown' | ./.agent/scripts/pandoc-helper.sh convert - pdf docs.pdf
-```
-
-### With AI Workflows
-
-```bash
-# Extract content for AI processing
-./.agent/scripts/crawl4ai-helper.sh crawl https://article.com markdown article.json
-CONTENT=$(jq -r '.results[0].markdown' article.json)
-echo "$CONTENT" | # Process with your AI pipeline
-```
-
-## Resources
-
-- **Helper Script**: `.agent/scripts/crawl4ai-helper.sh`
-- **Configuration**: `configs/crawl4ai-config.json.txt`
-- **MCP Setup**: `configs/mcp-templates/crawl4ai-mcp-config.json`
-- **Integration Guide**: `.agent/wiki/crawl4ai-integration.md`
-- **Official Docs**: https://docs.crawl4ai.com/
-
-## Success Checklist
-
-- [ ] Crawl4AI installed and running
-- [ ] Docker container started successfully
-- [ ] MCP integration configured
-- [ ] Basic crawling test completed
-- [ ] Structured extraction working
-- [ ] Dashboard accessible
-- [ ] API endpoints responding
-
-Use this guide to effectively leverage Crawl4AI's powerful web crawling and data extraction capabilities within your AI workflows.