aidevops 2.52.1

package/.agent/tools/browser/crawl4ai-integration.md

@@ -0,0 +1,422 @@
+---
+description: Crawl4AI MCP server integration setup
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Crawl4AI Integration Guide
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- Crawl4AI: #1 trending open-source web crawler for AI/LLM applications
+- Install: `./.agent/scripts/crawl4ai-helper.sh install`
+- Docker setup: `./.agent/scripts/crawl4ai-helper.sh docker-setup`
+- Start: `./.agent/scripts/crawl4ai-helper.sh docker-start`
+- MCP setup: `./.agent/scripts/crawl4ai-helper.sh mcp-setup`
+- URLs: Dashboard http://localhost:11235/dashboard, Playground /playground, API :11235
+- Crawl: `./.agent/scripts/crawl4ai-helper.sh crawl URL markdown output.json`
+- Extract: `./.agent/scripts/crawl4ai-helper.sh extract URL '{"schema"}' data.json`
+- Features: LLM-ready markdown, CSS/XPath/LLM extraction, async parallel crawling
+- MCP tools: crawl_url, crawl_multiple, extract_structured, take_screenshot, generate_pdf
+- Config: `configs/crawl4ai-config.json.txt`, `configs/mcp-templates/crawl4ai-mcp-config.json`
+- Debug: `./.agent/scripts/crawl4ai-helper.sh status`, `docker logs crawl4ai`
+<!-- AI-CONTEXT-END -->
+
+## Overview
+
+Crawl4AI is the #1 trending open-source web crawler on GitHub, specifically designed for AI and LLM applications. This integration provides comprehensive web crawling and data extraction capabilities for the AI DevOps Framework.
+
+### Key Features
+
+- **LLM-Ready Output**: Clean markdown generation perfect for RAG pipelines
+- **Structured Extraction**: CSS selectors, XPath, and LLM-based data extraction
+- **Advanced Browser Control**: Hooks, proxies, stealth modes, session management
+- **High Performance**: Parallel crawling, async operations, real-time processing
+- **AI Integration**: Native MCP support for AI assistants like Claude
+- **Enterprise Features**: Monitoring dashboard, job queues, webhook notifications
+
+## 🛠️ Installation & Setup
+
+### Quick Start
+
+```bash
+# Install Python package
+./.agent/scripts/crawl4ai-helper.sh install
+
+# Setup Docker deployment
+./.agent/scripts/crawl4ai-helper.sh docker-setup
+
+# Start Docker container with monitoring dashboard
+./.agent/scripts/crawl4ai-helper.sh docker-start
+
+# Setup MCP integration for AI assistants
+./.agent/scripts/crawl4ai-helper.sh mcp-setup
+```
+
+### Docker Deployment
+
+The Docker deployment includes:
+
+- **Real-time Monitoring Dashboard**: http://localhost:11235/dashboard
+- **Interactive Playground**: http://localhost:11235/playground
+- **REST API**: http://localhost:11235
+- **WebSocket Streaming**: Real-time crawl results
+- **Job Queue System**: Asynchronous processing with webhooks
+
+### MCP Integration
+
+Crawl4AI provides native MCP (Model Context Protocol) support for AI assistants:
+
+```json
+{
+  "crawl4ai": {
+    "command": "npx",
+    "args": ["crawl4ai-mcp-server@latest"],
+    "env": {
+      "CRAWL4AI_API_URL": "http://localhost:11235"
+    }
+  }
+}
+```
+
+## 🎯 Core Capabilities
+
+### 1. Web Crawling
+
+```bash
+# Basic crawling
+./.agent/scripts/crawl4ai-helper.sh crawl https://example.com markdown output.json
+
+# With structured extraction
+./.agent/scripts/crawl4ai-helper.sh extract https://example.com '{"title":"h1","content":".article"}' data.json
+```
+
+### 2. LLM-Powered Extraction
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, LLMExtractionStrategy, LLMConfig
+
+async def extract_with_llm():
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(
+            url="https://example.com",
+            extraction_strategy=LLMExtractionStrategy(
+                llm_config=LLMConfig(provider="openai/gpt-4o"),
+                instruction="Extract key information and summarize"
+            )
+        )
+        return result.extracted_content
+```
+
+### 3. Advanced Browser Control
+
+```python
+# Custom hooks for advanced control
+async def setup_hook(page, context, **kwargs):
+    # Block images for faster crawling
+    await context.route("**/*.{png,jpg,gif}", lambda r: r.abort())
+    # Set custom viewport
+    await page.set_viewport_size({"width": 1920, "height": 1080})
+    return page
+
+result = await crawler.arun(
+    url="https://example.com",
+    hooks={"on_page_context_created": setup_hook}
+)
+```
+
+### 4. Adaptive Crawling
+
+```python
+from crawl4ai import AdaptiveCrawler, AdaptiveConfig
+
+config = AdaptiveConfig(
+    confidence_threshold=0.7,
+    max_depth=5,
+    max_pages=20,
+    strategy="statistical"
+)
+
+adaptive_crawler = AdaptiveCrawler(crawler, config)
+state = await adaptive_crawler.digest(
+    start_url="https://news.example.com",
+    query="latest technology news"
+)
+```
+
+## 🔧 Configuration
+
+### Environment Variables
+
+```bash
+# LLM Provider Configuration
+OPENAI_API_KEY=sk-your-key
+ANTHROPIC_API_KEY=your-anthropic-key
+LLM_PROVIDER=openai/gpt-4o-mini
+LLM_TEMPERATURE=0.7
+
+# Crawl4AI Settings
+CRAWL4AI_MAX_PAGES=50
+CRAWL4AI_TIMEOUT=60
+CRAWL4AI_DEFAULT_FORMAT=markdown
+```
+
+### Browser Configuration
+
+```python
+browser_config = BrowserConfig(
+    headless=True,
+    viewport={"width": 1920, "height": 1080},
+    user_agent="Mozilla/5.0 (compatible; Crawl4AI/0.7.7)",
+    timeout=30000,
+    extra_args=["--disable-blink-features=AutomationControlled"]
+)
+```
+
+### Crawler Configuration
+
+```python
+crawler_config = CrawlerRunConfig(
+    cache_mode=CacheMode.ENABLED,
+    max_depth=3,
+    delay_between_requests=1.0,
+    respect_robots_txt=True,
+    follow_redirects=True,
+    extraction_strategy=JsonCssExtractionStrategy(schema=your_schema)
+)
+```
+
+## 📊 Monitoring & Analytics
+
+### Dashboard Features
+
+- **Real-time Metrics**: System health, memory usage, request tracking
+- **Browser Pool Management**: Active/hot/cold browser instances
+- **Request Analytics**: Success rates, response times, error tracking
+- **Resource Monitoring**: CPU, memory, network utilization
+
+### API Endpoints
+
+```bash
+# Health check
+curl http://localhost:11235/health
+
+# Prometheus metrics
+curl http://localhost:11235/metrics
+
+# API schema
+curl http://localhost:11235/schema
+```
+
+## 🔄 Job Queue & Webhooks
+
+### Asynchronous Processing
+
+```python
+# Submit crawl job
+response = requests.post("http://localhost:11235/crawl/job", json={
+    "urls": ["https://example.com"],
+    "webhook_config": {
+        "webhook_url": "https://your-app.com/webhook",
+        "webhook_data_in_payload": True
+    }
+})
+
+task_id = response.json()["task_id"]
+```
+
+### Webhook Notifications
+
+```python
+@app.route('/webhook', methods=['POST'])
+def handle_webhook():
+    payload = request.json
+    if payload['status'] == 'completed':
+        process_results(payload['data'])
+    return "OK", 200
+```
+
+## 🤖 AI Assistant Integration
+
+### Claude Desktop Setup
+
+Add to your Claude Desktop MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "crawl4ai": {
+      "command": "npx",
+      "args": ["crawl4ai-mcp-server@latest"]
+    }
+  }
+}
+```
+
+### Available MCP Tools
+
+- `crawl_url`: Crawl single URL with format options
+- `crawl_multiple`: Batch crawl multiple URLs
+- `extract_structured`: Extract data using CSS or LLM
+- `take_screenshot`: Capture webpage screenshots
+- `generate_pdf`: Convert webpages to PDF
+- `execute_javascript`: Run custom JavaScript on pages
+
+## 🔒 Security & Best Practices
+
+### Rate Limiting
+
+```yaml
+rate_limiting:
+  enabled: true
+  default_limit: "1000/minute"
+  trusted_proxies: []
+```
+
+### Security Headers
+
+```yaml
+security:
+  headers:
+    x_content_type_options: "nosniff"
+    x_frame_options: "DENY"
+    content_security_policy: "default-src 'self'"
+```
+
+### Hook Security
+
+- Never trust user-provided hook code
+- Validate and sandbox hook execution
+- Use timeouts to prevent infinite loops
+- Audit hook code before deployment
+
+## 📚 Use Cases
+
+### 1. Content Aggregation
+
+```python
+# News aggregation
+urls = ["https://news1.com", "https://news2.com", "https://news3.com"]
+results = await crawler.arun_many(urls, extraction_strategy=news_schema)
+```
+
+### 2. E-commerce Data
+
+```python
+# Product information extraction
+product_schema = {
+    "name": "h1.product-title",
+    "price": ".price",
+    "description": ".product-description",
+    "images": {"selector": "img.product-image", "type": "attribute", "attribute": "src"}
+}
+```
+
+### 3. Research & Analysis
+
+```python
+# Academic paper extraction
+paper_extraction = LLMExtractionStrategy(
+    instruction="Extract title, authors, abstract, and key findings",
+    schema=paper_schema
+)
+```
+
+### 4. SEO & Marketing
+
+```python
+# SEO data extraction
+seo_schema = {
+    "title": "title",
+    "meta_description": "meta[name='description']",
+    "headings": "h1, h2, h3",
+    "links": {"selector": "a", "type": "attribute", "attribute": "href"}
+}
+```
+
+## 🚀 Advanced Features
+
+### Virtual Scroll Support
+
+```python
+scroll_config = VirtualScrollConfig(
+    container_selector="[data-testid='feed']",
+    scroll_count=20,
+    scroll_by="container_height",
+    wait_after_scroll=1.0
+)
+```
+
+### Session Management
+
+```python
+# Persistent browser sessions
+browser_config = BrowserConfig(
+    use_persistent_context=True,
+    user_data_dir="/path/to/profile"
+)
+```
+
+### Proxy Support
+
+```python
+# Proxy configuration
+browser_config = BrowserConfig(
+    proxy={
+        "server": "http://proxy.example.com:8080",
+        "username": "user",
+        "password": "pass"
+    }
+)
+```
+
+## 🔧 Troubleshooting
+
+### Common Issues
+
+1. **Browser not starting**: Check Docker memory allocation (--shm-size=1g)
+2. **API not responding**: Verify container is running and port is accessible
+3. **Extraction failing**: Validate CSS selectors or LLM configuration
+4. **Memory issues**: Adjust browser pool size and cleanup intervals
+
+### Debug Commands
+
+```bash
+# Check service status
+./.agent/scripts/crawl4ai-helper.sh status
+
+# View container logs
+docker logs crawl4ai
+
+# Test API health
+curl http://localhost:11235/health
+```
+
+## 📖 Resources
+
+- **Official Documentation**: https://docs.crawl4ai.com/
+- **GitHub Repository**: https://github.com/unclecode/crawl4ai
+- **Framework Integration**: `.agent/scripts/crawl4ai-helper.sh`
+- **Configuration Templates**: `configs/crawl4ai-config.json.txt`
+- **MCP Configuration**: `configs/mcp-templates/crawl4ai-mcp-config.json`
+
+## 🎯 Next Steps
+
+1. **Install and Setup**: Run the helper script to get started
+2. **Explore Dashboard**: Visit http://localhost:11235/dashboard
+3. **Try Playground**: Test crawling at http://localhost:11235/playground
+4. **Setup MCP**: Integrate with your AI assistant
+5. **Build Applications**: Use the API for your specific use cases
+
+Crawl4AI transforms web data into AI-ready formats, making it perfect for RAG systems, data pipelines, and AI-powered applications.

package/.agent/tools/browser/crawl4ai-resources.md

@@ -0,0 +1,277 @@
+---
+description: Crawl4AI documentation and resource links
+mode: subagent
+tools:
+  read: true
+  write: false
+  edit: false
+  bash: false
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Crawl4AI Resources & Links
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Docs**: https://docs.crawl4ai.com/
+- **GitHub**: https://github.com/unclecode/crawl4ai
+- **Docker**: `unclecode/crawl4ai:latest`
+- **PyPI**: https://pypi.org/project/crawl4ai/
+- **MCP NPM**: `npx crawl4ai-mcp-server@latest`
+- **Discord**: https://discord.gg/jP8KfhDhyN
+- **CapSolver**: https://www.capsolver.com/ (CAPTCHA integration)
+- **API Endpoints**: `/crawl`, `/crawl/job`, `/health`, `/metrics`, `/screenshot`, `/pdf`
+- **Framework Files**: `.agent/scripts/crawl4ai-helper.sh`, `configs/crawl4ai-config.json.txt`
+- **Current Version**: v0.7.7 (November 2024)
+<!-- AI-CONTEXT-END -->
+
+## Official Resources
+
+### Primary Documentation
+
+- **Official Documentation**: https://docs.crawl4ai.com/
+- **GitHub Repository**: https://github.com/unclecode/crawl4ai
+- **Docker Hub**: https://hub.docker.com/r/unclecode/crawl4ai
+- **PyPI Package**: https://pypi.org/project/crawl4ai/
+
+### Community & Support
+
+- **Discord Community**: https://discord.gg/jP8KfhDhyN
+- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
+- **GitHub Discussions**: https://github.com/unclecode/crawl4ai/discussions
+- **Changelog**: https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md
+
+### CapSolver Integration
+
+- **CapSolver Homepage**: https://www.capsolver.com/
+- **CapSolver Dashboard**: https://dashboard.capsolver.com/dashboard/overview
+- **CapSolver Documentation**: https://docs.capsolver.com/
+- **Crawl4AI Partnership**: https://www.capsolver.com/blog/Partners/crawl4ai-capsolver/
+- **Chrome Extension**: https://chrome.google.com/webstore/detail/capsolver/pgojnojmmhpofjgdmaebadhbocahppod
+
+## Documentation Sections
+
+### Core Documentation
+
+- **Quick Start**: https://docs.crawl4ai.com/quick-start/
+- **Installation**: https://docs.crawl4ai.com/setup-installation/installation/
+- **Docker Deployment**: https://docs.crawl4ai.com/setup-installation/docker-deployment/
+- **API Reference**: https://docs.crawl4ai.com/api-reference/
+
+### Advanced Features
+
+- **Adaptive Crawling**: https://docs.crawl4ai.com/advanced/adaptive-strategies/
+- **Virtual Scroll**: https://docs.crawl4ai.com/advanced/virtual-scroll/
+- **Hooks & Authentication**: https://docs.crawl4ai.com/advanced/hooks-auth/
+- **Session Management**: https://docs.crawl4ai.com/advanced/session-management/
+
+### Extraction Strategies
+
+- **LLM-Free Strategies**: https://docs.crawl4ai.com/extraction/llm-free-strategies/
+- **LLM Strategies**: https://docs.crawl4ai.com/extraction/llm-strategies/
+- **Clustering Strategies**: https://docs.crawl4ai.com/extraction/clustering-strategies/
+- **Chunking**: https://docs.crawl4ai.com/extraction/chunking/
+
+## Framework Integration
+
+### Helper Scripts
+
+- **Main Helper**: `.agent/scripts/crawl4ai-helper.sh`
+- **Examples Script**: `.agent/scripts/crawl4ai-examples.sh`
+- **Configuration Template**: `configs/crawl4ai-config.json.txt`
+- **MCP Configuration**: `configs/mcp-templates/crawl4ai-mcp-config.json`
+
+### Documentation Files
+
+- **Main Guide**: `.agent/crawl4ai.md`
+- **Integration Guide**: `.agent/wiki/crawl4ai-integration.md`
+- **Usage Guide**: `.agent/spec/crawl4ai-usage.md`
+- **Resources**: `.agent/links/crawl4ai-resources.md` (this file)
+
+## MCP Integration
+
+### MCP Server
+
+- **NPM Package**: https://www.npmjs.com/package/crawl4ai-mcp-server
+- **Installation**: `npx crawl4ai-mcp-server@latest`
+- **Documentation**: https://docs.crawl4ai.com/core/docker-deployment/#mcp-model-context-protocol-support
+
+### Claude Desktop Integration
+
+```json
+{
+  "mcpServers": {
+    "crawl4ai": {
+      "command": "npx",
+      "args": ["crawl4ai-mcp-server@latest"]
+    }
+  }
+}
+```
+
+## Docker Resources
+
+### Docker Images
+
+- **Latest Release**: `unclecode/crawl4ai:latest`
+- **Specific Version**: `unclecode/crawl4ai:0.7.7`
+- **Multi-Architecture**: Supports AMD64 and ARM64
+
+### Docker Compose
+
+- **Example Compose**: https://github.com/unclecode/crawl4ai/blob/main/docker-compose.yml
+- **Environment Variables**: https://docs.crawl4ai.com/core/docker-deployment/#environment-setup-api-keys
+
+## Use Case Examples
+
+### Content Research
+
+- **News Aggregation**: Extract articles from multiple news sources
+- **Academic Papers**: Extract titles, authors, abstracts, and citations
+- **Documentation**: Process API docs and technical documentation
+
+### E-commerce Data
+
+- **Product Information**: Extract names, prices, descriptions, specifications
+- **Inventory Tracking**: Monitor stock levels and price changes
+- **Competitor Analysis**: Compare products across different sites
+
+### SEO & Marketing
+
+- **Content Analysis**: Extract headings, meta tags, and content structure
+- **Link Analysis**: Discover internal and external link patterns
+- **Performance Monitoring**: Track page changes and updates
+
+## API Endpoints
+
+### Core Endpoints
+
+- **Crawl**: `POST /crawl` - Synchronous crawling
+- **Crawl Job**: `POST /crawl/job` - Asynchronous crawling with webhooks
+- **LLM Job**: `POST /llm/job` - LLM extraction with webhooks
+- **Job Status**: `GET /job/{task_id}` - Check job status
+
+### Utility Endpoints
+
+- **Health**: `GET /health` - Service health check
+- **Metrics**: `GET /metrics` - Prometheus metrics
+- **Schema**: `GET /schema` - API schema documentation
+- **Dashboard**: `GET /dashboard` - Monitoring dashboard
+- **Playground**: `GET /playground` - Interactive testing interface
+
+### Media Endpoints
+
+- **Screenshot**: `POST /screenshot` - Capture page screenshots
+- **PDF**: `POST /pdf` - Generate PDF from webpage
+- **HTML**: `POST /html` - Extract raw HTML
+- **JavaScript**: `POST /js` - Execute JavaScript on page
+
+## Security Resources
+
+### Best Practices
+
+- **Rate Limiting**: Built-in protection against abuse
+- **User Agent**: Clear identification as Crawl4AI
+- **Robots.txt**: Respects robots.txt by default
+- **Timeout Protection**: Prevents hanging requests
+
+### Authentication
+
+- **JWT Support**: Optional JWT authentication for API access
+- **API Keys**: Secure API key management for LLM providers
+- **Webhook Security**: Custom headers for webhook authentication
+
+## Monitoring & Analytics
+
+### Dashboard Features
+
+- **System Metrics**: CPU, memory, network utilization
+- **Request Analytics**: Success rates, response times, error tracking
+- **Browser Pool**: Active/hot/cold browser instances management
+- **Job Queue**: Real-time job processing status
+
+### Metrics Integration
+
+- **Prometheus**: Native Prometheus metrics export
+- **Health Checks**: Comprehensive health monitoring
+- **Performance Tracking**: Request timing and resource usage
+
+## Performance Optimization
+
+### Configuration Tips
+
+- **Browser Pool Size**: Optimize based on available resources
+- **Concurrent Requests**: Balance speed vs resource usage
+- **Memory Management**: Configure cleanup intervals and thresholds
+- **Caching**: Use appropriate cache modes for your use case
+
+### Resource Management
+
+- **Docker Memory**: Allocate sufficient shared memory (--shm-size=1g)
+- **CPU Throttling**: Configure CPU limits for container
+- **Network Optimization**: Use appropriate timeouts and retry policies
+
+## Version Information
+
+### Current Version
+
+- **Latest Stable**: v0.7.7
+- **Release Date**: November 2024
+- **Breaking Changes**: Check CHANGELOG.md for migration notes
+
+### Version History
+
+- **v0.7.7**: Self-hosting platform with real-time monitoring
+- **v0.7.6**: Complete webhook infrastructure for job queue API
+- **v0.7.5**: Docker hooks system with function-based API
+- **v0.7.4**: Intelligent table extraction & performance updates
+
+## Learning Resources
+
+### Tutorials & Guides
+
+- **Video Tutorial**: Available on documentation homepage
+- **Code Examples**: https://github.com/unclecode/crawl4ai/tree/main/.agent/examples
+- **Blog Posts**: Check GitHub discussions for community tutorials
+
+### Community Examples
+
+- **GitHub Examples**: Real-world usage examples in repository
+- **Discord Discussions**: Community-shared patterns and solutions
+- **Stack Overflow**: Tagged questions and answers
+
+## Contributing
+
+### Development
+
+- **Contributing Guide**: https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md
+- **Code of Conduct**: https://github.com/unclecode/crawl4ai/blob/main/CODE_OF_CONDUCT.md
+- **Development Setup**: Local development instructions in README
+
+### Sponsorship
+
+- **GitHub Sponsors**: Support the project development
+- **Enterprise Support**: Commercial support options available
+- **Community Recognition**: Contributors acknowledged in project
+
+## Support Channels
+
+### Technical Support
+
+1. **GitHub Issues**: Bug reports and feature requests
+2. **Discord Community**: Real-time community support
+3. **Documentation**: Comprehensive guides and API reference
+4. **Stack Overflow**: Tag questions with `crawl4ai`
+
+### Enterprise Support
+
+- **Commercial Licensing**: Available for enterprise use
+- **Priority Support**: Dedicated support channels
+- **Custom Development**: Tailored solutions and integrations
+
+This resource collection provides comprehensive access to all Crawl4AI documentation, tools, and community resources for effective integration within the AI DevOps Framework.