aidevops 2.52.1

package/.agent/seo/serper.md

@@ -0,0 +1,278 @@
+---
+description: Serper MCP for Google Search API integration
+mode: subagent
+tools:
+  read: true
+  write: false
+  edit: false
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+---
+
+# Serper MCP Integration
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Purpose**: Google Search results via Serper API
+- **MCP Package**: `serper-mcp-server` (Python, community)
+- **Auth**: API Key (stored in `~/.config/aidevops/mcp-env.sh`)
+- **Env Var**: `SERPER_API_KEY`
+- **API Dashboard**: https://serper.dev/
+- **GitHub**: https://github.com/garylab/serper-mcp-server
+
+**Available Tools**:
+
+| Tool | Purpose |
+|------|---------|
+| `google_search` | Web search results |
+| `google_search_images` | Image search results |
+| `google_search_videos` | Video search results |
+| `google_search_places` | Local business/place search |
+| `google_search_maps` | Map search results |
+| `google_search_reviews` | Business reviews |
+| `google_search_news` | News search results |
+| `google_search_shopping` | Shopping/product search |
+| `google_search_lens` | Visual search (Google Lens) |
+| `google_search_scholar` | Academic/scholarly search |
+| `google_search_patents` | Patent search |
+| `google_search_autocomplete` | Search suggestions |
+| `webpage_scrape` | Scrape webpage content |
+
+<!-- AI-CONTEXT-END -->
+
+## Installation
+
+### Via setup.sh (Recommended)
+
+The aidevops `setup.sh` script automatically configures the Serper MCP server.
+
+### Manual Installation
+
+```bash
+# Using uv (recommended)
+uvx serper-mcp-server
+
+# Using pip
+pip install serper-mcp-server
+
+# Or install globally
+pip3 install serper-mcp-server
+```text
+
+## Configuration
+
+### Store Credentials
+
+```bash
+# Using the secure key management script
+bash ~/.aidevops/agents/scripts/setup-local-api-keys.sh set SERPER_API_KEY "your_api_key"
+```text
+
+### OpenCode Configuration
+
+Add to `~/.config/opencode/opencode.json`:
+
+```json
+{
+  "mcp": {
+    "serper": {
+      "type": "local",
+      "command": [
+        "/bin/bash",
+        "-c",
+        "source ~/.config/aidevops/mcp-env.sh && SERPER_API_KEY=$SERPER_API_KEY uvx serper-mcp-server"
+      ],
+      "enabled": true
+    }
+  }
+}
+```text
+
+### Claude Desktop Configuration
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "serper": {
+      "command": "uvx",
+      "args": ["serper-mcp-server"],
+      "env": {
+        "SERPER_API_KEY": "your_api_key"
+      }
+    }
+  }
+}
+```text
+
+### Cursor Configuration
+
+Add to `~/.cursor/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "serper": {
+      "command": "uvx",
+      "args": ["serper-mcp-server"],
+      "env": {
+        "SERPER_API_KEY": "your_api_key"
+      }
+    }
+  }
+}
+```text
+
+### Alternative: Using pip
+
+If you prefer pip over uv:
+
+```json
+{
+  "mcpServers": {
+    "serper": {
+      "command": "python3",
+      "args": ["-m", "serper_mcp_server"],
+      "env": {
+        "SERPER_API_KEY": "your_api_key"
+      }
+    }
+  }
+}
+```text
+
+## Usage Examples
+
+### Web Search
+
+```javascript
+// Basic Google search
+await serper.google_search({
+  q: "best seo tools 2024",
+  gl: "us",  // Country code
+  hl: "en",  // Language
+  num: 10    // Number of results
+});
+```text
+
+### Image Search
+
+```javascript
+// Search for images
+await serper.google_search_images({
+  q: "seo infographic",
+  gl: "us",
+  num: 20
+});
+```text
+
+### News Search
+
+```javascript
+// Get latest news
+await serper.google_search_news({
+  q: "google algorithm update",
+  gl: "us",
+  tbs: "qdr:w"  // Past week
+});
+```text
+
+### Local/Places Search
+
+```javascript
+// Find local businesses
+await serper.google_search_places({
+  q: "seo agency",
+  location: "New York, NY"
+});
+```text
+
+### Shopping Search
+
+```javascript
+// Product search
+await serper.google_search_shopping({
+  q: "seo software",
+  gl: "us"
+});
+```text
+
+### Scholar Search
+
+```javascript
+// Academic papers
+await serper.google_search_scholar({
+  q: "search engine optimization research",
+  num: 10
+});
+```text
+
+### Webpage Scraping
+
+```javascript
+// Scrape a webpage
+await serper.webpage_scrape({
+  url: "https://example.com/article"
+});
+```text
+
+## Search Parameters
+
+Common parameters for all search tools:
+
+| Parameter | Description | Example |
+|-----------|-------------|---------|
+| `q` | Search query | `"seo tools"` |
+| `gl` | Country code | `"us"`, `"uk"`, `"de"` |
+| `hl` | Language code | `"en"`, `"es"`, `"fr"` |
+| `num` | Number of results | `10`, `20`, `100` |
+| `page` | Page number | `1`, `2`, `3` |
+| `tbs` | Time filter | `"qdr:d"` (day), `"qdr:w"` (week), `"qdr:m"` (month) |
+
+## Verification
+
+Test the integration:
+
+```text
+Use the Serper MCP to search for "best seo tools 2024" in the United States
+```text
+
+Expected: Google search results with titles, URLs, snippets, and related data.
+
+## Debugging
+
+```bash
+# Using MCP inspector with uvx
+npx @modelcontextprotocol/inspector uvx serper-mcp-server
+
+# Or with local development
+git clone https://github.com/garylab/serper-mcp-server.git
+cd serper-mcp-server
+npx @modelcontextprotocol/inspector uv run serper-mcp-server -e SERPER_API_KEY=your_key
+```text
+
+## Comparison with DataForSEO
+
+| Feature | Serper | DataForSEO |
+|---------|--------|------------|
+| **Focus** | Google Search API | Comprehensive SEO data |
+| **SERP Data** | Yes | Yes |
+| **Keyword Research** | No | Yes |
+| **Backlinks** | No | Yes |
+| **On-Page Analysis** | No | Yes |
+| **Pricing** | Pay-per-search | Subscription |
+| **Best For** | Quick searches | Full SEO workflows |
+
+Use Serper for quick Google searches; use DataForSEO for comprehensive SEO analysis.
+
+## Resources
+
+- **API Dashboard**: https://serper.dev/
+- **GitHub**: https://github.com/garylab/serper-mcp-server
+- **PyPI**: https://pypi.org/project/serper-mcp-server/
+- **API Docs**: https://serper.dev/docs

package/.agent/seo/site-crawler.md

@@ -0,0 +1,387 @@
+---
+description: SEO site crawler with Screaming Frog-like capabilities
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Site Crawler - SEO Spider Agent
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Purpose**: Comprehensive SEO site auditing like Screaming Frog
+- **Helper**: `~/.aidevops/agents/scripts/site-crawler-helper.sh`
+- **Browser Tools**: `tools/browser/crawl4ai.md`, `tools/browser/playwriter.md`
+- **Output**: `~/Downloads/{domain}/{datestamp}/` with `_latest` symlink
+- **Formats**: CSV, XLSX, JSON, HTML reports
+
+**Commands**:
+
+```bash
+# Full site crawl
+site-crawler-helper.sh crawl https://example.com
+
+# Crawl with depth limit
+site-crawler-helper.sh crawl https://example.com --depth 3 --max-urls 500
+
+# Specific audits
+site-crawler-helper.sh audit-links https://example.com
+site-crawler-helper.sh audit-meta https://example.com
+site-crawler-helper.sh audit-redirects https://example.com
+
+# Export formats
+site-crawler-helper.sh crawl https://example.com --format xlsx
+site-crawler-helper.sh crawl https://example.com --format csv
+
+# JavaScript rendering
+site-crawler-helper.sh crawl https://example.com --render-js
+```text
+
+**Key Features**:
+
+- Broken link detection (4XX, 5XX errors)
+- Redirect chain analysis
+- Meta data auditing (titles, descriptions, robots)
+- Duplicate content detection
+- Structured data extraction
+- XML sitemap generation
+- Internal linking analysis
+- JavaScript rendering support
+
+<!-- AI-CONTEXT-END -->
+
+## Overview
+
+The Site Crawler agent provides Screaming Frog-like SEO auditing capabilities using
+Crawl4AI, Playwriter, and custom scripts. It crawls websites to identify technical
+SEO issues and exports findings to spreadsheets for analysis.
+
+## Crawl Capabilities
+
+### Core SEO Data Collection
+
+| Category | Data Collected |
+|----------|----------------|
+| **URLs** | Address, status code, content type, response time, file size |
+| **Page Titles** | Title text, length, missing/duplicate detection |
+| **Meta Descriptions** | Description text, length, missing/duplicate detection |
+| **Meta Robots** | Index/noindex, follow/nofollow, canonical, robots directives |
+| **Headings** | H1, H2 content, missing/duplicate/multiple detection |
+| **Links** | Internal/external, follow/nofollow, anchor text, broken links |
+| **Images** | URL, alt text, file size, missing alt detection |
+| **Redirects** | Type (301/302/307), chains, loops, final destination |
+| **Canonicals** | Canonical URL, self-referencing, conflicts |
+| **Hreflang** | Language codes, return links, conflicts |
+| **Structured Data** | JSON-LD, Microdata, RDFa extraction and validation |
+
+### Advanced Features
+
+| Feature | Description |
+|---------|-------------|
+| **JavaScript Rendering** | Crawl SPAs (React, Vue, Angular) via Chromium |
+| **Custom Extraction** | XPath, CSS selectors, regex for any HTML data |
+| **Robots.txt Analysis** | Blocked URLs, directives, crawl delays |
+| **XML Sitemap Analysis** | Parse sitemaps, find orphan/missing pages |
+| **Duplicate Detection** | MD5 hash for exact duplicates, similarity scoring |
+| **Crawl Depth** | Track URL depth in site architecture |
+| **Word Count** | Content length analysis per page |
+
+## Usage
+
+### Basic Site Crawl
+
+```bash
+# Crawl entire site (respects robots.txt)
+site-crawler-helper.sh crawl https://example.com
+
+# Output: ~/Downloads/example.com/2025-01-15_143022/
+#   - crawl-data.csv
+#   - crawl-data.xlsx
+#   - broken-links.csv
+#   - redirects.csv
+#   - meta-issues.csv
+#   - summary.json
+```text
+
+### Targeted Audits
+
+```bash
+# Broken links only
+site-crawler-helper.sh audit-links https://example.com
+
+# Meta data audit (titles, descriptions)
+site-crawler-helper.sh audit-meta https://example.com
+
+# Redirect audit
+site-crawler-helper.sh audit-redirects https://example.com
+
+# Duplicate content check
+site-crawler-helper.sh audit-duplicates https://example.com
+
+# Structured data validation
+site-crawler-helper.sh audit-schema https://example.com
+```text
+
+### Crawl Configuration
+
+```bash
+# Limit crawl scope
+site-crawler-helper.sh crawl https://example.com \
+  --depth 3 \
+  --max-urls 1000 \
+  --include "/blog/*" \
+  --exclude "/admin/*,/wp-json/*"
+
+# JavaScript rendering for SPAs
+site-crawler-helper.sh crawl https://spa-site.com --render-js
+
+# Custom user agent
+site-crawler-helper.sh crawl https://example.com --user-agent "Googlebot"
+
+# Respect/ignore robots.txt
+site-crawler-helper.sh crawl https://example.com --ignore-robots
+```text
+
+### Export Options
+
+```bash
+# CSV export (default)
+site-crawler-helper.sh crawl https://example.com --format csv
+
+# Excel export
+site-crawler-helper.sh crawl https://example.com --format xlsx
+
+# Both formats
+site-crawler-helper.sh crawl https://example.com --format all
+
+# Custom output location
+site-crawler-helper.sh crawl https://example.com --output ~/SEO-Audits/
+```text
+
+## Output Structure
+
+All crawl outputs are organized by domain and timestamp:
+
+```text
+~/Downloads/
+└── example.com/
+    ├── 2025-01-15_143022/
+    │   ├── crawl-data.xlsx          # Full crawl data
+    │   ├── crawl-data.csv           # Full crawl data (CSV)
+    │   ├── broken-links.csv         # 4XX/5XX errors
+    │   ├── redirects.csv            # All redirects with chains
+    │   ├── meta-issues.csv          # Title/description issues
+    │   ├── duplicate-content.csv    # Duplicate pages
+    │   ├── images.csv               # Image audit
+    │   ├── internal-links.csv       # Link structure
+    │   ├── external-links.csv       # Outbound links
+    │   ├── structured-data.json     # Schema.org data
+    │   └── summary.json             # Crawl statistics
+    ├── 2025-01-10_091500/
+    │   └── ...
+    └── _latest -> 2025-01-15_143022  # Symlink to latest
+```text
+
+## Spreadsheet Columns
+
+### Main Crawl Data (crawl-data.xlsx)
+
+| Column | Description |
+|--------|-------------|
+| URL | Full page URL |
+| Status Code | HTTP response code |
+| Status | OK, Redirect, Client Error, Server Error |
+| Content Type | MIME type |
+| Title | Page title |
+| Title Length | Character count |
+| Meta Description | Description content |
+| Description Length | Character count |
+| H1 | First H1 content |
+| H1 Count | Number of H1 tags |
+| H2 | First H2 content |
+| H2 Count | Number of H2 tags |
+| Canonical | Canonical URL |
+| Meta Robots | Robots directives |
+| Word Count | Text content word count |
+| Response Time | Server response in ms |
+| File Size | Page size in bytes |
+| Crawl Depth | Clicks from homepage |
+| Inlinks | Number of internal links to page |
+| Outlinks | Number of links from page |
+| External Links | Number of external links |
+| Images | Number of images |
+| Images Missing Alt | Images without alt text |
+
+### Broken Links Report
+
+| Column | Description |
+|--------|-------------|
+| Broken URL | The 4XX/5XX URL |
+| Status Code | Error code |
+| Source URL | Page containing the link |
+| Anchor Text | Link text |
+| Link Type | Internal/External |
+
+### Redirect Report
+
+| Column | Description |
+|--------|-------------|
+| Original URL | Starting URL |
+| Status Code | 301/302/307/308 |
+| Redirect URL | Target URL |
+| Final URL | End of chain |
+| Chain Length | Number of hops |
+| Chain | Full redirect path |
+
+## Integration with Other Agents
+
+### With E-E-A-T Score Agent
+
+```bash
+# Crawl site first
+site-crawler-helper.sh crawl https://example.com --format json
+
+# Then run E-E-A-T analysis on crawled pages
+eeat-score-helper.sh analyze ~/Downloads/example.com/_latest/crawl-data.json
+```text
+
+### With PageSpeed Agent
+
+```bash
+# Crawl and get performance data
+site-crawler-helper.sh crawl https://example.com --include-pagespeed
+```text
+
+### With Crawl4AI
+
+The site crawler uses Crawl4AI for:
+- JavaScript rendering
+- Structured data extraction
+- LLM-powered content analysis
+- CAPTCHA handling (with CapSolver)
+
+See `tools/browser/crawl4ai.md` for advanced configuration.
+
+## Browser Automation
+
+For sites requiring authentication or complex interactions:
+
+```bash
+# Use Playwriter for authenticated crawls
+site-crawler-helper.sh crawl https://example.com \
+  --auth-type form \
+  --login-url https://example.com/login \
+  --username user@example.com \
+  --password-env SITE_PASSWORD
+```text
+
+See `tools/browser/playwriter.md` for browser automation details.
+
+## XML Sitemap Generation
+
+```bash
+# Generate sitemap from crawl
+site-crawler-helper.sh generate-sitemap https://example.com
+
+# Output: ~/Downloads/example.com/_latest/sitemap.xml
+
+# With configuration
+site-crawler-helper.sh generate-sitemap https://example.com \
+  --changefreq weekly \
+  --priority-rules "/blog/*:0.8,/*:0.5" \
+  --exclude "/admin/*,/private/*"
+```text
+
+## Crawl Comparison
+
+Compare two crawls to track changes:
+
+```bash
+# Compare latest with previous
+site-crawler-helper.sh compare https://example.com
+
+# Compare specific crawls
+site-crawler-helper.sh compare \
+  ~/Downloads/example.com/2025-01-10_091500 \
+  ~/Downloads/example.com/2025-01-15_143022
+
+# Output: changes-report.xlsx with:
+#   - New URLs
+#   - Removed URLs
+#   - Changed titles/descriptions
+#   - New/fixed broken links
+#   - Redirect changes
+```text
+
+## Configuration File
+
+Create `~/.config/aidevops/site-crawler.json` for defaults:
+
+```json
+{
+  "default_depth": 10,
+  "max_urls": 10000,
+  "respect_robots": true,
+  "render_js": false,
+  "user_agent": "AIDevOps-Crawler/1.0",
+  "request_delay": 100,
+  "concurrent_requests": 5,
+  "timeout": 30,
+  "output_format": "xlsx",
+  "output_directory": "~/Downloads",
+  "exclude_patterns": [
+    "/wp-admin/*",
+    "/wp-json/*",
+    "*.pdf",
+    "*.zip"
+  ]
+}
+```text
+
+## Rate Limiting & Politeness
+
+The crawler respects website resources:
+
+- **Robots.txt**: Honored by default (override with `--ignore-robots`)
+- **Crawl-delay**: Respected from robots.txt
+- **Request delay**: Configurable delay between requests
+- **Concurrent requests**: Limited to avoid overwhelming servers
+
+## Troubleshooting
+
+### Common Issues
+
+| Issue | Solution |
+|-------|----------|
+| Crawl blocked | Check robots.txt, try different user-agent |
+| JavaScript not rendering | Use `--render-js` flag |
+| Missing pages | Increase `--depth` or check internal linking |
+| Slow crawl | Reduce `--concurrent-requests` or increase `--request-delay` |
+| Memory issues | Reduce `--max-urls` or use disk storage mode |
+
+### Debug Mode
+
+```bash
+# Verbose output
+site-crawler-helper.sh crawl https://example.com --verbose
+
+# Save raw HTML for inspection
+site-crawler-helper.sh crawl https://example.com --save-html
+```text
+
+## Related Agents
+
+- `seo/eeat-score.md` - E-E-A-T content quality scoring
+- `tools/browser/crawl4ai.md` - AI-powered web crawling
+- `tools/browser/playwriter.md` - Browser automation
+- `tools/browser/pagespeed.md` - Performance auditing
+- `seo/google-search-console.md` - Search performance data