aidevops 2.52.1

package/.agent/seo/google-search-console.md

@@ -0,0 +1,366 @@
+---
+description: Google Search Console MCP setup and usage
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Google Search Console MCP Setup & Usage
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **MCP Integration**: Google Search Console API for AI assistants
+- **Credentials**: `~/.config/aidevops/gsc-credentials.json` (service account JSON key)
+- **Setup**: Google Cloud Project → Enable Search Console API → Service Account → Add to GSC properties
+- **Capabilities**: Search analytics, URL inspection, indexing requests, sitemap management
+- **Key Methods**: `getSearchAnalytics()`, `getTopPages()`, `getTopQueries()`, `getCoreWebVitals()`
+- **Metrics**: clicks, impressions, ctr, position
+- **Dimensions**: query, page, country, device, searchAppearance
+- **Bulk Setup**: Use Playwright to add service account to all GSC properties automatically
+
+## Setup Steps
+
+### 1. Google Cloud Project Setup
+
+1. Go to [Google Cloud Console](https://console.cloud.google.com)
+2. Create or select a project
+3. Enable the **Google Search Console API**
+4. Go to **Credentials** → **Create Credentials** → **Service Account**
+5. Name it (e.g., `aidevops`) and create
+6. Go to **Keys** tab → **Add Key** → **Create new key** → **JSON**
+7. Save the downloaded file to `~/.config/aidevops/gsc-credentials.json`
+8. Set permissions: `chmod 600 ~/.config/aidevops/gsc-credentials.json`
+
+### 2. Add Service Account to GSC Properties
+
+The service account email (e.g., `aidevops@project-id.iam.gserviceaccount.com`) must be added as a user to each GSC property.
+
+**Manual method**: GSC → Property → Settings → Users and permissions → Add user
+
+**Automated method**: Use Playwright to bulk-add to all properties (see below)
+
+### 3. Verify Access
+
+```bash
+# Test GSC MCP connection
+opencode mcp list | grep -i search
+```
+<!-- AI-CONTEXT-END -->
+
+## Automated Bulk Setup with Playwright
+
+Add the service account to all GSC properties automatically:
+
+```javascript
+// Save as gsc-add-service-account.js
+import { chromium } from 'playwright';
+
+const SERVICE_ACCOUNT = "your-service-account@project.iam.gserviceaccount.com";
+
+async function main() {
+    // Launch Chrome with user profile (logged into Google)
+    const browser = await chromium.launchPersistentContext(
+        '/Users/USERNAME/Library/Application Support/Google/Chrome/Default',
+        { headless: false, channel: 'chrome' }
+    );
+    
+    const page = await browser.newPage();
+    await page.goto("https://search.google.com/search-console", { waitUntil: 'networkidle' });
+    await page.waitForTimeout(2000);
+    
+    // Get all domains from the page
+    const html = await page.content();
+    const domainRegex = /sc-domain:([a-z0-9.-]+)/g;
+    const matches = [...html.matchAll(domainRegex)];
+    const domains = [...new Set(matches.map(m => m[1]))];
+    
+    console.log(`Found ${domains.length} properties`);
+    
+    for (const domain of domains) {
+        console.log(`Processing ${domain}...`);
+        
+        try {
+            await page.goto(`https://search.google.com/search-console/users?resource_id=sc-domain:${domain}`, 
+                { waitUntil: 'networkidle' });
+            await page.waitForTimeout(400);
+            
+            const content = await page.content();
+            
+            // Skip if no access or already added
+            if (content.includes("don't have access")) {
+                console.log(`  ⏭ No access`);
+                continue;
+            }
+            if (content.includes(SERVICE_ACCOUNT)) {
+                console.log(`  ⏭ Already added`);
+                continue;
+            }
+            
+            // Click ADD USER, type email, press Enter
+            await page.click('text=ADD USER');
+            await page.waitForTimeout(400);
+            await page.keyboard.type(SERVICE_ACCOUNT, { delay: 5 });
+            await page.keyboard.press('Enter');
+            await page.waitForTimeout(1000);
+            
+            console.log(`  ✓ Added`);
+        } catch (error) {
+            console.error(`  ✗ ${error.message}`);
+        }
+    }
+    
+    await browser.close();
+}
+
+main().catch(console.error);
+```
+
+Run with: `node gsc-add-service-account.js`
+
+**Requirements**:
+- Playwright installed: `npm install playwright`
+- Chrome browser with logged-in Google session
+- User must have Owner access to GSC properties
+
+## Search Performance Analysis
+
+### **Query Performance Metrics**
+
+```javascript
+// Get search performance data for specific queries
+await googleSearchConsole.getSearchAnalytics({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-01-01",
+  endDate: "2024-12-31",
+  dimensions: ["query", "page", "country", "device"],
+  metrics: ["clicks", "impressions", "ctr", "position"]
+});
+```
+
+### **Top Performing Pages**
+
+```javascript
+// Analyze top performing pages by clicks
+await googleSearchConsole.getTopPages({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  orderBy: "clicks",
+  limit: 50
+});
+```
+
+## Keyword Research & Analysis
+
+### **Top Search Queries**
+
+```javascript
+// Get top search queries driving traffic
+await googleSearchConsole.getTopQueries({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-10-01",
+  endDate: "2024-10-31",
+  orderBy: "impressions",
+  limit: 100
+});
+```
+
+### **Query Position Tracking**
+
+```javascript
+// Track specific keyword positions over time
+await googleSearchConsole.trackKeywordPositions({
+  siteUrl: "https://your-website.com",
+  queries: ["ai assisted devops", "automation framework", "mcp integration"],
+  startDate: "2024-01-01",
+  endDate: "2024-12-31",
+  groupBy: "month"
+});
+```
+
+## Device & Geographic Analysis
+
+### **Device Performance Breakdown**
+
+```javascript
+// Analyze performance across different devices
+await googleSearchConsole.getDevicePerformance({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  devices: ["desktop", "mobile", "tablet"],
+  metrics: ["clicks", "impressions", "ctr", "position"]
+});
+```
+
+### **Geographic Performance**
+
+```javascript
+// Get performance data by country
+await googleSearchConsole.getGeographicPerformance({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  countries: ["USA", "GBR", "CAN", "AUS"],
+  orderBy: "clicks"
+});
+```
+
+## Technical SEO Monitoring
+
+### **Index Coverage Analysis**
+
+```javascript
+// Check index coverage status
+await googleSearchConsole.getIndexCoverage({
+  siteUrl: "https://your-website.com",
+  category: "all", // or "error", "valid", "excluded", "warning"
+  platform: "web" // or "mobile"
+});
+```
+
+### **Core Web Vitals Monitoring**
+
+```javascript
+// Monitor Core Web Vitals performance
+await googleSearchConsole.getCoreWebVitals({
+  siteUrl: "https://your-website.com",
+  category: "all", // or "good", "needs_improvement", "poor"
+  platform: "desktop" // or "mobile"
+});
+```
+
+## Competitive Analysis
+
+### **Search Appearance Features**
+
+```javascript
+// Analyze search appearance features (rich snippets, etc.)
+await googleSearchConsole.getSearchAppearance({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  searchAppearance: ["richSnippet", "ampBlueLink", "ampNonRichResult"]
+});
+```
+
+### **Click-Through Rate Optimization**
+
+```javascript
+// Identify pages with high impressions but low CTR
+await googleSearchConsole.getCTROpportunities({
+  siteUrl: "https://your-website.com",
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  minImpressions: 100,
+  maxCTR: 0.05, // 5% CTR threshold
+  orderBy: "impressions"
+});
+```
+
+## Issue Detection & Monitoring
+
+### **Manual Actions Check**
+
+```javascript
+// Check for manual actions against the site
+await googleSearchConsole.getManualActions({
+  siteUrl: "https://your-website.com"
+});
+```
+
+### **Security Issues Monitoring**
+
+```javascript
+// Monitor security issues
+await googleSearchConsole.getSecurityIssues({
+  siteUrl: "https://your-website.com"
+});
+```
+
+## Reporting & Analytics
+
+### **Monthly Performance Report**
+
+```javascript
+// Generate comprehensive monthly report
+await googleSearchConsole.generateMonthlyReport({
+  siteUrl: "https://your-website.com",
+  month: "2024-11",
+  includeMetrics: ["clicks", "impressions", "ctr", "position"],
+  includeDimensions: ["query", "page", "country", "device"],
+  exportFormat: "json" // or "csv"
+});
+```
+
+### **Competitor Comparison**
+
+```javascript
+// Compare performance with competitor keywords
+await googleSearchConsole.compareWithCompetitors({
+  siteUrl: "https://your-website.com",
+  competitorQueries: ["devops automation", "infrastructure management"],
+  startDate: "2024-11-01",
+  endDate: "2024-11-30",
+  metrics: ["position", "clicks", "impressions"]
+});
+```
+
+## Troubleshooting
+
+### Empty Results from API
+
+If the API returns empty results `{}`:
+- Service account not added to any GSC properties
+- Use the Playwright script above to bulk-add access
+
+### "No access to property" Error
+
+- The service account email must be added as a user (Full or Owner permission)
+- Domain must be verified in GSC first
+
+### Connection Issues
+
+```bash
+# Check credentials file exists and has correct permissions
+ls -la ~/.config/aidevops/gsc-credentials.json
+
+# Verify service account email
+cat ~/.config/aidevops/gsc-credentials.json | grep client_email
+
+# Test MCP connection
+opencode mcp list
+```
+
+### Chrome Profile Path (for Playwright)
+
+- **macOS**: `/Users/USERNAME/Library/Application Support/Google/Chrome/Default`
+- **Linux**: `~/.config/google-chrome/Default`
+- **Windows**: `%LOCALAPPDATA%\Google\Chrome\User Data\Default`
+
+## MCP Configuration
+
+Add to your MCP config (`~/.config/opencode/mcp.json` or similar):
+
+```json
+{
+  "mcpServers": {
+    "google-search-console": {
+      "command": "npx",
+      "args": ["-y", "@anthropic/google-search-console-mcp"],
+      "env": {
+        "GOOGLE_APPLICATION_CREDENTIALS": "~/.config/aidevops/gsc-credentials.json"
+      }
+    }
+  }
+}
+```

package/.agent/seo/gsc-sitemaps.md

@@ -0,0 +1,282 @@
+---
+description: Google Search Console sitemap submission via Playwright browser automation
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Google Search Console Sitemap Submission
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Purpose**: Automate sitemap submissions to Google Search Console
+- **Method**: Playwright browser automation with persistent Chrome profile
+- **Script**: `~/.aidevops/agents/scripts/gsc-sitemap-helper.sh`
+- **Config**: `~/.config/aidevops/gsc-config.json`
+- **Profile**: `~/.aidevops/.agent-workspace/chrome-gsc-profile/`
+- **Screenshots**: `/tmp/gsc-screenshots/` (verification)
+
+**Commands**:
+
+```bash
+# Submit sitemap for single domain
+~/.aidevops/agents/scripts/gsc-sitemap-helper.sh submit example.com
+
+# Submit sitemaps for multiple domains
+~/.aidevops/agents/scripts/gsc-sitemap-helper.sh submit example.com example.net example.org
+
+# Submit from file (one domain per line)
+~/.aidevops/agents/scripts/gsc-sitemap-helper.sh submit --file domains.txt
+
+# Check sitemap status
+~/.aidevops/agents/scripts/gsc-sitemap-helper.sh status example.com
+
+# List all sitemaps for a domain
+~/.aidevops/agents/scripts/gsc-sitemap-helper.sh list example.com
+```
+
+**Note**: Add `~/.aidevops/agents/scripts` to your PATH for shorter commands, or use the full path as shown above.
+
+**Prerequisites**:
+- Domain verified in Google Search Console
+- Sitemap accessible at URL (test with `curl https://example.com/sitemap.xml`)
+- User logged into Google in the Chrome profile
+- Node.js and Playwright installed
+
+<!-- AI-CONTEXT-END -->
+
+## When to Use
+
+- After deploying a new site
+- After adding `sitemap.xml` to a domain
+- When setting up multiple domains (like a portfolio of sites)
+- After major site restructuring that changes sitemap content
+
+## How It Works
+
+1. Opens Chrome with persistent profile (preserves Google login)
+2. Navigates to GSC sitemaps page for each domain
+3. Fills sitemap URL in "Add a new sitemap" input
+4. Clicks SUBMIT button (finds it relative to input, not sidebar feedback button)
+5. Verifies success via screenshot
+6. Handles domains that already have sitemaps submitted
+
+## Technical Details
+
+### Chrome Profile Setup
+
+Uses `chromium.launchPersistentContext()` with a dedicated profile directory and stealth flags to avoid "browser isn't secure" warnings:
+
+```javascript
+{
+    ignoreDefaultArgs: ['--enable-automation'],
+    args: [
+        '--disable-blink-features=AutomationControlled',
+        '--disable-infobars',
+        '--no-first-run',
+        '--no-default-browser-check'
+    ]
+}
+```
+
+### GSC UI Specifics
+
+- **SUBMIT button**: A `div[role="button"]` not a `<button>` element
+- **Sidebar conflict**: There's also a "Submit feedback" link - must avoid clicking that
+- **Button location**: Found relative to input field (walk up DOM to find container)
+- **Input format**: Requires FULL URL: `https://www.domain.com/sitemap.xml` not just `sitemap.xml`
+- **Button state**: Disabled until input has valid content
+
+### Finding the Correct Submit Button
+
+```javascript
+const submitBtn = await input.evaluateHandle(el => {
+    let parent = el.parentElement;
+    for (let i = 0; i < 10; i++) {
+        if (!parent) break;
+        const btn = parent.querySelector('[role="button"]');
+        if (btn && btn.textContent.trim().toUpperCase() === 'SUBMIT') {
+            return btn;
+        }
+        parent = parent.parentElement;
+    }
+    return null;
+});
+```
+
+### Detecting Already Submitted
+
+Look for sitemap in table (not just page content, as "sitemap.xml" appears in input placeholder):
+
+```javascript
+const sitemapInTable = await page.$('table:has-text("sitemap.xml")') ||
+                       await page.$('tr:has-text("sitemap.xml")') ||
+                       await page.$('[role="row"]:has-text("sitemap.xml")');
+```
+
+## Configuration
+
+Store in `~/.config/aidevops/gsc-config.json`:
+
+```json
+{
+  "chrome_profile_dir": "~/.aidevops/.agent-workspace/chrome-gsc-profile",
+  "default_sitemap_path": "sitemap.xml",
+  "screenshot_dir": "/tmp/gsc-screenshots",
+  "timeout_ms": 60000,
+  "headless": false
+}
+```
+
+## First-Time Setup
+
+1. **Install dependencies**:
+
+   ```bash
+   ~/.aidevops/agents/scripts/gsc-sitemap-helper.sh setup
+   ```
+
+2. **Login to Google** (first run opens browser for manual login):
+
+   ```bash
+   ~/.aidevops/agents/scripts/gsc-sitemap-helper.sh login
+   ```
+
+3. **Verify access**:
+
+   ```bash
+   ~/.aidevops/agents/scripts/gsc-sitemap-helper.sh list example.com
+   ```
+
+## Usage Examples
+
+### Single Domain
+
+```bash
+# Submit sitemap for one domain
+gsc-sitemap-helper.sh submit example.com
+
+# With custom sitemap path
+gsc-sitemap-helper.sh submit example.com --sitemap news-sitemap.xml
+```
+
+### Multiple Domains
+
+```bash
+# Submit to multiple domains
+gsc-sitemap-helper.sh submit example.com example.net example.org
+
+# From file
+echo -e "example.com\nexample.net\nexample.org" > domains.txt
+gsc-sitemap-helper.sh submit --file domains.txt
+```
+
+### Status Checking
+
+```bash
+# Check if sitemap is submitted
+gsc-sitemap-helper.sh status example.com
+
+# List all sitemaps for a domain
+gsc-sitemap-helper.sh list example.com
+```
+
+### Batch Operations
+
+```bash
+# Dry run (show what would be done)
+gsc-sitemap-helper.sh submit --dry-run example.com example.net
+
+# Skip already-submitted domains
+gsc-sitemap-helper.sh submit --skip-existing example.com example.net
+```
+
+## Troubleshooting
+
+### "Browser isn't secure" Warning
+
+The script uses stealth flags to prevent this. If it still appears:
+
+1. Close all Chrome instances
+2. Delete the profile: `rm -rf ~/.aidevops/.agent-workspace/chrome-gsc-profile`
+3. Run `gsc-sitemap-helper.sh login` to create fresh profile
+
+### "No access" Error
+
+- Domain not verified in GSC for this Google account
+- Check GSC manually: https://search.google.com/search-console
+
+### Submit Button Not Clicking
+
+- Check screenshot in `/tmp/gsc-screenshots/`
+- May be clicking feedback button instead of submit
+- Script uses DOM traversal to find correct button
+
+### Session Expired
+
+```bash
+# Re-login to refresh session
+gsc-sitemap-helper.sh login
+```
+
+### Sitemap Not Accessible
+
+Before submitting, verify sitemap is accessible:
+
+```bash
+curl -I https://www.example.com/sitemap.xml
+# Should return 200 OK with Content-Type: application/xml
+```
+
+## Integration with Other Tools
+
+### With Site Crawler
+
+After crawling a site, submit its sitemap:
+
+```bash
+# Crawl site
+site-crawler-helper.sh crawl https://example.com
+
+# Submit sitemap
+gsc-sitemap-helper.sh submit example.com
+```
+
+### With MainWP (WordPress Fleet)
+
+Submit sitemaps for all managed WordPress sites:
+
+```bash
+# Get domains from MainWP
+mainwp-helper.sh list-sites | awk '{print $2}' > wp-domains.txt
+
+# Submit sitemaps
+gsc-sitemap-helper.sh submit --file wp-domains.txt
+```
+
+### With Coolify Deployments
+
+After deploying a new site:
+
+```bash
+# Deploy site
+coolify-helper.sh deploy my-app
+
+# Submit sitemap
+gsc-sitemap-helper.sh submit my-app.example.com
+```
+
+## Related
+
+- `seo/google-search-console.md` - GSC API integration (analytics, not sitemaps)
+- `tools/browser/playwright.md` - Playwright automation patterns
+- `seo/site-crawler.md` - Site auditing and crawling