aidevops 2.52.1

package/.agent/tools/conversion/pandoc.md

@@ -0,0 +1,304 @@
+---
+description: Pandoc document format conversion
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: false
+  bash: true
+  glob: true
+  grep: true
+  webfetch: false
+  task: true
+---
+
+# Pandoc Document Conversion for AI DevOps
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Purpose**: Convert documents to markdown for AI processing
+- **Install**: `brew install pandoc poppler` (macOS) or `apt install pandoc poppler-utils`
+- **Helper**: `.agent/scripts/pandoc-helper.sh`
+- **Commands**: `convert [file]` | `batch [dir] [output] [pattern]` | `formats` | `detect [file]`
+- **Supported**: DOCX, PDF, HTML, EPUB, ODT, RTF, LaTeX, JSON, CSV, RST, Org-mode
+- **Output**: Markdown with ATX headers, no line wrapping, preserved structure
+- **Config**: `configs/pandoc-config.json`
+<!-- AI-CONTEXT-END -->
+
+**Convert any document format to markdown for optimal AI assistant processing**
+
+## Overview
+
+The Pandoc integration in AI DevOps Framework enables seamless conversion of various document formats to markdown, making them easily accessible and processable by AI assistants. This dramatically improves the ability to work with legacy documents, presentations, PDFs, and other formats.
+
+## Installation
+
+### **Install Pandoc**
+
+```bash
+# macOS
+brew install pandoc poppler
+
+# Ubuntu/Debian
+sudo apt-get update
+sudo apt-get install pandoc poppler-utils
+
+# CentOS/RHEL
+sudo yum install pandoc poppler-utils
+
+# Windows (Chocolatey)
+choco install pandoc
+
+# Windows (Scoop)
+scoop install pandoc
+```
+
+### **Verify Installation**
+
+```bash
+pandoc --version
+pdftotext -v  # For PDF support
+```
+
+## Quick Start
+
+### **Single File Conversion**
+
+```bash
+# Convert Word document to markdown
+bash .agent/scripts/pandoc-helper.sh convert document.docx
+
+# Convert PDF with custom output name
+bash .agent/scripts/pandoc-helper.sh convert report.pdf analysis.md
+
+# Convert with specific format and options
+bash .agent/scripts/pandoc-helper.sh convert file.html output.md html "--extract-media=./images"
+```
+
+### **Batch Conversion**
+
+```bash
+# Convert all Word documents in a directory
+bash .agent/scripts/pandoc-helper.sh batch ./documents ./markdown "*.docx"
+
+# Convert all supported formats
+bash .agent/scripts/pandoc-helper.sh batch ./input ./output "*"
+
+# Convert with specific pattern
+bash .agent/scripts/pandoc-helper.sh batch ./reports ./markdown "*.{pdf,docx,html}"
+```
+
+## Supported Formats
+
+### Document Formats
+
+- **Microsoft Word**: `.docx`, `.doc`
+- **PDF**: `.pdf` (requires pdftotext)
+- **OpenDocument**: `.odt`
+- **Rich Text**: `.rtf`
+- **LaTeX**: `.tex`, `.latex`
+
+### Web & eBook Formats
+
+- **HTML**: `.html`, `.htm`
+- **EPUB**: `.epub`
+- **MediaWiki**: `.mediawiki`
+- **TWiki**: `.twiki`
+
+### Data Formats
+
+- **JSON**: `.json`
+- **CSV**: `.csv`
+- **TSV**: `.tsv`
+- **XML**: `.xml`
+
+### Markup Formats
+
+- **reStructuredText**: `.rst`
+- **Org-mode**: `.org`
+- **Textile**: `.textile`
+- **OPML**: `.opml`
+
+### Presentation Formats
+
+- **PowerPoint**: `.pptx`, `.ppt` (limited support)
+- **Excel**: `.xlsx`, `.xls` (limited support)
+
+## Advanced Usage
+
+### **Format Detection**
+
+```bash
+# Automatically detect file format
+bash .agent/scripts/pandoc-helper.sh detect unknown_file.ext
+
+# Show all supported formats
+bash .agent/scripts/pandoc-helper.sh formats
+```
+
+### **Custom Conversion Options**
+
+```bash
+# Extract images and media
+bash .agent/scripts/pandoc-helper.sh convert document.docx output.md docx "--extract-media=./media"
+
+# Include table of contents
+bash .agent/scripts/pandoc-helper.sh convert document.html output.md html "--toc"
+
+# Create standalone document
+bash .agent/scripts/pandoc-helper.sh convert document.rst output.md rst "--standalone"
+
+# Set custom metadata
+bash .agent/scripts/pandoc-helper.sh convert document.tex output.md latex "--metadata title='My Document'"
+```
+
+## AI Assistant Integration
+
+### **Why Convert to Markdown?**
+
+1. **Consistent Formatting**: Standardized structure for AI processing
+2. **Easy Parsing**: Simple syntax that AI can understand and manipulate
+3. **Preserved Structure**: Maintains headings, lists, and formatting
+4. **Lightweight**: Fast processing and analysis
+5. **Version Control**: Git-friendly format for tracking changes
+6. **Cross-Platform**: Works everywhere without special software
+
+### **Optimal AI Workflows**
+
+```bash
+# 1. Convert documents for analysis
+bash .agent/scripts/pandoc-helper.sh batch ./project-docs ./markdown "*.{docx,pdf,html}"
+
+# 2. Process converted files with AI
+# AI can now easily read and analyze all documents
+
+# 3. Generate summaries, extract information, or create new content
+# Based on the converted markdown files
+```
+
+## Configuration
+
+### **Default Settings**
+
+The framework uses optimized settings for AI processing:
+
+- **Output Format**: Markdown with ATX headers (`# ## ###`)
+- **Line Wrapping**: None (preserves formatting)
+- **Media Extraction**: Automatic for supported formats
+- **Structure Preservation**: Maintains document hierarchy
+- **Metadata Addition**: Includes source file information
+
+### **Customization**
+
+Edit `configs/pandoc-config.json` to customize:
+
+```json
+{
+  "conversion_settings": {
+    "default_output_format": "markdown",
+    "wrap_mode": "none",
+    "header_style": "atx",
+    "include_toc": false,
+    "extract_media": true
+  }
+}
+```
+
+## Quality Assurance
+
+### **Conversion Validation**
+
+The helper script automatically:
+
+- ✅ **Validates output**: Checks for successful conversion
+- ✅ **Shows preview**: Displays first 10 lines of converted content
+- ✅ **Reports metrics**: File size and line count
+- ✅ **Error handling**: Clear error messages for failed conversions
+
+### **Best Practices**
+
+1. **Test conversions**: Always review output quality
+2. **Backup originals**: Keep source files safe
+3. **Check encoding**: Ensure proper character encoding
+4. **Validate structure**: Verify headings and formatting
+5. **Extract media**: Save images and attachments separately
+
+## Troubleshooting
+
+### **Common Issues**
+
+#### **PDF Conversion Problems**
+
+```bash
+# Install PDF support
+brew install poppler          # macOS
+sudo apt install poppler-utils # Ubuntu
+```
+
+#### **Encoding Issues**
+
+```bash
+# Specify input encoding
+pandoc -f html -t markdown --from=html+smart input.html -o output.md
+```
+
+#### **Large File Processing**
+
+```bash
+# For large files, consider splitting or using specific options
+pandoc --verbose input.pdf -o output.md
+```
+
+### **Format-Specific Notes**
+
+- **PDF**: Quality depends on source document structure
+- **PowerPoint**: Limited support, best for text content
+- **Excel**: Basic table conversion only
+- **HTML**: May need cleanup for complex layouts
+- **Word**: Generally excellent conversion quality
+
+## Performance Tips
+
+1. **Batch Processing**: Convert multiple files at once
+2. **Format Selection**: Use specific input formats when known
+3. **Media Extraction**: Extract images to separate directory
+4. **Parallel Processing**: Use multiple terminal sessions for large batches
+5. **Cleanup**: Remove temporary files after conversion
+
+## Integration Examples
+
+### **With AI Assistants**
+
+```bash
+# Convert project documentation
+bash .agent/scripts/pandoc-helper.sh batch ./docs ./ai-ready "*.{docx,pdf}"
+
+# Now AI can process all documentation:
+# "Analyze all the converted documentation and create a project summary"
+# "Extract all requirements from the converted specifications"
+# "Generate a comprehensive README from all the documentation"
+```
+
+### **With Version Control**
+
+```bash
+# Convert and commit to Git
+bash .agent/scripts/pandoc-helper.sh batch ./documents ./markdown
+git add markdown/
+git commit -m "📄 Add converted documentation for AI processing"
+```
+
+## Benefits for AI DevOps
+
+- **Legacy Document Access**: Convert old formats for modern AI processing
+- **Format Standardization**: Unified markdown format across all documents
+- **AI Optimization**: Perfect format for AI analysis and manipulation
+- **Batch Processing**: Handle large document collections efficiently
+- **Content Discovery**: Make all documents searchable and analyzable
+- **Documentation Modernization**: Update legacy docs to current standards
+
+---
+
+**Transform any document into AI-ready markdown with the AI DevOps Pandoc integration!**

package/.agent/tools/credentials/api-key-management.md

@@ -0,0 +1,154 @@
+---
+description: API key management and rotation guide
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: false
+  task: true
+---
+
+# API Key Management Guide
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Primary Method**: Environment variables (`export CODACY_API_TOKEN="..."`)
+- **Local Storage**: `configs/*-config.json` (gitignored), `~/.config/coderabbit/api_key`
+- **CI/CD**: GitHub Secrets (`SONAR_TOKEN`, `CODACY_API_TOKEN`, `GITHUB_TOKEN`)
+- **Helper Script**: `.agent/scripts/setup-local-api-keys.sh` (set, load, list)
+- **Token Sources**: Codacy (app.codacy.com/account/api-tokens), SonarCloud (sonarcloud.io/account/security)
+- **Security**: 600 permissions, never commit, regular rotation (90 days)
+- **If Compromised**: Revoke immediately → Generate new → Update local + GitHub secrets → Verify
+- **Test**: `echo "${CODACY_API_TOKEN:0:10}..."` to verify without exposing
+<!-- AI-CONTEXT-END -->
+
+## Secure API Key Storage Locations
+
+### **1. Environment Variables (Primary Method)**
+
+```bash
+# Set for current session
+export CODACY_API_TOKEN="YOUR_CODACY_API_TOKEN_HERE"
+export SONAR_TOKEN="YOUR_SONAR_TOKEN_HERE"
+
+# Add to shell profile for persistence
+echo 'export CODACY_API_TOKEN="YOUR_CODACY_API_TOKEN_HERE"' >> ~/.bashrc
+echo 'export SONAR_TOKEN="YOUR_SONAR_TOKEN_HERE"' >> ~/.bashrc
+```
+
+### 2. Local Configuration Files (Gitignored)
+
+```text
+# Repository configs (gitignored)
+configs/codacy-config.json          # Codacy API configuration
+configs/sonar-config.json           # SonarCloud configuration (if needed)
+
+# CLI-specific storage
+~/.config/coderabbit/api_key         # CodeRabbit CLI token
+~/.codacy/config                     # Codacy CLI configuration (if used)
+```
+
+### 3. GitHub Repository Secrets
+
+```text
+# Required for GitHub Actions
+SONAR_TOKEN                          # SonarCloud analysis
+CODACY_API_TOKEN                     # Codacy analysis
+GITHUB_TOKEN                         # Automatic (provided by GitHub)
+```
+
+## Current API Key Status
+
+### **✅ CONFIGURED:**
+
+- **Codacy API Token**: `[CONFIGURED LOCALLY]` (Local environment + config file)
+- **CodeRabbit CLI**: Stored in `~/.config/coderabbit/api_key`
+
+### **❌ MISSING (CAUSING GITHUB ACTION FAILURES):**
+
+- **SONAR_TOKEN**: Not set in GitHub Secrets
+- **CODACY_API_TOKEN**: Not set in GitHub Secrets
+
+## Setup Instructions
+
+### **1. Get SonarCloud Token**
+
+1. Go to: https://sonarcloud.io/account/security
+2. Generate new token with project analysis permissions
+3. Copy the token value
+
+### **2. Add GitHub Secrets**
+
+1. Go to: https://github.com/marcusquinn/aidevops/settings/secrets/actions
+2. Click "New repository secret"
+3. Add:
+   - Name: `SONAR_TOKEN`, Value: [Your SonarCloud token]
+   - Name: `CODACY_API_TOKEN`, Value: [Your Codacy API token]
+
+### **3. Set Local API Keys Securely**
+
+```bash
+# Use secure local storage (RECOMMENDED)
+bash .agent/scripts/setup-local-api-keys.sh set codacy YOUR_CODACY_API_TOKEN
+bash .agent/scripts/setup-local-api-keys.sh set sonar YOUR_SONAR_TOKEN
+
+# Load all API keys into environment when needed
+bash .agent/scripts/setup-local-api-keys.sh load
+
+# List configured services
+bash .agent/scripts/setup-local-api-keys.sh list
+```
+
+### **4. Test Configuration**
+
+```bash
+# Test Codacy CLI
+cd git/aidevops
+bash .agent/scripts/codacy-cli.sh analyze
+
+# Test environment variables
+echo "Codacy token: ${CODACY_API_TOKEN:0:10}..."
+echo "Sonar token: ${SONAR_TOKEN:0:10}..."
+```
+
+## Security Audit Checklist
+
+### **✅ SECURE STORAGE:**
+
+- [ ] API keys in environment variables (not hardcoded)
+- [ ] Local config files are gitignored
+- [ ] GitHub Secrets configured for CI/CD
+- [ ] No API keys in commit messages or code
+
+### **✅ ACCESS CONTROL:**
+
+- [ ] Minimal required permissions for each token
+- [ ] Regular token rotation (every 90 days)
+- [ ] Revoke old tokens immediately after replacement
+- [ ] Monitor token usage and access logs
+
+### **✅ BACKUP & RECOVERY:**
+
+- [ ] Document token sources and regeneration procedures
+- [ ] Secure backup of configuration templates
+- [ ] Emergency token revocation procedures
+- [ ] Team access management for shared tokens
+
+## Emergency Procedures
+
+### **If API Key is Compromised:**
+
+1. **IMMEDIATE**: Revoke the compromised key at provider
+2. **IMMEDIATE**: Generate new API key
+3. **UPDATE**: Local environment variables
+4. **UPDATE**: GitHub repository secrets
+5. **VERIFY**: All systems working with new key
+6. **DOCUMENT**: Incident and lessons learned
+
+This guide ensures secure, professional API key management across all platforms and environments.

package/.agent/tools/credentials/api-key-setup.md

@@ -0,0 +1,224 @@
+---
+description: API key setup with secure local storage
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: false
+  task: true
+---
+
+# API Key Setup Guide - Secure Local Storage
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Secrets Location**: `~/.config/aidevops/mcp-env.sh` (600 permissions)
+- **Working Dirs**: `~/.aidevops/` (agno, stagehand, reports)
+- **Setup**: `bash ~/Git/aidevops/.agent/scripts/setup-local-api-keys.sh setup`
+
+**Commands**:
+- `set <service-name> <VALUE>` - Store API key (converts to UPPER_CASE export)
+- `add 'export VAR="value"'` - Parse and store export command
+- `get <service-name>` - Retrieve key value
+- `list` - Show configured services (keys redacted)
+
+**Common Services**: codacy-project-token, sonar-token, coderabbit-api-key, hcloud-token-*, openai-api-key
+
+**Security**: Shell startup auto-sources mcp-env.sh, never commit keys to repo
+<!-- AI-CONTEXT-END -->
+
+## Directory Structure
+
+AI DevOps uses two directories for different purposes:
+
+| Location | Purpose | Permissions |
+|----------|---------|-------------|
+| `~/.config/aidevops/` | **Secrets & credentials** | 700 (dir), 600 (files) |
+| `~/.aidevops/` | **Working directories** (agno, stagehand, reports) | Standard |
+
+## Security Principle
+
+**API keys are stored ONLY in `~/.config/aidevops/mcp-env.sh`, NEVER in repository files.**
+
+This file is automatically sourced by your shell (zsh and bash) on startup.
+
+## Setup Instructions
+
+### 1. Initialize Secure Storage
+
+```bash
+bash ~/Git/aidevops/.agent/scripts/setup-local-api-keys.sh setup
+```
+
+This will:
+
+- Create `~/.config/aidevops/` with secure permissions
+- Create `mcp-env.sh` for storing API keys
+- Add sourcing to your shell configs (`.zshrc`, `.bashrc`, `.bash_profile`)
+
+### 2. Store API Keys
+
+#### Method A: Using the helper script
+
+```bash
+# Service name format (converted to UPPER_CASE)
+bash .agent/scripts/setup-local-api-keys.sh set vercel-token YOUR_TOKEN
+# Result: export VERCEL_TOKEN="YOUR_TOKEN"
+
+bash .agent/scripts/setup-local-api-keys.sh set sonar YOUR_TOKEN
+# Result: export SONAR="YOUR_TOKEN"
+```
+
+#### Method B: Paste export commands from services
+
+Many services give you an export command like:
+
+```bash
+export VERCEL_TOKEN="abc123"
+```
+
+Use the `add` command to parse and store it:
+
+```bash
+bash .agent/scripts/setup-local-api-keys.sh add 'export VERCEL_TOKEN="abc123"'
+```
+
+#### Method C: Direct env var name
+
+```bash
+bash .agent/scripts/setup-local-api-keys.sh set SUPABASE_KEY abc123
+# Result: export SUPABASE_KEY="abc123"
+```
+
+### 3. Common Services
+
+```bash
+# Codacy - https://app.codacy.com/account/api-tokens
+bash .agent/scripts/setup-local-api-keys.sh set codacy-project-token YOUR_TOKEN
+
+# SonarCloud - https://sonarcloud.io/account/security
+bash .agent/scripts/setup-local-api-keys.sh set sonar-token YOUR_TOKEN
+
+# CodeRabbit - https://app.coderabbit.ai/settings
+bash .agent/scripts/setup-local-api-keys.sh set coderabbit-api-key YOUR_KEY
+
+# Hetzner Cloud - https://console.hetzner.cloud/projects/*/security/tokens
+bash .agent/scripts/setup-local-api-keys.sh set hcloud-token-projectname YOUR_TOKEN
+
+# OpenAI - https://platform.openai.com/api-keys
+bash .agent/scripts/setup-local-api-keys.sh set openai-api-key YOUR_KEY
+```
+
+### 4. Verify Storage
+
+```bash
+# List configured services (keys are not shown)
+bash .agent/scripts/setup-local-api-keys.sh list
+
+# Get a specific key
+bash .agent/scripts/setup-local-api-keys.sh get sonar-token
+
+# View the file directly (redacted)
+cat ~/.config/aidevops/mcp-env.sh | sed 's/=.*/=<REDACTED>/'
+```
+
+## How It Works
+
+1. **mcp-env.sh** contains all API keys as shell exports:
+
+   ```bash
+   export SONAR_TOKEN="xxx"
+   export OPENAI_API_KEY="xxx"
+   ```
+
+2. **Shell startup** sources this file automatically:
+
+   ```bash
+   # In ~/.zshrc and ~/.bashrc:
+   [[ -f ~/.config/aidevops/mcp-env.sh ]] && source ~/.config/aidevops/mcp-env.sh
+   ```
+
+3. **All processes** (terminals, scripts, MCPs) get access to the env vars
+
+## Storage Locations
+
+### Secrets (Secure - 600 permissions)
+
+- `~/.config/aidevops/mcp-env.sh` - All API keys and tokens
+
+### Working Directories (Standard permissions)
+
+- `~/.aidevops/agno/` - Agno AI framework
+- `~/.aidevops/agent-ui/` - Agent UI frontend
+- `~/.aidevops/stagehand/` - Browser automation
+- `~/.aidevops/reports/` - Generated reports
+- `~/.aidevops/mcp/` - MCP configurations
+
+### NEVER Store In
+
+- Repository files (any file in `~/Git/aidevops/`)
+- Documentation or code examples
+- Git-tracked configuration files
+
+## Security Features
+
+### File Permissions
+
+```bash
+# Verify permissions
+ls -la ~/.config/aidevops/
+# drwx------ (700) for directory
+# -rw------- (600) for mcp-env.sh
+```
+
+### Fix Permissions
+
+```bash
+chmod 700 ~/.config/aidevops
+chmod 600 ~/.config/aidevops/mcp-env.sh
+```
+
+## Troubleshooting
+
+### Key Not Found
+
+```bash
+# Check if stored
+bash .agent/scripts/setup-local-api-keys.sh get service-name
+
+# Check environment
+echo $SERVICE_NAME
+
+# Re-add if missing
+bash .agent/scripts/setup-local-api-keys.sh set service-name YOUR_KEY
+```
+
+### Changes Not Taking Effect
+
+```bash
+# Reload shell config
+source ~/.zshrc  # or ~/.bashrc
+
+# Or restart terminal
+```
+
+### Shell Integration Missing
+
+```bash
+# Re-run setup to add sourcing to shell configs
+bash .agent/scripts/setup-local-api-keys.sh setup
+```
+
+## Best Practices
+
+1. **Single source** - Always add keys via `setup-local-api-keys.sh`, never paste directly into `.zshrc`
+2. **Regular rotation** - Rotate API keys every 90 days
+3. **Minimal permissions** - Use tokens with minimal required scopes
+4. **Monitor usage** - Check API usage in provider dashboards
+5. **Never commit** - API keys should never appear in git history