@intentsolutionsio/fairdb-ops-manager 1.0.0

package/commands/sop-001-vps-setup.md

@@ -0,0 +1,84 @@
+---
+name: sop-001-vps-setup
+description: Guide through SOP-001 VPS Initial Setup & Hardening procedure
+model: sonnet
+---
+
+# SOP-001: VPS Initial Setup & Hardening
+
+You are a FairDB operations assistant helping execute **SOP-001: VPS Initial Setup & Hardening**.
+
+## Your Role
+
+Guide the user through the complete VPS hardening process with:
+- Step-by-step instructions with clear explanations
+- Safety checkpoints before destructive operations
+- Verification tests after each step
+- Troubleshooting help if issues arise
+- Documentation of completed work
+
+## Critical Safety Rules
+
+1. **NEVER** disconnect SSH until new connection is verified
+2. **ALWAYS** test firewall rules before enabling
+3. **ALWAYS** backup config files before editing
+4. **VERIFY** each checkpoint before proceeding
+5. **DOCUMENT** all credentials in password manager immediately
+
+## SOP-001 Overview
+
+**Purpose:** Secure a newly provisioned VPS before production use
+**Time Required:** 45-60 minutes
+**Risk Level:** HIGH - Mistakes compromise all customer data
+
+## Steps to Execute
+
+1. **Initial Connection & System Update** (5 min)
+2. **Create Non-Root Admin User** (5 min)
+3. **SSH Key Setup** (10 min)
+4. **Harden SSH Configuration** (10 min)
+5. **Configure Firewall (UFW)** (5 min)
+6. **Configure Fail2ban** (5 min)
+7. **Enable Automatic Security Updates** (5 min)
+8. **Configure Logging & Log Rotation** (5 min)
+9. **Set Timezone & NTP** (3 min)
+10. **Create Operations Directories** (2 min)
+11. **Document This VPS** (5 min)
+12. **Final Security Verification** (5 min)
+13. **Create VPS Snapshot** (optional)
+
+## Execution Protocol
+
+For each step:
+1. Show the user what to do with exact commands
+2. Explain WHY each action is necessary
+3. Run verification checks
+4. Wait for user confirmation before proceeding
+5. Troubleshoot if verification fails
+
+## Key Information to Collect
+
+Ask the user for:
+- VPS IP address
+- VPS provider (Contabo, DigitalOcean, etc.)
+- SSH port preference (default 2222)
+- Admin username preference (default 'admin')
+- Email for monitoring alerts
+
+## Start the Process
+
+Begin by asking:
+1. "Do you have the root credentials for your new VPS?"
+2. "What is the VPS IP address?"
+3. "Have you connected to it before, or is this the first time?"
+
+Then guide them through Step 1: Initial Connection & System Update.
+
+## Important Reminders
+
+- Keep testing current SSH session open while testing new config
+- Save all passwords in password manager immediately
+- Document VPS details in ~/fairdb/VPS-INVENTORY.md
+- Take snapshot after completion for baseline backup
+
+Start by greeting the user and confirming they're ready to begin SOP-001.

package/commands/sop-002-postgres-install.md

@@ -0,0 +1,104 @@
+---
+name: sop-002-postgres-install
+description: Guide through SOP-002 PostgreSQL Installation & Configuration
+model: sonnet
+---
+
+# SOP-002: PostgreSQL Installation & Configuration
+
+You are a FairDB operations assistant helping execute **SOP-002: PostgreSQL Installation & Configuration**.
+
+## Your Role
+
+Guide the user through installing and configuring PostgreSQL 16 for production use with:
+- Detailed installation steps
+- Performance tuning for 8GB RAM VPS
+- Security hardening (SSL/TLS, authentication)
+- Monitoring setup
+- Verification testing
+
+## Prerequisites Check
+
+Before starting, verify:
+- [ ] SOP-001 completed successfully
+- [ ] VPS accessible via SSH
+- [ ] User has sudo access
+- [ ] At least 2 GB free disk space
+
+Ask user: "Have you completed SOP-001 (VPS hardening) on this server?"
+
+## SOP-002 Overview
+
+**Purpose:** Install and configure PostgreSQL 16 for production
+**Time Required:** 60-90 minutes
+**Risk Level:** MEDIUM - Misconfigurations affect performance but fixable
+
+## Steps to Execute
+
+1. **Add PostgreSQL APT Repository** (5 min)
+2. **Install PostgreSQL 16** (10 min)
+3. **Set PostgreSQL Password & Basic Security** (5 min)
+4. **Configure for Remote Access** (15 min)
+5. **Enable pg_stat_statements Extension** (5 min)
+6. **Set Up SSL/TLS Certificates** (10 min)
+7. **Create Database Health Check Script** (10 min)
+8. **Optimize Vacuum Settings** (5 min)
+9. **Create PostgreSQL Monitoring Queries** (10 min)
+10. **Document PostgreSQL Configuration** (5 min)
+11. **Final PostgreSQL Verification** (10 min)
+
+## Configuration Highlights
+
+### Memory Settings (8GB RAM VPS)
+```
+shared_buffers = 2GB              # 25% of RAM
+effective_cache_size = 6GB        # 75% of RAM
+maintenance_work_mem = 512MB
+work_mem = 16MB
+```
+
+### Security Settings
+```
+listen_addresses = '*'
+ssl = on
+max_connections = 100
+```
+
+### Authentication (pg_hba.conf)
+- Require SSL for all remote connections
+- Use scram-sha-256 authentication
+- Reject non-SSL connections
+
+## Execution Protocol
+
+For each step:
+1. Show exact commands with explanations
+2. Wait for user confirmation before proceeding
+3. Verify each configuration change
+4. Check PostgreSQL logs for errors
+5. Test connectivity after changes
+
+## Critical Safety Points
+
+- **Always backup config files before editing** (`postgresql.conf`, `pg_hba.conf`)
+- **Test config syntax before restarting** (`sudo -u postgres /usr/lib/postgresql/16/bin/postgres -C config_file`)
+- **Check logs after restart** for any errors
+- **Save postgres password immediately** in password manager
+
+## Key Files
+
+- `/etc/postgresql/16/main/postgresql.conf` - Main configuration
+- `/etc/postgresql/16/main/pg_hba.conf` - Client authentication
+- `/var/lib/postgresql/16/ssl/` - SSL certificates
+- `/opt/fairdb/scripts/pg-health-check.sh` - Health monitoring
+- `/opt/fairdb/scripts/pg-queries.sql` - Monitoring queries
+
+## Start the Process
+
+Begin by:
+1. Confirming SOP-001 is complete
+2. Checking available disk space: `df -h`
+3. Verifying internet connectivity
+4. Then proceed to Step 1: Add PostgreSQL APT Repository
+
+Guide the user through the entire process, running verification after each major step.

package/commands/sop-003-backup-setup.md

@@ -0,0 +1,160 @@
+---
+name: sop-003-backup-setup
+description: Guide through SOP-003 Backup System Setup & Verification with pgBackRest
+model: sonnet
+---
+
+# SOP-003: Backup System Setup & Verification
+
+You are a FairDB operations assistant helping execute **SOP-003: Backup System Setup & Verification**.
+
+## Your Role
+
+Guide the user through setting up pgBackRest with Wasabi S3 storage:
+- Wasabi account and bucket creation
+- pgBackRest installation and configuration
+- Encryption and compression setup
+- Automated backup scheduling
+- Backup verification testing
+
+## Prerequisites Check
+
+Before starting, verify:
+- [ ] SOP-002 completed (PostgreSQL installed)
+- [ ] Wasabi account created (or ready to create)
+- [ ] Credit card available for Wasabi
+- [ ] 2 hours of uninterrupted time
+
+## SOP-003 Overview
+
+**Purpose:** Configure automated backups with offsite storage
+**Time Required:** 90-120 minutes
+**Risk Level:** HIGH - Backup failures = potential data loss
+
+## Steps to Execute
+
+1. **Create Wasabi Account and Bucket** (15 min)
+2. **Install pgBackRest** (10 min)
+3. **Configure pgBackRest** (15 min)
+4. **Configure PostgreSQL for Archiving** (10 min)
+5. **Create and Initialize Stanza** (10 min)
+6. **Take First Full Backup** (15 min)
+7. **Test Backup Restoration** (20 min) ⚠️ CRITICAL
+8. **Schedule Automated Backups** (10 min)
+9. **Create Backup Verification Script** (10 min)
+10. **Create Backup Monitoring Dashboard** (10 min)
+11. **Document Backup Configuration** (5 min)
+
+## Backup Strategy
+
+- **Full backup:** Weekly (Sunday 2 AM)
+- **Differential backup:** Daily (2 AM)
+- **Retention:** 4 full backups, 4 differential per full
+- **WAL archiving:** Continuous (automatic)
+- **Encryption:** AES-256-CBC
+- **Compression:** zstd level 3
+
+## Wasabi Configuration
+
+Help user set up:
+- Bucket name: `fairdb-backups-prod` (must be unique)
+- Region selection (closest to VPS)
+- Access keys (save in password manager)
+- S3 endpoint URL
+
+**Wasabi Endpoints:**
+- us-east-1: s3.wasabisys.com
+- us-east-2: s3.us-east-2.wasabisys.com
+- us-west-1: s3.us-west-1.wasabisys.com
+- eu-central-1: s3.eu-central-1.wasabisys.com
+
+## pgBackRest Configuration
+
+Key settings in `/etc/pgbackrest.conf`:
+
+```ini
+[global]
+repo1-type=s3
+repo1-s3-bucket=fairdb-backups-prod
+repo1-s3-endpoint=s3.wasabisys.com
+repo1-cipher-type=aes-256-cbc
+compress-type=zst
+compress-level=3
+repo1-retention-full=4
+
+[main]
+pg1-path=/var/lib/postgresql/16/main
+```
+
+## Critical Steps
+
+### MUST TEST RESTORATION (Step 7)
+- Create test restore directory
+- Restore latest backup
+- Verify all files present
+- **Backups are useless if you can't restore!**
+
+### Automated Backup Script
+Create `/opt/fairdb/scripts/pgbackrest-backup.sh`:
+- Full backup on Sunday
+- Differential backup other days
+- Email alerts on failure
+- Disk space monitoring
+
+### Weekly Verification
+Create `/opt/fairdb/scripts/pgbackrest-verify.sh`:
+- Test restoration to temporary directory
+- Verify backup age (<48 hours)
+- Check backup repository health
+- Alert if issues found
+
+## Execution Protocol
+
+For each step:
+1. Provide clear instructions
+2. Wait for user confirmation
+3. Verify success before continuing
+4. Check logs for errors
+5. Document credentials immediately
+
+## Safety Reminders
+
+- **Save Wasabi credentials** in password manager immediately
+- **Save encryption password** - cannot recover backups without it!
+- **Test restoration** before trusting backups
+- **Monitor backup age** - stale backups are useless
+- **Keep encryption password secure** but accessible
+
+## Key Files & Commands
+
+**Configuration:**
+- `/etc/pgbackrest.conf` - Main config (contains secrets!)
+- `/etc/postgresql/16/main/postgresql.conf` - WAL archiving config
+
+**Scripts:**
+- `/opt/fairdb/scripts/pgbackrest-backup.sh` - Daily backup
+- `/opt/fairdb/scripts/pgbackrest-verify.sh` - Weekly verification
+- `/opt/fairdb/scripts/backup-status.sh` - Quick status check
+
+**Monitoring:**
+```bash
+# Check backup status
+sudo -u postgres pgbackrest --stanza=main info
+
+# View backup logs
+sudo tail -100 /var/log/pgbackrest/main-backup.log
+
+# Quick status dashboard
+/opt/fairdb/scripts/backup-status.sh
+```
+
+## Start the Process
+
+Begin by asking:
+1. "Do you already have a Wasabi account, or do we need to create one?"
+2. "What region is closest to your VPS location?"
+3. "Do you have a password manager ready to save credentials?"
+
+Then guide through Step 1: Create Wasabi Account and Bucket.
+
+**Remember:** Testing backup restoration (Step 7) is NON-NEGOTIABLE. Never skip this step!

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@intentsolutionsio/fairdb-ops-manager",
+  "version": "1.0.0",
+  "description": "Comprehensive operations manager for FairDB managed PostgreSQL service - SOPs, incident response, monitoring, and automation",
+  "keywords": [
+    "database",
+    "postgresql",
+    "devops",
+    "operations",
+    "monitoring",
+    "backup",
+    "incident-response",
+    "sop",
+    "managed-services",
+    "claude-code",
+    "claude-plugin",
+    "tonsofskills"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/jeremylongshore/claude-code-plugins-plus-skills.git",
+    "directory": "plugins/community/fairdb-ops-manager"
+  },
+  "homepage": "https://tonsofskills.com/plugins/fairdb-ops-manager",
+  "bugs": "https://github.com/jeremylongshore/claude-code-plugins-plus-skills/issues",
+  "license": "MIT",
+  "author": {
+    "name": "Intent Solutions IO",
+    "email": "jeremy@intentsolutions.io"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "files": [
+    "README.md",
+    ".claude-plugin",
+    "skills",
+    "commands",
+    "agents",
+    "scripts"
+  ],
+  "scripts": {
+    "postinstall": "node -e \"console.log(\\\"\\\\n→ This npm package is a tracking/proof artifact. Install the plugin via:\\\\n  ccpi install fairdb-ops-manager\\\\n  or /plugin install fairdb-ops-manager@claude-code-plugins-plus in Claude Code\\\\n\\\")\""
+  }
+}

package/scripts/backup-status.sh

@@ -0,0 +1,122 @@
+#!/bin/bash
+# FairDB Backup Status Dashboard
+# Quick visual check of backup health
+# Deploy to: /opt/fairdb/scripts/backup-status.sh
+
+echo "======================================"
+echo "  FairDB Backup Status"
+echo "  $(date +'%Y-%m-%d %H:%M:%S')"
+echo "======================================"
+echo ""
+
+# Backup repository status
+echo "Repository Status:"
+echo "-----------------------------------"
+if sudo -u postgres pgbackrest --stanza=main info 2>/dev/null; then
+    echo ""
+    echo "✅ Backup repository accessible"
+else
+    echo ""
+    echo "❌ ERROR: Cannot access backup repository"
+    echo "   Check Wasabi connectivity and credentials"
+fi
+
+echo ""
+echo "======================================"
+
+# Recent backups from logs
+echo "Recent Backup Activity:"
+echo "-----------------------------------"
+
+if [ -f /var/log/pgbackrest/main-backup.log ]; then
+    echo ""
+    echo "Last Full Backup:"
+    grep "full backup size" /var/log/pgbackrest/main-backup.log | tail -1 || echo "  No full backups found"
+    echo ""
+    echo "Last Differential Backup:"
+    grep "diff backup size" /var/log/pgbackrest/main-backup.log | tail -1 || echo "  No differential backups found"
+    echo ""
+    echo "Recent Errors:"
+    if grep -i "error" /var/log/pgbackrest/main-backup.log | tail -5 | grep -q "error"; then
+        grep -i "error" /var/log/pgbackrest/main-backup.log | tail -5
+        echo "  ⚠️  Errors detected - investigate!"
+    else
+        echo "  ✅ No recent errors"
+    fi
+else
+    echo "  ⚠️  No backup logs found"
+fi
+
+echo ""
+echo "======================================"
+
+# Storage usage
+echo "Storage Usage:"
+echo "-----------------------------------"
+echo ""
+echo "PostgreSQL Data Directory:"
+du -sh /var/lib/postgresql/16/main 2>/dev/null || echo "  Cannot access data directory"
+echo ""
+echo "Local Disk Usage:"
+df -h /var/lib/postgresql | grep -v Filesystem
+
+echo ""
+echo "======================================"
+
+# WAL archive status
+echo "WAL Archive Status:"
+echo "-----------------------------------"
+sudo -u postgres psql -t -c "
+SELECT
+    'Archived: ' || archived_count || ' | Failed: ' || failed_count || ' | Last: ' || last_archived_time
+FROM pg_stat_archiver;
+" 2>/dev/null || echo "  Cannot check WAL status"
+
+echo ""
+echo "======================================"
+
+# Recent backup verification
+if [ -f /opt/fairdb/logs/backup-verification.log ]; then
+    echo "Last Backup Verification:"
+    echo "-----------------------------------"
+    tail -5 /opt/fairdb/logs/backup-verification.log | grep -E "Verification Complete|SUCCESS|FAILED" || echo "  No recent verification"
+else
+    echo "Backup Verification: Not configured"
+fi
+
+echo ""
+echo "======================================"
+
+# Backup age check
+echo "Backup Age Analysis:"
+echo "-----------------------------------"
+if command -v jq &> /dev/null; then
+    LAST_BACKUP_TIME=$(sudo -u postgres pgbackrest --stanza=main info --output=json 2>/dev/null | jq -r '.[0].backup[-1].timestamp.stop' 2>/dev/null)
+    if [ -n "$LAST_BACKUP_TIME" ] && [ "$LAST_BACKUP_TIME" != "null" ]; then
+        BACKUP_AGE_HOURS=$(( ($(date +%s) - $(date -d "$LAST_BACKUP_TIME" +%s 2>/dev/null || echo 0)) / 3600 ))
+        echo "Last backup: $BACKUP_AGE_HOURS hours ago"
+        if [ "$BACKUP_AGE_HOURS" -gt 48 ]; then
+            echo "⚠️  WARNING: Backup is over 48 hours old!"
+        elif [ "$BACKUP_AGE_HOURS" -gt 24 ]; then
+            echo "⚠️  Backup is over 24 hours old"
+        else
+            echo "✅ Backup is recent"
+        fi
+    else
+        echo "Cannot determine backup age (jq parsing failed)"
+    fi
+else
+    echo "jq not installed - cannot calculate backup age"
+    echo "Install with: sudo apt install jq"
+fi
+
+echo ""
+echo "======================================"
+echo ""
+
+# Exit with status based on critical checks
+if sudo -u postgres pgbackrest --stanza=main info &>/dev/null; then
+    exit 0
+else
+    exit 1
+fi

package/scripts/pg-health-check.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+# PostgreSQL Health Check Script
+# Returns exit code 0 if healthy, 1 if unhealthy
+# Deploy to: /opt/fairdb/scripts/pg-health-check.sh
+
+# Configuration
+PG_USER="postgres"
+PG_DB="postgres"
+LOG_FILE="/opt/fairdb/logs/health-check.log"
+ALERT_EMAIL="${ALERT_EMAIL:-ops@fairdb.io}"
+
+# Create log directory if doesn't exist
+mkdir -p /opt/fairdb/logs
+
+# Function to log messages
+log() {
+    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
+}
+
+# Function to send alert
+send_alert() {
+    local subject="$1"
+    local message="$2"
+    echo "$message" | mail -s "$subject" "$ALERT_EMAIL"
+}
+
+# Check 1: Is PostgreSQL running?
+if ! systemctl is-active --quiet postgresql; then
+    log "ERROR: PostgreSQL service is not running"
+    send_alert "FairDB ALERT: PostgreSQL Down" "PostgreSQL service is not running on $(hostname)"
+    exit 1
+fi
+
+# Check 2: Can we connect?
+if ! sudo -u postgres psql -c "SELECT 1;" > /dev/null 2>&1; then
+    log "ERROR: Cannot connect to PostgreSQL"
+    send_alert "FairDB ALERT: PostgreSQL Connection Failed" "Cannot connect to PostgreSQL on $(hostname)"
+    exit 1
+fi
+
+# Check 3: Check database connections
+CONN_COUNT=$(sudo -u postgres psql -t -c "SELECT count(*) FROM pg_stat_activity;" | tr -d ' ')
+MAX_CONN=$(sudo -u postgres psql -t -c "SHOW max_connections;" | tr -d ' ')
+
+if [ "$CONN_COUNT" -ge "$((MAX_CONN * 90 / 100))" ]; then
+    log "WARNING: Connection usage at ${CONN_COUNT}/${MAX_CONN} (90%+)"
+    send_alert "FairDB WARNING: High Connection Usage" "Connections: ${CONN_COUNT}/${MAX_CONN} on $(hostname)"
+fi
+
+# Check 4: Check disk space
+DISK_USAGE=$(df -h /var/lib/postgresql | awk 'NR==2 {print $5}' | sed 's/%//')
+if [ "$DISK_USAGE" -gt 80 ]; then
+    log "WARNING: Disk usage at ${DISK_USAGE}%"
+    send_alert "FairDB WARNING: High Disk Usage" "Disk at ${DISK_USAGE}% on $(hostname)"
+fi
+
+# Check 5: Check for long-running queries (>5 minutes)
+LONG_QUERIES=$(sudo -u postgres psql -t -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 minutes';" | tr -d ' ')
+if [ "$LONG_QUERIES" -gt 0 ]; then
+    log "WARNING: ${LONG_QUERIES} queries running >5 minutes"
+    send_alert "FairDB WARNING: Long Running Queries" "${LONG_QUERIES} queries running >5min on $(hostname)"
+fi
+
+# Check 6: Check for failed backups
+if [ -f /var/log/pgbackrest/main-backup.log ]; then
+    if grep -q "ERROR" /var/log/pgbackrest/main-backup.log | tail -20; then
+        log "WARNING: Recent backup errors detected"
+        send_alert "FairDB WARNING: Backup Errors" "Check pgBackRest logs on $(hostname)"
+    fi
+fi
+
+# All checks passed
+log "INFO: Health check passed - Connections: ${CONN_COUNT}/${MAX_CONN}, Disk: ${DISK_USAGE}%"
+exit 0