@adaptic/maestro 1.0.0

package/docs/runbooks/recovery-and-failover.md

@@ -0,0 +1,588 @@
+# Recovery and Failover Runbook
+
+This document covers all recovery procedures for the CEO Brain system. Use it when something has gone wrong and you need to restore normal operations.
+
+---
+
+## Quick Reference: Recovery Decision Tree
+
+```
+System not producing outputs?
+  |
+  +-- Is the Mac mini powered on?
+  |     No --> Power on. Auto-login should resume operations. Go to "System Crash Recovery".
+  |     Yes |
+  |         +-- Is the orchestrator process running?
+  |         |     No --> Go to "Application Crash Recovery: Orchestrator".
+  |         |     Yes |
+  |         |         +-- Is the network working?
+  |         |         |     No --> Go to "Network Interruption Handling".
+  |         |         |     Yes |
+  |         |         |         +-- Are required apps running (Slack, Safari, WhatsApp)?
+  |         |         |         |     No --> Go to "Application Crash Recovery: Desktop Apps".
+  |         |         |         |     Yes --> Go to "State Reconstruction from Logs".
+```
+
+---
+
+## 1. System Crash Recovery
+
+A system crash means the Mac mini lost power, kernel panicked, or was otherwise forced to restart.
+
+### Symptoms
+
+- Mac mini was off or rebooted unexpectedly
+- No outputs generated since the crash time
+- launchd agents not running
+
+### Recovery Procedure
+
+```bash
+# Step 1: Verify the system is up and you are logged in
+whoami
+uptime
+
+# Step 2: Check if auto-login worked
+#   If you are at a login screen, log in manually.
+#   Then configure auto-login: System Settings > Users & Groups > Automatic Login
+
+# Step 3: Check for emergency stop flag
+if [ -f ~/maestro/logs/EMERGENCY_STOP ]; then
+    echo "EMERGENCY STOP flag is set. System was intentionally stopped."
+    echo "Remove the flag with: rm ~/maestro/logs/EMERGENCY_STOP"
+    echo "Do NOT resume until you have reviewed the reason for the stop."
+    exit 1
+fi
+
+# Step 4: Check launchd agents
+launchctl list | grep com.adaptic
+```
+
+If no agents are listed:
+
+```bash
+# Step 5: Reload all agents
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro.plist
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro-healthcheck.plist
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro-logrotate.plist
+
+# Step 6: Verify they loaded
+launchctl list | grep com.adaptic
+```
+
+```bash
+# Step 7: Check state integrity
+cd ~/maestro
+git status
+
+for f in config/priorities.yaml config/contacts.yaml knowledge/memory/executive-memory.yaml; do
+    yq eval '.' "$f" > /dev/null 2>&1 && echo "$f: OK" || echo "$f: CORRUPT"
+done
+```
+
+If state files are corrupt, see "State Reconstruction from Logs" below.
+
+```bash
+# Step 8: Restart required applications
+open -a Slack
+open -a Safari
+open -a WhatsApp
+
+# Step 9: Verify permissions are still granted
+#   System Settings > Privacy & Security > Accessibility
+#   System Settings > Privacy & Security > Screen Recording
+#   System Settings > Privacy & Security > Automation
+#   macOS sometimes resets permissions after a crash or OS update
+
+# Step 10: Run health check
+./scripts/healthcheck.sh 2>/dev/null || echo "Health check script not available -- verify manually"
+
+# Step 11: Log the recovery
+echo "$(date): System crash recovery complete. Uptime: $(uptime)" >> ~/maestro/logs/maintenance.log
+```
+
+### Estimating Data Loss
+
+Check the last successful outputs:
+
+```bash
+# Last morning brief
+ls -la ~/maestro/outputs/briefs/daily/*/ceo-morning-brief.md 2>/dev/null | tail -3
+
+# Last evening wrap
+ls -la ~/maestro/outputs/briefs/daily/*/ceo-evening-wrap.md 2>/dev/null | tail -3
+
+# Last comms triage
+ls -la ~/maestro/outputs/followups/daily/*/routing-log-*.md 2>/dev/null | tail -5
+
+# Last git commit (state snapshot)
+cd ~/maestro && git log --oneline -5
+```
+
+Workflows missed during downtime are not retroactively executed (stale data risk). The next scheduled run picks up from current state. Note any gaps in the morning brief.
+
+---
+
+## 2. Application Crash Recovery
+
+### Orchestrator Crash
+
+The orchestrator is the main process that coordinates all workflows.
+
+**Symptoms**: No workflows executing. `launchctl list | grep maestro` shows no PID or a non-zero exit status.
+
+```bash
+# Check current status
+launchctl list | grep com.adaptic.maestro
+
+# Check error logs
+tail -50 ~/maestro/logs/orchestrator-stderr.log
+```
+
+**Common causes and fixes**:
+
+| Cause              | Symptom in Logs                                   | Fix                                      |
+| ------------------ | ------------------------------------------------- | ---------------------------------------- |
+| API key expired    | "ANTHROPIC_API_KEY not set" or "401 Unauthorized" | Update key in `~/.maestro-env`, reload |
+| Disk full          | "No space left on device"                         | Run log rotation, clear old outputs      |
+| Corrupt state file | "YAML parse error"                                | See "State Reconstruction"               |
+| Dependency missing | "command not found"                               | Reinstall via Homebrew or npm            |
+| Crash loop         | Repeated restarts in stderr                       | Unload, fix root cause, reload           |
+
+```bash
+# If in a crash loop, stop the restarts first
+launchctl unload ~/Library/LaunchAgents/com.adaptic.maestro.plist
+
+# Fix the issue based on log analysis
+
+# Then reload
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro.plist
+
+# Verify
+launchctl list | grep com.adaptic.maestro
+```
+
+### Desktop App Crashes (Slack, Safari, WhatsApp)
+
+**Symptoms**: Follow-up messages not sent, comms triage returning empty results, desktop control errors in logs.
+
+**General recovery pattern**: Detect failure --> Log --> Attempt restart --> Verify --> Retry (max 3) --> Escalate
+
+```bash
+# Check which apps are running
+pgrep -x Slack && echo "Slack: running" || echo "Slack: NOT RUNNING"
+pgrep -x Safari && echo "Safari: running" || echo "Safari: NOT RUNNING"
+pgrep -x WhatsApp && echo "WhatsApp: running" || echo "WhatsApp: NOT RUNNING"
+
+# Restart any crashed apps
+pgrep -x Slack || open -a Slack
+pgrep -x Safari || open -a Safari
+pgrep -x WhatsApp || open -a WhatsApp
+
+# Wait for apps to launch
+sleep 15
+```
+
+**Slack-specific recovery**:
+
+1. `open -a Slack` --> wait 10 seconds --> verify via screenshot or `pgrep`.
+2. If unresponsive: `killall Slack && sleep 5 && open -a Slack`.
+3. If still failing after 3 attempts: log error, continue without Slack, notify via email.
+4. If re-authentication required: manual sign-in to Adaptic.ai workspace.
+5. Do NOT attempt more than 3 restarts in 10 minutes.
+
+**WhatsApp-specific recovery**:
+
+1. Same pattern as Slack.
+2. If the device link has expired: open WhatsApp on Mac, scan QR code from Sophie's phone.
+3. Wait for chat history to sync before resuming operations.
+
+**Safari/Gmail-specific recovery**:
+
+1. `open -a Safari` --> navigate to Gmail tab.
+2. If browser unresponsive: `killall Safari && sleep 5 && open -a Safari`.
+3. If Gmail requires re-authentication: manual sign-in as sophie@adaptic.ai.
+
+### Claude Code Process Hangs
+
+**Symptoms**: A workflow running far beyond its timeout. No new output files being created.
+
+```bash
+# Find Claude Code processes
+ps aux | grep claude | grep -v grep
+
+# Check how long the process has been running
+ps -o etime,command -p $(pgrep -f "claude.*workflow")
+
+# Kill the hung process (launchd will restart the orchestrator)
+pkill -f "claude.*workflow"
+
+# Wait for restart
+sleep 35
+launchctl list | grep com.adaptic.maestro
+```
+
+---
+
+## 3. Network Interruption Handling
+
+### Symptoms
+
+- Workflows fail at API-dependent steps
+- Health check logs show connectivity failure
+- `curl -s --max-time 5 https://api.anthropic.com` fails
+
+### Diagnosis
+
+```bash
+# Test basic connectivity
+ping -c 3 8.8.8.8
+
+# Test DNS resolution
+nslookup api.anthropic.com
+
+# Test HTTPS
+curl -s --max-time 10 -o /dev/null -w "%{http_code}" https://api.anthropic.com
+
+# Check network interface
+ifconfig en0   # Wired Ethernet
+ifconfig en1   # Wi-Fi
+```
+
+### Operations During Network Outage
+
+| Can Do                     | Cannot Do           |
+| -------------------------- | ------------------- |
+| Read local files and repos | Web searches        |
+| Update knowledge base      | Send emails         |
+| Draft communications       | Send Slack messages |
+| Run local analysis         | Fetch market data   |
+| Update decision log        | Access cloud APIs   |
+| Log rotation               | Push to git remote  |
+
+### Recovery by Cause
+
+**Wired connection down**:
+
+1. Check the Ethernet cable.
+2. Check the router/switch.
+3. `sudo ifconfig en0 down && sudo ifconfig en0 up`
+
+**DNS failing**:
+
+```bash
+sudo dscacheutil -flushcache
+sudo killall -HUP mDNSResponder
+nslookup api.anthropic.com
+```
+
+**ISP outage**:
+
+1. System retries per built-in logic.
+2. Local operations continue normally.
+3. Network-dependent workflows fail and are logged.
+4. When connectivity returns, next scheduled workflow runs normally.
+5. No retroactive execution.
+
+**Anthropic API outage**:
+
+1. Check https://status.anthropic.com.
+2. System logs failures and waits for the next scheduled run.
+3. Non-Claude workflow steps may still complete.
+
+### Post-Network-Recovery
+
+```bash
+# Verify connectivity
+curl -s --max-time 5 https://api.anthropic.com && echo "API reachable"
+
+# Check for stale outputs
+ls -lt ~/maestro/outputs/briefs/daily/ | head -5
+
+# Push any uncommitted state that was queued during outage
+cd ~/maestro
+git push origin main
+```
+
+---
+
+## 4. State Reconstruction from Logs
+
+If state files (`executive-memory.yaml`, `priorities.yaml`) become corrupted or lost, reconstruct them using the following hierarchy of sources.
+
+### Source 1: Git History (Fastest)
+
+```bash
+cd ~/maestro
+
+# Find last known good commit for the file
+git log --oneline -- knowledge/memory/executive-memory.yaml | head -10
+
+# Restore from a specific commit
+git checkout <commit-hash> -- knowledge/memory/executive-memory.yaml
+
+# Or from the previous commit
+git checkout HEAD~1 -- knowledge/memory/executive-memory.yaml
+
+# Verify
+yq eval '.' knowledge/memory/executive-memory.yaml > /dev/null && echo "OK"
+```
+
+### Source 2: Local Backups
+
+```bash
+# Check for backups
+ls knowledge/memory/backups/ 2>/dev/null
+ls ~/maestro-backup-*/ 2>/dev/null
+
+# Restore from backup
+cp knowledge/memory/backups/executive-memory-latest.yaml knowledge/memory/executive-memory.yaml
+```
+
+### Source 3: Write-Ahead Log
+
+```bash
+# Review state change log
+tail -50 ~/maestro/logs/state-changes.log
+# Manually reconstruct from the logged changes
+```
+
+### Source 4: Output Files (Last Resort)
+
+```bash
+# Morning briefs contain open loops and workstream status
+LATEST_DIR=$(ls -d ~/maestro/outputs/briefs/daily/*/ 2>/dev/null | tail -1)
+cat "${LATEST_DIR}ceo-morning-brief.md"
+
+# Evening wraps contain completed and still-open items
+cat "${LATEST_DIR}ceo-evening-wrap.md"
+
+# Follow-up logs contain pending actions
+ls ~/maestro/outputs/followups/daily/*/
+```
+
+### Source 5: Fresh Clone from Remote
+
+If the entire local repo is lost:
+
+```bash
+cd ~
+mv maestro maestro-corrupt-$(date +%Y%m%d)
+git clone git@github.com:adapticai/maestro.git
+cd maestro
+
+source ~/.maestro-env
+
+# Salvage audit logs from corrupt copy (DFSA requirement)
+cp -r ~/maestro-corrupt-*/logs/comms/ ~/maestro/logs/comms/ 2>/dev/null
+cp -r ~/maestro-corrupt-*/logs/actions/ ~/maestro/logs/actions/ 2>/dev/null
+
+# Reload launchd agents
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro.plist
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro-healthcheck.plist
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro-logrotate.plist
+```
+
+---
+
+## 5. Manual Intervention Procedures
+
+### When to Intervene Manually
+
+- Health check reporting CRITICAL for more than 30 minutes
+- Multiple consecutive workflow failures
+- State corruption detected
+- Permissions revoked by macOS update
+- Disk space critically low (< 2GB)
+- System sending incorrect communications (highest priority -- stop immediately)
+
+### Remote Access
+
+```bash
+# SSH (if configured)
+ssh sophie@<mac-mini-ip>
+
+# Screen sharing
+# Enable: System Settings > General > Sharing > Screen Sharing
+# Connect: vnc://<mac-mini-ip>
+```
+
+### Manual Workflow Execution
+
+```bash
+# Via launchctl
+launchctl start com.adaptic.ceobrain.morning-brief
+
+# Via orchestrator script
+cd ~/maestro && source ~/.maestro-env
+./scripts/orchestrator.sh --workflow morning-brief
+
+# Via Claude Code directly
+claude --workflow ~/maestro/workflows/daily/morning-brief.yaml
+```
+
+### Manually Approving Queued Communications
+
+If the automated approval system is down:
+
+```bash
+# Review queued items
+cat ~/maestro/outputs/followups/daily/*/routing-log-*.md | tail -50
+
+# For each item: approve, reject, or defer
+# If approving: note approval and trigger send workflow manually
+```
+
+### Clearing a Stuck Queue
+
+```bash
+cd ~/maestro
+
+# Review the current queue
+yq eval '.open_loops' knowledge/memory/executive-memory.yaml
+
+# Remove a specific stale item
+yq eval 'del(.open_loops[] | select(.id == "stale-item-id"))' \
+  knowledge/memory/executive-memory.yaml > executive-memory.yaml.tmp
+mv executive-memory.yaml.tmp knowledge/memory/executive-memory.yaml
+
+git add knowledge/memory/executive-memory.yaml
+git commit -m "manual: cleared stale queue item"
+```
+
+---
+
+## 6. Emergency Stop Procedures
+
+### When to Use Emergency Stop
+
+- Uncontrolled autonomous messaging detected
+- System sending communications without proper approval
+- Desktop control acting erratically
+- Security breach suspected
+- Any behaviour that feels wrong -- err on the side of stopping
+
+### Immediate Stop (All Operations)
+
+```bash
+# Method 1: Emergency stop flag (detected within 15 min by health check)
+touch ~/maestro/logs/EMERGENCY_STOP
+
+# Method 2: Kill everything immediately
+launchctl unload ~/Library/LaunchAgents/com.adaptic.maestro.plist
+pkill -f "maestro"
+pkill -f "claude.*workflow"
+
+echo "$(date): EMERGENCY STOP activated" >> ~/maestro/logs/maintenance.log
+```
+
+### Stop Communications Only
+
+If workflows should continue but no external messages should be sent:
+
+```bash
+cd ~/maestro
+echo "emergency_block: true  # TEMPORARY" >> policies/communications-policy.yaml
+git add policies/communications-policy.yaml
+git commit -m "emergency: block all outbound communications"
+```
+
+### After Emergency Stop
+
+1. **Investigate**: Review `logs/orchestrator-stderr.log` and `logs/alerts/critical.log`.
+2. **Assess**: Determine if any unintended actions were taken (check `logs/comms/` and `logs/actions/`).
+3. **Remediate**: If messages were sent inappropriately, notify affected parties.
+4. **Fix**: Address the root cause.
+5. **Test**: Run the fixed workflow in isolation before resuming all operations.
+6. **Resume**: Remove flags, reload agents, monitor for 1 hour.
+
+```bash
+rm -f ~/maestro/logs/EMERGENCY_STOP
+# Remove emergency_block from communications-policy.yaml if set
+launchctl load ~/Library/LaunchAgents/com.adaptic.maestro.plist
+tail -f ~/maestro/logs/orchestrator-stdout.log
+```
+
+---
+
+## 7. Post-Incident Review
+
+After any significant incident (system down > 1 hour, incorrect communications sent, data loss), conduct a post-incident review.
+
+### Review Template
+
+```markdown
+# Post-Incident Review: {incident_title}
+
+**Date**: {date}
+**Duration**: {start_time} to {end_time} ({total_duration})
+**Severity**: CRITICAL / HIGH / MEDIUM
+**Detected by**: Health check / Manual observation / CEO report
+
+## Timeline
+
+- {time}: {event}
+- {time}: {event}
+- {time}: {resolution}
+
+## Root Cause
+
+{description of what went wrong and why}
+
+## Impact
+
+- Workflows missed: {list}
+- Communications affected: {list}
+- Data loss: {description or "none"}
+
+## Recovery Actions Taken
+
+1. {action}
+2. {action}
+
+## Preventive Measures
+
+1. {measure to prevent recurrence}
+2. {measure to detect earlier}
+
+## Follow-Up Items
+
+- [ ] {action item with owner and deadline}
+```
+
+### Storage
+
+Save the review to `knowledge/decisions/incidents/{date}-{incident-name}.md` and commit to git.
+
+### Review Timing
+
+- **CRITICAL** incidents: Review within 24 hours
+- **HIGH** incidents: Review within 1 week
+- **MEDIUM** incidents: Review in next monthly retrospective
+
+Update this runbook and related documentation with any lessons learned.
+
+---
+
+## Appendix: Common Error Messages and Solutions
+
+| Error                              | Likely Cause                     | Solution                                                      |
+| ---------------------------------- | -------------------------------- | ------------------------------------------------------------- |
+| "ANTHROPIC_API_KEY not set"        | Environment not sourced          | `source ~/.maestro-env`                                     |
+| "401 Unauthorized" on API calls    | API key expired or invalid       | Generate new key, update `~/.maestro-env`                   |
+| "Connection refused"               | Network down or API outage       | See "Network Interruption Handling"                           |
+| "Permission denied" on AppleScript | Accessibility permission revoked | Re-grant in System Settings > Privacy & Security              |
+| "No space left on device"          | Disk full                        | Run `./scripts/log-rotate.sh` manually, clear old outputs     |
+| "YAML parse error" on state file   | Corrupt state file               | See "State Reconstruction from Logs"                          |
+| "Process exited with status 137"   | OOM kill (out of memory)         | See memory management in perpetual-operations.md              |
+| "Slack: NOT RUNNING"               | Slack crashed                    | `open -a Slack`, verify sign-in                               |
+| Repeated rapid restarts in stderr  | Crash loop                       | Unload agent, diagnose, fix root cause, reload                |
+| "EMERGENCY_STOP file found"        | Previous emergency stop          | Investigate reason, then `rm ~/maestro/logs/EMERGENCY_STOP` |
+
+---
+
+Related documents:
+
+- `docs/runbooks/perpetual-operations.md` -- Normal operations, monitoring, and maintenance
+- `docs/runbooks/mac-mini-bootstrap.md` -- Initial setup (for full rebuild scenarios)
+- `docs/workflows/executive-cadence.md` -- Expected workflow schedule