@cohortapp/agent-sdk 2.3.0

package/docs/runbooks/perpetual-operations.md

@@ -0,0 +1,509 @@
+# Perpetual Operations Runbook
+
+This document describes how the the agent operating system system runs perpetually on the Mac mini. It covers process supervision, health monitoring, log management, memory management, state persistence, crash recovery, graceful shutdown, and upgrade procedures.
+
+---
+
+## Architecture Overview
+
+The the agent operating system runs as a set of launchd-managed processes on macOS. The primary orchestrator process coordinates all workflow execution according to the cadence defined in `docs/workflows/executive-cadence.md`. Supporting processes handle log rotation and health monitoring.
+
+```
+launchd
+  |
+  +-- ai.maestro.the-agent-repo (orchestrator)
+  |     |
+  |     +-- morning-brief (06:00 daily)
+  |     +-- comms-triage (08:00, 12:00, 17:00 daily)
+  |     +-- slack-followup-sweep (12:00, 17:00 daily)
+  |     +-- evening-wrap (18:00 daily)
+  |     +-- rollup-pipeline-review (Wed 10:00)
+  |     +-- strategic-memo (Fri 14:00)
+  |     +-- board-readiness (1st Mon monthly)
+  |     +-- scenario-analysis (1st Mon quarterly)
+  |
+  +-- ai.maestro.the-agent-repo-logrotate (daily at midnight)
+  +-- ai.maestro.the-agent-repo-healthcheck (every 15 min)
+```
+
+---
+
+## Process Supervision (launchd)
+
+The the agent operating system uses macOS `launchd` as its process supervisor. This ensures:
+
+- Workflows restart automatically after crashes
+- Scheduled jobs survive reboots (when auto-login is configured)
+- Process health is monitored by the OS
+- Restart throttling prevents crash loops
+
+### Primary Orchestrator
+
+The orchestrator is managed by launchd via `~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist`.
+
+Key configuration:
+
+- **RunAtLoad**: true -- starts when the user logs in
+- **KeepAlive.SuccessfulExit**: false -- restarts if the process exits with a non-zero code
+- **ThrottleInterval**: 30 seconds -- prevents rapid restart loops
+
+### Per-Workflow Scheduling (Cadence Bus, 1.8+)
+
+As of maestro 1.8, scheduled cadence ticks no longer spawn `claude --print` per tick. Each scheduled workflow has a launchd `.plist` at `scripts/local-triggers/plists/` that calls the lightweight Node cadence enqueue script. That script drops a JSON event onto `state/cadence-bus/inbox/`. The persistent daemon (`scripts/daemon/maestro-daemon.mjs`) hosts the consumer that drains the bus and decides whether to handle each tick inline or escalate to a managed sub-session.
+
+Example trigger plist (generated by `scripts/local-triggers/generate-plists.sh`):
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- maestro-plist-arch: cadence-bus v1 -->
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
+  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>Label</key>
+    <string>ai.adaptic.<agent>-inbox-processor</string>
+    <key>ProgramArguments</key>
+    <array>
+        <string>/Users/<agent>/<agent>-ai/scripts/cadence/launchd-cadence-wrapper.sh</string>
+        <string>/Users/<agent>/<agent>-ai/scripts/cadence/enqueue-cadence-tick.mjs</string>
+        <string>inbox-processor</string>
+        <string>--source=launchd</string>
+        <string>--quiet</string>
+    </array>
+    <key>StartInterval</key>
+    <integer>300</integer>
+    <key>StandardOutPath</key>
+    <string>/Users/<agent>/<agent>-ai/logs/daemon/launchd-stdout.log</string>
+    <key>StandardErrorPath</key>
+    <string>/Users/<agent>/<agent>-ai/logs/daemon/launchd-stderr.log</string>
+</dict>
+</plist>
+```
+
+The bus consumer's per-tick decisions are logged to `logs/cadence-bus/<date>.jsonl`. Bus depth and consumer heartbeat are available via `npm run cadence:status`. The consumer writes a heartbeat to `state/cadence-bus/health.json` every ~15 seconds; doctor flags it as stale if it's older than 5 minutes.
+
+**Legacy plists (run-trigger.sh):** Plists that still call `scripts/local-triggers/run-trigger.sh` directly are detected by `maestro doctor` and auto-migrated by `maestro upgrade` (backed up under `.maestro/backup/plists/<utc-timestamp>/`). `run-trigger.sh` itself is preserved for manual one-shot operator use only.
+
+### Managing Jobs
+
+```bash
+# Load a job
+launchctl load ~/the-agent-repo/schedules/ai.maestro.morning-brief.plist
+
+# Unload a job
+launchctl unload ~/the-agent-repo/schedules/ai.maestro.morning-brief.plist
+
+# List active jobs
+launchctl list | grep ai.maestro
+
+# Run a job immediately (for testing)
+launchctl start ai.maestro.morning-brief
+
+# Check status
+launchctl list | grep the-agent-repo
+```
+
+### Auto-Login Configuration
+
+To ensure the Mac mini resumes operations after a power cycle:
+
+1. System Settings > Users & Groups > Automatic Login > select the operating user.
+2. System Settings > Lock Screen > set "Require password" to a reasonable timeout.
+3. System Settings > Energy Saver > prevent sleep when display is off.
+4. System Settings > Energy Saver > start up automatically after a power failure.
+
+---
+
+## Health Check Intervals
+
+### Scheduled Health Check (Every 15 Minutes)
+
+The health check runs via launchd (`ai.maestro.the-agent-repo-healthcheck`) with a StartInterval of 900 seconds.
+
+The check script (`scripts/healthcheck.sh`) monitors:
+
+| Check                      | Method                                           | Alert Threshold                                             |
+| -------------------------- | ------------------------------------------------ | ----------------------------------------------------------- |
+| Orchestrator process alive | `launchctl list \| grep the-agent-repo`               | Process not found                                           |
+| Emergency stop flag        | Check for `logs/EMERGENCY_STOP` file             | File exists                                                 |
+| Environment variables set  | Verify `$ANTHROPIC_API_KEY` is non-empty         | Variable missing                                            |
+| Disk space                 | `df -h ~`                                        | Less than 10GB free                                         |
+| Memory pressure            | `memory_pressure` command                        | "WARN" or "CRITICAL" level                                  |
+| Log directory size         | `du -s ~/the-agent-repo/logs/`                        | Greater than 1GB                                            |
+| Last successful workflow   | Check latest output file timestamps              | More than 2 hours stale during business hours (06:00-18:00) |
+| Network connectivity       | `curl -s --max-time 5 https://api.anthropic.com` | Connection failure                                          |
+| Slack app running          | `pgrep -x Slack`                                 | Not running during business hours                           |
+| Safari running             | `pgrep -x Safari`                                | Not running during business hours                           |
+| WhatsApp running           | `pgrep -x WhatsApp`                              | Not running during business hours                           |
+| Directory writability      | Touch test in `logs/`, `outputs/`                | Write failure                                               |
+| Config file validity       | `yq eval '.' config/*.yaml`                      | Parse failure                                               |
+
+### Alert Levels and Routing
+
+- **CRITICAL** (orchestrator down, emergency stop, network failure): Log to `logs/alerts/critical.log`. Attempt Slack notification to #ceo-office if possible.
+- **DEGRADED** (disk space low, memory pressure, app not running): Log to `logs/alerts/warning.log`. Include in next morning brief. Continue operating with limitations.
+- **HEALTHY**: Log to `logs/healthcheck.log` only. No action needed.
+
+---
+
+## Log Rotation
+
+### Strategy
+
+Logs are rotated daily at midnight via the `ai.maestro.the-agent-repo-logrotate` launchd agent. The rotation script compresses old logs and removes logs older than the retention period.
+
+### Retention Periods
+
+| Log Category               | Location                  | Retention       | Notes                                            |
+| -------------------------- | ------------------------- | --------------- | ------------------------------------------------ |
+| Operations logs            | `logs/daily/`             | 30 days         | Compressed after 1 day                           |
+| Orchestrator stdout/stderr | `logs/orchestrator-*.log` | Rotated at 10MB | Archived copies kept 30 days                     |
+| Health check logs          | `logs/healthcheck.log`    | 30 days         |                                                  |
+| Alert logs                 | `logs/alerts/`            | 90 days         | Longer retention for incident review             |
+| Communication audit logs   | `logs/comms/`             | 6 years         | the regulator regulatory requirement -- never auto-delete |
+| Action audit logs          | `logs/actions/`           | 6 years         | the regulator regulatory requirement -- never auto-delete |
+| Maintenance logs           | `logs/maintenance.log`    | 1 year          |                                                  |
+
+### Rotation Script
+
+The log rotation script (`scripts/log-rotate.sh`) performs:
+
+```bash
+#!/bin/bash
+# Log rotation for the agent operating system
+LOG_DIR="$HOME/the-agent-repo/logs"
+RETENTION_DAYS=30
+ALERT_RETENTION_DAYS=90
+
+# Compress operations logs older than 1 day (exclude comms/ and actions/)
+find "$LOG_DIR" -path "$LOG_DIR/comms" -prune -o \
+                -path "$LOG_DIR/actions" -prune -o \
+                -name "*.log" -mtime +1 -not -name "*.gz" -exec gzip {} \;
+
+# Remove compressed operations logs older than retention period
+find "$LOG_DIR" -path "$LOG_DIR/comms" -prune -o \
+                -path "$LOG_DIR/actions" -prune -o \
+                -name "*.log.gz" -mtime +$RETENTION_DAYS -delete
+
+# Remove compressed alert logs older than alert retention
+find "$LOG_DIR/alerts" -name "*.log.gz" -mtime +$ALERT_RETENTION_DAYS -delete
+
+# Rotate large orchestrator logs (truncate and archive)
+for logfile in orchestrator-stdout.log orchestrator-stderr.log healthcheck.log; do
+    filepath="$LOG_DIR/$logfile"
+    if [ -f "$filepath" ] && [ "$(stat -f%z "$filepath" 2>/dev/null)" -gt 10485760 ]; then
+        cp "$filepath" "$LOG_DIR/${logfile%.log}-$(date +%Y%m%d).log"
+        : > "$filepath"
+    fi
+done
+
+echo "$(date): Log rotation complete" >> "$LOG_DIR/maintenance.log"
+```
+
+### Log Directory Structure
+
+```
+logs/
+  orchestrator-stdout.log        # Current orchestrator stdout
+  orchestrator-stderr.log        # Current orchestrator stderr
+  healthcheck.log                # Health check results
+  maintenance.log                # Rotation, upgrades, shutdowns
+  daily/
+    {date}/
+      slack-sends.log            # Slack messages sent
+      workflow-execution.log     # Workflow step timings and results
+  alerts/
+    critical.log                 # Critical alerts
+    warning.log                  # Warning-level alerts
+  comms/                         # the regulator-regulated: 6 year retention
+    {date}/
+      email-log.yaml
+      slack-log.yaml
+      whatsapp-log.yaml
+  actions/                       # the regulator-regulated: 6 year retention
+    {date}/
+      action-audit.yaml
+```
+
+---
+
+## Memory Management
+
+### Claude Code Process Memory
+
+Claude Code processes can accumulate memory over long sessions. Mitigation strategies:
+
+1. **Per-workflow process isolation**: Each workflow invocation spawns a fresh Claude Code process. This prevents memory accumulation across workflows.
+
+2. **Memory monitoring**: The health check monitors system memory pressure every 15 minutes:
+
+   ```bash
+   memory_pressure | grep "System-wide memory free percentage"
+   ```
+
+3. **Thresholds and actions**:
+   | Free Memory | Status | Action |
+   |-------------|--------|--------|
+   | > 30% | Normal | Continue all operations |
+   | 15-30% | Warning | Log warning. Defer non-critical workflows if below 20%. |
+   | < 15% | Critical | Kill non-essential processes. Defer all non-critical workflows. Alert. |
+
+4. **macOS memory management**: Swap compression is enabled by default on macOS. Do not disable it.
+
+### Knowledge Base Growth
+
+- `knowledge/memory/executive-memory.yaml` is the primary persistent state. Monitor its size.
+- Archive completed decision records older than 1 year to `knowledge/archive/`.
+- Compress synthesis documents older than 6 months.
+- Keep entity map and source registry lean by removing stale entries quarterly.
+
+### Application Memory
+
+Slack, Safari, and WhatsApp accumulate memory over continuous operation. Schedule a weekly restart:
+
+```bash
+# Weekly app restart (Sunday 03:00 via launchd or cron)
+osascript -e 'tell application "Slack" to quit'
+sleep 10
+open -a Slack
+
+osascript -e 'tell application "Safari" to quit'
+sleep 10
+open -a Safari
+
+osascript -e 'tell application "WhatsApp" to quit'
+sleep 10
+open -a WhatsApp
+```
+
+### Session State
+
+- Desktop control state cached in `desktop-control/state/`.
+- Cleared on system restart.
+- Rebuilt from application observation on resume.
+
+---
+
+## State Persistence
+
+### Critical State Files
+
+| File                                     | Purpose                                    | Backup Strategy                                |
+| ---------------------------------------- | ------------------------------------------ | ---------------------------------------------- |
+| `knowledge/memory/executive-memory.yaml` | Open loops, action items, workstream state | Write-ahead log + git commit after every write |
+| `config/priorities.yaml`                 | Active strategic priorities                | Git commit after every update                  |
+| `config/contacts.yaml`                   | Contact information                        | Weekly git commit                              |
+| `knowledge/entities/entity-map.yaml`     | Pipeline entity data                       | Git commit after every update                  |
+| `knowledge/decisions/`                   | Decision log and history                   | Git commit after every decision recorded       |
+| `outputs/followups/`                     | Follow-up queue                            | Git commit with periodic snapshots             |
+
+### What Must Persist Across Restarts
+
+1. Executive memory (`knowledge/memory/executive-memory.yaml`)
+2. Decision log (`knowledge/decisions/`)
+3. Follow-up queue (`outputs/followups/`)
+4. Priority configuration (`config/priorities.yaml`)
+5. Contact registry (`config/contacts.yaml`)
+6. Audit logs (`logs/comms/`, `logs/actions/`)
+
+### What Is Rebuilt on Restart
+
+1. Desktop control state (fresh app observation)
+2. Working context (knowledge base read)
+3. Workflow scheduling (launchd reload)
+
+### Persistence Mechanisms
+
+1. **Git as primary persistence**: All state changes are committed to the repository. This provides version history, recovery points, and remote backup.
+
+2. **Write-ahead logging**: Before modifying any state file, the intended change is written to `logs/state-changes.log` with a timestamp. If the write fails, the log enables recovery.
+
+3. **Atomic writes**: Use write-to-temp-then-rename to prevent partial writes:
+
+   ```bash
+   yq eval '.open_loops += [{"item": "new"}]' file.yaml > file.yaml.tmp
+   mv file.yaml.tmp file.yaml
+   ```
+
+4. **Periodic git commits**: Auto-commit state changes every 4 hours:
+   ```bash
+   cd ~/the-agent-repo
+   git add knowledge/ config/ outputs/
+   git commit -m "auto: state snapshot $(date +%Y-%m-%dT%H:%M)"
+   git push origin main
+   ```
+
+---
+
+## Recovery from Crashes
+
+### Automatic Recovery (launchd)
+
+When the orchestrator process crashes:
+
+1. launchd detects the non-zero exit code.
+2. After ThrottleInterval (30 seconds), launchd restarts the process.
+3. The orchestrator reads its schedule and determines the next workflow to run.
+4. Missed workflows during downtime are logged but not retroactively executed (stale data risk).
+
+### Manual Recovery
+
+If auto-recovery fails (crash loop detected):
+
+```bash
+# 1. Stop the restart loop
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+
+# 2. Diagnose
+tail -100 ~/the-agent-repo/logs/orchestrator-stderr.log
+
+# 3. Fix the issue
+
+# 4. Restart
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+```
+
+See `docs/runbooks/recovery-and-failover.md` for detailed recovery procedures covering system crashes, application crashes, network interruptions, and state reconstruction.
+
+---
+
+## Graceful Shutdown
+
+### Standard Shutdown
+
+```bash
+# 1. Check for in-progress workflows
+tail -1 ~/the-agent-repo/logs/daily/$(date +%Y-%m-%d)/workflow-execution.log
+
+# 2. Wait for completion (or skip if no workflow is running)
+
+# 3. Unload all launchd agents
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo-healthcheck.plist
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo-logrotate.plist
+
+# 4. Commit state
+cd ~/the-agent-repo
+git add -A
+git commit -m "manual: graceful shutdown $(date +%Y-%m-%dT%H:%M)"
+git push origin main
+
+# 5. Log the shutdown
+echo "$(date): Graceful shutdown by operator" >> ~/the-agent-repo/logs/maintenance.log
+```
+
+### Emergency Shutdown
+
+```bash
+# Create emergency stop flag (health check will detect this)
+touch ~/the-agent-repo/logs/EMERGENCY_STOP
+
+# Kill all processes
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+pkill -f "the-agent-repo"
+
+echo "$(date): EMERGENCY SHUTDOWN" >> ~/the-agent-repo/logs/maintenance.log
+```
+
+### Resuming After Shutdown
+
+```bash
+# 1. Remove emergency stop flag if present
+rm -f ~/the-agent-repo/logs/EMERGENCY_STOP
+
+# 2. Verify state integrity
+cd ~/the-agent-repo
+git status
+yq eval '.' config/priorities.yaml > /dev/null && echo "priorities OK"
+yq eval '.' knowledge/memory/executive-memory.yaml > /dev/null && echo "memory OK"
+
+# 3. Reload all agents
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo-healthcheck.plist
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo-logrotate.plist
+
+# 4. Verify
+launchctl list | grep the-agent-repo
+
+# 5. Log the resume
+echo "$(date): System resumed" >> ~/the-agent-repo/logs/maintenance.log
+```
+
+---
+
+## Upgrade Procedures
+
+### Upgrading Claude Code CLI
+
+```bash
+# 1. Graceful shutdown
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+
+# 2. Upgrade
+npm update -g @anthropic-ai/claude-code
+
+# 3. Verify
+claude --version
+
+# 4. Resume
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+```
+
+### Upgrading the-agent-repo Repository
+
+```bash
+# 1. Graceful shutdown
+launchctl unload ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+
+# 2. Commit local state
+cd ~/the-agent-repo
+git add -A
+git commit -m "pre-upgrade: state snapshot $(date +%Y-%m-%dT%H:%M)"
+
+# 3. Pull latest
+git pull origin main
+
+# 4. Install new dependencies if needed
+npm install
+
+# 5. Validate workflows
+for f in workflows/**/*.yaml; do
+  yq eval '.' "$f" > /dev/null 2>&1 || echo "INVALID: $f"
+done
+
+# 6. Resume
+launchctl load ~/Library/LaunchAgents/ai.maestro.the-agent-repo.plist
+
+# 7. Monitor for 15 minutes
+tail -f ~/the-agent-repo/logs/orchestrator-stdout.log
+```
+
+### Major Upgrades
+
+For major changes (new workflow structure, config format changes):
+
+1. Create a full backup: `cp -r ~/the-agent-repo ~/the-agent-repo-backup-$(date +%Y%m%d)`
+2. Graceful shutdown.
+3. Apply changes.
+4. Run health check: `./scripts/healthcheck.sh`
+5. Test one workflow manually before resuming all.
+6. Resume operations.
+7. Monitor for one full cadence cycle (24 hours).
+8. Remove backup after 7 days if stable.
+
+### Upgrading macOS
+
+1. Graceful shutdown and push all state to git.
+2. Run the macOS upgrade.
+3. After reboot: verify auto-login, launchd agents loaded, permissions intact.
+4. Re-grant any permissions reset by the upgrade (Accessibility, Screen Recording, Automation).
+5. Run health check.
+6. Monitor for one full cadence cycle.
+
+---
+
+Related documents:
+
+- `docs/runbooks/mac-mini-bootstrap.md` -- Initial setup
+- `docs/runbooks/recovery-and-failover.md` -- Detailed recovery procedures
+- `docs/workflows/executive-cadence.md` -- Full operating rhythm