@cohortapp/agent-sdk 2.3.0

package/docs/guides/poller-daemon-setup.md

@@ -0,0 +1,563 @@
+# Poller, Daemon & Trigger Setup Guide
+
+How the agent's autonomous event loop works: the lightweight poller, the reactive daemon, session management, launchd trigger scheduling, and the memory watchdog. This is the nervous system that makes the agent perpetually operational.
+
+**Prerequisites**: Complete the [Mac Mini Bootstrap](../runbooks/mac-mini-bootstrap.md) and configure at least one communication channel (Slack, Gmail, etc.).
+
+---
+
+## Architecture Overview
+
+The agent has three concurrent execution modes, powered by different subsystems:
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│  MODE 1: REACTIVE — Respond to Events                              │
+│                                                                     │
+│  ┌──────────┐  ┌─────────────┐  ┌────────────────────────────────┐ │
+│  │  launchd  │─▶│  Poller     │─▶│  state/inbox/{slack,gmail,    │ │
+│  │  (60s)    │  │  index.mjs  │  │   calendar,sms,whatsapp}/*.yaml│ │
+│  └──────────┘  └──────┬──────┘  └─────────────┬──────────────────┘ │
+│                        │                       │                    │
+│                  priority item?           ┌────▼────────────┐      │
+│                        │                  │ Inbox Processor  │      │
+│                        ▼                  │ (scheduled       │      │
+│                  triggerSophie()          │  trigger, 5 min) │      │
+│                  (immediate session)      └─────────────────┘      │
+├─────────────────────────────────────────────────────────────────────┤
+│  MODE 1b: REACTIVE DAEMON (Alternative to Poller)                  │
+│                                                                     │
+│  ┌────────────────────────────────────────────────────────────┐    │
+│  │  sophie-daemon.mjs (persistent Node.js process)            │    │
+│  │                                                             │    │
+│  │  Poll loop (60s):                                           │    │
+│  │    Slack + Gmail + Calendar → classify (Haiku) → dispatch  │    │
+│  │                                                             │    │
+│  │  Backlog sweep (2min):                                      │    │
+│  │    Scan queues → pick actionable items → dispatch           │    │
+│  │                                                             │    │
+│  │  Health check (1min):                                       │    │
+│  │    Write health dashboard, log metrics                      │    │
+│  │                                                             │    │
+│  │  Dispatcher:                                                │    │
+│  │    Up to 10 parallel claude --print sessions                │    │
+│  └────────────────────────────────────────────────────────────┘    │
+├─────────────────────────────────────────────────────────────────────┤
+│  MODE 2: SCHEDULED — Run Cadence Workflows via the Cadence Bus     │
+│                                                                     │
+│  ┌──────────┐  ┌────────────────────┐  ┌─────────────────────────┐ │
+│  │  launchd  │─▶│ enqueue-cadence-    │─▶│ state/cadence-bus/     │ │
+│  │ (schedule)│  │ tick.mjs (~10ms)    │  │   inbox/<event>.json    │ │
+│  └──────────┘  └────────────────────┘  └────────────┬────────────┘ │
+│                                                       │              │
+│                                            ┌──────────▼───────────┐ │
+│                                            │ cadence-consumer.mjs  │ │
+│                                            │ (inside daemon)       │ │
+│                                            │                       │ │
+│                                            │ ┌─────┐  ┌──────────┐ │ │
+│                                            │ │inline│  │sub-session│ │ │
+│                                            │ └─────┘  └──────────┘ │ │
+│                                            └──────────────────────┘ │
+│                                                                     │
+│  Per-cadence policy in scripts/daemon/cadence-handlers.mjs:        │
+│    inline    → housekeeping (heartbeat, recovery)                  │
+│    guarded   → cheap pre-check; escalate only if work pending     │
+│    escalate  → spawn claude --print with schedules/triggers/*.md   │
+│                                                                     │
+│  Cadences: inbox-processor, backlog-executor, meeting-prep,        │
+│            meeting-action-capture, daily-*, weekly-*, quarterly-*  │
+├─────────────────────────────────────────────────────────────────────┤
+│  MODE 3: PROACTIVE — Execute the Backlog                          │
+│                                                                     │
+│  backlog-executor trigger (every 10 min):                          │
+│    Read all queues → select top 3-5 items → spawn parallel agents │
+│    → review results → update queues → pick next batch             │
+├─────────────────────────────────────────────────────────────────────┤
+│  SAFETY                                                            │
+│                                                                     │
+│  memory-watchdog.sh (every 30s):                                   │
+│    Monitor RAM + process count → kill runaways → emergency stop   │
+│                                                                     │
+│  emergency-stop.sh:                                                │
+│    Creates .emergency-stop file → all triggers abort on check     │
+│                                                                     │
+│  resume-operations.sh:                                             │
+│    Removes .emergency-stop → triggers resume                       │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 1. The Poller (`scripts/poller/`)
+
+The poller is a lightweight Node.js script that runs every 60 seconds via launchd. It checks all inbound channels and writes new items to the inbox.
+
+### 1.1 Entry Point: `index.mjs`
+
+Runs four service pollers in sequence:
+
+| Service | Module | What it Polls | Credentials |
+|---|---|---|---|
+| Slack | `slack-poller.mjs` | DMs, channels, mentions | `SLACK_USER_TOKEN` |
+| Gmail | `gmail-poller.mjs` | Agent's inbox (UNSEEN) | `GMAIL_APP_PASSWORD` |
+| CEO Gmail | `secondary-gmail-poller.mjs` | Principal's inbox | `SECONDARY_GMAIL_APP_PASSWORD` |
+| Calendar | `calendar-poller.mjs` | Upcoming events, changes | Google Calendar API |
+
+### 1.2 IMAP Client (`imap-client.mjs`)
+
+Shared IMAP connection wrapper used by both Gmail pollers:
+- Manages connection pooling and reconnection
+- Handles IMAP search queries (UNSEEN, date ranges)
+- Parses email headers for threading information
+
+### 1.3 Priority Detection (`utils.mjs`)
+
+The `isPriorityItem()` function detects items that need immediate processing:
+- CEO DMs (principal's Slack ID)
+- Messages mentioning the agent by name
+- Urgent keywords (urgent, ASAP, emergency)
+- Messages tagged with priority emoji
+
+When a priority item is detected, `triggerSophie()` spawns an immediate Claude session rather than waiting for the inbox processor cycle.
+
+### 1.4 Trigger Module (`trigger.mjs`)
+
+Spawns a Claude Code session for priority items:
+- Runs `claude --print --dangerously-skip-permissions` with a targeted prompt
+- The prompt includes the priority item's content and relevant context
+- Session output is logged to `logs/polling/`
+
+### 1.5 Intra-Session Check (`intra-session-check.mjs`)
+
+Allows a running Claude session to check for new priority events mid-execution. Called by long-running sessions (like backlog executor) to detect urgent interrupts.
+
+### 1.6 Poller Logs
+
+- `logs/polling/YYYY-MM-DD-poller.jsonl` — Per-run summary (items found, errors, duration)
+- Each run logs: timestamp, items per service, errors, whether a priority trigger fired
+
+---
+
+## 2. The Reactive Daemon (`scripts/daemon/`)
+
+The daemon is an alternative to the poller that provides faster response times. Instead of running every 60 seconds as a separate process, it's a persistent Node.js process with built-in polling, classification, and dispatching.
+
+### 2.1 Entry Point: `maestro-daemon.mjs`
+
+- Agent-name-agnostic wrapper that sets `AGENT_DIR` environment variable
+- Acquires a singleton lock (via `~/maestro/lib/singleton.js`) to prevent duplicate daemons
+- Imports and runs `sophie-daemon.mjs`
+
+### 2.2 Core Daemon: `sophie-daemon.mjs`
+
+Three concurrent loops:
+
+| Loop | Interval | Purpose |
+|---|---|---|
+| Poll | 60s | Check Slack, Gmail, Calendar for new items |
+| Backlog | 2 min | Sweep queues for actionable items |
+| Health | 1 min | Write health dashboard, log metrics |
+
+**Poll loop flow:**
+1. Run all four service pollers (same as standalone poller)
+2. For each new item, classify it via `classifier.mjs`
+3. Check if it's directed at the agent via `isDirectedAtSophie()`
+4. For items needing response: build prompt → dispatch session
+5. For quick replies: use `responder.mjs` for immediate response
+
+### 2.3 Classifier (`classifier.mjs`)
+
+Classifies incoming items using Claude Haiku (fast, cheap):
+
+- **Priority**: critical / high / medium / low
+- **Type**: question, request, notification, FYI, greeting, spam
+- **Directed at agent**: yes / no / unclear
+- **Quick reply**: yes (can respond immediately) / no (needs full session)
+
+Classification takes ~0.5-1 second via Haiku API.
+
+### 2.4 Dispatcher (`dispatcher.mjs`)
+
+Manages parallel Claude Code sessions:
+
+- Up to 10 concurrent sessions (`claude --print`)
+- Tracks active sessions with PIDs and start times
+- `availableSlots()` returns how many sessions can be spawned
+- `canDispatchBacklog()` checks if there's capacity for proactive work
+- `resetActiveSessions()` cleans up orphaned session records
+- Each session gets a targeted prompt from `prompt-builder.mjs`
+
+### 2.5 Prompt Builder (`prompt-builder.mjs`)
+
+Constructs context-rich prompts for dispatched sessions:
+
+- Includes the classified item (message content, sender, channel)
+- Loads relevant user profile from `memory/profiles/`
+- Includes standing instructions from the user profile
+- Includes relevant queue context if the item references tracked work
+- Adds the session protocol (what to do at session start/end)
+
+### 2.6 Responder (`responder.mjs`)
+
+Handles quick replies without spawning a full Claude session:
+
+- `isQuickReply()` detects items that can be answered immediately (greetings, acknowledgments)
+- `sendQuickResponse()` sends a fast reply via the appropriate channel
+- `sendHoldingMessage()` sends "Got it, working on this" for complex requests
+
+### 2.7 Session Lock (`session-lock.mjs`)
+
+File-based locking to prevent duplicate processing:
+
+- `acquireLock(itemId)` — claim exclusive processing of an item
+- `acquireThreadLock(channel, thread)` — claim a thread for response
+- `claimRequest(requestId)` — claim a specific request
+- `updateLock(lockId, status)` — update lock with processing status
+- `scanStaleLocks()` — find and clean locks older than TTL
+- Lock directory: `state/locks/daemon/`
+
+### 2.8 Health Monitor (`health.mjs`)
+
+Tracks daemon performance:
+
+- `recordPoll(results)` — log poll cycle results
+- `recordClassification(item, classification)` — log classification outcomes
+- `recordSession(sessionInfo)` — log dispatched session metrics
+- `writeHealthDashboard()` — write `state/dashboards/daemon-health.yaml`
+
+### 2.9 Context Compiler (`context-compiler.mjs`)
+
+Pre-compiles session context to reduce prompt size:
+
+- Reads recent interactions, queue state, and active items
+- Compresses context to fit within token limits
+- Enabled via `DAEMON_CONTEXT_COMPILER=1` in `.env`
+
+---
+
+## 3. Session Management
+
+### 3.1 Spawning Sessions (`spawn-session.sh`)
+
+Creates a new Claude Code session:
+
+```bash
+./scripts/spawn-session.sh "Process this incoming request" --session-id "req-12345"
+```
+
+- Sets `SOPHIE_SESSION_ID` for dedup tracking
+- Checks `.emergency-stop` before proceeding
+- Logs session start to `logs/sessions/`
+
+### 3.2 Session Start (`session-start.sh`)
+
+Initialises session state:
+
+```bash
+./scripts/session-start.sh
+```
+
+Called at the beginning of each Claude session to:
+- Load the executive summary dashboard
+- Check for unprocessed inbox items
+- Load pending decisions from the decision queue
+- Set up the session environment
+
+### 3.3 Emergency Stop (`emergency-stop.sh`)
+
+```bash
+./scripts/emergency-stop.sh
+```
+
+Creates a `.emergency-stop` file in the repo root. All triggers check for this file before executing and abort if present. Use this to immediately halt all autonomous operations.
+
+### 3.4 Resume Operations (`resume-operations.sh`)
+
+```bash
+./scripts/resume-operations.sh
+```
+
+Removes the `.emergency-stop` file, allowing triggers to resume.
+
+---
+
+## 4. LaunchD Triggers (`scripts/local-triggers/`)
+
+### 4.1 How Triggers Work
+
+Triggers are Markdown prompts in `schedules/triggers/` that launchd runs on a schedule. Each trigger invokes a Claude Code session with the prompt as input.
+
+**Execution chain**: launchd → `run-trigger.sh <trigger-name>` → reads `schedules/triggers/<trigger-name>.md` → runs `claude --print --dangerously-skip-permissions <prompt>`
+
+### 4.2 Run a Trigger Manually
+
+```bash
+./scripts/local-triggers/run-trigger.sh morning-brief
+```
+
+### 4.3 Generate Plist Files (`generate-plists.sh`)
+
+Generates all launchd plist files from the agent's configuration:
+
+```bash
+./scripts/local-triggers/generate-plists.sh
+```
+
+This script:
+1. Reads `config/agent.ts` to extract the agent's name
+2. Generates plists for: daemon, poller, all scheduled triggers, watchdog
+3. Uses the agent name in plist labels (e.g., `ai.maestro.agent.morning-brief`)
+4. Sets correct working directory, Node.js path, and log paths
+5. Saves plists to `scripts/local-triggers/plists/`
+
+### 4.4 Install All Triggers (`install-all.sh`)
+
+```bash
+./scripts/local-triggers/install-all.sh
+```
+
+Copies all generated plists to `~/Library/LaunchAgents/` and loads them.
+
+### 4.5 Standard Trigger Schedule
+
+| Trigger | Cadence | Time | Purpose |
+|---|---|---|---|
+| `morning-brief` | Daily | 06:00 | CEO morning brief |
+| `midday-sweep` | Daily | 12:00 | SLA check, loop closure |
+| `evening-wrap` | Daily | 18:00 | End-of-day summary, queue cleanup |
+| `backlog-executor` | Every 10 min | Continuous | Execute top queue items |
+| `inbox-processor` | Every 5 min | Continuous | Classify and route inbox items |
+| `meeting-prep` | Every 15 min | Continuous | Pre-meeting brief generation |
+| `meeting-action-capture` | Every 30 min | Continuous | Extract actions from meetings |
+| `weekly-priorities` | Weekly (Mon) | 09:00 | Priority review |
+| `weekly-execution` | Weekly (Wed) | 09:00 | Execution review |
+| `weekly-strategic-memo` | Weekly (Fri) | 15:00 | Strategic memo to principal |
+| `weekly-hiring` | Weekly (Mon) | 11:00 | Hiring pipeline review |
+| `weekly-engineering-health` | Weekly (Wed) | 08:00 | Engineering health check |
+| `quarterly-self-assessment` | Quarterly | First Mon | Agent self-assessment |
+
+Times are in the agent's configured timezone from `config/agent.ts`.
+
+---
+
+## 5. Memory Watchdog (`scripts/watchdog/`)
+
+### 5.1 What It Does
+
+The watchdog monitors system resources and prevents runaway Claude processes from consuming all RAM:
+
+| Threshold | Level | Action |
+|---|---|---|
+| >60% RAM (Claude processes) | Warning | Log warning |
+| >75% RAM (Claude processes) | Critical | Kill oldest subagent processes |
+| >85% total system memory pressure | Emergency | Emergency stop all operations |
+| >8 concurrent claude processes | Process limit | Kill excess processes |
+
+### 5.2 Configuration
+
+Environment variables (with defaults):
+
+```bash
+WATCHDOG_WARN_PERCENT=60        # Warning threshold
+WATCHDOG_CRITICAL_PERCENT=75    # Critical threshold (kill runaways)
+WATCHDOG_EMERGENCY_PERCENT=85   # Emergency threshold (full stop)
+WATCHDOG_MAX_CLAUDE_PROCS=8     # Max concurrent claude processes
+WATCHDOG_MIN_AGE=60             # Minimum age (seconds) before killing a process
+```
+
+### 5.3 Running the Watchdog
+
+```bash
+# Normal run (takes action if thresholds exceeded)
+./scripts/watchdog/memory-watchdog.sh
+
+# Check status only (no actions)
+./scripts/watchdog/memory-watchdog.sh --check
+
+# Dry run (show what would be killed)
+./scripts/watchdog/memory-watchdog.sh --dry-run
+```
+
+### 5.4 Force Reboot
+
+For severe cases where the watchdog can't recover:
+
+```bash
+./scripts/watchdog/force-reboot.sh
+```
+
+Kills all claude processes and restarts the daemon.
+
+### 5.5 Watchdog Plist
+
+The watchdog runs every 30 seconds via `ai.maestro.memory-watchdog.plist`. Generated automatically by `generate-plists.sh`.
+
+---
+
+## 6. Health Monitoring
+
+### 6.1 Healthcheck (`healthcheck.sh`)
+
+Quick system health verification:
+
+```bash
+./scripts/healthcheck.sh
+```
+
+Checks:
+- launchd agents are loaded and running
+- Disk space available
+- Memory usage
+- Log directory sizes
+- Last successful trigger runs
+- Daemon process status
+
+### 6.2 System Verify (`system-verify.sh`)
+
+Deep configuration verification:
+
+```bash
+./scripts/system-verify.sh
+```
+
+Validates all config files, environment variables, directory structure, and permissions.
+
+### 6.3 Continuous Monitor (`continuous-monitor.sh`)
+
+Long-running monitoring process:
+
+```bash
+./scripts/continuous-monitor.sh
+```
+
+Watches system metrics over time and logs trends.
+
+### 6.4 Communications Monitor (`comms-monitor.sh`)
+
+Monitors communication channel health:
+
+```bash
+./scripts/comms-monitor.sh
+```
+
+Verifies Slack tokens, Gmail connectivity, Twilio status, and tunnel health.
+
+---
+
+## 7. Choosing: Poller vs Daemon
+
+| Aspect | Standalone Poller | Reactive Daemon |
+|---|---|---|
+| Response latency | 60s + inbox processor cycle (~5 min total) | ~2 min (poll + classify + dispatch) |
+| Resource usage | Low (runs briefly every 60s) | Moderate (persistent Node.js process) |
+| Complexity | Simple (one-shot script) | Complex (8 interconnected modules) |
+| Parallel sessions | None (relies on triggers) | Up to 10 concurrent sessions |
+| Built-in classification | No (inbox processor does it) | Yes (Haiku classifier) |
+| Backlog execution | Separate trigger (every 10 min) | Built-in sweep (every 2 min) |
+| Recommended for | Low-volume agents, simple roles | High-volume agents, executive roles |
+
+**Default**: Use the daemon for production agents that need fast response times. Use the standalone poller for development/testing or low-volume agents.
+
+---
+
+## 8. Testing
+
+| # | Test | How to Verify |
+|---|---|---|
+| 1 | Poller runs | `node scripts/poller/index.mjs` — should poll all services |
+| 2 | Daemon starts | `node scripts/daemon/maestro-daemon.mjs` — should show poll loop starting |
+| 3 | Singleton guard | Start daemon twice — second should exit with "Already running" |
+| 4 | Priority detection | Send CEO DM during poller run — should trigger immediate session |
+| 5 | Trigger execution | `./scripts/local-triggers/run-trigger.sh morning-brief` |
+| 6 | Emergency stop | `./scripts/emergency-stop.sh` → try running a trigger → should abort |
+| 7 | Resume | `./scripts/resume-operations.sh` → trigger should work again |
+| 8 | Watchdog check | `./scripts/watchdog/memory-watchdog.sh --check` |
+| 9 | Plist generation | `./scripts/local-triggers/generate-plists.sh` → check `plists/` directory |
+| 10 | Healthcheck | `./scripts/healthcheck.sh` |
+
+---
+
+## 9. Troubleshooting
+
+### Daemon won't start: "Already running"
+
+1. Check for stale singleton lock: `ls /tmp/maestro-daemon.lock`
+2. If the previous daemon crashed without releasing the lock, remove it: `rm /tmp/maestro-daemon.lock`
+3. Check for orphaned node processes: `pgrep -f maestro-daemon`
+
+### Triggers not firing
+
+1. Verify plists are loaded: `launchctl list | grep -E 'maestro|adaptic'`
+2. Check plist logs: `cat ~/Library/LaunchAgents/ai.maestro.AGENT-*.plist`
+3. Verify working directory in plist points to the right repo
+4. Check `.emergency-stop` doesn't exist
+5. Check trigger prompt file exists: `ls schedules/triggers/morning-brief.md`
+
+### High memory usage
+
+1. Run watchdog: `./scripts/watchdog/memory-watchdog.sh --check`
+2. Count claude processes: `pgrep -f claude | wc -l` (should be ≤8)
+3. Kill orphaned processes: `./scripts/watchdog/memory-watchdog.sh` (auto-kills above threshold)
+4. Reduce `MAX_PARALLEL_SESSIONS` in daemon config if persistent issue
+
+### Poller errors for a specific service
+
+1. Check service credentials in `.env` (e.g., `SLACK_USER_TOKEN` for Slack)
+2. Check poller logs: `tail logs/polling/$(date +%Y-%m-%d)-poller.jsonl`
+3. Test the specific poller: import and run `pollSlack()` directly
+4. Check for rate limiting (especially Slack — 60s interval helps)
+
+### Sessions dispatched but no response sent
+
+1. Check daemon logs: `tail logs/daemon/$(date +%Y-%m-%d)-sessions.jsonl`
+2. Check session locks: `ls state/locks/daemon/`
+3. Verify Claude CLI is accessible: `claude --version`
+4. Check session output in logs for errors
+
+---
+
+## Key Files
+
+| File | Purpose |
+|---|---|
+| **Poller** | |
+| `scripts/poller/index.mjs` | Main poller entry point |
+| `scripts/poller/slack-poller.mjs` | Slack event polling |
+| `scripts/poller/gmail-poller.mjs` | Gmail inbox polling |
+| `scripts/poller/calendar-poller.mjs` | Calendar event polling |
+| `scripts/poller/secondary-gmail-poller.mjs` | CEO inbox polling |
+| `scripts/poller/imap-client.mjs` | IMAP connection wrapper |
+| `scripts/poller/trigger.mjs` | Priority item session spawner |
+| `scripts/poller/utils.mjs` | Shared utilities (priority detection) |
+| `scripts/poller/intra-session-check.mjs` | Mid-session event check |
+| **Daemon** | |
+| `scripts/daemon/maestro-daemon.mjs` | Generic daemon entry point |
+| `scripts/daemon/sophie-daemon.mjs` | Core daemon logic (poll + backlog + health) |
+| `scripts/daemon/classifier.mjs` | Haiku-based message classifier |
+| `scripts/daemon/dispatcher.mjs` | Parallel session manager |
+| `scripts/daemon/prompt-builder.mjs` | Context-rich prompt construction |
+| `scripts/daemon/responder.mjs` | Quick reply and holding messages |
+| `scripts/daemon/session-lock.mjs` | File-based dedup locking |
+| `scripts/daemon/health.mjs` | Health monitoring and dashboard |
+| `scripts/daemon/context-compiler.mjs` | Session context pre-compilation |
+| **Triggers** | |
+| `scripts/local-triggers/run-trigger.sh` | Execute a scheduled trigger |
+| `scripts/local-triggers/generate-plists.sh` | Generate launchd plist files |
+| `scripts/local-triggers/install-all.sh` | Install all plists to launchd |
+| **Session** | |
+| `scripts/spawn-session.sh` | Spawn a new Claude Code session |
+| `scripts/session-start.sh` | Session initialisation |
+| `scripts/emergency-stop.sh` | Emergency halt all operations |
+| `scripts/resume-operations.sh` | Resume after emergency stop |
+| **Watchdog** | |
+| `scripts/watchdog/memory-watchdog.sh` | Resource monitoring and process management |
+| `scripts/watchdog/force-reboot.sh` | Kill all claude processes and restart |
+
+---
+
+## Related Documents
+
+- [Mac Mini Bootstrap](../runbooks/mac-mini-bootstrap.md) — Initial machine and launchd setup
+- [Perpetual Operations](../runbooks/perpetual-operations.md) — How the system runs 24/7
+- [Recovery and Failover](../runbooks/recovery-and-failover.md) — What to do when things break
+- [Agent Persona Setup](agent-persona-setup.md) — Trigger schedule configuration