@intent-systems/nexus 2026.1.5-3

package/skills/onboarding/docs/skill-deep-dives/cron-and-heartbeat.md

@@ -0,0 +1,576 @@
+# Conceptual Deep-Dive: Cron & Heartbeat
+
+**Feature Type**: Nexus Gateway Feature (Not a Skill)
+**Category**: Automation (Task Scheduling)
+**Setup Difficulty**: 🟡 Medium
+**Time Estimate**: ~10 minutes
+**Platform**: Cross-platform (macOS, Windows, Linux)
+
+---
+
+## What This Is
+
+Cron and Heartbeat are **Nexus Gateway features**, not standalone skills. They allow you to schedule tasks and create periodic checks for your Nexus workspace.
+
+### Key Concepts
+
+| Feature | What It Does | Example |
+|---------|-------------|---------|
+| **Cron** | Schedule tasks to run at specific times | "Every day at 9am, sync my messages" |
+| **Heartbeat** | Periodic health checks and monitoring | "Every 5 minutes, check if services are running" |
+
+---
+
+## Cron vs Heartbeat
+
+| Aspect | Cron | Heartbeat |
+|--------|------|-----------|
+| **Purpose** | Schedule tasks | Monitor system health |
+| **Frequency** | Any schedule (minute, hour, day, etc.) | Usually frequent (seconds to minutes) |
+| **Use case** | Automation (backups, syncs, reports) | Health checks (service status, disk space) |
+| **Failure handling** | Log errors | Alert on failures |
+
+---
+
+## When To Use
+
+### Use Cron For:
+- Daily backups (`nexus-cloud push`)
+- Hourly message sync (`eve sync`, `wacli sync`)
+- Morning briefings (7am summary)
+- Weekly reports
+- Scheduled cleanups
+
+### Use Heartbeat For:
+- Service health checks (is Ollama running?)
+- Disk space monitoring
+- Certificate expiration checks
+- API endpoint availability
+- Database connection tests
+
+---
+
+## How to Use: nexus cron
+
+The `nexus cron` command manages scheduled tasks through the Nexus Gateway.
+
+### Basic Commands
+
+```bash
+# List all cron jobs
+nexus cron list
+
+# Add a new cron job
+nexus cron add --schedule "0 9 * * *" --command "eve sync" --name "Morning sync"
+
+# Remove a cron job
+nexus cron remove <job-id>
+
+# Enable/disable a job
+nexus cron disable <job-id>
+nexus cron enable <job-id>
+
+# View job logs
+nexus cron logs <job-id>
+```
+
+### Schedule Format
+
+Cron uses standard cron syntax: `minute hour day month weekday`
+
+```bash
+# Examples:
+"0 9 * * *"      # Every day at 9:00 AM
+"*/30 * * * *"   # Every 30 minutes
+"0 */6 * * *"    # Every 6 hours
+"0 0 * * 0"      # Every Sunday at midnight
+"0 12 * * 1-5"   # Weekdays at noon
+"0 2 1 * *"      # First day of month at 2 AM
+```
+
+### Common Use Cases
+
+#### Daily Message Sync
+
+```bash
+# Sync iMessage daily at 2 AM
+nexus cron add \
+  --schedule "0 2 * * *" \
+  --command "eve sync" \
+  --name "Daily iMessage sync" \
+  --description "Sync Messages.app to eve database"
+
+# Sync WhatsApp hourly
+nexus cron add \
+  --schedule "0 * * * *" \
+  --command "wacli sync" \
+  --name "Hourly WhatsApp sync"
+```
+
+#### Daily Backups
+
+```bash
+# Backup workspace daily at 3 AM
+nexus cron add \
+  --schedule "0 3 * * *" \
+  --command "nexus-cloud push -m 'Daily backup'" \
+  --name "Daily workspace backup"
+```
+
+#### Morning Briefing
+
+```bash
+# Morning briefing at 7 AM on weekdays
+nexus cron add \
+  --schedule "0 7 * * 1-5" \
+  --command "nexus briefing --today" \
+  --name "Morning briefing"
+```
+
+#### Weekly Reports
+
+```bash
+# Weekly summary every Monday at 9 AM
+nexus cron add \
+  --schedule "0 9 * * 1" \
+  --command "nexus report --week" \
+  --name "Weekly summary"
+```
+
+#### Index Updates
+
+```bash
+# Update qmd index every hour
+nexus cron add \
+  --schedule "0 * * * *" \
+  --command "qmd update" \
+  --name "Update search index"
+```
+
+---
+
+## How to Use: Heartbeat
+
+Heartbeat configuration is typically done in the Nexus Gateway config file.
+
+### Configuration File Location
+
+```bash
+# Gateway config (varies by installation)
+~/.config/nexus/gateway/config.yaml
+# or
+~/nexus/home/.config/gateway.yaml
+```
+
+### Heartbeat Configuration
+
+```yaml
+# Example gateway config
+heartbeats:
+  - name: "Ollama health"
+    interval: 60  # seconds
+    command: "curl -sf http://localhost:11434/api/tags > /dev/null"
+    alert_on_failure: true
+    alert_after: 2  # Alert after 2 consecutive failures
+
+  - name: "Disk space check"
+    interval: 300  # 5 minutes
+    command: "df -h / | awk 'NR==2 {print $5}' | sed 's/%//' | awk '{if ($1 > 90) exit 1}'"
+    alert_on_failure: true
+
+  - name: "Database connectivity"
+    interval: 120
+    command: "sqlite3 ~/.eve/eve.db 'SELECT 1'"
+    alert_on_failure: true
+    alert_after: 3
+```
+
+### Heartbeat Commands
+
+```bash
+# List active heartbeats
+nexus heartbeat list
+
+# View heartbeat status
+nexus heartbeat status <heartbeat-name>
+
+# View heartbeat logs
+nexus heartbeat logs <heartbeat-name>
+
+# Manually trigger a heartbeat
+nexus heartbeat run <heartbeat-name>
+
+# Disable/enable heartbeat
+nexus heartbeat disable <heartbeat-name>
+nexus heartbeat enable <heartbeat-name>
+```
+
+### Common Heartbeat Checks
+
+#### Service Availability
+
+```yaml
+heartbeats:
+  - name: "Ollama running"
+    interval: 60
+    command: "curl -sf http://localhost:11434/api/tags > /dev/null"
+    alert_on_failure: true
+
+  - name: "PostgreSQL up"
+    interval: 120
+    command: "pg_isready -h localhost -U postgres"
+    alert_on_failure: true
+
+  - name: "Redis responding"
+    interval: 60
+    command: "redis-cli ping | grep -q PONG"
+    alert_on_failure: true
+```
+
+#### System Health
+
+```yaml
+heartbeats:
+  - name: "Disk space"
+    interval: 300
+    command: "df -h / | awk 'NR==2 {print $5}' | sed 's/%//' | awk '{if ($1 > 90) exit 1}'"
+    alert_on_failure: true
+
+  - name: "Memory usage"
+    interval: 300
+    command: "free | awk '/Mem:/ {if ($3/$2 > 0.9) exit 1}'"
+    alert_on_failure: true
+
+  - name: "CPU load"
+    interval: 60
+    command: "uptime | awk '{if ($10 > 4.0) exit 1}'"  # Load > 4.0
+    alert_on_failure: true
+```
+
+#### Application Health
+
+```yaml
+heartbeats:
+  - name: "Messages.app running"
+    interval: 120
+    command: "pgrep -x Messages"
+    alert_on_failure: true
+
+  - name: "API endpoint"
+    interval: 60
+    command: "curl -sf https://api.example.com/health"
+    alert_on_failure: true
+
+  - name: "Database backup age"
+    interval: 3600  # hourly
+    command: "find ~/backups -name 'db-*.sql' -mtime -1 | grep -q ."
+    alert_on_failure: true
+```
+
+---
+
+## Integration with Other Skills
+
+| Skill | Cron Use Case | Heartbeat Use Case |
+|-------|---------------|-------------------|
+| **eve** | Hourly message sync | Check Messages.app is running |
+| **wacli** | Hourly WhatsApp sync | Check WhatsApp connection |
+| **gog** | Daily Gmail sync | Check OAuth token validity |
+| **nexus-cloud** | Daily workspace backup | Check cloud connectivity |
+| **qmd** | Hourly index update | Check Ollama availability |
+| **comms** | Hourly unified sync | Check all adapters healthy |
+
+---
+
+## Common Failure Points
+
+### Failure: Cron job not executing
+
+**Symptom**: Scheduled task doesn't run at expected time
+
+**Cause**: Incorrect schedule syntax, gateway not running, or command error
+
+**Recovery**:
+```bash
+# Check gateway status
+nexus status
+
+# View cron job configuration
+nexus cron list
+
+# Check job logs for errors
+nexus cron logs <job-id>
+
+# Test command manually
+<your-command>
+
+# Fix schedule syntax (use crontab.guru for help)
+nexus cron remove <job-id>
+nexus cron add --schedule "0 9 * * *" --command "<fixed-command>" --name "<name>"
+```
+
+**Prevention**: Test commands manually before scheduling, verify schedule syntax
+
+---
+
+### Failure: Heartbeat false alarms
+
+**Symptom**: Heartbeat alerts even when service is healthy
+
+**Cause**: Check command too strict, network issues, or timing problems
+
+**Recovery**:
+```yaml
+# Adjust failure threshold
+heartbeats:
+  - name: "Service check"
+    interval: 60
+    command: "curl -sf http://localhost:8000/health"
+    alert_on_failure: true
+    alert_after: 3  # Only alert after 3 consecutive failures
+    timeout: 10  # Allow 10 seconds for response
+```
+
+**Prevention**: Add buffer for transient issues, increase timeout, require multiple failures
+
+---
+
+### Failure: Gateway not running
+
+**Symptom**: No cron jobs executing, no heartbeat checks
+
+**Cause**: Nexus Gateway stopped or not started
+
+**Recovery**:
+```bash
+# Check gateway status
+nexus status
+
+# Start gateway
+nexus gateway start
+
+# Enable auto-start
+nexus gateway enable
+
+# Check logs
+nexus gateway logs
+```
+
+**Prevention**: Enable gateway auto-start, monitor gateway health
+
+---
+
+## Best Practices
+
+### Cron Best Practices
+
+1. **Test commands first**: Run manually before scheduling
+2. **Use absolute paths**: Don't rely on PATH environment
+3. **Redirect output**: Capture stdout/stderr for debugging
+4. **Avoid overlaps**: Don't schedule heavy tasks too frequently
+5. **Monitor logs**: Regularly check job logs for errors
+6. **Use meaningful names**: Descriptive job names for easy management
+
+```bash
+# Good cron job
+nexus cron add \
+  --schedule "0 3 * * *" \
+  --command "/usr/local/bin/nexus-cloud push -m 'Daily backup' >> /tmp/backup.log 2>&1" \
+  --name "Daily workspace backup" \
+  --description "Backs up entire Nexus workspace to cloud storage"
+
+# Bad cron job (vague name, no logging)
+nexus cron add --schedule "0 3 * * *" --command "backup" --name "backup"
+```
+
+### Heartbeat Best Practices
+
+1. **Set appropriate intervals**: Balance monitoring vs overhead
+2. **Require multiple failures**: Avoid false alarms
+3. **Use timeouts**: Don't let checks hang forever
+4. **Monitor critical services only**: Don't over-monitor
+5. **Escalate properly**: Alert humans for critical failures
+
+```yaml
+# Good heartbeat
+heartbeats:
+  - name: "Critical DB health"
+    interval: 60
+    command: "timeout 5 pg_isready -h localhost"
+    alert_on_failure: true
+    alert_after: 3
+    timeout: 10
+    escalate_after: 5  # Human alert after 5 failures
+
+# Bad heartbeat (too frequent, no buffer)
+heartbeats:
+  - name: "DB"
+    interval: 5
+    command: "pg_isready"
+    alert_on_failure: true
+```
+
+---
+
+## Automation Examples
+
+### Complete Morning Routine
+
+```bash
+# Setup morning automation stack
+
+# 1. Sync messages overnight
+nexus cron add --schedule "0 2 * * *" --command "eve sync" --name "Overnight iMessage sync"
+nexus cron add --schedule "0 2 * * *" --command "wacli sync" --name "Overnight WhatsApp sync"
+
+# 2. Update indices
+nexus cron add --schedule "0 3 * * *" --command "qmd update" --name "Overnight index update"
+
+# 3. Backup workspace
+nexus cron add --schedule "0 4 * * *" --command "nexus-cloud push -m 'Daily backup'" --name "Daily backup"
+
+# 4. Generate briefing at 7 AM
+nexus cron add --schedule "0 7 * * 1-5" --command "nexus briefing --today" --name "Morning briefing"
+```
+
+### Health Monitoring Stack
+
+```yaml
+# Complete health monitoring configuration
+
+heartbeats:
+  # Service health
+  - name: "Ollama availability"
+    interval: 60
+    command: "curl -sf http://localhost:11434/api/tags > /dev/null"
+    alert_on_failure: true
+    alert_after: 2
+
+  - name: "Messages.app running"
+    interval: 300
+    command: "pgrep -x Messages"
+    alert_on_failure: true
+    alert_after: 1
+
+  # System health
+  - name: "Disk space check"
+    interval: 600
+    command: "df -h / | awk 'NR==2 {print $5}' | sed 's/%//' | awk '{if ($1 > 85) exit 1}'"
+    alert_on_failure: true
+    alert_after: 2
+
+  - name: "Memory usage"
+    interval: 300
+    command: "vm_stat | grep 'Pages free' | awk '{if ($3 < 10000) exit 1}'"
+    alert_on_failure: true
+    alert_after: 3
+
+  # Application health
+  - name: "Database accessible"
+    interval: 120
+    command: "sqlite3 ~/.eve/eve.db 'SELECT 1'"
+    alert_on_failure: true
+    alert_after: 2
+
+  - name: "Cloud connectivity"
+    interval: 300
+    command: "nexus-cloud status > /dev/null"
+    alert_on_failure: true
+    alert_after: 3
+```
+
+---
+
+## Testing Checklist
+
+### Cron Testing
+
+- [ ] Gateway running: `nexus status`
+- [ ] Add test job: `nexus cron add --schedule "*/1 * * * *" --command "date >> /tmp/test.log" --name "Test"`
+- [ ] Wait 1 minute
+- [ ] Check execution: `cat /tmp/test.log`
+- [ ] Check logs: `nexus cron logs <job-id>`
+- [ ] Remove test job: `nexus cron remove <job-id>`
+
+### Heartbeat Testing
+
+- [ ] Gateway running: `nexus status`
+- [ ] Add test heartbeat (config file)
+- [ ] Reload gateway: `nexus gateway reload`
+- [ ] View heartbeat status: `nexus heartbeat status <name>`
+- [ ] Manually trigger: `nexus heartbeat run <name>`
+- [ ] Check logs: `nexus heartbeat logs <name>`
+- [ ] Test failure: Break service, verify alert
+
+---
+
+## Notes for Onboarding Agent
+
+- **Gateway feature, not a skill**: Emphasize this is built into Nexus Gateway
+- **Setup requires gateway**: User must have gateway running
+- **When to recommend**:
+  - User wants automation (daily syncs, backups)
+  - User needs monitoring (service health checks)
+  - User has repetitive tasks
+  - User wants unattended operations
+- **When NOT to recommend**:
+  - User doesn't have gateway setup
+  - User prefers manual operations
+  - Tasks are infrequent/one-off
+- **Value proposition**: "Automate your Nexus tasks and monitor system health"
+- **Time estimate**: 10 minutes (add several cron jobs + heartbeats)
+- **Prerequisites**: Gateway must be installed and running
+- **Testing is critical**: Always test commands manually before scheduling
+- **Start simple**: Begin with one or two cron jobs, add more over time
+
+---
+
+## Gateway Documentation Reference
+
+For more details on Nexus Gateway features, see:
+- Gateway setup guide: `~/nexus/gateway/README.md`
+- Gateway configuration: `~/.config/nexus/gateway/config.yaml`
+- Gateway CLI reference: `nexus gateway --help`
+- Cron syntax helper: https://crontab.guru
+
+---
+
+## Advanced: Custom Alert Handlers
+
+You can configure custom alert handlers for heartbeat failures:
+
+```yaml
+# gateway config
+alert_handlers:
+  - type: "log"
+    path: "/var/log/nexus/alerts.log"
+
+  - type: "email"
+    to: "admin@example.com"
+    smtp_server: "smtp.gmail.com"
+
+  - type: "webhook"
+    url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
+
+  - type: "command"
+    command: "say 'Service down'"  # macOS text-to-speech
+
+heartbeats:
+  - name: "Critical service"
+    interval: 60
+    command: "curl -sf http://critical-service/health"
+    alert_on_failure: true
+    alert_handlers: ["email", "webhook"]
+```
+
+---
+
+## Changelog
+
+| Date | Change |
+|------|--------|
+| 2026-01-12 | Initial conceptual deep-dive created |
+
+---
+
+*Template version: 1.0*