@intent-systems/nexus 2026.1.5-3

package/skills/onboarding/docs/skill-deep-dives/computer-use.md

@@ -0,0 +1,599 @@
+# Skill Deep-Dive: computer-use
+
+**Skill**: `computer-use`
+**Category**: Automation
+**Setup Difficulty**: 🟡 Medium
+**Time Estimate**: ~5-10 minutes
+**Platform**: macOS + Linux
+
+---
+
+## What It Enables
+
+| Capability | Description | Example |
+|------------|-------------|---------|
+| Browser automation | Navigate websites, fill forms, complete OAuth flows | `interpreter "Set up OAuth credentials"` |
+| Desktop UI automation | Control any macOS app with mouse/keyboard | `peekaboo see --annotate` |
+| Screenshot analysis | Capture and analyze screen contents | `peekaboo see --analyze "What buttons are visible?"` |
+| Form filling | Automate repetitive data entry | Navigate to signup page and fill form |
+| Web scraping | Extract data from websites | Navigate and extract information |
+| OAuth setup automation | Complete OAuth consent screens | Set up Google Cloud OAuth for skills |
+| Scripted workflows | Reliable automation for known flows | Peekaboo scripts for repetitive tasks |
+| AI-driven workflows | Intelligent automation that adapts | Open Interpreter for complex tasks |
+
+---
+
+## Tool Comparison
+
+| Tool | Best For | LLM Required | Complexity | Platform |
+|------|----------|--------------|------------|----------|
+| **Peekaboo** | Scripted macOS UI automation | No (you drive) | Low | macOS only |
+| **Open Interpreter** | Full computer control with AI reasoning | Yes (OpenAI/Anthropic) | Medium | Cross-platform |
+| **browser-use** | Browser-only automation | Yes | Medium | Cross-platform |
+| **Claude Computer Use** | Anthropic's official solution | Claude | High (beta) | Cross-platform |
+
+---
+
+## Dependencies
+
+### Hard Dependencies (Required)
+- [ ] `uv` or `python` — For installing Open Interpreter
+- [ ] `brew` (macOS) — For installing Peekaboo
+- [ ] OpenAI or Anthropic API key — For LLM-driven tools (Open Interpreter)
+
+### Soft Dependencies (Recommended)
+- [ ] `1password` skill — For storing API keys securely
+- [ ] `tmux` skill — For background task execution
+
+### This Skill Enables
+- [ ] Autonomous OAuth setup — Skills like gog, nexus-cloud can auto-configure credentials
+- [ ] Form automation — Any repetitive UI task
+- [ ] Agent autonomy — Run complex UI flows without manual intervention
+
+---
+
+## Installation
+
+### Prerequisites Check
+
+```bash
+# Check for Python (for Open Interpreter)
+which python3 && echo "✅ Python available"
+
+# Check for uv (recommended installer)
+which uv && echo "✅ uv available" || echo "⚠️  Consider installing uv: curl -LsSf https://astral.sh/uv/install.sh | sh"
+
+# Check for Homebrew (for Peekaboo on macOS)
+which brew && echo "✅ Homebrew available"
+
+# Check platform
+uname -s
+```
+
+### Installation Steps
+
+#### Open Interpreter (Cross-platform)
+
+```bash
+# Install via uv (recommended)
+uv pip install open-interpreter --system
+
+# Or install via pip
+pip install open-interpreter
+
+# Verify installation
+interpreter --version
+```
+
+#### Peekaboo (macOS only)
+
+```bash
+# Install via Homebrew
+brew install steipete/tap/peekaboo
+
+# Verify installation
+peekaboo --version
+```
+
+### Verification
+
+```bash
+# Verify Open Interpreter
+interpreter --version
+
+# Verify Peekaboo (macOS)
+peekaboo --version
+
+# Check API keys for LLM access
+echo $OPENAI_API_KEY | grep -q . && echo "✅ OpenAI key set"
+echo $ANTHROPIC_API_KEY | grep -q . && echo "✅ Anthropic key set"
+```
+
+---
+
+## Configuration
+
+### Open Interpreter Configuration
+
+```bash
+# Set API key for OpenAI
+export OPENAI_API_KEY="your-key-here"
+
+# Or use 1Password
+export OPENAI_API_KEY=$(op read "op://Personal/OpenAI API Key/credential")
+
+# Or use Anthropic
+export ANTHROPIC_API_KEY="your-key-here"
+export ANTHROPIC_API_KEY=$(op read "op://Personal/Anthropic API Key/credential")
+
+# Configure default model
+interpreter --model gpt-4o
+# or
+interpreter --model claude-3-5-sonnet-20241022
+```
+
+### Peekaboo Configuration (macOS)
+
+```bash
+# Grant Accessibility permissions (MANUAL - Required for UI control)
+# System Settings → Privacy & Security → Accessibility → Add Terminal
+
+# Grant Screen Recording permission (MANUAL - Required for screenshots)
+# System Settings → Privacy & Security → Screen Recording → Add Terminal
+
+# Test permissions
+peekaboo see --mode frontmost --path /tmp/test.png
+```
+
+---
+
+## User Actions Required
+
+1. **Grant Accessibility permission** (Peekaboo): System Settings → Privacy & Security → Accessibility → Add Terminal
+2. **Grant Screen Recording permission** (Peekaboo): System Settings → Privacy & Security → Screen Recording → Add Terminal
+3. **Restart Terminal after granting permissions**
+4. **Set API key** (Open Interpreter): Create OpenAI or Anthropic account and get API key
+
+---
+
+## Common Failure Points
+
+### Failure: "Accessibility permission denied" (Peekaboo)
+
+**Symptom**: `peekaboo click` or `peekaboo paste` fails with permission error
+
+**Cause**: Terminal doesn't have Accessibility permission
+
+**Recovery**:
+1. Open System Settings → Privacy & Security → Accessibility
+2. Click the lock icon and authenticate
+3. Add Terminal (or your terminal app)
+4. **Restart Terminal**
+5. Try again: `peekaboo click --coords 100,100`
+
+**Prevention**: Complete permission setup before running peekaboo commands
+
+---
+
+### Failure: "Screen Recording permission denied" (Peekaboo)
+
+**Symptom**: `peekaboo see` fails or returns black screenshots
+
+**Cause**: Terminal doesn't have Screen Recording permission
+
+**Recovery**:
+1. System Settings → Privacy & Security → Screen Recording
+2. Enable Terminal
+3. Restart Terminal
+4. Try again: `peekaboo see --path /tmp/test.png`
+
+**Prevention**: Grant Screen Recording permission during initial setup
+
+---
+
+### Failure: "API key not found" (Open Interpreter)
+
+**Symptom**: `interpreter` command fails with authentication error
+
+**Cause**: API key environment variable not set
+
+**Recovery**:
+```bash
+# Check if key is set
+echo $OPENAI_API_KEY
+
+# Set it
+export OPENAI_API_KEY="sk-..."
+
+# Or load from 1Password
+export OPENAI_API_KEY=$(op read "op://Personal/OpenAI API Key/credential")
+
+# Add to shell profile for persistence
+echo 'export OPENAI_API_KEY=$(op read "op://Personal/OpenAI API Key/credential")' >> ~/.zshrc
+source ~/.zshrc
+```
+
+**Prevention**: Set up API key before running interpreter
+
+---
+
+### Failure: QR code expired or OAuth timeout
+
+**Symptom**: OAuth flow times out or QR code expires during automation
+
+**Cause**: Automation took too long, or unexpected UI state
+
+**Recovery**:
+```bash
+# For Open Interpreter: Re-run with explicit guidance
+interpreter --model gpt-4o -y "Complete OAuth flow, QR code expires in 20 seconds so work quickly"
+
+# For Peekaboo: Reduce sleep times in script
+# Change: sleep 10
+# To: sleep 2
+```
+
+**Prevention**: Test OAuth flows manually first to understand timing
+
+---
+
+### Failure: Element not found or wrong coordinates
+
+**Symptom**: Peekaboo clicks wrong location or can't find element
+
+**Cause**: UI changed, window resized, or incorrect coordinates
+
+**Recovery**:
+```bash
+# Take annotated screenshot to find new coordinates
+peekaboo see --annotate --path /tmp/debug.png
+
+# Open image to see element numbers
+open /tmp/debug.png
+
+# Update script with correct coordinates or element IDs
+```
+
+**Prevention**: Use annotated screenshots during script development
+
+---
+
+## Automation Opportunities
+
+| Step | Automatable? | Tool | Notes |
+|------|--------------|------|-------|
+| Install Open Interpreter | ✅ Yes | bash | `uv pip install open-interpreter --system` |
+| Install Peekaboo | ✅ Yes | bash | `brew install steipete/tap/peekaboo` |
+| Grant Accessibility | ❌ No | - | Requires manual GUI interaction |
+| Grant Screen Recording | ❌ No | - | Requires manual GUI interaction |
+| Set API key | ✅ Yes | bash | Environment variable + 1Password |
+| Run automation task | ✅ Yes | bash | Both tools fully scriptable |
+| Background execution | ✅ Yes | tmux | Dispatch long-running tasks |
+
+### Automation Script: OAuth Flow Example
+
+```bash
+#!/bin/bash
+# Automate Google Cloud OAuth setup using Peekaboo
+
+set -e
+
+echo "Starting OAuth setup automation..."
+
+# Open Chrome to Google Cloud Console
+peekaboo app switch --to "Google Chrome"
+peekaboo hotkey --keys "cmd,l"  # Focus address bar
+peekaboo paste --text "https://console.cloud.google.com/apis/credentials"
+peekaboo press return
+sleep 3
+
+# Take screenshot to see current state
+peekaboo see --annotate --path /tmp/oauth-step1.png
+echo "Screenshot saved to /tmp/oauth-step1.png"
+
+# Click "Create Credentials" button
+peekaboo click --coords 300,93
+sleep 2
+
+# Select "OAuth client ID"
+peekaboo click --on "OAuth client ID"
+sleep 2
+
+# Select "Desktop app"
+peekaboo click --on "Desktop app"
+
+# Enter app name
+peekaboo paste --text "gog-cli"
+peekaboo press tab
+
+# Click Create
+peekaboo click --on "Create"
+sleep 2
+
+# Download JSON
+peekaboo click --on "Download JSON"
+sleep 2
+
+echo "✅ OAuth setup complete! Check ~/Downloads for credentials.json"
+```
+
+### Open Interpreter: AI-Driven Automation
+
+```bash
+#!/bin/bash
+# Use Open Interpreter for complex OAuth flows
+
+interpreter --model gpt-4o -y "
+Navigate to console.cloud.google.com/apis/credentials
+Click 'Create credentials' > 'OAuth client ID'
+If prompted, configure consent screen:
+  - User type: External
+  - App name: gog-cli
+  - Scopes: Gmail API
+Select 'Desktop app'
+Name: gog-cli
+Click Create
+Download the JSON file
+Save to ~/.config/gog/credentials.json
+"
+```
+
+---
+
+## Wow Moments
+
+| Moment | What Happens | When to Demo |
+|--------|--------------|--------------|
+| Auto-OAuth setup | Watch agent navigate Google Cloud Console and set up OAuth | When setting up gog skill |
+| Screenshot analysis | Agent sees screen and describes what's visible | When debugging UI issues |
+| Form auto-fill | Agent fills out complex multi-page forms | When doing repetitive data entry |
+| Adaptive automation | Open Interpreter handles unexpected dialogs | When UI changes unexpectedly |
+| Background execution | Dispatch OAuth setup, continue working | During onboarding automation |
+
+### Demo Script
+
+```
+Agent: "Let me show you computer-use in action. I'll automate setting up OAuth credentials for the gog skill."
+
+# Take screenshot first
+peekaboo see --annotate --path /tmp/oauth-demo.png
+
+Agent: "I can see the current screen. Now I'll navigate to Google Cloud Console and create OAuth credentials."
+
+# Run automation
+interpreter --model gpt-4o -y "Set up OAuth credentials for gog-cli at console.cloud.google.com"
+
+Agent: "The OAuth setup is now complete! The credentials are saved to ~/.config/gog/credentials.json. This entire process would normally take 10 manual steps and 5 minutes. With computer-use, it's automated."
+
+Agent: "For simpler, repetitive tasks, I can use Peekaboo scripts that run instantly without AI costs."
+```
+
+---
+
+## Use Cases
+
+### 1. OAuth Setup Automation
+
+**Problem**: Setting up OAuth for skills like gog requires 10+ manual steps in web browser
+
+**Solution**: Use Open Interpreter to navigate browser and complete OAuth flow
+
+```bash
+interpreter --model gpt-4o -y "Set up Gmail OAuth credentials for gog-cli"
+```
+
+### 2. Form Filling
+
+**Problem**: Need to fill out the same form 50 times with different data
+
+**Solution**: Use Peekaboo script to automate form entry
+
+```bash
+# form-fill.sh
+for user in users.txt; do
+  peekaboo app switch --to "Google Chrome"
+  peekaboo click --on "name_field"
+  peekaboo paste --text "$user"
+  # ... continue for all fields
+done
+```
+
+### 3. Screenshot-Based Debugging
+
+**Problem**: User reports UI issue but can't describe it clearly
+
+**Solution**: Use Peekaboo to capture and analyze screen
+
+```bash
+peekaboo see --analyze "What error messages are visible?" --path /tmp/debug.png
+```
+
+### 4. Website Navigation
+
+**Problem**: Need to extract data from paginated website
+
+**Solution**: Use Open Interpreter to navigate and extract
+
+```bash
+interpreter "Go to example.com, navigate through all pages, extract all product names and prices"
+```
+
+---
+
+## Integration with Other Skills
+
+| Skill | Integration | Notes |
+|-------|-------------|-------|
+| **gog** | Automate OAuth setup | computer-use can set up Google Cloud credentials |
+| **nexus-cloud** | Automate Cloudflare setup | Deploy workers via browser automation |
+| **1password** | Store API keys | computer-use needs API keys for LLM access |
+| **tmux** | Background execution | Run long automations in background |
+| **onboarding** | Autonomous setup | computer-use enables fully automated skill setup |
+
+---
+
+## Testing Checklist
+
+- [ ] Fresh install: `uv pip install open-interpreter --system`
+- [ ] Fresh install: `brew install steipete/tap/peekaboo` (macOS)
+- [ ] Version check: `interpreter --version`
+- [ ] Version check: `peekaboo --version` (macOS)
+- [ ] API key set: `echo $OPENAI_API_KEY`
+- [ ] Accessibility permission granted (macOS)
+- [ ] Screen Recording permission granted (macOS)
+- [ ] Terminal restarted after permissions
+- [ ] Open Interpreter runs: `interpreter --model gpt-4o "Print hello"`
+- [ ] Peekaboo screenshot works: `peekaboo see --path /tmp/test.png`
+- [ ] Peekaboo annotate works: `peekaboo see --annotate --path /tmp/test.png`
+- [ ] OAuth automation test (safe environment)
+- [ ] Background execution: Dispatch task via tmux
+
+---
+
+## Notes for Onboarding Agent
+
+- **Permission requirements**: Always warn macOS users about Accessibility and Screen Recording permissions
+- **API costs**: Warn that Open Interpreter uses API tokens (costs money)
+- **Security awareness**: Never let agent see/handle passwords directly
+- **When to recommend**:
+  - User needs OAuth setup for skills (gog, nexus-cloud)
+  - User mentions repetitive UI tasks
+  - User wants autonomous onboarding
+- **When NOT to recommend**:
+  - User uncomfortable with giving CLI access to UI control
+  - Task is faster/safer to do manually
+  - No API key available (Open Interpreter)
+- **Tool selection guidance**:
+  - **Peekaboo**: Simple, known workflows (fast, no AI cost)
+  - **Open Interpreter**: Complex, adaptive workflows (slower, AI cost)
+- **Security reminders**:
+  - Sandbox first (incognito browser)
+  - Watch the first run
+  - Never automate production actions without review
+- **Time estimates**:
+  - Installation: 5 minutes
+  - Permission setup (macOS): 2 minutes + restart
+  - OAuth automation: 2-5 minutes per service
+  - Form filling: Depends on complexity
+
+---
+
+## Architecture: How It Works
+
+### Peekaboo (macOS Native)
+
+```
+Peekaboo CLI
+  ↓
+macOS Accessibility API
+  ↓
+Screen: Screenshot, OCR
+Mouse: Click, drag
+Keyboard: Type, paste, hotkeys
+Apps: Switch, launch
+```
+
+### Open Interpreter (LLM-Driven)
+
+```
+User instruction
+  ↓
+Open Interpreter
+  ↓
+LLM (GPT-4, Claude, etc.)
+  ↓
+Generated actions:
+  - Browser automation (Playwright/Selenium)
+  - Desktop automation (PyAutoGUI, AppleScript)
+  - Code execution (Python, JavaScript)
+  ↓
+System execution
+```
+
+---
+
+## Security & Privacy
+
+- **Peekaboo**: Runs locally, no data sent to cloud
+- **Open Interpreter**: Sends task descriptions to LLM API
+- **API keys**: Store in 1Password, not plaintext files
+- **Screen recording**: Peekaboo can capture everything on screen
+- **Accessibility**: Full keyboard/mouse control when granted
+- **Sandboxing**: Test in incognito/isolated environments first
+- **Credentials**: Never expose passwords to automation tools
+
+---
+
+## Limitations
+
+1. **Interactive sessions**: OAuth may require 2FA - agent will wait/prompt
+2. **CAPTCHAs**: Require human intervention (automation detection)
+3. **Rate limits**: Some sites detect and block automation
+4. **UI changes**: Scripts break when websites redesign
+5. **Peekaboo macOS only**: Not available on Linux/Windows
+6. **API costs**: Open Interpreter uses LLM tokens ($$)
+7. **Speed**: LLM-driven automation slower than scripted
+
+---
+
+## Best Practices
+
+1. **Test in sandbox**: Use incognito browser, test accounts
+2. **Watch first run**: Monitor automation to catch issues
+3. **Checkpoint often**: Save state after each major step
+4. **Error handling**: Add retries and fallbacks
+5. **Annotate screenshots**: Use `--annotate` during development
+6. **Keep scripts simple**: Complex logic → harder to debug
+7. **Document coordinates**: Comment what each click does
+8. **Version control**: Store automation scripts in Git
+9. **API key rotation**: Rotate keys regularly
+10. **Fallback to manual**: Some flows are faster by hand
+
+---
+
+## Advanced: Subagent Pattern
+
+For long-running automations, use background execution:
+
+```bash
+# Create task file
+cat > /tmp/oauth-task.sh << 'EOF'
+#!/bin/bash
+interpreter --model gpt-4o -y "Set up OAuth for gog-cli"
+EOF
+chmod +x /tmp/oauth-task.sh
+
+# Run in background via tmux
+tmux new -d -s oauth-setup "/tmp/oauth-task.sh"
+
+# Check status
+tmux ls
+
+# Attach to see progress
+tmux attach -t oauth-setup
+
+# View output later
+tmux capture-pane -t oauth-setup -p
+```
+
+---
+
+## Related Skills
+
+- `peekaboo` - macOS UI automation (part of this skill)
+- `tmux` - Background task execution
+- `1password` - API key storage
+- `gog` - Benefits from automated OAuth setup
+- `nexus-cloud` - Benefits from automated Cloudflare setup
+
+---
+
+## Changelog
+
+| Date | Change |
+|------|--------|
+| 2026-01-12 | Initial deep-dive created |
+
+---
+
+*Template version: 1.0*