@adaptic/maestro 1.1.7 → 1.4.1

package/docs/guides/email-setup.md

@@ -0,0 +1,399 @@
+# Email Setup Guide
+
+How to enable Gmail-based email for a Maestro agent: IMAP polling (reading inbound mail), SMTP sending (plain, threaded, with attachments, as principal), email thread deduplication, signature management, and email archival.
+
+**Prerequisites**: Complete the [Mac Mini Bootstrap](../runbooks/mac-mini-bootstrap.md) and have the agent's `.env` file created.
+
+---
+
+## Architecture Overview
+
+```
+┌──────────────────────────────────────────────────────────────────────┐
+│  INBOUND                                                              │
+│  Gmail IMAP ──▶ gmail-poller.mjs ──▶ state/inbox/gmail/*.yaml        │
+│  (every 60s)     imap-client.mjs      (inbox processor routes)       │
+│                                                                       │
+│  Optional: secondary inbox (e.g. CEO inbox)                          │
+│  Gmail IMAP ──▶ mehran-gmail-poller.mjs ──▶ state/inbox/gmail/*.yaml │
+├──────────────────────────────────────────────────────────────────────┤
+│  OUTBOUND                                                             │
+│                                                                       │
+│  ┌─────────────────┐  ┌───────────────────────┐  ┌───────────────┐  │
+│  │ send-email.sh   │  │ send-email-threaded.py │  │ send-email-   │  │
+│  │ (simple HTML)   │  │ (thread-aware + dedup) │  │ with-attach.py│  │
+│  └───────┬─────────┘  └──────────┬────────────┘  └──────┬────────┘  │
+│          │                       │                       │           │
+│          ▼                       ▼                       ▼           │
+│  ┌────────────────────────────────────────────────────────────────┐  │
+│  │  Pre-send pipeline:                                            │  │
+│  │  1. validate_outbound.py  (factual checks — blocks if issues)  │  │
+│  │  2. llm_email_dedup.py    (LLM asks: already addressed?)      │  │
+│  │  3. outbound_dedup.py     (content-hash dedup)                │  │
+│  │  4. pre_draft_lookup.py   (context enrichment — advisory)     │  │
+│  │  5. email_quote_thread.py (include quoted original in reply)  │  │
+│  └────────────────────────────────────────────────────────────────┘  │
+│          │                                                           │
+│          ▼                                                           │
+│     msmtp / Gmail SMTP ──▶ recipient                                │
+│     (with branded HTML signature auto-appended)                     │
+├──────────────────────────────────────────────────────────────────────┤
+│  DEDUP & THREADING                                                   │
+│  email_thread_dedup.py  — Hash-based thread dedup                   │
+│  llm_email_dedup.py     — LLM semantic dedup (Claude Haiku)         │
+│  email_quote_thread.py  — Fetch and format quoted reply chain       │
+├──────────────────────────────────────────────────────────────────────┤
+│  SEARCH                                                              │
+│  search-mehran-inbox.py — Search CEO inbox via IMAP                  │
+└──────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 1. Gmail Account & App Password
+
+Maestro uses IMAP for reading email and SMTP for sending. Both use Gmail app passwords (not OAuth) for simplicity and reliability on headless Mac minis.
+
+### 1.1 Enable 2-Factor Authentication
+
+App passwords require 2FA:
+
+1. Go to https://myaccount.google.com/security
+2. Enable 2-Step Verification if not already active
+
+### 1.2 Generate an App Password
+
+1. Go to https://myaccount.google.com/apppasswords
+2. Select app: "Mail", device: "Mac"
+3. Click "Generate"
+4. Copy the 16-character password (no spaces)
+
+### 1.3 Configure Environment Variables
+
+Add to `.env`:
+
+```bash
+# Agent's Gmail account
+GMAIL_APP_PASSWORD=xxxxxxxxxxxxxxxx
+
+# Optional: secondary inbox (e.g. CEO email) for monitoring
+SECONDARY_GMAIL_APP_PASSWORD=xxxxxxxxxxxxxxxx
+```
+
+### 1.4 Configure msmtp (SMTP Client)
+
+The `send-email.sh` script uses `msmtp` to send email via Gmail SMTP:
+
+```bash
+# Install msmtp
+brew install msmtp
+
+# Create configuration
+cat > ~/.msmtprc << 'EOF'
+defaults
+auth           on
+tls            on
+tls_trust_file /etc/ssl/cert.pem
+logfile        ~/maestro/logs/msmtp.log
+
+account adaptic
+host           smtp.gmail.com
+port           587
+from           agent@company.com
+user           agent@company.com
+password       YOUR_GMAIL_APP_PASSWORD
+EOF
+
+chmod 600 ~/.msmtprc
+```
+
+Replace `agent@company.com` and the password with your agent's credentials.
+
+**Alternative**: The Python send scripts (`send-email-threaded.py`, `send-email-with-attachment.py`) use `smtplib` directly with Gmail SMTP, so they don't need msmtp. Only the shell `send-email.sh` requires msmtp.
+
+---
+
+## 2. Inbound Email — IMAP Polling
+
+The poller checks Gmail via IMAP on every polling cycle (default: 60 seconds) and writes new messages to the inbox.
+
+### 2.1 How It Works
+
+1. `scripts/poller/gmail-poller.mjs` connects via `scripts/poller/imap-client.mjs`
+2. Searches for UNSEEN messages in INBOX
+3. Parses each message: sender, subject, body, attachments, threading headers
+4. Writes a YAML file to `state/inbox/gmail/` for the inbox processor
+5. Marks messages as SEEN to avoid reprocessing
+
+### 2.2 IMAP Configuration
+
+The poller reads credentials from `.env`:
+
+| Variable | Purpose |
+|---|---|
+| `GMAIL_APP_PASSWORD` | Agent's own inbox |
+| `SECONDARY_GMAIL_APP_PASSWORD` | CEO/secondary inbox |
+
+IMAP server settings are hardcoded to Gmail defaults (`imap.gmail.com:993` with SSL).
+
+### 2.3 Secondary Inbox Monitoring
+
+If the agent monitors a second inbox (e.g. the CEO's email for triage):
+
+- `scripts/poller/mehran-gmail-poller.mjs` handles the secondary inbox
+- Uses `SECONDARY_GMAIL_APP_PASSWORD`
+- Writes to the same `state/inbox/gmail/` directory with source annotation
+
+### 2.4 Verify IMAP Access
+
+```bash
+# Test IMAP connectivity (uses Python imaplib)
+python3 -c "
+import imaplib
+m = imaplib.IMAP4_SSL('imap.gmail.com')
+m.login('agent@company.com', 'YOUR_APP_PASSWORD')
+m.select('INBOX')
+_, msgs = m.search(None, 'UNSEEN')
+print(f'Connected OK. {len(msgs[0].split())} unread messages.')
+m.logout()
+"
+```
+
+---
+
+## 3. Outbound Email — Sending
+
+Three send scripts handle different use cases:
+
+### 3.1 Simple HTML Email (`send-email.sh`)
+
+Sends a one-shot HTML email with branded signature via `msmtp`.
+
+```bash
+./scripts/send-email.sh "recipient@example.com" "Subject line" "Body text here"
+
+# With CC and threading headers:
+./scripts/send-email.sh "to@example.com" "Re: Subject" "Reply body" "cc@example.com" "<message-id>" "<references>"
+```
+
+**Features**:
+- HTML signature with name, title, logo, phone, address auto-appended
+- Content-hash dedup via `outbound-dedup.sh`
+- Markdown-to-HTML conversion for body
+
+### 3.2 Threaded Email (`send-email-threaded.py`)
+
+The primary send script for production use. Thread-aware with full dedup pipeline.
+
+```bash
+python3 scripts/send-email-threaded.py "to@example.com" "Subject" "Body" \
+  --reply-to-subject "Original thread subject" \
+  --attachment /path/to/file.pdf
+```
+
+**Features**:
+- IMAP lookup for real Message-IDs (proper Gmail threading)
+- 3-layer dedup: LLM semantic check → Gmail Sent folder check → content-hash
+- Pre-draft context enrichment (advisory — looks up recipient before sending)
+- Factual validation (blocks if critical issues found)
+- Quoted reply chain inclusion (`email_quote_thread.py`)
+- Branded HTML signature auto-appended
+- `--force` flag to bypass dedup in exceptional cases
+
+### 3.3 Email with Attachments (`send-email-with-attachment.py`)
+
+```bash
+python3 scripts/send-email-with-attachment.py "to@example.com" "Subject" "Body" \
+  --attachment /path/to/file.pdf \
+  --attachment /path/to/image.png \
+  --cc "cc@example.com" \
+  --reply-to-subject "Thread subject"
+```
+
+**Features**:
+- MIME multipart with proper content-type detection
+- Supports multiple attachments
+- Full dedup pipeline (LLM + content-hash)
+- Pre-draft context enrichment
+- Same branded signature
+
+### 3.4 Sending as Principal (`send-email-as-mehran.py`)
+
+Sends email in the CEO/principal's voice from their email account:
+
+```bash
+python3 scripts/send-email-as-mehran.py "to@example.com" "Subject" "Body" \
+  --cc "cc@example.com"
+```
+
+This uses the secondary Gmail credentials and the principal's signature block.
+
+---
+
+## 4. Email Signatures
+
+### 4.1 Signature Files
+
+Two HTML signature templates live in `scripts/`:
+
+| File | Used By | Identity |
+|---|---|---|
+| `email-signature.html` | Agent's own emails | Agent name, title, logo, phone, address |
+| `email-signature-mehran.html` | Principal's emails | CEO name, title, logo, phone, address |
+
+### 4.2 Signature Behaviour
+
+- **All send scripts auto-append the HTML signature** — do not include a text sign-off in the email body
+- The signature includes: name, title, Adaptic logo (hosted on Google), phone number, office address, confidentiality disclaimer
+- Logo URL points to a Google-hosted image (no local file dependency)
+
+### 4.3 Customising for a New Agent
+
+Update `email-signature.html`:
+- Replace the agent name, title, and phone number
+- Keep the logo URL, address, and disclaimer structure
+
+---
+
+## 5. Email Thread Deduplication
+
+Email dedup prevents the agent from sending duplicate replies to the same thread. Three layers work together:
+
+### 5.1 Layer 1 — LLM Semantic Dedup (`llm_email_dedup.py`)
+
+Uses Claude Haiku to check if a topic has already been addressed in recent conversation history with the recipient.
+
+- Reads recent sent emails to the same recipient via IMAP
+- Asks Claude: "Has this topic already been addressed?"
+- Returns `DEDUP_SKIP` if the LLM determines it's a duplicate
+- Replaced the old subject+recipient hash lock (removed per CEO directive — was causing false positives on follow-up emails with the same subject)
+
+### 5.2 Layer 2 — Gmail Sent Folder Check
+
+Checks the actual Gmail Sent folder via IMAP to see if a similar email was recently sent.
+
+### 5.3 Layer 3 — Content-Hash Dedup (`outbound_dedup.py`)
+
+Atomic mkdir-based locking using SHA-256 hash of `to + subject + first 100 chars of body`. Prevents concurrent sessions from sending identical emails.
+
+- Lock TTL: 12 hours
+- Fail-open: if the lock system errors, the email sends anyway
+
+### 5.4 Email Quote Threading (`email_quote_thread.py`)
+
+When replying to an existing thread:
+1. Fetches the original email chain via IMAP
+2. Formats quoted HTML with attribution headers ("On [date], [sender] wrote:")
+3. Appends quoted chain below the new reply
+
+---
+
+## 6. Email Search
+
+### 6.1 Inbox Search (`search-mehran-inbox.py`)
+
+Searches the CEO's (or agent's) Gmail via IMAP for context retrieval:
+
+```bash
+python3 scripts/search-mehran-inbox.py --query "DFSA submission" --limit 5
+```
+
+Used by the pre-draft context system to find relevant email history before composing messages.
+
+---
+
+## 7. Email Archival
+
+### 7.1 Archive Script (`archive-email.sh`)
+
+Archives processed emails:
+
+```bash
+./scripts/archive-email.sh
+```
+
+Moves processed emails from active inbox directories to date-partitioned archive directories.
+
+---
+
+## 8. Testing
+
+| # | Test | How to Verify |
+|---|---|---|
+| 1 | IMAP connectivity | Run the Python IMAP test from section 2.4 |
+| 2 | msmtp configuration | `echo "test" \| msmtp -a adaptic your@email.com` |
+| 3 | Simple send | `./scripts/send-email.sh "your@email.com" "Test" "Body"` |
+| 4 | Threaded send | Send a reply with `--reply-to-subject` and verify Gmail threads it |
+| 5 | Attachment send | Send with `--attachment` and verify file arrives |
+| 6 | Dedup working | Send identical email twice; second should show `DEDUP_SKIP` |
+| 7 | LLM dedup | Send two different emails about same topic; second should warn |
+| 8 | Signature rendering | Check received email has branded HTML signature |
+| 9 | Poller picking up mail | Send email to agent, check `state/inbox/gmail/` |
+| 10 | Audit logging | Check `logs/audit/YYYY-MM-DD-actions.jsonl` for send entries |
+
+---
+
+## 9. Troubleshooting
+
+### IMAP connection refused
+
+1. Verify 2FA is enabled on the Google account
+2. Verify the app password is correct (no spaces)
+3. Check that IMAP is enabled: Gmail Settings → Forwarding and POP/IMAP → Enable IMAP
+4. Check firewall allows outbound connections to `imap.gmail.com:993`
+
+### msmtp authentication failure
+
+1. Verify `~/.msmtprc` has correct username and app password
+2. Verify file permissions: `chmod 600 ~/.msmtprc`
+3. Test: `echo "test" | msmtp -a adaptic --debug your@email.com`
+4. Check `logs/msmtp.log` for detailed error
+
+### Emails not threading in Gmail
+
+1. Verify `In-Reply-To` and `References` headers are set correctly
+2. Use `send-email-threaded.py` (not `send-email.sh`) for replies — it does IMAP lookup for real Message-IDs
+3. Gmail threads by `In-Reply-To` + matching `References` + same subject (with `Re:` prefix)
+
+### LLM dedup false positives
+
+1. If legitimate follow-ups are being blocked, use `--force` flag
+2. Review the LLM dedup logic in `llm_email_dedup.py` — it uses Claude Haiku for semantic comparison
+3. The LLM checks recent sent emails only (last 48 hours) — old conversations won't trigger
+
+### Duplicate emails being sent
+
+1. Verify `outbound-dedup.sh` is executable: `chmod +x scripts/outbound-dedup.sh`
+2. Check lock directory exists: `ls state/locks/outbound/email/`
+3. Verify lock TTL hasn't been set too low (default: 720 minutes / 12 hours)
+4. Run cleanup: `./scripts/outbound-dedup-cleanup.sh`
+
+---
+
+## Key Files
+
+| File | Purpose |
+|---|---|
+| `scripts/send-email.sh` | Simple HTML email via msmtp |
+| `scripts/send-email-threaded.py` | Thread-aware email with full dedup pipeline |
+| `scripts/send-email-with-attachment.py` | MIME email with file attachments |
+| `scripts/send-email-as-mehran.py` | Send as principal (CEO voice) |
+| `scripts/email_thread_dedup.py` | Hash-based email thread deduplication |
+| `scripts/llm_email_dedup.py` | LLM semantic deduplication (Claude Haiku) |
+| `scripts/email_quote_thread.py` | Fetch and format quoted reply chains |
+| `scripts/archive-email.sh` | Email archival workflow |
+| `scripts/email-signature.html` | Agent's branded HTML signature |
+| `scripts/email-signature-mehran.html` | Principal's branded HTML signature |
+| `scripts/search-mehran-inbox.py` | IMAP email search for context retrieval |
+| `scripts/poller/gmail-poller.mjs` | Inbound email polling (IMAP) |
+| `scripts/poller/imap-client.mjs` | IMAP client wrapper |
+| `scripts/poller/mehran-gmail-poller.mjs` | Secondary inbox polling |
+
+---
+
+## Related Documents
+
+- [Voice & SMS Setup](voice-sms-setup.md) — Phone and SMS capabilities
+- [Outbound Governance Setup](outbound-governance-setup.md) — Dedup, validation, information barriers
+- [Poller & Daemon Setup](poller-daemon-setup.md) — How email polling integrates with the event loop
+- [Agent Persona Setup](agent-persona-setup.md) — Configuring agent identity for email signature