@bobfrankston/mailx 1.0.146 → 1.0.150

package/README.md

@@ -13,245 +13,355 @@ npm install -g @bobfrankston/mailx
 mailx
 ```
 
-Requires Node.js 22 or later (uses built-in `node:sqlite`).
+Requires Node.js 22 or later.
 
-On Windows, the native WebView2 app launches automatically. On Linux/Mac, it opens in your default browser at `http://127.0.0.1:9333`.
+On Windows, the native WebView2 app launches automatically via IPC (no HTTP server needed). Use `mailx --server` for browser mode at `http://127.0.0.1:9333`.
 
 ## First-Time Setup
 
-### Option A: Existing user on a new machine (shared settings on OneDrive)
+On first run, mailx shows a setup form. Enter your name and email address.
 
-If you already have mailx configured on another machine with settings on OneDrive, just install and run. mailx auto-detects shared settings at:
+### Gmail / Google Workspace
 
-- `%OneDrive%/home/.mailx/settings.jsonc`
-- `%OneDriveConsumer%/home/.mailx/settings.jsonc`
-- `~/OneDrive/home/.mailx/settings.jsonc`
+No password needed. mailx detects Gmail via MX records, opens a browser for OAuth2 sign-in, and auto-configures Google Drive for settings sync. Custom domains hosted on Google Workspace are detected automatically.
 
-Once OneDrive syncs, mailx picks up your accounts automatically.
+### Outlook / Hotmail / Live
 
-### Option B: New user -- single machine, local settings
+No password needed. OAuth2 sign-in opens in your browser.
 
-**Step 1.** Create `~/.mailx/settings.jsonc` with your account(s):
+### Yahoo
 
-(On Windows `~` means `%USERPROFILE%`, e.g., `C:\Users\You\.mailx\settings.jsonc`)
+Use an **app password** (not your regular password):
+1. Go to **Yahoo Account Settings** > **Account Security**
+2. Click **Generate app password**
+3. Paste the generated password into mailx
 
-**Gmail -- minimal config (just your email):**
-```jsonc
-{
-  "accounts": [
-    { "email": "you@gmail.com" }
-  ]
-}
-```
+### AOL
 
-mailx auto-fills Gmail's IMAP/SMTP/OAuth settings. First run opens a browser for OAuth consent.
+Use an **app password**:
+1. Go to **AOL Account Settings** > **Account Security**
+2. Click **Generate app password**
+3. Paste the generated password into mailx
 
-**Standard IMAP -- just host and password:**
-```jsonc
-{
-  "accounts": [
-    {
-      "email": "you@example.com",
-      "password": "your-password",
-      "imap": { "host": "imap.example.com" },
-      "smtp": { "host": "smtp.example.com" }
-    }
-  ]
-}
-```
+### iCloud
 
-Defaults: port 993/587, TLS on, auth password, username = email address. Only provide fields that differ from defaults.
+Use an **app-specific password**:
+1. Go to [appleid.apple.com](https://appleid.apple.com) > **Sign-In and Security**
+2. Click **App-Specific Passwords** > **Generate**
+3. Paste the generated password into mailx
+
+### Other Providers (Dovecot, Fastmail, etc.)
+
+Enter your IMAP password. mailx auto-detects server settings via MX records and DNS. If detection fails, use **Manual setup (advanced)** in the setup form, or create an accounts file and import it (see below).
+
+### Adding Accounts via File
+
+You can create an `accounts.jsonc` file and import it into mailx. This is useful for adding accounts that aren't easily configured through the setup form (custom IMAP servers, non-standard ports, etc.).
+
+**Step 1.** Create a file (e.g., `my-accounts.jsonc`) with your account(s):
 
-**Full config with all optional fields:**
 ```jsonc
 {
   "accounts": [
     {
-      "id": "mymail",                 // Internal ID (default: domain name)
-      "name": "Your Name",            // From: header name (default: local part of email)
-      "label": "Work",                // UI display label (default: name)
       "email": "you@example.com",
+      "name": "Your Name",
+      "label": "Work",                   // Display name in folder tree
       "imap": {
-        "host": "imap.example.com",   // Default: imap.{domain}
-        "port": 993,                  // Default: 993
-        "tls": true,                  // Default: true
-        "auth": "password",           // Default: "password" (or "oauth2" for Gmail/Outlook)
-        "user": "you@example.com",    // Default: email address
+        "host": "imap.example.com",      // IMAP server hostname
+        "port": 993,                     // 993 for TLS (default)
+        "tls": true,
+        "user": "you@example.com",       // IMAP username (default: email)
         "password": "your-password"
       },
       "smtp": {
-        "host": "smtp.example.com",   // Default: smtp.{domain}
-        "port": 587,                  // Default: 587
-        "tls": true,                  // Default: true
-        "auth": "password",
+        "host": "smtp.example.com",      // SMTP server hostname
+        "port": 587,                     // 587 for STARTTLS (default)
+        "tls": true,
         "user": "you@example.com",
         "password": "your-password"
-      },
-      "enabled": true,                // Default: true
-      "defaultSend": true,            // Use this account when From doesn't match
-      "relayDomains": [],             // Domains to skip in Delivered-To chain
-      "deliveredToPrefix": []         // Prefixes to strip from Delivered-To alias
+      }
     }
-  ],
-  "sync": {
-    "intervalMinutes": 5,             // Default: 5
-    "historyDays": 0                  // Default: 30 (0 = all)
-  },
-  "ui": {
-    "theme": "system"                 // "system" (default), "dark", or "light"
-  }
+  ]
 }
 ```
 
-**Known providers with automatic defaults:** Gmail, Google, Outlook, Hotmail, Yahoo, iCloud. For these, only `email` is required.
-
-No `config.jsonc` needed -- when it doesn't exist, mailx reads settings directly from `~/.mailx/`.
-
-**Step 2.** Run `mailx` and open `http://127.0.0.1:9333` in your browser.
-
-### Option C: Multi-machine with shared settings (OneDrive/Dropbox)
-
-To share settings across machines, put `settings.jsonc` in a cloud-synced folder and create a local pointer file.
-
-**Step 1.** Move `settings.jsonc` to a shared location:
-
-```
-# Example: OneDrive
-mkdir "%OneDrive%\home\.mailx"
-move "%USERPROFILE%\.mailx\settings.jsonc" "%OneDrive%\home\.mailx\settings.jsonc"
-```
-
-**Step 2.** Create `~/.mailx/config.jsonc` pointing to the shared location:
+For known providers, most fields are optional:
 
 ```jsonc
 {
-  // Where the shared settings live (OneDrive, Dropbox, network share, etc.)
-  "sharedDir": "C:/Users/You/OneDrive/home/.mailx",
-
-  // Local-only settings (not synced):
-  "storePath": "C:/Users/You/.mailx/mailxstore",   // cached message bodies
-  "historyDays": 90                                  // override sync depth per machine
+  "accounts": [
+    { "email": "you@gmail.com", "name": "Your Name" },
+    { "email": "you@yahoo.com", "name": "Your Name", "password": "app-password-here" },
+    {
+      "email": "you@yourserver.com",
+      "name": "Your Name",
+      "label": "Personal",
+      "imap": { "host": "imap.yourserver.com", "user": "you", "password": "pass" },
+      "smtp": { "host": "smtp.yourserver.com", "user": "you", "password": "pass" }
+    }
+  ]
 }
 ```
 
-**How it works:**
-- `config.jsonc` is local to each machine -- never synced
-- `sharedDir` points to the cloud folder containing `settings.jsonc`, `allowlist.jsonc`, `preferences.jsonc`
-- mailx caches shared files locally for offline use
-- `historyDays` in `config.jsonc` overrides the shared default per machine
-- On new machines, if `config.jsonc` doesn't exist, mailx auto-detects OneDrive at `%OneDrive%/home/.mailx/`
+**Step 2.** Import into mailx:
 
-### Gmail OAuth setup
+```bash
+mailx --import my-accounts.jsonc
+```
+
+This merges the accounts into your Google Drive settings (deduplicates by email address). Existing accounts are preserved.
 
-Gmail accounts auto-configure -- just add `{ "email": "you@gmail.com" }` to accounts. But OAuth requires a one-time Google Cloud setup:
+### Gmail OAuth Prerequisites
 
+Gmail OAuth requires a one-time Google Cloud setup:
 1. Go to [Google Cloud Console](https://console.cloud.google.com/) and create a project
 2. Enable the **Gmail API** (and **People API** for contacts)
 3. Create **OAuth 2.0 credentials** (Desktop app type)
 4. Download `credentials.json` to the iflow package directory
-5. First connection opens a browser for OAuth consent. Tokens are cached and refresh automatically.
+5. First connection opens a browser for consent. Tokens refresh automatically.
 
 ## Usage
 
 ### Reading Mail
-- **Folder tree** (left) -- click to view, expand/collapse subfolders
-- **Message list** (center) -- click to preview, scroll for more
-- **Preview pane** (right) -- shows message content
-- **All Inboxes** -- unified view across all accounts
-- **Folder search** -- filter box above folder tree to find folders quickly
-- **Message search** -- search bar with scope: This folder / All folders / IMAP server
+
+- Click a **folder** to see its messages
+- Click a **message** to read it in the preview pane
+- **All Inboxes** combines inboxes from all accounts (appears with 2+ accounts)
+- Unread counts show on folders; sub-folder counts bubble up to collapsed parents
+- Use the **folder search** box to find folders by name
 
 ### Composing
-- **Compose** -- toolbar or Ctrl+N
-- **Reply / Reply All / Forward** -- Ctrl+R / Ctrl+Shift+R / toolbar
-- **Send** -- Ctrl+Enter
-- **From field** -- dropdown with all accounts + "Other..." for custom addresses
-- **Address autocomplete** -- type in To/Cc/Bcc, Tab accepts match
-- **Auto-save** -- drafts saved to IMAP every 5 seconds
+
+- **Ctrl+N** -- New message
+- **Ctrl+R** -- Reply
+- **Ctrl+Shift+R** -- Reply All
+- **Ctrl+F** -- Forward
+- From dropdown lets you pick which account to send from
+- Contact autocomplete searches Google Contacts as you type in To/Cc/Bcc
+- Drafts auto-save every 5 seconds to your Drafts folder
 
 ### Managing Messages
-- **Delete** -- Del key or trash button, moves to Trash. Ctrl+Z to undo (30s).
-- **Flag/unflag** -- click the star
-- **Drag and drop** -- drag messages to folders to move them. Shift/Ctrl+click for multi-select.
-- **Attachments** -- click the chip to open PDFs, images, etc. in the browser
-- **Remote content** -- blocked by default. "Load once" or "Always" buttons on the banner.
-- **Unsubscribe** -- button appears in header when List-Unsubscribe header is present
-- **View Source** -- Source button copies .eml file path to clipboard
-- **Link preview** -- hover over links in email body to see URL in status bar
-
-### Managing Folders
-- **Right-click any folder** for context menu:
-  - Mark all read
-  - New subfolder
-  - Rename folder
-  - Delete folder
-  - Empty folder (Trash/Junk only -- permanently deletes all messages)
-- Special folders (Inbox, Sent, Trash, etc.) cannot be renamed or deleted
+
+- **Delete** or **Ctrl+D** -- Delete selected messages (moves to Trash)
+- **Ctrl+Z** -- Undo last delete
+- **Ctrl+A** -- Select all messages in the list
+- **Drag and drop** -- Move messages to a folder by dragging them
+- Click the **star** column to flag/unflag a message
+- **Unsubscribe** button appears when the message has a List-Unsubscribe header
+
+### Searching
+
+Type in the search bar and press **Enter**. Use the dropdown to search:
+- **All folders** -- Every account and folder
+- **This folder** -- Current folder only (filters as you type, server search on Enter)
+- **IMAP server** -- Server-side search (useful for messages older than your sync window)
+
+Supports regex: wrap pattern in `/slashes/`.
+
+### Folder Operations
+
+Right-click a folder for options:
+- Mark all read
+- New subfolder
+- Rename / Delete folder
+- Empty (Trash and Junk only)
+
+### View Options
+
+Under **View** in the toolbar:
+- **Two-line view** -- Show preview snippet below each subject
+- **Preview pane** -- Toggle the message reader panel
+- **Preview snippets** -- Show snippet text in message rows
+- **Flagged only** -- Show only starred messages
+- **Folder counts** -- Show total message counts on all folders
+
+### Settings
+
+Under **Settings** in the toolbar:
+- **Editor** -- Choose between Quill (default) and tiptap for compose
+- **AI autocomplete** -- Enable LLM-powered writing suggestions (Ollama, Claude, or OpenAI)
 
 ### Keyboard Shortcuts
+
 | Key | Action |
 |-----|--------|
 | Ctrl+N | New message |
 | Ctrl+R | Reply |
 | Ctrl+Shift+R | Reply All |
-| Del / Ctrl+D | Delete |
+| Ctrl+F | Forward |
+| Delete / Ctrl+D | Delete |
 | Ctrl+Z | Undo delete |
-| Ctrl+Enter | Send (compose) |
-| Escape | Clear search / Discard compose |
+| Ctrl+A | Select all |
 | F5 | Sync all folders |
+| Escape | Clear search / close menus |
 
-### View Options (View menu)
-- **Two-line view** -- compact message rows
-- **Preview pane** -- toggle on/off
-- **Flagged only** -- show only starred messages
+## Command Line
 
-## Data Storage
+```
+mailx                        Start the app (native window via IPC)
+mailx --server               Start HTTP server mode (http://localhost:9333)
+mailx --no-browser            Server mode without opening browser
+mailx --verbose               Show log output in terminal (default: log file only)
+mailx --import <file>         Import accounts.jsonc into Google Drive
+
+mailx -kill                   Kill running mailx processes + clean up WAL files
+mailx -repair                 Re-sync message metadata (fix garbled subjects)
+mailx -rebuild                Wipe local cache and re-download everything from IMAP
+mailx -setup                  Interactive first-time setup (CLI)
+mailx -test                   Test IMAP/SMTP connectivity for all accounts
+mailx -v                      Show version
+```
+
+### -repair vs -rebuild
+
+**-repair** clears message metadata (subjects, flags, sender names) from the database but preserves your downloaded `.eml` message files. Folder sync state resets so mailx re-fetches all envelopes on next run. Use this when subjects show garbled characters.
+
+**-rebuild** deletes the entire database and message store, then re-downloads everything from IMAP. Accounts and settings are preserved. Use this for a clean start or to reclaim disk space.
 
-All data lives in `~/.mailx/` (e.g., `C:\Users\You\.mailx\`):
+## Configuration
 
-| File | Shared? | Purpose |
+All config files live in `~/.mailx/` (on Windows: `C:\Users\You\.mailx\`).
+
+### config.jsonc (per-machine, not synced)
+
+Points to shared settings and controls local behavior. Created automatically for Gmail accounts.
+
+```jsonc
+{
+  "sharedDir": { "provider": "gdrive", "path": "mailx", "folderId": "..." },
+  "storePath": "C:/Users/You/.mailx/mailxstore",
+  "historyDays": 90,
+  "accountOverrides": {
+    "work": { "historyDays": 7 },
+    "archive": { "enabled": false }
+  }
+}
+```
+
+**sharedDir** can be:
+- `{ "provider": "gdrive", "path": "mailx" }` -- Google Drive via API (auto-configured for Gmail)
+- `"C:/Users/You/OneDrive/home/.mailx"` -- Local/network path
+
+### accounts.jsonc (shared via Google Drive or local)
+
+For known providers, only `email` is required -- IMAP/SMTP settings fill automatically.
+
+```jsonc
+{
+  "accounts": [
+    { "email": "you@gmail.com", "name": "Your Name" },
+    { "email": "you@icloud.com", "name": "Your Name", "password": "xxxx-xxxx-xxxx-xxxx" },
+    {
+      "email": "you@example.com",
+      "name": "Your Name",
+      "label": "Work",
+      "imap": { "host": "imap.example.com", "port": 993, "tls": true, "user": "you", "password": "..." },
+      "smtp": { "host": "smtp.example.com", "port": 587, "tls": true, "user": "you", "password": "..." }
+    }
+  ]
+}
+```
+
+**Auto-detected providers:**
+
+| Domain | IMAP | SMTP | Auth | Label |
+|--------|------|------|------|-------|
+| gmail.com | imap.gmail.com:993 | smtp.gmail.com:587 | OAuth2 | Gmail |
+| outlook.com, hotmail.com | outlook.office365.com:993 | smtp.office365.com:587 | OAuth2 | Outlook |
+| yahoo.com | imap.mail.yahoo.com:993 | smtp.mail.yahoo.com:587 | App password | Yahoo |
+| aol.com | imap.aol.com:993 | smtp.aol.com:587 | App password | AOL |
+| icloud.com | imap.mail.me.com:993 | smtp.mail.me.com:587 | App password | iCloud |
+
+Google Workspace and Microsoft 365 custom domains are detected automatically via MX records.
+
+### preferences.jsonc (shared)
+
+```jsonc
+{
+  "ui": { "theme": "dark", "editor": "quill", "fontSize": 15 },
+  "sync": { "intervalMinutes": 5, "historyDays": 30 },
+  "autocomplete": {
+    "enabled": false,
+    "provider": "ollama",
+    "ollamaUrl": "http://localhost:11434",
+    "ollamaModel": "qwen2.5-coder:1.5b"
+  }
+}
+```
+
+### allowlist.jsonc (shared)
+
+Remote content (images, tracking pixels) is blocked by default. Whitelist trusted senders:
+
+```jsonc
+{
+  "senders": ["boss@example.com"],
+  "domains": ["example.com"],
+  "recipients": ["team@our-group.com"]
+}
+```
+
+### Multi-Machine Setup
+
+Settings sync via Google Drive API. Each machine has its own `config.jsonc`:
+
+**Desktop** (full history):
+```jsonc
+{ "sharedDir": { "provider": "gdrive", "path": "mailx" }, "historyDays": 0 }
+```
+
+**Laptop** (recent only):
+```jsonc
+{ "sharedDir": { "provider": "gdrive", "path": "mailx" }, "historyDays": 30,
+  "accountOverrides": { "archive": { "enabled": false } } }
+```
+
+Both machines share accounts, preferences, and allowlist via Google Drive.
+
+## Data Storage
+
+| Path | Synced? | Purpose |
 |------|---------|---------|
-| config.jsonc | No | Points to shared settings + local overrides |
-| settings.jsonc | Yes | Account configs, sync, UI prefs (or split files below) |
+| config.jsonc | No | Local config pointer + overrides |
 | accounts.jsonc | Yes | IMAP/SMTP account configs |
 | preferences.jsonc | Yes | UI and sync settings |
-| allowlist.jsonc | Yes | Remote content sender/domain allow-list |
+| allowlist.jsonc | Yes | Remote content allow-list |
 | mailx.db | No | SQLite metadata (headers, contacts, sync state) |
 | mailxstore/ | No | Cached message bodies (.eml per message) |
-| logs/ | No | Server logs (auto-deleted after 7 days) |
-| window.json | No | Window position (per machine) |
-| webview2/ | No | WebView2 data (Windows native app) |
+| logs/ | No | Daily log files |
 
-**Shared** files can live on OneDrive/Dropbox. **Local** files stay on the machine.
+**Safe to delete:** mailxstore/ (re-downloaded on sync), mailx.db (re-created on startup), logs/ (anytime).
 
-### Safe to Delete
+## Logs
 
-- **mailxstore/** -- re-downloaded during sync. Delete to reclaim space.
-- **mailx.db** -- re-created on startup, triggers full re-sync.
-- **logs/** -- safe to delete anytime.
-- **window.json** -- resets window position.
+Daily log files: `~/.mailx/logs/mailx-YYYY-MM-DD.log`
 
-## Development
+Use `mailx --verbose` to see logs in the terminal instead.
 
-```bash
-git clone https://github.com/BobFrankston/mailx.git
-cd mailx
-npm install
-npm start                          # Server with --watch (auto-restart)
-# Open http://127.0.0.1:9333
-
-launch.ps1                        # Windows native WebView2 app
-launch.ps1 -restart                # Kill existing and restart
-launch.ps1 -clear                  # Clear WebView2 cache
-```
+## Troubleshooting
+
+**Blank window or no response** -- Run `mailx -kill` then `mailx` to restart.
+
+**OAuth timeout** -- If the browser auth window doesn't appear or times out (5 minutes), close mailx and try again.
+
+**Garbled subjects** -- Run `mailx -repair` to re-sync metadata from IMAP.
+
+**Sync stuck or corrupt data** -- Run `mailx -rebuild` for a clean start. Accounts and settings are preserved.
+
+**"Too many connections"** -- mailx uses up to 5 IMAP connections per account. Close other email clients if you hit your server's limit.
+
+**App password rejected (Yahoo/AOL/iCloud)** -- Make sure you're using an app-specific password, not your regular login. Generate a new one from your account security settings.
 
 ## Architecture
 
-- **Local-first** -- changes update local DB immediately, background worker syncs to IMAP
-- **Offline reading** -- full message bodies cached as .eml files during sync
-- **IMAP IDLE** -- instant new mail notifications + periodic sync fallback
-- **Outbox** -- queued in IMAP Outbox folder with multi-machine interlock via flags
-- **Remote content blocking** -- HTML sanitized server-side, CSP in iframe, per-sender/domain allow-list
-- **Search** -- SQLite FTS5 full-text index, IMAP server search, regex filtering
-- **Built-in SQLite** -- uses Node.js built-in node:sqlite (no native compilation needed)
-- **Cross-platform** -- Windows (WebView2), Linux (webkit2gtk), any platform (browser mode)
+- **IPC-first** -- Default mode uses msger (Rust/WebView2) with bidirectional IPC. No TCP server, no CLOSE_WAIT zombies.
+- **HTTP fallback** -- `--server` flag starts Express for browser/remote access.
+- **Local-first** -- Changes update local DB immediately; background worker syncs to IMAP.
+- **Offline reading** -- Full message bodies cached as .eml files.
+- **IMAP IDLE** -- Instant new-mail notifications + periodic sync fallback.
+- **Outbox** -- Queued in IMAP Outbox folder with multi-machine interlock via flags.
+- **Remote content blocking** -- HTML sanitized server-side, CSP in iframe, per-sender allow-list.
+- **Search** -- SQLite FTS5 full-text index + IMAP server search + regex filtering.
+- **Connection management** -- Semaphore limits 5 connections per account; exponential backoff on server limits.
+- **Cloud sync** -- Google Drive API for settings portability (no filesystem mount required).