@lengelhard/imap-email-mcp 1.8.0 → 1.9.0

package/.env.example

@@ -21,6 +21,7 @@ IMAP_HOST=imap.example.com
 # INDEX_FOLDERS=            (comma list; default: all folders except Trash and Junk)
 # INDEX_EXCLUDE_FOLDERS=    (comma list, added to the default exclusions)
 # INDEX_SYNC_INTERVAL_MINUTES=5
+# INDEX_SYNC_MAX_MINUTES=15     (wall-clock cap on one sync run; 0 disables)
 
 # Optional - SMTP Settings (defaults to IMAP values if not set)
 # SMTP_HOST=smtp.example.com

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 1.9.0
+
+### Added
+- `find_split_threads`: lists conversations whose messages sit in more than one folder, threaded on the In-Reply-To and References headers rather than on subject. A reply that was renamed ("Onboarding - Accrete" becoming "Onboarding - Accrete: data room request list") stays in its thread because its References still chain back to the root, and two unrelated conversations that happen to share a subject ("Catch-up call") never merge. `scope: "split"` (the default) reports only threads spread across the searched folders; `scope: "all"` lists every thread in scope. Each thread carries `messages`, `distinct_messages`, `by_folder`, `oldest`, `newest`, `participants` and its `message_list`; a thread whose only spread is the same Message-ID copied into two folders is marked `copies_only`, which is a duplicate for `find_duplicates`, not a split. Paged with `limit` and `offset`. Requires the local index.
+- `in_reply_to` and `references` on the results of `list_emails`, `search_emails`, `search_index` and `get_email`, angle brackets stripped, folded headers unfolded, oldest first.
+- The index now stores In-Reply-To and References. Rows indexed by an earlier version are backfilled by the background sync, up to 3,000 rows per folder per run, without a rebuild. `index_status` reports the number still waiting as `thread_headers_pending`, and `find_split_threads` reports it under `coverage.without_thread_headers` and marks any thread that touches such rows `threading: "partial"`, so a report on a half-backfilled index says so instead of looking complete.
+
+### Changed
+- Opening a database written by 1.8.x adds the three threading columns in place. Searches keep working while the backfill runs.
+
+## 1.8.1
+
+### Fixed
+- A sync could still hang forever after a connection dropped. The folder LIST at the start of every run was the one IMAP command not under the 1.7.3 time limit, and node-imap never fails a command whose socket has died, so a run that started on a silent connection (a sleeping laptop, a dropped session) waited indefinitely with `sync_running: true`, blocking every later sync and every `sync_index` call. Seen on 18, 20 and 21 Sep 2026. The LIST now runs under the same limit as every other command.
+- As a backstop, every sync run has a wall-clock cap (`INDEX_SYNC_MAX_MINUTES`, default 15). A run that outlives it is aborted, its connection is dropped and its lease released, and the next scheduled sync starts clean. A run cut loose can no longer write to the index.
+- The next background sync is scheduled from the end of the previous run instead of a fixed timer, so a slow run cannot queue a pile-up behind itself.
+
+### Added
+- `index_status` reports `sync_running_since`, `sync_running_seconds` and `sync_max_minutes`, so a stuck sync is a visible number rather than a flag.
+- `sync_index` takes `force: true`, which aborts a running sync before starting a fresh one. The index is read-only against the server, so this is always safe.
+- `find_duplicates` takes `offset` and returns `has_more`, so more than 500 groups can be paged through instead of partitioned by date.
+
 ## 1.8.0
 
 ### Added

package/README.md

@@ -105,7 +105,8 @@ Add to your config file:
 | `INDEX_DB_PATH` | see below | Location of the index database |
 | `INDEX_FOLDERS` | all but Trash/Junk | Comma-separated folders to index |
 | `INDEX_EXCLUDE_FOLDERS` | none | Comma-separated folders to skip, on top of Trash/Junk |
-| `INDEX_SYNC_INTERVAL_MINUTES` | `5` | Background sync interval |
+| `INDEX_SYNC_INTERVAL_MINUTES` | `5` | Background sync interval, measured from the end of the previous run |
+| `INDEX_SYNC_MAX_MINUTES` | `15` | Wall-clock cap on one sync run; a run that outlives it is aborted and its connection dropped (`0` disables) |
 
 ### Local search index
 
@@ -128,6 +129,16 @@ Mail clients that file by tag sometimes append a fresh copy of a message instead
 - **Honest coverage.** A message with no Message-ID header cannot be compared. It is counted in `coverage.without_message_id` rather than quietly skipped, and `index.complete` says whether the index had seen everything.
 - **Nothing is deleted.** The tool only reports. Acting on it means calling `delete_email`, which moves to Trash unless you pass `expunge: true`.
 
+### Finding split threads
+
+A conversation can end up spread across folders without any message being duplicated: the root is filed in one place, a later reply in another. `find_split_threads` finds those by following the In-Reply-To and References headers, which every mail client writes when replying.
+
+- **Why headers, not subject.** References lists the Message-IDs of every earlier message in the conversation, so a reply keeps its thread even after someone renames the subject, and two unrelated conversations with the same subject line stay apart. Subject matching gets both of those wrong.
+- **Scope.** `scope: "split"` (default) reports only threads present in more than one of the searched folders; `scope: "all"` lists every thread. The folders you pass decide what counts as split: a thread with three messages in Delegated and two in Hold is split when both folders are searched and not when only Delegated is.
+- **Copies are not splits.** A thread whose only spread is the same Message-ID present in two folders is marked `copies_only`. That is a duplicate, and `find_duplicates` is the tool for it.
+- **Honest coverage.** Rows indexed before 1.9.0 have no threading headers until the background sync backfills them (3,000 rows per folder per run). `coverage.without_thread_headers` and `index_status.thread_headers_pending` show how many are still waiting, and a thread touching such rows is marked `threading: "partial"`.
+- **Nothing is moved.** The tool only reports. `move_email` or `move_thread` are the tools for acting on it.
+
 ### Provider Settings
 
 | Provider | IMAP_HOST | SMTP_HOST | Notes |
@@ -147,9 +158,10 @@ Mail clients that file by tag sometimes append a fresh copy of a message instead
 | `get_email` | Get an email by UID. Text by default; HTML (without inline images) on request |
 | `search_emails` | Live IMAP search across one or more folders: subject, from, to, cc, body, text, dates, flags |
 | `search_index` | Full-text search of the local index (when `INDEX_ENABLED=true`); `index.warning` flags a stale index or folder errors |
-| `index_status` | Per-folder index coverage, last sync and errors |
-| `sync_index` | Run an index sync now |
+| `index_status` | Per-folder index coverage, last sync and errors; `sync_running_seconds` shows how long the current sync has run |
+| `sync_index` | Run an index sync now; `force: true` aborts a stuck sync first |
 | `find_duplicates` | Messages that exist more than once, matched on Message-ID: filed twice in one folder, or spread across folders |
+| `find_split_threads` | Conversations spread across folders, threaded on In-Reply-To and References so renamed replies stay with their thread |
 | `list_drafts` | List all draft emails |
 | `get_draft` | Get a specific draft by UID |
 | `create_draft` | Create a new email draft |

package/index.js

@@ -31,7 +31,7 @@ import {
   buildCriteria
 } from './lib/imap-session.js';
 import { htmlToText, stripDataUris, leafParts } from './lib/mime-text.js';
-import { LocalIndex, normalizeMessageId } from './lib/local-index.js';
+import { LocalIndex, normalizeMessageId, parseMessageIdList } from './lib/local-index.js';
 
 const PKG_VERSION = createRequire(import.meta.url)('./package.json').version;
 
@@ -445,9 +445,53 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
             required: ['query']
           }
         },
+        {
+          name: 'find_split_threads',
+          description: 'List conversations whose messages sit in more than one folder, threaded by Message-ID chains (In-Reply-To and References), the same way a mail client threads. Exact where the headers exist: a renamed subject stays in its thread and two conversations that share a subject never merge. A thread flagged copies_only is the same message copied into several folders, a duplicate rather than a split; see find_duplicates for those. Check coverage.without_thread_headers: messages indexed before 1.9.0 thread only by their own Message-ID until the backfill reaches them. Requires the local index.',
+          inputSchema: {
+            type: 'object',
+            properties: {
+              folders: {
+                type: 'array',
+                items: { type: 'string' },
+                description: 'Only these folders (default: all indexed folders). A thread counts as split only across folders in this list.'
+              },
+              scope: {
+                type: 'string',
+                enum: ['split', 'all'],
+                description: 'split: threads spanning more than one folder (default); all: every thread',
+                default: 'split'
+              },
+              since_date: {
+                type: 'string',
+                description: 'Only messages delivered on or after this date (YYYY-MM-DD)'
+              },
+              before_date: {
+                type: 'string',
+                description: 'Only messages delivered before this date (YYYY-MM-DD)'
+              },
+              limit: {
+                type: 'number',
+                description: 'Maximum threads to return (default: 50, max: 500). total_threads reports the full count.',
+                default: 50
+              },
+              offset: {
+                type: 'number',
+                description: 'Skip this many threads, for paging (default: 0). has_more says whether another page exists.',
+                default: 0
+              },
+              refresh: {
+                type: 'boolean',
+                description: 'Pick up new and moved mail before searching (default: true). Skipped while a background sync is running.',
+                default: true
+              }
+            },
+            required: []
+          }
+        },
         {
           name: 'index_status',
-          description: 'Show the local index state: per-folder message counts on the server and in the index, whether backfill is complete, last sync time and errors.',
+          description: 'Show the local index state: per-folder message counts on the server and in the index, whether backfill is complete, last sync time and errors. thread_headers_pending counts messages still waiting for their threading headers (1.9.0 backfill). sync_running_seconds says how long the current sync has been going; a value far beyond sync_max_minutes means it is stuck and sync_index with force: true will cut it loose.',
           inputSchema: { type: 'object', properties: {}, required: [] }
         },
         {
@@ -480,6 +524,11 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
                 description: 'Maximum groups to return (default: 50, max: 500). total_groups reports the full count.',
                 default: 50
               },
+              offset: {
+                type: 'number',
+                description: 'Skip this many groups, for paging (default: 0). has_more says whether another page exists.',
+                default: 0
+              },
               refresh: {
                 type: 'boolean',
                 description: 'Pick up new and moved mail before searching (default: true). Skipped while a background sync is running.',
@@ -491,7 +540,7 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
         },
         {
           name: 'sync_index',
-          description: 'Run an index sync now and wait for it (up to wait_seconds). The index is read-only: syncing never changes anything on the mail server.',
+          description: 'Run an index sync now and wait for it (up to wait_seconds). The index is read-only: syncing never changes anything on the mail server. If a sync is already running the call waits on that one; pass force: true to abort it first and start fresh, which is the remedy for a sync that index_status shows as running far longer than sync_max_minutes.',
           inputSchema: {
             type: 'object',
             properties: {
@@ -504,6 +553,11 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
                 type: 'number',
                 description: 'How long to wait before returning while the sync continues in the background (default: 60)',
                 default: 60
+              },
+              force: {
+                type: 'boolean',
+                description: 'Abort a sync that is already running before starting this one (default: false). Safe: the index is read-only against the server. Use when index_status shows sync_running_seconds far beyond sync_max_minutes.',
+                default: false
               }
             },
             required: []
@@ -856,7 +910,7 @@ async function connectIMAP() {
   return await sessions.acquire();
 }
 
-const LIST_HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID)';
+const LIST_HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID IN-REPLY-TO REFERENCES)';
 const MAX_LIMIT = 200;
 
 function clampLimit(value, fallback = 20) {
@@ -885,6 +939,10 @@ async function summarize(connection, folder, uids, { withFlags = true } = {}) {
     if (h.cc?.length) item.cc = h.cc.join(', ');
     const messageId = normalizeMessageId(h['message-id']?.[0]);
     if (messageId) item.message_id = messageId;
+    const inReplyTo = parseMessageIdList(h['in-reply-to'])[0];
+    if (inReplyTo) item.in_reply_to = inReplyTo;
+    const references = parseMessageIdList(h.references);
+    if (references.length) item.references = references;
     if (withFlags) item.flags = rec.attributes.flags;
     out.push(item);
   }
@@ -1064,6 +1122,8 @@ async function handleTool(name, args) {
                 subject: parsed.subject,
                 date: parsed.date,
                 ...(normalizeMessageId(parsed.messageId) && { message_id: normalizeMessageId(parsed.messageId) }),
+                ...(parseMessageIdList(parsed.inReplyTo)[0] && { in_reply_to: parseMessageIdList(parsed.inReplyTo)[0] }),
+                ...(parseMessageIdList(parsed.references).length && { references: parseMessageIdList(parsed.references) }),
                 text: (htmlOnly || !parsed.text) && parsed.html ? htmlToText(parsed.html) : (parsed.text || ''),
                 ...(htmlOnly && { text_source: 'converted from HTML' }),
                 ...(args.include_html && parsed.html && { html: stripDataUris(parsed.html) }),
@@ -1982,7 +2042,39 @@ async function handleTool(name, args) {
           scope: args.scope || 'all',
           since_date: args.since_date || null,
           before_date: args.before_date || null,
-          limit: args.limit
+          limit: args.limit,
+          offset: args.offset
+        });
+        if (refresh) found.index = { ...found.index, refresh };
+        return { content: [{ type: 'text', text: JSON.stringify(found, null, 2) }] };
+      }
+
+      case 'find_split_threads': {
+        if (!localIndex) throw new Error(localIndexError || 'Local index is not enabled (set INDEX_ENABLED=true)');
+        let refresh = null;
+        if (args.refresh !== false) {
+          if (localIndex.running) {
+            refresh = 'skipped: background sync in progress';
+          } else {
+            const quick = localIndex.sync({ maxPerFolder: 300, passes: 1 });
+            const outcome = await Promise.race([
+              quick.then((r) => (r && r.skipped ? 'skipped' : 'done')),
+              new Promise((resolve) => setTimeout(() => resolve('timeout'), 20000))
+            ]);
+            refresh = {
+              done: 'done',
+              skipped: 'skipped: another server process is syncing the index',
+              timeout: 'partial: refresh still running in background'
+            }[outcome];
+          }
+        }
+        const found = localIndex.findThreads({
+          folders: args.folders || null,
+          scope: args.scope || 'split',
+          since_date: args.since_date || null,
+          before_date: args.before_date || null,
+          limit: args.limit,
+          offset: args.offset
         });
         if (refresh) found.index = { ...found.index, refresh };
         return { content: [{ type: 'text', text: JSON.stringify(found, null, 2) }] };
@@ -1991,7 +2083,16 @@ async function handleTool(name, args) {
       case 'sync_index': {
         if (!localIndex) throw new Error(localIndexError || 'Local index is not enabled (set INDEX_ENABLED=true)');
         const waitMs = Math.max(0, Number(args.wait_seconds ?? 60)) * 1000;
-        const alreadyRunning = !!localIndex.running;
+        let alreadyRunning = !!localIndex.running;
+        let forced = false;
+        if (alreadyRunning && args.force) {
+          const stuckFor = localIndex.runningSince ? Math.round((Date.now() - localIndex.runningSince) / 1000) : null;
+          forced = await Promise.race([
+            localIndex.abortSync(`forced by sync_index after ${stuckFor}s`),
+            new Promise((resolve) => setTimeout(() => resolve('timeout'), 15000))
+          ]);
+          alreadyRunning = !!localIndex.running;
+        }
         const run = localIndex.sync(args.folders?.length ? { folders: args.folders } : {});
         const outcome = await Promise.race([
           run.then((summary) => ({ finished: true, summary })),
@@ -2002,7 +2103,9 @@ async function handleTool(name, args) {
           content: [{
             type: 'text',
             text: JSON.stringify({
-              ...(alreadyRunning && { note: 'A sync was already running; waited on that one.' }),
+              ...(forced === true && { note: 'A running sync was aborted first (force: true) and a fresh one started.' }),
+              ...(forced === 'timeout' && { note: 'force: true was requested but the running sync did not release within 15s; waited on it instead.' }),
+              ...(alreadyRunning && !forced && { note: 'A sync was already running; waited on that one.' }),
               ...(skipped && { note: `Not started: ${outcome.summary.reason}. It keeps the index current; check index_status.` }),
               finished: skipped ? false : outcome.finished,
               ...(outcome.summary && !skipped && { summary: outcome.summary }),
@@ -2038,7 +2141,8 @@ async function main() {
         dbPath: process.env.INDEX_DB_PATH || undefined,
         folders: list(process.env.INDEX_FOLDERS),
         excludeFolders: list(process.env.INDEX_EXCLUDE_FOLDERS),
-        intervalMinutes: parseFloat(process.env.INDEX_SYNC_INTERVAL_MINUTES || '5')
+        intervalMinutes: parseFloat(process.env.INDEX_SYNC_INTERVAL_MINUTES || '5'),
+        maxRunMinutes: parseFloat(process.env.INDEX_SYNC_MAX_MINUTES || '15')
       }).open();
       localIndex.start();
       console.error(`Local index enabled: ${localIndex.dbPath}`);

package/lib/imap-session.js

@@ -89,6 +89,9 @@ export class SessionManager {
     this.lastUsedAt = 0;
     this.tail = Promise.resolve();
     this.idleTimer = null;
+    this.activeRelease = null;
+    this.generation = 0;
+    this.abortConnect = null;
   }
 
   _isUsable(conn) {
@@ -113,15 +116,34 @@ export class SessionManager {
   async _connect() {
     if (this._isUsable(this.conn)) return this.conn;
     if (!this.connecting) {
-      this.connecting = (async () => {
+      const generation = this.generation;
+      // dropCurrent() can reject this attempt at once, so a caller queued on a
+      // socket that will never answer does not wait out the connect timeout.
+      let abort;
+      const aborted = new Promise((_, reject) => { abort = reject; });
+      aborted.catch(() => {});
+      this.abortConnect = abort;
+      const dial = (async () => {
         const conn = await imaps.connect(this.config);
+        if (generation !== this.generation) {
+          // The caller that wanted this socket is gone; do not hand it to anyone.
+          try { conn.imap.destroy(); } catch { /* ignore */ }
+          throw new Error('connection attempt superseded');
+        }
         // Without an 'error' listener an idle socket error would crash the process.
         conn.on('error', (err) => this._drop(conn, `error: ${err && err.message}`));
         conn.imap.once('close', () => { if (this.conn === conn) this.conn = null; });
         conn.imap.once('end', () => { if (this.conn === conn) this.conn = null; });
         this.conn = conn;
         return conn;
-      })().finally(() => { this.connecting = null; });
+      })();
+      dial.catch(() => {});
+      const attempt = Promise.race([dial, aborted]);
+      attempt.then(() => {}, () => {}).then(() => {
+        if (this.connecting === attempt) this.connecting = null;
+        if (this.abortConnect === abort) this.abortConnect = null;
+      });
+      this.connecting = attempt;
     }
     return this.connecting;
   }
@@ -172,6 +194,7 @@ export class SessionManager {
       const release = () => {
         if (released) return;
         released = true;
+        if (this.activeRelease === release) this.activeRelease = null;
         this.lastUsedAt = Date.now();
         clearTimeout(watchdog);
         if (!this._isUsable(conn)) this._drop(conn, 'unusable after call');
@@ -185,6 +208,7 @@ export class SessionManager {
         release();
       }, this.leaseTimeoutMs);
       if (watchdog.unref) watchdog.unref();
+      this.activeRelease = release;
 
       return new Proxy(conn, {
         get(target, prop) {
@@ -196,12 +220,41 @@ export class SessionManager {
     });
   }
 
+  /**
+   * Drop the current connection (if any) so the next acquire() opens a fresh
+   * one. Used to cut a sync loose from a socket that has gone silent.
+   */
+  dropCurrent(reason) {
+    if (this.conn) this._drop(this.conn, reason);
+    // A connect attempt still pending belongs to the run being cut loose:
+    // fail it now and detach it, so the next acquire() starts its own.
+    this.generation++;
+    const abort = this.abortConnect;
+    this.abortConnect = null;
+    this.connecting = null;
+    if (abort) abort(new Error(`connection attempt aborted: ${reason}`));
+    // Release the lease the dropped connection was handed out under, so the
+    // next acquire() does not wait behind an owner that will never return.
+    if (this.activeRelease) this.activeRelease();
+  }
+
   close() {
     clearTimeout(this.idleTimer);
     if (this.conn) this._drop(this.conn, null);
   }
 }
 
+/** LIST all mailboxes, under the command time limit. */
+export function listBoxes(connection) {
+  return withTimeout(connection, 'list folders', new Promise((resolve, reject) => {
+    try {
+      connection.imap.getBoxes((err, boxes) => (err ? reject(err) : resolve(boxes)));
+    } catch (err) {
+      reject(err);
+    }
+  }));
+}
+
 /** Open a mailbox. readOnly uses EXAMINE, which never changes flags. */
 export function openBox(connection, name, readOnly = false) {
   return withTimeout(connection, `open ${name}`, new Promise((resolve, reject) => {

package/lib/local-index.js

@@ -19,7 +19,7 @@ import fs from 'fs';
 import os from 'os';
 import path from 'path';
 import crypto from 'crypto';
-import { SessionManager, openBox, fetchMessages, headerOf, localDate, uidSearchAll, getOpTimeout } from './imap-session.js';
+import { SessionManager, openBox, fetchMessages, headerOf, localDate, uidSearchAll, listBoxes, getOpTimeout } from './imap-session.js';
 import { planParts, partsToText } from './mime-text.js';
 
 const SCHEMA_VERSION = '1';
@@ -29,6 +29,7 @@ const REQUEST_RETRY_MS = 15 * 1000;
 const STALL_MS = 5 * 60 * 1000;
 const REQUEST_MAX_ATTEMPTS = 6;
 const STALE_AFTER_MS = 15 * 60 * 1000;
+const DEFAULT_MAX_RUN_MINUTES = 15;
 
 function describeAge(ms) {
   const minutes = Math.round(ms / 60000);
@@ -55,7 +56,8 @@ function pidAlive(pid) {
     return err.code === 'EPERM';
   }
 }
-const HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID)';
+const HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID IN-REPLY-TO REFERENCES)';
+const THREAD_BACKFILL_PER_FOLDER = 3000;
 const DEFAULT_EXCLUDE = ['trash', 'junk', 'spam', 'deleted items', 'deleted messages', 'bin'];
 const SPECIAL_EXCLUDE = ['\\trash', '\\junk'];
 
@@ -133,18 +135,40 @@ export function normalizeMessageId(value) {
   return id || null;
 }
 
+/**
+ * Every Message-ID in an In-Reply-To or References header, in order, without
+ * duplicates. A References header can be folded across lines and can carry
+ * several ids; a header with no angle brackets is treated as one bare id.
+ */
+export function parseMessageIdList(value) {
+  const text = Array.isArray(value) ? value.join(' ') : String(value ?? '');
+  const out = [];
+  const seen = new Set();
+  const push = (id) => { const n = normalizeMessageId(id); if (n && !seen.has(n)) { seen.add(n); out.push(n); } };
+  const bracketed = text.match(/<[^<>]+>/g);
+  if (bracketed) bracketed.forEach(push);
+  else text.split(/\s+/).forEach(push);
+  return out;
+}
+
 export class LocalIndex {
-  constructor({ imapConfig, dbPath, folders = null, excludeFolders = [], intervalMinutes = 5, batchSize = 100, maxBodyChars = 100000, log = console.error }) {
+  constructor({ imapConfig, dbPath, folders = null, excludeFolders = [], intervalMinutes = 5, maxRunMinutes = DEFAULT_MAX_RUN_MINUTES, batchSize = 100, maxBodyChars = 100000, log = console.error }) {
     this.imapConfig = imapConfig;
     this.dbPath = dbPath || defaultDbPath(imapConfig.imap.user, imapConfig.imap.host);
     this.folders = folders && folders.length ? folders : null;
     this.excludeFolders = excludeFolders.map((f) => f.toLowerCase());
     this.intervalMs = Math.max(1, intervalMinutes) * 60 * 1000;
+    // Wall-clock cap on one sync run. 0 disables it. Fractions are allowed for tests.
+    const cap = Number(maxRunMinutes);
+    this.maxRunMs = Number.isFinite(cap) && cap >= 0 ? cap * 60 * 1000 : DEFAULT_MAX_RUN_MINUTES * 60 * 1000;
     this.batchSize = batchSize;
     this.maxBodyChars = maxBodyChars;
     this.log = log;
     this.db = null;
     this.running = null;
+    this.runningSince = 0;
+    this.abortRun = null;
+    this.stopped = false;
     this.timer = null;
     this.progressAt = 0;
     this.debounce = null;
@@ -200,25 +224,44 @@ export class LocalIndex {
         tokenize = 'unicode61 remove_diacritics 2'
       );
     `);
+    // 1.9.0: threading headers. Rows indexed by an older version have
+    // thread_headers = 0 and are backfilled by the next syncs, headers only.
+    const columns = new Set(this.db.prepare('PRAGMA table_info(messages)').all().map((c) => c.name));
+    if (!columns.has('in_reply_to')) this.db.exec('ALTER TABLE messages ADD COLUMN in_reply_to TEXT');
+    if (!columns.has('references_ids')) this.db.exec('ALTER TABLE messages ADD COLUMN references_ids TEXT');
+    if (!columns.has('thread_headers')) this.db.exec('ALTER TABLE messages ADD COLUMN thread_headers INTEGER NOT NULL DEFAULT 0');
+    this.db.exec('CREATE INDEX IF NOT EXISTS messages_thread_headers ON messages (thread_headers)');
     const version = this.db.prepare('SELECT value FROM meta WHERE key = ?').get('schema_version');
     if (!version) this.db.prepare('INSERT INTO meta (key, value) VALUES (?, ?)').run('schema_version', SCHEMA_VERSION);
     return this;
   }
 
   start() {
-    const kick = () => {
-      const last = Number(this._meta('last_full_sync') || 0);
-      if (Date.now() - last < this.intervalMs / 2) return;
-      this.sync().catch((err) => this.log(`[index] sync failed: ${err.message}`));
+    // The next scheduled sync is armed when the previous one ends, so a slow
+    // or stuck run can never queue a pile-up behind itself.
+    const arm = (delay) => {
+      clearTimeout(this.timer);
+      if (this.stopped) return;
+      this.timer = setTimeout(kick, delay);
+      if (this.timer.unref) this.timer.unref();
+    };
+    const kick = async () => {
+      try {
+        const last = Number(this._meta('last_full_sync') || 0);
+        if (Date.now() - last >= this.intervalMs / 2) await this.sync();
+      } catch (err) {
+        this.log(`[index] sync failed: ${err.message}`);
+      } finally {
+        arm(this.intervalMs);
+      }
     };
-    const first = setTimeout(kick, 5000);
-    if (first.unref) first.unref();
-    this.timer = setInterval(kick, this.intervalMs);
-    if (this.timer.unref) this.timer.unref();
+    this.stopped = false;
+    arm(5000);
   }
 
   stop() {
-    clearInterval(this.timer);
+    this.stopped = true;
+    clearTimeout(this.timer);
     clearTimeout(this.debounce);
     this.releaseLock();
     this.session.close();
@@ -332,7 +375,7 @@ export class LocalIndex {
   }
 
   async _targetFolders(connection) {
-    const boxes = flattenBoxes(await connection.getBoxes());
+    const boxes = flattenBoxes(await listBoxes(connection));
     const selectable = boxes.filter((b) => !b.attribs.includes('\\noselect') && !b.attribs.includes('\\nonexistent'));
     if (this.folders) {
       const wanted = new Set(this.folders.map((f) => f.toLowerCase()));
@@ -353,6 +396,7 @@ export class LocalIndex {
    */
   sync({ folders = null, maxPerFolder = 1000, passes = Infinity } = {}) {
     if (this.running) return this.running;
+    this.runningSince = Date.now();
     this.running = (async () => {
       if (!this._acquireLock()) {
         return { skipped: true, reason: `another server process (pid ${this.lockHolder}) is syncing the index` };
@@ -373,7 +417,7 @@ export class LocalIndex {
       }, LOCK_HEARTBEAT_MS);
       if (heartbeat.unref) heartbeat.unref();
       try {
-        const summary = await this._sync({ folders, maxPerFolder, passes });
+        const summary = await this._runCapped(() => this._sync({ folders, maxPerFolder, passes }));
         if (this.db) {
           if (!folders && passes === Infinity && !summary.aborted) this._setMeta('last_full_sync', Date.now());
           this._setMeta('last_run', JSON.stringify({
@@ -391,15 +435,58 @@ export class LocalIndex {
         clearInterval(heartbeat);
         this.releaseLock();
       }
-    })().finally(() => { this.running = null; });
+    })().finally(() => { this.running = null; this.runningSince = 0; this.abortRun = null; });
     return this.running;
   }
 
+  /**
+   * Run one sync body under a wall-clock cap and an abort handle. A run that
+   * outlives the cap, or is aborted through abortSync(), is cut loose: its
+   * connection is dropped so any command it is waiting on fails, the caller
+   * releases the lock, and the orphaned body cannot touch the index again
+   * because _assertLock() throws once the lock is gone.
+   */
+  _runCapped(body) {
+    const started = Date.now();
+    let capTimer = null;
+    const cut = (reason) => ({ added: 0, removed: 0, folders: {}, aborted: reason, duration_ms: Date.now() - started, cut: true });
+    const stop = new Promise((resolve) => {
+      this.abortRun = (reason) => resolve(cut(reason));
+      if (this.maxRunMs > 0) {
+        const minutes = this.maxRunMs / 60000;
+        capTimer = setTimeout(() => this.abortRun(`sync exceeded ${minutes} minute${minutes === 1 ? '' : 's'}, aborted`), this.maxRunMs);
+        if (capTimer.unref) capTimer.unref();
+      }
+    });
+    const guarded = body().catch((err) => ({ error: err }));
+    return Promise.race([guarded, stop]).then((summary) => {
+      clearTimeout(capTimer);
+      if (summary.error) throw summary.error;
+      if (summary.cut) {
+        delete summary.cut;
+        this.session.dropCurrent(summary.aborted);
+        this.log(`[index] sync stopped: ${summary.aborted}`);
+      }
+      return summary;
+    });
+  }
+
+  /**
+   * Abort a running sync so a fresh one can start. Resolves once the run has
+   * been released. Returns false when nothing was running.
+   */
+  async abortSync(reason = 'aborted') {
+    if (!this.running) return false;
+    if (this.abortRun) this.abortRun(reason);
+    try { await this.running; } catch { /* the aborted run's own error is not interesting */ }
+    return true;
+  }
+
   async _sync({ folders, maxPerFolder, passes }) {
     const started = Date.now();
     const summary = { added: 0, removed: 0, folders: {} };
     let targets;
-    {
+    try {
       const conn = await this.session.acquire();
       try {
         const all = await this._targetFolders(conn);
@@ -408,6 +495,14 @@ export class LocalIndex {
       } finally {
         conn.end();
       }
+    } catch (err) {
+      if (err.code !== 'IMAP_TIMEOUT') throw err;
+      // The folder LIST on a dead connection: the run ends here, recorded like
+      // any other timed-out command, and the next scheduled sync reconnects.
+      summary.aborted = err.message;
+      summary.duration_ms = Date.now() - started;
+      this.log(`[index] sync stopped: ${err.message}`);
+      return summary;
     }
 
     let pass = 0;
@@ -522,6 +617,8 @@ export class LocalIndex {
         added += await this._indexBatch(conn, folder, uidvalidity, todo.slice(i, i + this.batchSize));
       }
 
+      await this._backfillThreadHeaders(conn, folder);
+
       const pending = missing.length - todo.length;
       const indexed = this.db.prepare('SELECT count(*) AS n FROM messages WHERE folder = ?').get(folder).n;
       this.db.prepare(`INSERT INTO folders (name, uidvalidity, server_count, indexed_count, backfill_complete, last_sync, last_error)
@@ -536,8 +633,47 @@ export class LocalIndex {
     }
   }
 
+  /**
+   * Fetch In-Reply-To and References for rows indexed before 1.9.0, headers
+   * only, in batches, up to THREAD_BACKFILL_PER_FOLDER rows per run so a large
+   * Archive is spread over a few syncs rather than one long one.
+   */
+  async _backfillThreadHeaders(conn, folder) {
+    const rows = this.db.prepare('SELECT uid FROM messages WHERE folder = ? AND thread_headers = 0 ORDER BY uid DESC LIMIT ?')
+      .all(folder, THREAD_BACKFILL_PER_FOLDER);
+    if (!rows.length) return 0;
+    const parseHeader = conn.imap.constructor.parseHeader;
+    const upd = this.db.prepare('UPDATE messages SET in_reply_to = ?, references_ids = ?, thread_headers = 1 WHERE folder = ? AND uid = ?');
+    let done = 0;
+    for (let i = 0; i < rows.length; i += this.batchSize) {
+      this._assertLock();
+      this.progressAt = Date.now();
+      const uids = rows.slice(i, i + this.batchSize).map((r) => Number(r.uid));
+      const fetched = await fetchMessages(conn, uids, { bodies: ['HEADER.FIELDS (IN-REPLY-TO REFERENCES)'] });
+      this._assertLock();
+      this.db.exec('BEGIN');
+      try {
+        for (const uid of uids) {
+          const rec = fetched.get(uid);
+          // A uid the server no longer has is marked done; the next set
+          // difference removes the row.
+          const h = rec ? headerOf(rec, parseHeader) : {};
+          upd.run(parseMessageIdList(h['in-reply-to'])[0] || null, parseMessageIdList(h.references).join(' ') || null, folder, uid);
+          done++;
+        }
+        this.db.exec('COMMIT');
+      } catch (err) {
+        this.db.exec('ROLLBACK');
+        throw err;
+      }
+    }
+    if (done) this.log(`[index] ${folder}: threading headers backfilled for ${done} message(s)`);
+    return done;
+  }
+
   async _indexBatch(conn, folder, uidvalidity, uids) {
     const meta = await fetchMessages(conn, uids, { bodies: [HEADER_FIELDS], struct: true, size: true });
+    this._assertLock(); // a run cut loose by the cap must not write after the fact
     const parseHeader = conn.imap.constructor.parseHeader;
 
     // Group messages by the exact set of text parts they need, so each group
@@ -557,10 +693,12 @@ export class LocalIndex {
       const fetched = await fetchMessages(conn, groupUids, { bodies: key.split(',') });
       for (const [uid, rec] of fetched) bodies.set(uid, rec.parts);
     }
+    this._assertLock();
 
     const insMsg = this.db.prepare(`INSERT OR IGNORE INTO messages
-      (folder, uid, uidvalidity, internal_date, sent_date, from_addr, to_addr, cc_addr, subject, message_id, size, attachments, body_chars)
-      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`);
+      (folder, uid, uidvalidity, internal_date, sent_date, from_addr, to_addr, cc_addr, subject, message_id, size, attachments, body_chars,
+       in_reply_to, references_ids, thread_headers)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)`);
     const insFts = this.db.prepare('INSERT INTO fts (rowid, subject, from_addr, to_addr, attachments, body) VALUES (?, ?, ?, ?, ?, ?)');
 
     let added = 0;
@@ -582,7 +720,8 @@ export class LocalIndex {
         const attachments = plan.attachments.join(', ');
         const internal = rec.attributes.date instanceof Date ? rec.attributes.date.getTime() : null;
         const res = insMsg.run(folder, uid, uidvalidity, internal, (h.date || [null])[0], from, addressText(h.to), addressText(h.cc),
-          subject, (h['message-id'] || [null])[0], rec.attributes.size ?? null, attachments, body.length);
+          subject, (h['message-id'] || [null])[0], rec.attributes.size ?? null, attachments, body.length,
+          parseMessageIdList(h['in-reply-to'])[0] || null, parseMessageIdList(h.references).join(' ') || null);
         if (res.changes) {
           insFts.run(Number(res.lastInsertRowid), subject, from, to, attachments, body);
           added++;
@@ -619,10 +758,16 @@ export class LocalIndex {
       db_size_mb: Math.round(sizeBytes / 1024 / 102.4) / 10,
       this_pid: process.pid,
       sync_running: !!this.running || lockLive,
+      ...(this.running && this.runningSince && {
+        sync_running_since: new Date(this.runningSince).toISOString(),
+        sync_running_seconds: Math.round((Date.now() - this.runningSince) / 1000)
+      }),
       ...(lockLive && lock.pid !== process.pid && { sync_running_in_pid: lock.pid }),
       sync_interval_minutes: this.intervalMs / 60000,
+      sync_max_minutes: this.maxRunMs / 60000,
       command_timeout_seconds: getOpTimeout() / 1000,
       complete: folders.length > 0 && folders.every((f) => f.complete && !f.last_error),
+      thread_headers_pending: this.db.prepare('SELECT count(*) AS n FROM messages WHERE thread_headers = 0').get().n,
       last_full_sync: lastFull ? new Date(lastFull).toISOString() : null,
       last_run: lastRun,
       folders
@@ -673,7 +818,7 @@ export class LocalIndex {
    * Messages with no Message-ID header cannot be compared and are reported
    * under coverage rather than silently ignored.
    */
-  findDuplicates({ folders = null, scope = 'all', since_date = null, before_date = null, limit = 50 } = {}) {
+  findDuplicates({ folders = null, scope = 'all', since_date = null, before_date = null, limit = 50, offset = 0 } = {}) {
     if (!['all', 'same_folder', 'cross_folder'].includes(scope)) {
       throw new Error(`Unknown scope: ${scope}. Use all, same_folder or cross_folder.`);
     }
@@ -755,6 +900,8 @@ export class LocalIndex {
     }
     out.sort((a, b) => (b.copies - a.copies) || String(b.date || '').localeCompare(String(a.date || '')));
     const cap = Math.min(Math.max(1, Number(limit) || 50), 500);
+    const skip = Math.max(0, Number(offset) || 0);
+    const page = out.slice(skip, skip + cap);
     return {
       scope,
       folders_searched: folderList.length ? folderList : 'all indexed folders',
@@ -765,8 +912,133 @@ export class LocalIndex {
         ...(inScope > withId && { note: 'messages without a Message-ID header cannot be compared and are not included' })
       },
       total_groups: out.length,
-      returned: Math.min(out.length, cap),
-      groups: out.slice(0, cap),
+      returned: page.length,
+      offset: skip,
+      has_more: skip + page.length < out.length,
+      groups: page,
+      index: this._indexHealth(folderList)
+    };
+  }
+
+  /**
+   * Group messages into conversations by their Message-ID chains.
+   *
+   * Two messages are in the same thread when their Message-ID, In-Reply-To
+   * and References ids overlap, transitively. That is how mail clients thread,
+   * and it is exact: a renamed subject stays in its thread, two conversations
+   * that share a subject do not merge. A message with no chain at all (no
+   * In-Reply-To, no References, and nothing pointing at it) is a thread of one.
+   *
+   * scope:
+   *   split   threads whose messages sit in more than one of the searched
+   *           folders (default). A thread whose spread is only the same
+   *           message copied into two folders is flagged copies_only, since
+   *           that is a duplicate, not a split.
+   *   all     every thread in scope.
+   */
+  findThreads({ folders = null, scope = 'split', since_date = null, before_date = null, limit = 50, offset = 0 } = {}) {
+    if (!['split', 'all'].includes(scope)) throw new Error(`Unknown scope: ${scope}. Use split or all.`);
+    const folderList = Array.isArray(folders) ? folders.filter(Boolean) : (folders ? [folders] : []);
+    const where = [];
+    const params = [];
+    if (folderList.length) {
+      where.push(`folder IN (${folderList.map(() => '?').join(', ')})`);
+      params.push(...folderList);
+    }
+    if (since_date) { where.push('internal_date >= ?'); params.push(localDate(since_date).getTime()); }
+    if (before_date) { where.push('internal_date < ?'); params.push(localDate(before_date).getTime()); }
+    const scopeSql = where.length ? `WHERE ${where.join(' AND ')}` : '';
+    const rows = this.db.prepare(`
+      SELECT id, folder, uid, message_id, in_reply_to, references_ids, subject, from_addr, internal_date, sent_date, thread_headers
+      FROM messages ${scopeSql}`).all(...params);
+
+    // Union-find over ids. Each message joins its own Message-ID with every id
+    // it references; a message without a Message-ID gets a private key.
+    const parent = new Map();
+    const find = (k) => {
+      let r = k;
+      while (parent.get(r) !== r) r = parent.get(r);
+      let c = k;
+      while (parent.get(c) !== r) { const n = parent.get(c); parent.set(c, r); c = n; }
+      return r;
+    };
+    const add = (k) => { if (!parent.has(k)) parent.set(k, k); };
+    const union = (a, b) => { add(a); add(b); const ra = find(a); const rb = find(b); if (ra !== rb) parent.set(ra, rb); };
+    const keyOf = new Map();
+    let withHeaders = 0;
+    for (const r of rows) {
+      const own = normalizeMessageId(r.message_id) || `row:${r.id}`;
+      keyOf.set(r.id, own);
+      add(own);
+      if (r.thread_headers) withHeaders++;
+      if (r.in_reply_to) union(own, r.in_reply_to);
+      if (r.references_ids) for (const ref of r.references_ids.split(' ')) if (ref) union(own, ref);
+    }
+
+    const threads = new Map();
+    for (const r of rows) {
+      const root = find(keyOf.get(r.id));
+      if (!threads.has(root)) threads.set(root, []);
+      threads.get(root).push(r);
+    }
+
+    const dateOf = (r) => Number(r.internal_date) || Date.parse(r.sent_date || '') || 0;
+    const out = [];
+    for (const [root, msgs] of threads) {
+      const byFolder = {};
+      for (const m of msgs) byFolder[m.folder] = (byFolder[m.folder] || 0) + 1;
+      const folderCount = Object.keys(byFolder).length;
+      if (scope === 'split' && folderCount < 2) continue;
+      const sorted = [...msgs].sort((a, b) => dateOf(a) - dateOf(b));
+      const distinctIds = new Set(sorted.map((m) => normalizeMessageId(m.message_id) || `row:${m.id}`));
+      // Spread that exists only because the same messages sit in two folders.
+      let copiesOnly = false;
+      if (folderCount > 1) {
+        const perId = new Map();
+        for (const m of sorted) {
+          const id = normalizeMessageId(m.message_id) || `row:${m.id}`;
+          if (!perId.has(id)) perId.set(id, new Set());
+          perId.get(id).add(m.folder);
+        }
+        copiesOnly = [...perId.values()].every((set) => set.size === folderCount);
+      }
+      const newest = sorted[sorted.length - 1];
+      const oldest = sorted[0];
+      const participants = [...new Set(sorted.map((m) => m.from_addr).filter(Boolean))].slice(0, 12);
+      out.push({
+        thread_key: root.startsWith('row:') ? null : root,
+        subject: oldest.subject || '',
+        latest_subject: newest.subject !== oldest.subject ? newest.subject : undefined,
+        messages: msgs.length,
+        distinct_messages: distinctIds.size,
+        by_folder: byFolder,
+        ...(folderCount > 1 && { copies_only: copiesOnly }),
+        oldest: { folder: oldest.folder, uid: Number(oldest.uid), date: oldest.sent_date || new Date(dateOf(oldest)).toISOString(), from: oldest.from_addr },
+        newest: { folder: newest.folder, uid: Number(newest.uid), date: newest.sent_date || new Date(dateOf(newest)).toISOString(), from: newest.from_addr },
+        participants,
+        threading: sorted.every((m) => m.thread_headers) ? 'headers' : 'partial: some messages still lack threading headers',
+        message_list: sorted.slice(-50).map((m) => ({ folder: m.folder, uid: Number(m.uid), date: m.sent_date || new Date(dateOf(m)).toISOString(), from: m.from_addr }))
+      });
+    }
+    out.sort((a, b) => String(b.newest.date).localeCompare(String(a.newest.date)));
+    const cap = Math.min(Math.max(1, Number(limit) || 50), 500);
+    const skip = Math.max(0, Number(offset) || 0);
+    const page = out.slice(skip, skip + cap);
+    return {
+      scope,
+      folders_searched: folderList.length ? folderList : 'all indexed folders',
+      coverage: {
+        messages_in_scope: rows.length,
+        with_thread_headers: withHeaders,
+        without_thread_headers: rows.length - withHeaders,
+        ...(rows.length > withHeaders && { note: 'messages without threading headers are still being backfilled; they thread only by their own Message-ID until then' }),
+        threads_in_scope: threads.size
+      },
+      total_threads: out.length,
+      returned: page.length,
+      offset: skip,
+      has_more: skip + page.length < out.length,
+      threads: page,
       index: this._indexHealth(folderList)
     };
   }
@@ -799,6 +1071,7 @@ export class LocalIndex {
     const total = this.db.prepare(`SELECT count(*) AS n FROM fts JOIN messages m ON m.id = fts.rowid WHERE ${whereSql}`).get(...params).n;
     const rows = this.db.prepare(`
       SELECT m.folder, m.uid, m.internal_date, m.sent_date, m.from_addr, m.to_addr, m.subject, m.attachments, m.message_id,
+             m.in_reply_to, m.references_ids,
              snippet(fts, -1, '[', ']', '...', 16) AS snippet
       FROM fts JOIN messages m ON m.id = fts.rowid
       WHERE ${whereSql}
@@ -818,6 +1091,8 @@ export class LocalIndex {
         to: r.to_addr,
         subject: r.subject,
         message_id: normalizeMessageId(r.message_id) || undefined,
+        in_reply_to: r.in_reply_to || undefined,
+        references: r.references_ids ? r.references_ids.split(' ') : undefined,
         attachments: r.attachments || undefined,
         snippet: r.snippet
       }))

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lengelhard/imap-email-mcp",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "description": "MCP server for Claude Code that provides email capabilities through IMAP/SMTP. Read, search, compose, and manage emails from any IMAP provider.",
   "type": "module",
   "main": "index.js",