@lengelhard/imap-email-mcp 1.8.1 → 1.9.1

package/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## 1.9.1
+
+### Fixed
+- The threading-header backfill no longer runs inside each folder's sync. On a server that reads In-Reply-To and References from disk at a few messages per second (seen on Migadu: about 3 per second on a large Archive), the 3,000-row slice took longer than the run cap, so every scheduled run was cut off in Archive and the folders after it were never synced. The backfill now runs once per scheduled run, after every folder has had its normal sync and `last_full_sync` has been stamped, under its own time budget (`INDEX_BACKFILL_SECONDS`, default 90, `0` disables). Folders with the fewest pending rows go first, so the working folders reach full coverage long before a large Archive. `sync_index` and refresh-triggered syncs never backfill, so they stay quick.
+- `last_run` records `thread_headers_backfilled` for each scheduled run.
+
+## 1.9.0
+
+### Added
+- `find_split_threads`: lists conversations whose messages sit in more than one folder, threaded on the In-Reply-To and References headers rather than on subject. A reply that was renamed ("Onboarding - Accrete" becoming "Onboarding - Accrete: data room request list") stays in its thread because its References still chain back to the root, and two unrelated conversations that happen to share a subject ("Catch-up call") never merge. `scope: "split"` (the default) reports only threads spread across the searched folders; `scope: "all"` lists every thread in scope. Each thread carries `messages`, `distinct_messages`, `by_folder`, `oldest`, `newest`, `participants` and its `message_list`; a thread whose only spread is the same Message-ID copied into two folders is marked `copies_only`, which is a duplicate for `find_duplicates`, not a split. Paged with `limit` and `offset`. Requires the local index.
+- `in_reply_to` and `references` on the results of `list_emails`, `search_emails`, `search_index` and `get_email`, angle brackets stripped, folded headers unfolded, oldest first.
+- The index now stores In-Reply-To and References. Rows indexed by an earlier version are backfilled by the background sync, up to 3,000 rows per folder per run, without a rebuild. `index_status` reports the number still waiting as `thread_headers_pending`, and `find_split_threads` reports it under `coverage.without_thread_headers` and marks any thread that touches such rows `threading: "partial"`, so a report on a half-backfilled index says so instead of looking complete.
+
+### Changed
+- Opening a database written by 1.8.x adds the three threading columns in place. Searches keep working while the backfill runs.
+
 ## 1.8.1
 
 ### Fixed

package/README.md

@@ -107,6 +107,7 @@ Add to your config file:
 | `INDEX_EXCLUDE_FOLDERS` | none | Comma-separated folders to skip, on top of Trash/Junk |
 | `INDEX_SYNC_INTERVAL_MINUTES` | `5` | Background sync interval, measured from the end of the previous run |
 | `INDEX_SYNC_MAX_MINUTES` | `15` | Wall-clock cap on one sync run; a run that outlives it is aborted and its connection dropped (`0` disables) |
+| `INDEX_BACKFILL_SECONDS` | `90` | Time budget per scheduled run for backfilling threading headers on rows indexed before 1.9.0 (`0` disables) |
 
 ### Local search index
 
@@ -129,6 +130,16 @@ Mail clients that file by tag sometimes append a fresh copy of a message instead
 - **Honest coverage.** A message with no Message-ID header cannot be compared. It is counted in `coverage.without_message_id` rather than quietly skipped, and `index.complete` says whether the index had seen everything.
 - **Nothing is deleted.** The tool only reports. Acting on it means calling `delete_email`, which moves to Trash unless you pass `expunge: true`.
 
+### Finding split threads
+
+A conversation can end up spread across folders without any message being duplicated: the root is filed in one place, a later reply in another. `find_split_threads` finds those by following the In-Reply-To and References headers, which every mail client writes when replying.
+
+- **Why headers, not subject.** References lists the Message-IDs of every earlier message in the conversation, so a reply keeps its thread even after someone renames the subject, and two unrelated conversations with the same subject line stay apart. Subject matching gets both of those wrong.
+- **Scope.** `scope: "split"` (default) reports only threads present in more than one of the searched folders; `scope: "all"` lists every thread. The folders you pass decide what counts as split: a thread with three messages in Delegated and two in Hold is split when both folders are searched and not when only Delegated is.
+- **Copies are not splits.** A thread whose only spread is the same Message-ID present in two folders is marked `copies_only`. That is a duplicate, and `find_duplicates` is the tool for it.
+- **Honest coverage.** Rows indexed before 1.9.0 have no threading headers until the background sync backfills them, which it does at the end of each scheduled run for up to `INDEX_BACKFILL_SECONDS`, smallest folders first, without ever delaying the indexing of new mail. `coverage.without_thread_headers` and `index_status.thread_headers_pending` show how many are still waiting, and a thread touching such rows is marked `threading: "partial"`.
+- **Nothing is moved.** The tool only reports. `move_email` or `move_thread` are the tools for acting on it.
+
 ### Provider Settings
 
 | Provider | IMAP_HOST | SMTP_HOST | Notes |
@@ -151,6 +162,7 @@ Mail clients that file by tag sometimes append a fresh copy of a message instead
 | `index_status` | Per-folder index coverage, last sync and errors; `sync_running_seconds` shows how long the current sync has run |
 | `sync_index` | Run an index sync now; `force: true` aborts a stuck sync first |
 | `find_duplicates` | Messages that exist more than once, matched on Message-ID: filed twice in one folder, or spread across folders |
+| `find_split_threads` | Conversations spread across folders, threaded on In-Reply-To and References so renamed replies stay with their thread |
 | `list_drafts` | List all draft emails |
 | `get_draft` | Get a specific draft by UID |
 | `create_draft` | Create a new email draft |

package/index.js

@@ -31,7 +31,7 @@ import {
   buildCriteria
 } from './lib/imap-session.js';
 import { htmlToText, stripDataUris, leafParts } from './lib/mime-text.js';
-import { LocalIndex, normalizeMessageId } from './lib/local-index.js';
+import { LocalIndex, normalizeMessageId, parseMessageIdList } from './lib/local-index.js';
 
 const PKG_VERSION = createRequire(import.meta.url)('./package.json').version;
 
@@ -445,9 +445,53 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
             required: ['query']
           }
         },
+        {
+          name: 'find_split_threads',
+          description: 'List conversations whose messages sit in more than one folder, threaded by Message-ID chains (In-Reply-To and References), the same way a mail client threads. Exact where the headers exist: a renamed subject stays in its thread and two conversations that share a subject never merge. A thread flagged copies_only is the same message copied into several folders, a duplicate rather than a split; see find_duplicates for those. Check coverage.without_thread_headers: messages indexed before 1.9.0 thread only by their own Message-ID until the backfill reaches them. Requires the local index.',
+          inputSchema: {
+            type: 'object',
+            properties: {
+              folders: {
+                type: 'array',
+                items: { type: 'string' },
+                description: 'Only these folders (default: all indexed folders). A thread counts as split only across folders in this list.'
+              },
+              scope: {
+                type: 'string',
+                enum: ['split', 'all'],
+                description: 'split: threads spanning more than one folder (default); all: every thread',
+                default: 'split'
+              },
+              since_date: {
+                type: 'string',
+                description: 'Only messages delivered on or after this date (YYYY-MM-DD)'
+              },
+              before_date: {
+                type: 'string',
+                description: 'Only messages delivered before this date (YYYY-MM-DD)'
+              },
+              limit: {
+                type: 'number',
+                description: 'Maximum threads to return (default: 50, max: 500). total_threads reports the full count.',
+                default: 50
+              },
+              offset: {
+                type: 'number',
+                description: 'Skip this many threads, for paging (default: 0). has_more says whether another page exists.',
+                default: 0
+              },
+              refresh: {
+                type: 'boolean',
+                description: 'Pick up new and moved mail before searching (default: true). Skipped while a background sync is running.',
+                default: true
+              }
+            },
+            required: []
+          }
+        },
         {
           name: 'index_status',
-          description: 'Show the local index state: per-folder message counts on the server and in the index, whether backfill is complete, last sync time and errors. sync_running_seconds says how long the current sync has been going; a value far beyond sync_max_minutes means it is stuck and sync_index with force: true will cut it loose.',
+          description: 'Show the local index state: per-folder message counts on the server and in the index, whether backfill is complete, last sync time and errors. thread_headers_pending counts messages still waiting for their threading headers (1.9.0 backfill). sync_running_seconds says how long the current sync has been going; a value far beyond sync_max_minutes means it is stuck and sync_index with force: true will cut it loose.',
           inputSchema: { type: 'object', properties: {}, required: [] }
         },
         {
@@ -866,7 +910,7 @@ async function connectIMAP() {
   return await sessions.acquire();
 }
 
-const LIST_HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID)';
+const LIST_HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID IN-REPLY-TO REFERENCES)';
 const MAX_LIMIT = 200;
 
 function clampLimit(value, fallback = 20) {
@@ -895,6 +939,10 @@ async function summarize(connection, folder, uids, { withFlags = true } = {}) {
     if (h.cc?.length) item.cc = h.cc.join(', ');
     const messageId = normalizeMessageId(h['message-id']?.[0]);
     if (messageId) item.message_id = messageId;
+    const inReplyTo = parseMessageIdList(h['in-reply-to'])[0];
+    if (inReplyTo) item.in_reply_to = inReplyTo;
+    const references = parseMessageIdList(h.references);
+    if (references.length) item.references = references;
     if (withFlags) item.flags = rec.attributes.flags;
     out.push(item);
   }
@@ -1074,6 +1122,8 @@ async function handleTool(name, args) {
                 subject: parsed.subject,
                 date: parsed.date,
                 ...(normalizeMessageId(parsed.messageId) && { message_id: normalizeMessageId(parsed.messageId) }),
+                ...(parseMessageIdList(parsed.inReplyTo)[0] && { in_reply_to: parseMessageIdList(parsed.inReplyTo)[0] }),
+                ...(parseMessageIdList(parsed.references).length && { references: parseMessageIdList(parsed.references) }),
                 text: (htmlOnly || !parsed.text) && parsed.html ? htmlToText(parsed.html) : (parsed.text || ''),
                 ...(htmlOnly && { text_source: 'converted from HTML' }),
                 ...(args.include_html && parsed.html && { html: stripDataUris(parsed.html) }),
@@ -1999,6 +2049,37 @@ async function handleTool(name, args) {
         return { content: [{ type: 'text', text: JSON.stringify(found, null, 2) }] };
       }
 
+      case 'find_split_threads': {
+        if (!localIndex) throw new Error(localIndexError || 'Local index is not enabled (set INDEX_ENABLED=true)');
+        let refresh = null;
+        if (args.refresh !== false) {
+          if (localIndex.running) {
+            refresh = 'skipped: background sync in progress';
+          } else {
+            const quick = localIndex.sync({ maxPerFolder: 300, passes: 1 });
+            const outcome = await Promise.race([
+              quick.then((r) => (r && r.skipped ? 'skipped' : 'done')),
+              new Promise((resolve) => setTimeout(() => resolve('timeout'), 20000))
+            ]);
+            refresh = {
+              done: 'done',
+              skipped: 'skipped: another server process is syncing the index',
+              timeout: 'partial: refresh still running in background'
+            }[outcome];
+          }
+        }
+        const found = localIndex.findThreads({
+          folders: args.folders || null,
+          scope: args.scope || 'split',
+          since_date: args.since_date || null,
+          before_date: args.before_date || null,
+          limit: args.limit,
+          offset: args.offset
+        });
+        if (refresh) found.index = { ...found.index, refresh };
+        return { content: [{ type: 'text', text: JSON.stringify(found, null, 2) }] };
+      }
+
       case 'sync_index': {
         if (!localIndex) throw new Error(localIndexError || 'Local index is not enabled (set INDEX_ENABLED=true)');
         const waitMs = Math.max(0, Number(args.wait_seconds ?? 60)) * 1000;
@@ -2061,7 +2142,8 @@ async function main() {
         folders: list(process.env.INDEX_FOLDERS),
         excludeFolders: list(process.env.INDEX_EXCLUDE_FOLDERS),
         intervalMinutes: parseFloat(process.env.INDEX_SYNC_INTERVAL_MINUTES || '5'),
-        maxRunMinutes: parseFloat(process.env.INDEX_SYNC_MAX_MINUTES || '15')
+        maxRunMinutes: parseFloat(process.env.INDEX_SYNC_MAX_MINUTES || '15'),
+        backfillSeconds: parseFloat(process.env.INDEX_BACKFILL_SECONDS || '90')
       }).open();
       localIndex.start();
       console.error(`Local index enabled: ${localIndex.dbPath}`);

package/lib/local-index.js

@@ -56,7 +56,10 @@ function pidAlive(pid) {
     return err.code === 'EPERM';
   }
 }
-const HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID)';
+const HEADER_FIELDS = 'HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID IN-REPLY-TO REFERENCES)';
+const DEFAULT_BACKFILL_SECONDS = 90;
+// Small batches for the backfill: on a slow server a 100-row header fetch has taken longer than the command timeout.
+const BACKFILL_BATCH = 25;
 const DEFAULT_EXCLUDE = ['trash', 'junk', 'spam', 'deleted items', 'deleted messages', 'bin'];
 const SPECIAL_EXCLUDE = ['\\trash', '\\junk'];
 
@@ -134,8 +137,24 @@ export function normalizeMessageId(value) {
   return id || null;
 }
 
+/**
+ * Every Message-ID in an In-Reply-To or References header, in order, without
+ * duplicates. A References header can be folded across lines and can carry
+ * several ids; a header with no angle brackets is treated as one bare id.
+ */
+export function parseMessageIdList(value) {
+  const text = Array.isArray(value) ? value.join(' ') : String(value ?? '');
+  const out = [];
+  const seen = new Set();
+  const push = (id) => { const n = normalizeMessageId(id); if (n && !seen.has(n)) { seen.add(n); out.push(n); } };
+  const bracketed = text.match(/<[^<>]+>/g);
+  if (bracketed) bracketed.forEach(push);
+  else text.split(/\s+/).forEach(push);
+  return out;
+}
+
 export class LocalIndex {
-  constructor({ imapConfig, dbPath, folders = null, excludeFolders = [], intervalMinutes = 5, maxRunMinutes = DEFAULT_MAX_RUN_MINUTES, batchSize = 100, maxBodyChars = 100000, log = console.error }) {
+  constructor({ imapConfig, dbPath, folders = null, excludeFolders = [], intervalMinutes = 5, maxRunMinutes = DEFAULT_MAX_RUN_MINUTES, backfillSeconds = DEFAULT_BACKFILL_SECONDS, batchSize = 100, maxBodyChars = 100000, log = console.error }) {
     this.imapConfig = imapConfig;
     this.dbPath = dbPath || defaultDbPath(imapConfig.imap.user, imapConfig.imap.host);
     this.folders = folders && folders.length ? folders : null;
@@ -144,6 +163,9 @@ export class LocalIndex {
     // Wall-clock cap on one sync run. 0 disables it. Fractions are allowed for tests.
     const cap = Number(maxRunMinutes);
     this.maxRunMs = Number.isFinite(cap) && cap >= 0 ? cap * 60 * 1000 : DEFAULT_MAX_RUN_MINUTES * 60 * 1000;
+    // Time budget for the threading-header backfill at the end of each scheduled run. 0 disables it.
+    const bf = Number(backfillSeconds);
+    this.backfillMs = Number.isFinite(bf) && bf >= 0 ? bf * 1000 : DEFAULT_BACKFILL_SECONDS * 1000;
     this.batchSize = batchSize;
     this.maxBodyChars = maxBodyChars;
     this.log = log;
@@ -207,6 +229,13 @@ export class LocalIndex {
         tokenize = 'unicode61 remove_diacritics 2'
       );
     `);
+    // 1.9.0: threading headers. Rows indexed by an older version have
+    // thread_headers = 0 and are backfilled by the next syncs, headers only.
+    const columns = new Set(this.db.prepare('PRAGMA table_info(messages)').all().map((c) => c.name));
+    if (!columns.has('in_reply_to')) this.db.exec('ALTER TABLE messages ADD COLUMN in_reply_to TEXT');
+    if (!columns.has('references_ids')) this.db.exec('ALTER TABLE messages ADD COLUMN references_ids TEXT');
+    if (!columns.has('thread_headers')) this.db.exec('ALTER TABLE messages ADD COLUMN thread_headers INTEGER NOT NULL DEFAULT 0');
+    this.db.exec('CREATE INDEX IF NOT EXISTS messages_thread_headers ON messages (thread_headers)');
     const version = this.db.prepare('SELECT value FROM meta WHERE key = ?').get('schema_version');
     if (!version) this.db.prepare('INSERT INTO meta (key, value) VALUES (?, ?)').run('schema_version', SCHEMA_VERSION);
     return this;
@@ -224,7 +253,7 @@ export class LocalIndex {
     const kick = async () => {
       try {
         const last = Number(this._meta('last_full_sync') || 0);
-        if (Date.now() - last >= this.intervalMs / 2) await this.sync();
+        if (Date.now() - last >= this.intervalMs / 2) await this.sync({ backfill: true });
       } catch (err) {
         this.log(`[index] sync failed: ${err.message}`);
       } finally {
@@ -370,7 +399,7 @@ export class LocalIndex {
    * Run a sync. Single-flight: concurrent callers share the running sync.
    * maxPerFolder caps new messages per folder per pass; passes caps the loop.
    */
-  sync({ folders = null, maxPerFolder = 1000, passes = Infinity } = {}) {
+  sync({ folders = null, maxPerFolder = 1000, passes = Infinity, backfill = false } = {}) {
     if (this.running) return this.running;
     this.runningSince = Date.now();
     this.running = (async () => {
@@ -393,7 +422,7 @@ export class LocalIndex {
       }, LOCK_HEARTBEAT_MS);
       if (heartbeat.unref) heartbeat.unref();
       try {
-        const summary = await this._runCapped(() => this._sync({ folders, maxPerFolder, passes }));
+        const summary = await this._runCapped(() => this._sync({ folders, maxPerFolder, passes, backfill }));
         if (this.db) {
           if (!folders && passes === Infinity && !summary.aborted) this._setMeta('last_full_sync', Date.now());
           this._setMeta('last_run', JSON.stringify({
@@ -403,6 +432,7 @@ export class LocalIndex {
             added: summary.added,
             removed: summary.removed,
             duration_ms: summary.duration_ms,
+            ...(summary.backfilled != null && { thread_headers_backfilled: summary.backfilled }),
             ...(summary.aborted && { aborted: summary.aborted })
           }));
         }
@@ -458,7 +488,7 @@ export class LocalIndex {
     return true;
   }
 
-  async _sync({ folders, maxPerFolder, passes }) {
+  async _sync({ folders, maxPerFolder, passes, backfill = false }) {
     const started = Date.now();
     const summary = { added: 0, removed: 0, folders: {} };
     let targets;
@@ -527,6 +557,21 @@ export class LocalIndex {
         }
       }
     }
+    // The index is current at this point; last_full_sync is stamped on it by
+    // the caller. The threading-header backfill runs after that, under its
+    // own time budget, so it can never hold up the indexing of new mail.
+    if (backfill && !summary.aborted && this.backfillMs > 0) {
+      try {
+        summary.backfilled = await this._backfillThreadHeaders(targets, Date.now() + this.backfillMs);
+      } catch (err) {
+        if (err.code === 'IMAP_TIMEOUT' || err.code === 'LOCK_LOST' || !this.db) {
+          summary.aborted = err.message;
+          this.log(`[index] backfill stopped: ${err.message}`);
+        } else {
+          this.log(`[index] backfill: ${err.message}`);
+        }
+      }
+    }
     summary.duration_ms = Date.now() - started;
     return summary;
   }
@@ -607,6 +652,60 @@ export class LocalIndex {
     }
   }
 
+  /**
+   * Fetch In-Reply-To and References for rows indexed before 1.9.0, headers
+   * only, in batches, until the deadline. Some servers read these headers
+   * from disk at a few messages per second, so the work is spread over many
+   * scheduled runs. Folders with the fewest pending rows go first, so the
+   * small working folders reach full coverage long before a large Archive.
+   */
+  async _backfillThreadHeaders(folders, deadline) {
+    const counts = this.db.prepare('SELECT folder, count(*) AS n FROM messages WHERE thread_headers = 0 GROUP BY folder ORDER BY n ASC').all()
+      .filter((r) => folders.includes(r.folder));
+    if (!counts.length) return 0;
+    const upd = this.db.prepare('UPDATE messages SET in_reply_to = ?, references_ids = ?, thread_headers = 1 WHERE folder = ? AND uid = ?');
+    let total = 0;
+    for (const { folder } of counts) {
+      if (Date.now() >= deadline) break;
+      this._assertLock();
+      const conn = await this.session.acquire();
+      let done = 0;
+      try {
+        await openBox(conn, folder, true);
+        const parseHeader = conn.imap.constructor.parseHeader;
+        while (Date.now() < deadline) {
+          this._assertLock();
+          this.progressAt = Date.now();
+          const uids = this.db.prepare('SELECT uid FROM messages WHERE folder = ? AND thread_headers = 0 ORDER BY uid DESC LIMIT ?')
+            .all(folder, Math.min(this.batchSize, BACKFILL_BATCH)).map((r) => Number(r.uid));
+          if (!uids.length) break;
+          const fetched = await fetchMessages(conn, uids, { bodies: ['HEADER.FIELDS (IN-REPLY-TO REFERENCES)'] });
+          this._assertLock();
+          this.db.exec('BEGIN');
+          try {
+            for (const uid of uids) {
+              const rec = fetched.get(uid);
+              // A uid the server no longer has is marked done; the next set
+              // difference removes the row.
+              const h = rec ? headerOf(rec, parseHeader) : {};
+              upd.run(parseMessageIdList(h['in-reply-to'])[0] || null, parseMessageIdList(h.references).join(' ') || null, folder, uid);
+              done++;
+            }
+            this.db.exec('COMMIT');
+          } catch (err) {
+            this.db.exec('ROLLBACK');
+            throw err;
+          }
+        }
+      } finally {
+        conn.end();
+        if (done) this.log(`[index] ${folder}: threading headers backfilled for ${done} message(s)`);
+        total += done;
+      }
+    }
+    return total;
+  }
+
   async _indexBatch(conn, folder, uidvalidity, uids) {
     const meta = await fetchMessages(conn, uids, { bodies: [HEADER_FIELDS], struct: true, size: true });
     this._assertLock(); // a run cut loose by the cap must not write after the fact
@@ -632,8 +731,9 @@ export class LocalIndex {
     this._assertLock();
 
     const insMsg = this.db.prepare(`INSERT OR IGNORE INTO messages
-      (folder, uid, uidvalidity, internal_date, sent_date, from_addr, to_addr, cc_addr, subject, message_id, size, attachments, body_chars)
-      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`);
+      (folder, uid, uidvalidity, internal_date, sent_date, from_addr, to_addr, cc_addr, subject, message_id, size, attachments, body_chars,
+       in_reply_to, references_ids, thread_headers)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)`);
     const insFts = this.db.prepare('INSERT INTO fts (rowid, subject, from_addr, to_addr, attachments, body) VALUES (?, ?, ?, ?, ?, ?)');
 
     let added = 0;
@@ -655,7 +755,8 @@ export class LocalIndex {
         const attachments = plan.attachments.join(', ');
         const internal = rec.attributes.date instanceof Date ? rec.attributes.date.getTime() : null;
         const res = insMsg.run(folder, uid, uidvalidity, internal, (h.date || [null])[0], from, addressText(h.to), addressText(h.cc),
-          subject, (h['message-id'] || [null])[0], rec.attributes.size ?? null, attachments, body.length);
+          subject, (h['message-id'] || [null])[0], rec.attributes.size ?? null, attachments, body.length,
+          parseMessageIdList(h['in-reply-to'])[0] || null, parseMessageIdList(h.references).join(' ') || null);
         if (res.changes) {
           insFts.run(Number(res.lastInsertRowid), subject, from, to, attachments, body);
           added++;
@@ -701,6 +802,7 @@ export class LocalIndex {
       sync_max_minutes: this.maxRunMs / 60000,
       command_timeout_seconds: getOpTimeout() / 1000,
       complete: folders.length > 0 && folders.every((f) => f.complete && !f.last_error),
+      thread_headers_pending: this.db.prepare('SELECT count(*) AS n FROM messages WHERE thread_headers = 0').get().n,
       last_full_sync: lastFull ? new Date(lastFull).toISOString() : null,
       last_run: lastRun,
       folders
@@ -853,6 +955,129 @@ export class LocalIndex {
     };
   }
 
+  /**
+   * Group messages into conversations by their Message-ID chains.
+   *
+   * Two messages are in the same thread when their Message-ID, In-Reply-To
+   * and References ids overlap, transitively. That is how mail clients thread,
+   * and it is exact: a renamed subject stays in its thread, two conversations
+   * that share a subject do not merge. A message with no chain at all (no
+   * In-Reply-To, no References, and nothing pointing at it) is a thread of one.
+   *
+   * scope:
+   *   split   threads whose messages sit in more than one of the searched
+   *           folders (default). A thread whose spread is only the same
+   *           message copied into two folders is flagged copies_only, since
+   *           that is a duplicate, not a split.
+   *   all     every thread in scope.
+   */
+  findThreads({ folders = null, scope = 'split', since_date = null, before_date = null, limit = 50, offset = 0 } = {}) {
+    if (!['split', 'all'].includes(scope)) throw new Error(`Unknown scope: ${scope}. Use split or all.`);
+    const folderList = Array.isArray(folders) ? folders.filter(Boolean) : (folders ? [folders] : []);
+    const where = [];
+    const params = [];
+    if (folderList.length) {
+      where.push(`folder IN (${folderList.map(() => '?').join(', ')})`);
+      params.push(...folderList);
+    }
+    if (since_date) { where.push('internal_date >= ?'); params.push(localDate(since_date).getTime()); }
+    if (before_date) { where.push('internal_date < ?'); params.push(localDate(before_date).getTime()); }
+    const scopeSql = where.length ? `WHERE ${where.join(' AND ')}` : '';
+    const rows = this.db.prepare(`
+      SELECT id, folder, uid, message_id, in_reply_to, references_ids, subject, from_addr, internal_date, sent_date, thread_headers
+      FROM messages ${scopeSql}`).all(...params);
+
+    // Union-find over ids. Each message joins its own Message-ID with every id
+    // it references; a message without a Message-ID gets a private key.
+    const parent = new Map();
+    const find = (k) => {
+      let r = k;
+      while (parent.get(r) !== r) r = parent.get(r);
+      let c = k;
+      while (parent.get(c) !== r) { const n = parent.get(c); parent.set(c, r); c = n; }
+      return r;
+    };
+    const add = (k) => { if (!parent.has(k)) parent.set(k, k); };
+    const union = (a, b) => { add(a); add(b); const ra = find(a); const rb = find(b); if (ra !== rb) parent.set(ra, rb); };
+    const keyOf = new Map();
+    let withHeaders = 0;
+    for (const r of rows) {
+      const own = normalizeMessageId(r.message_id) || `row:${r.id}`;
+      keyOf.set(r.id, own);
+      add(own);
+      if (r.thread_headers) withHeaders++;
+      if (r.in_reply_to) union(own, r.in_reply_to);
+      if (r.references_ids) for (const ref of r.references_ids.split(' ')) if (ref) union(own, ref);
+    }
+
+    const threads = new Map();
+    for (const r of rows) {
+      const root = find(keyOf.get(r.id));
+      if (!threads.has(root)) threads.set(root, []);
+      threads.get(root).push(r);
+    }
+
+    const dateOf = (r) => Number(r.internal_date) || Date.parse(r.sent_date || '') || 0;
+    const out = [];
+    for (const [root, msgs] of threads) {
+      const byFolder = {};
+      for (const m of msgs) byFolder[m.folder] = (byFolder[m.folder] || 0) + 1;
+      const folderCount = Object.keys(byFolder).length;
+      if (scope === 'split' && folderCount < 2) continue;
+      const sorted = [...msgs].sort((a, b) => dateOf(a) - dateOf(b));
+      const distinctIds = new Set(sorted.map((m) => normalizeMessageId(m.message_id) || `row:${m.id}`));
+      // Spread that exists only because the same messages sit in two folders.
+      let copiesOnly = false;
+      if (folderCount > 1) {
+        const perId = new Map();
+        for (const m of sorted) {
+          const id = normalizeMessageId(m.message_id) || `row:${m.id}`;
+          if (!perId.has(id)) perId.set(id, new Set());
+          perId.get(id).add(m.folder);
+        }
+        copiesOnly = [...perId.values()].every((set) => set.size === folderCount);
+      }
+      const newest = sorted[sorted.length - 1];
+      const oldest = sorted[0];
+      const participants = [...new Set(sorted.map((m) => m.from_addr).filter(Boolean))].slice(0, 12);
+      out.push({
+        thread_key: root.startsWith('row:') ? null : root,
+        subject: oldest.subject || '',
+        latest_subject: newest.subject !== oldest.subject ? newest.subject : undefined,
+        messages: msgs.length,
+        distinct_messages: distinctIds.size,
+        by_folder: byFolder,
+        ...(folderCount > 1 && { copies_only: copiesOnly }),
+        oldest: { folder: oldest.folder, uid: Number(oldest.uid), date: oldest.sent_date || new Date(dateOf(oldest)).toISOString(), from: oldest.from_addr },
+        newest: { folder: newest.folder, uid: Number(newest.uid), date: newest.sent_date || new Date(dateOf(newest)).toISOString(), from: newest.from_addr },
+        participants,
+        threading: sorted.every((m) => m.thread_headers) ? 'headers' : 'partial: some messages still lack threading headers',
+        message_list: sorted.slice(-50).map((m) => ({ folder: m.folder, uid: Number(m.uid), date: m.sent_date || new Date(dateOf(m)).toISOString(), from: m.from_addr }))
+      });
+    }
+    out.sort((a, b) => String(b.newest.date).localeCompare(String(a.newest.date)));
+    const cap = Math.min(Math.max(1, Number(limit) || 50), 500);
+    const skip = Math.max(0, Number(offset) || 0);
+    const page = out.slice(skip, skip + cap);
+    return {
+      scope,
+      folders_searched: folderList.length ? folderList : 'all indexed folders',
+      coverage: {
+        messages_in_scope: rows.length,
+        with_thread_headers: withHeaders,
+        without_thread_headers: rows.length - withHeaders,
+        ...(rows.length > withHeaders && { note: 'messages without threading headers are still being backfilled; they thread only by their own Message-ID until then' }),
+        threads_in_scope: threads.size
+      },
+      total_threads: out.length,
+      returned: page.length,
+      offset: skip,
+      has_more: skip + page.length < out.length,
+      threads: page,
+      index: this._indexHealth(folderList)
+    };
+  }
+
   search({ query, folders = null, from = null, since_date = null, before_date = null, field = 'any', match = 'all', prefix = false, sort = 'date', limit = 20, offset = 0 }) {
     const where = ['fts MATCH ?'];
     const params = [ftsQuery(query, { match, prefix, field })];
@@ -881,6 +1106,7 @@ export class LocalIndex {
     const total = this.db.prepare(`SELECT count(*) AS n FROM fts JOIN messages m ON m.id = fts.rowid WHERE ${whereSql}`).get(...params).n;
     const rows = this.db.prepare(`
       SELECT m.folder, m.uid, m.internal_date, m.sent_date, m.from_addr, m.to_addr, m.subject, m.attachments, m.message_id,
+             m.in_reply_to, m.references_ids,
              snippet(fts, -1, '[', ']', '...', 16) AS snippet
       FROM fts JOIN messages m ON m.id = fts.rowid
       WHERE ${whereSql}
@@ -900,6 +1126,8 @@ export class LocalIndex {
         to: r.to_addr,
         subject: r.subject,
         message_id: normalizeMessageId(r.message_id) || undefined,
+        in_reply_to: r.in_reply_to || undefined,
+        references: r.references_ids ? r.references_ids.split(' ') : undefined,
         attachments: r.attachments || undefined,
         snippet: r.snippet
       }))

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lengelhard/imap-email-mcp",
-  "version": "1.8.1",
+  "version": "1.9.1",
   "description": "MCP server for Claude Code that provides email capabilities through IMAP/SMTP. Read, search, compose, and manage emails from any IMAP provider.",
   "type": "module",
   "main": "index.js",