apple-tools-mcp 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +133 -23
- package/index.js +5 -46
- package/indexer.js +61 -30
- package/lib/audit.js +13 -20
- package/lib/validators.js +2 -2
- package/package.json +28 -55
- package/search.js +1 -1
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# apple-tools-mcp
|
|
2
2
|
|
|
3
|
-
An MCP (Model Context Protocol) server that provides semantic search across Apple Mail, Messages, and
|
|
3
|
+
An MCP (Model Context Protocol) server that provides semantic search across Apple Mail, Messages, Calendar, and Contacts on macOS. Use natural language to search your emails, iMessages, calendar events, and contacts directly from Claude.
|
|
4
4
|
|
|
5
5
|
## Features
|
|
6
6
|
|
|
@@ -25,17 +25,44 @@ An MCP (Model Context Protocol) server that provides semantic search across Appl
|
|
|
25
25
|
npm install -g apple-tools-mcp
|
|
26
26
|
```
|
|
27
27
|
|
|
28
|
+
Or from source:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
git clone https://github.com/sfls1397/Apple-Tools-MCP.git
|
|
32
|
+
cd Apple-Tools-MCP
|
|
33
|
+
npm install
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
If you installed from source, point Claude Desktop at the local `index.js` instead of `npx` in step 3:
|
|
37
|
+
|
|
38
|
+
```json
|
|
39
|
+
"command": "node",
|
|
40
|
+
"args": ["/absolute/path/to/Apple-Tools-MCP/index.js"]
|
|
41
|
+
```
|
|
42
|
+
|
|
28
43
|
### 2. Grant Full Disk Access
|
|
29
44
|
|
|
30
45
|
The MCP server needs access to read your Mail, Messages, and Calendar databases.
|
|
31
46
|
|
|
32
|
-
1.
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
47
|
+
1. First, find your Node.js path by running in Terminal:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
which node
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
This will output something like `/opt/homebrew/bin/node` or `/usr/local/bin/node`
|
|
54
|
+
|
|
55
|
+
2. Open **System Settings** → **Privacy & Security** → **Full Disk Access**
|
|
56
|
+
|
|
57
|
+
3. Click the **+** button
|
|
58
|
+
|
|
59
|
+
4. Press **Cmd+Shift+G** to open the "Go to Folder" dialog
|
|
60
|
+
|
|
61
|
+
5. Paste the path from step 1 (e.g., `/opt/homebrew/bin/node`) and press Enter
|
|
62
|
+
|
|
63
|
+
6. Select the `node` file and click **Open**
|
|
64
|
+
|
|
65
|
+
7. Ensure the toggle for Node.js is enabled
|
|
39
66
|
|
|
40
67
|
### 3. Configure Claude Desktop
|
|
41
68
|
|
|
@@ -60,16 +87,16 @@ Quit and reopen Claude Desktop to load the MCP server.
|
|
|
60
87
|
|
|
61
88
|
## Building the Index
|
|
62
89
|
|
|
63
|
-
On first use, the server will automatically build a vector index of your
|
|
90
|
+
On first use, the server will automatically build a vector index of your emails, messages, and calendar events. Email history is unlimited by default. This may take a while depending on the volume of data.
|
|
64
91
|
|
|
65
92
|
You can manually rebuild the index:
|
|
66
93
|
|
|
67
94
|
```bash
|
|
68
|
-
# Index
|
|
69
|
-
|
|
95
|
+
# Index all email history (default)
|
|
96
|
+
npm run build-index
|
|
70
97
|
|
|
71
|
-
#
|
|
72
|
-
APPLE_TOOLS_INDEX_DAYS_BACK=
|
|
98
|
+
# Optional: cap email lookback (e.g. for a faster test rebuild)
|
|
99
|
+
APPLE_TOOLS_INDEX_DAYS_BACK=30 npm run build-index
|
|
73
100
|
```
|
|
74
101
|
|
|
75
102
|
The index is stored in `~/.apple-tools-mcp/vector-index/`.
|
|
@@ -78,15 +105,57 @@ The index is stored in `~/.apple-tools-mcp/vector-index/`.
|
|
|
78
105
|
|
|
79
106
|
Once configured, Claude can use these tools:
|
|
80
107
|
|
|
108
|
+
### Universal Search
|
|
109
|
+
|
|
110
|
+
| Tool | Description |
|
|
111
|
+
|------|-------------|
|
|
112
|
+
| `smart_search` | Intelligent search across all sources - automatically determines which to search |
|
|
113
|
+
| `person_search` | Find ALL communication with a person across Mail, Messages, and Calendar |
|
|
114
|
+
|
|
115
|
+
### Email Tools
|
|
116
|
+
|
|
81
117
|
| Tool | Description |
|
|
82
118
|
|------|-------------|
|
|
83
|
-
| `
|
|
84
|
-
| `
|
|
85
|
-
| `
|
|
86
|
-
| `
|
|
87
|
-
| `
|
|
88
|
-
| `
|
|
89
|
-
|
|
119
|
+
| `mail_search` | Semantic search for emails with filters (sender, recipient, attachments, mailbox) |
|
|
120
|
+
| `mail_recent` | Get most recent emails (supports unread filter) |
|
|
121
|
+
| `mail_date` | Get emails from a specific date ("today", "yesterday", "Nov 13") |
|
|
122
|
+
| `mail_read` | Read full email content by file path |
|
|
123
|
+
| `mail_senders` | List most frequent email senders |
|
|
124
|
+
| `mail_thread` | Get all emails in a conversation thread |
|
|
125
|
+
|
|
126
|
+
### Messages Tools
|
|
127
|
+
|
|
128
|
+
| Tool | Description |
|
|
129
|
+
|------|-------------|
|
|
130
|
+
| `messages_search` | Semantic search for iMessages/SMS with filters |
|
|
131
|
+
| `messages_recent` | Get most recent messages |
|
|
132
|
+
| `messages_conversation` | Get full conversation history with a contact |
|
|
133
|
+
| `messages_contacts` | List all contacts you've messaged |
|
|
134
|
+
|
|
135
|
+
### Calendar Tools
|
|
136
|
+
|
|
137
|
+
| Tool | Description |
|
|
138
|
+
|------|-------------|
|
|
139
|
+
| `calendar_search` | Semantic search for events with filters |
|
|
140
|
+
| `calendar_date` | Get events on a specific date |
|
|
141
|
+
| `calendar_upcoming` | Get next N upcoming events |
|
|
142
|
+
| `calendar_week` | Get all events for current or future week |
|
|
143
|
+
| `calendar_free_time` | Find available time slots on a date |
|
|
144
|
+
| `calendar_recurring` | List recurring events |
|
|
145
|
+
|
|
146
|
+
### Contacts Tools
|
|
147
|
+
|
|
148
|
+
| Tool | Description |
|
|
149
|
+
|------|-------------|
|
|
150
|
+
| `contacts_search` | Search contacts by name, email, phone, or organization |
|
|
151
|
+
| `contacts_lookup` | Look up a specific contact's full details |
|
|
152
|
+
|
|
153
|
+
### Admin Tools
|
|
154
|
+
|
|
155
|
+
| Tool | Description |
|
|
156
|
+
|------|-------------|
|
|
157
|
+
| `rebuild_index` | Rebuild search index for one or all sources |
|
|
158
|
+
| `audit_index` | Audit index health and coverage |
|
|
90
159
|
|
|
91
160
|
## Example Queries
|
|
92
161
|
|
|
@@ -97,6 +166,8 @@ Ask Claude things like:
|
|
|
97
166
|
- "When is my next dentist appointment?"
|
|
98
167
|
- "Search for emails about the AWS bill from November"
|
|
99
168
|
- "Find all calendar events with Zoom links"
|
|
169
|
+
- "What's Sarah's phone number?"
|
|
170
|
+
- "Show me all communication with David from last month"
|
|
100
171
|
|
|
101
172
|
## Privacy & Security
|
|
102
173
|
|
|
@@ -114,7 +185,7 @@ Ensure Node.js has Full Disk Access (see Installation step 2).
|
|
|
114
185
|
### Empty search results
|
|
115
186
|
|
|
116
187
|
1. Check that the index was built: `ls ~/.apple-tools-mcp/vector-index/`
|
|
117
|
-
2. Rebuild the index if needed: `
|
|
188
|
+
2. Rebuild the index if needed: `npm run build-index`
|
|
118
189
|
|
|
119
190
|
### Server not appearing in Claude
|
|
120
191
|
|
|
@@ -122,19 +193,58 @@ Ensure Node.js has Full Disk Access (see Installation step 2).
|
|
|
122
193
|
2. Restart Claude Desktop completely (Cmd+Q, then reopen)
|
|
123
194
|
3. Check Claude's MCP logs for errors
|
|
124
195
|
|
|
196
|
+
### Force rebuild the index
|
|
197
|
+
|
|
198
|
+
If the index becomes corrupted or out of sync:
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
# Remove existing index files
|
|
202
|
+
rm -rf ~/.apple-tools-mcp/vector-index
|
|
203
|
+
rm -f ~/.apple-tools-mcp/index-meta.json
|
|
204
|
+
rm -f ~/.apple-tools-mcp/indexer.lock
|
|
205
|
+
|
|
206
|
+
# Restart Claude Desktop to trigger a fresh rebuild
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### Monitor indexing progress
|
|
210
|
+
|
|
211
|
+
Watch the MCP server logs in real-time:
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
tail -f ~/Library/Logs/Claude/mcp-server-apple-tools.log
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
### Audit the index
|
|
218
|
+
|
|
219
|
+
Check index health and coverage:
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
# Quick audit
|
|
223
|
+
npm run audit
|
|
224
|
+
|
|
225
|
+
# Detailed audit saved to file
|
|
226
|
+
npm run audit -- --reporter=verbose > audit-report.txt
|
|
227
|
+
```
|
|
228
|
+
|
|
125
229
|
## Development
|
|
126
230
|
|
|
127
231
|
```bash
|
|
128
232
|
# Clone the repo
|
|
129
|
-
git clone https://github.com/sfls1397/
|
|
130
|
-
cd
|
|
233
|
+
git clone https://github.com/sfls1397/Apple-Tools-MCP.git
|
|
234
|
+
cd Apple-Tools-MCP
|
|
131
235
|
|
|
132
236
|
# Install dependencies
|
|
133
237
|
npm install
|
|
134
238
|
|
|
239
|
+
# Install test dependencies
|
|
240
|
+
npm install -D vitest @vitest/coverage-v8 fast-check
|
|
241
|
+
|
|
135
242
|
# Run tests
|
|
136
243
|
npm test
|
|
137
244
|
|
|
245
|
+
# Run tests with verbose coverage report
|
|
246
|
+
npx vitest run --coverage --reporter=verbose
|
|
247
|
+
|
|
138
248
|
# Build index with debug output
|
|
139
249
|
npm run build-index
|
|
140
250
|
|
package/index.js
CHANGED
|
@@ -89,35 +89,6 @@ function releaseLock() {
|
|
|
89
89
|
}
|
|
90
90
|
}
|
|
91
91
|
|
|
92
|
-
// Kill any zombie MCP processes on startup (except this one)
|
|
93
|
-
function cleanupZombieProcesses() {
|
|
94
|
-
try {
|
|
95
|
-
const { execSync } = require('child_process');
|
|
96
|
-
// Find all apple-tools-mcp index.js processes
|
|
97
|
-
const psOutput = execSync('ps aux | grep "apple-tools-mcp/index.js" | grep -v grep || true', { encoding: 'utf-8' });
|
|
98
|
-
const lines = psOutput.trim().split('\n').filter(l => l);
|
|
99
|
-
|
|
100
|
-
for (const line of lines) {
|
|
101
|
-
const parts = line.trim().split(/\s+/);
|
|
102
|
-
const pid = parseInt(parts[1]);
|
|
103
|
-
|
|
104
|
-
// Skip this process
|
|
105
|
-
if (pid === process.pid) continue;
|
|
106
|
-
|
|
107
|
-
// Check if process is still running and kill it
|
|
108
|
-
try {
|
|
109
|
-
process.kill(pid, 0); // Check if exists
|
|
110
|
-
console.error(`Killing zombie MCP process: ${pid}`);
|
|
111
|
-
process.kill(pid, 'SIGTERM');
|
|
112
|
-
} catch {
|
|
113
|
-
// Process already dead
|
|
114
|
-
}
|
|
115
|
-
}
|
|
116
|
-
} catch (e) {
|
|
117
|
-
// Ignore errors - cleanup is best-effort
|
|
118
|
-
}
|
|
119
|
-
}
|
|
120
|
-
|
|
121
92
|
// Clean up lock and timer on exit
|
|
122
93
|
process.on("exit", () => {
|
|
123
94
|
stopBackgroundIndexing();
|
|
@@ -235,16 +206,6 @@ function getIndexingMessage() {
|
|
|
235
206
|
return "Indexing new data. Please try again in a moment.";
|
|
236
207
|
}
|
|
237
208
|
|
|
238
|
-
// Timeout wrapper for promises
|
|
239
|
-
function withTimeout(promise, timeoutMs, operation = "Operation") {
|
|
240
|
-
return Promise.race([
|
|
241
|
-
promise,
|
|
242
|
-
new Promise((_, reject) =>
|
|
243
|
-
setTimeout(() => reject(new Error(`${operation} timed out after ${timeoutMs}ms`)), timeoutMs)
|
|
244
|
-
)
|
|
245
|
-
]);
|
|
246
|
-
}
|
|
247
|
-
|
|
248
209
|
// Run a single indexing cycle (called by background timer)
|
|
249
210
|
function runIndexCycle() {
|
|
250
211
|
if (indexingInProgress) {
|
|
@@ -356,10 +317,11 @@ function stopBackgroundIndexing() {
|
|
|
356
317
|
async function initializeIndexing() {
|
|
357
318
|
isFirstEverRun = await checkIfFirstRun();
|
|
358
319
|
|
|
359
|
-
// Try to acquire lock - if another instance is
|
|
320
|
+
// Try to acquire lock - if another instance is indexing, skip background
|
|
321
|
+
// indexing but keep the MCP server running so search still works.
|
|
360
322
|
if (!acquireLock()) {
|
|
361
|
-
console.error("Another apple-tools-mcp instance is
|
|
362
|
-
|
|
323
|
+
console.error("Another apple-tools-mcp instance is indexing. Server will run without background indexing.");
|
|
324
|
+
return;
|
|
363
325
|
}
|
|
364
326
|
|
|
365
327
|
// Start background indexing
|
|
@@ -1490,12 +1452,9 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
|
|
1490
1452
|
|
|
1491
1453
|
// Start the server
|
|
1492
1454
|
async function main() {
|
|
1493
|
-
// Kill any zombie processes from previous sessions
|
|
1494
|
-
cleanupZombieProcesses();
|
|
1495
|
-
|
|
1496
1455
|
const transport = new StdioServerTransport();
|
|
1497
1456
|
await server.connect(transport);
|
|
1498
|
-
console.error("Apple Tools MCP server running (
|
|
1457
|
+
console.error("Apple Tools MCP server running (v1.1.0)");
|
|
1499
1458
|
// Background indexing runs automatically on startup and every INDEX_INTERVAL
|
|
1500
1459
|
}
|
|
1501
1460
|
|
package/indexer.js
CHANGED
|
@@ -1,7 +1,5 @@
|
|
|
1
1
|
import fs from "fs";
|
|
2
2
|
import path from "path";
|
|
3
|
-
import { exec } from "child_process";
|
|
4
|
-
import { promisify } from "util";
|
|
5
3
|
import * as lancedb from "@lancedb/lancedb";
|
|
6
4
|
import { pipeline } from "@xenova/transformers";
|
|
7
5
|
import {
|
|
@@ -13,7 +11,7 @@ import {
|
|
|
13
11
|
escapeSQL,
|
|
14
12
|
stripHtmlTags
|
|
15
13
|
} from "./lib/validators.js";
|
|
16
|
-
import { safeSqlite3Json, safeOsascript } from "./lib/shell.js";
|
|
14
|
+
import { safeSqlite3Json, safeOsascript, safeFind } from "./lib/shell.js";
|
|
17
15
|
|
|
18
16
|
// Re-export contact functions for use by other modules
|
|
19
17
|
export {
|
|
@@ -28,16 +26,17 @@ export {
|
|
|
28
26
|
getContactStats
|
|
29
27
|
} from "./contacts.js";
|
|
30
28
|
|
|
31
|
-
const execAsync = promisify(exec);
|
|
32
|
-
|
|
33
29
|
// Support env var overrides for testing with separate index
|
|
34
30
|
export const INDEX_DIR = process.env.APPLE_TOOLS_INDEX_DIR ||
|
|
35
31
|
path.join(process.env.HOME, ".apple-tools-mcp", "vector-index");
|
|
36
32
|
const META_FILE = process.env.APPLE_TOOLS_META_FILE ||
|
|
37
33
|
path.join(process.env.HOME, ".apple-tools-mcp", "index-meta.json");
|
|
38
|
-
//
|
|
39
|
-
|
|
40
|
-
|
|
34
|
+
// Optional date filter for email indexing. Default is unlimited (index all emails).
|
|
35
|
+
// Set APPLE_TOOLS_INDEX_DAYS_BACK=30 to cap the window (tests do this).
|
|
36
|
+
// Set APPLE_TOOLS_INDEX_DAYS_BACK=0 to explicitly disable filtering.
|
|
37
|
+
const DAYS_BACK = process.env.APPLE_TOOLS_INDEX_DAYS_BACK !== undefined
|
|
38
|
+
? parseInt(process.env.APPLE_TOOLS_INDEX_DAYS_BACK, 10) || null // 0 means null (no filter)
|
|
39
|
+
: null; // Default: no date filter
|
|
41
40
|
const MAIL_DIR = path.join(process.env.HOME, "Library", "Mail");
|
|
42
41
|
|
|
43
42
|
// Load/save index metadata (timestamps, etc.)
|
|
@@ -59,8 +58,8 @@ function saveIndexMeta(meta) {
|
|
|
59
58
|
}
|
|
60
59
|
const MESSAGES_DB = path.join(process.env.HOME, "Library", "Messages", "chat.db");
|
|
61
60
|
const CALENDAR_DB = path.join(process.env.HOME, "Library", "Group Containers", "group.com.apple.calendar", "Calendar.sqlitedb");
|
|
62
|
-
const BATCH_SIZE =
|
|
63
|
-
const BATCH_DELAY_MS =
|
|
61
|
+
const BATCH_SIZE = 64; // Optimized for batch embedding throughput (increased for faster indexing)
|
|
62
|
+
const BATCH_DELAY_MS = 0; // No delay needed - benchmarks showed no thermal throttling
|
|
64
63
|
|
|
65
64
|
// Mac Absolute Time epoch: Jan 1, 2001 00:00:00 UTC
|
|
66
65
|
const MAC_ABSOLUTE_EPOCH = 978307200;
|
|
@@ -89,11 +88,14 @@ const EMBEDDING_DIM = 384; // all-MiniLM-L6-v2 dimension
|
|
|
89
88
|
|
|
90
89
|
// Timeout wrapper for promises
|
|
91
90
|
function withTimeout(promise, timeoutMs, operation = "Operation") {
|
|
91
|
+
let timer;
|
|
92
|
+
const timeoutPromise = new Promise((_, reject) => {
|
|
93
|
+
timer = setTimeout(() => reject(new Error(`${operation} timed out after ${timeoutMs}ms`)), timeoutMs);
|
|
94
|
+
if (timer.unref) timer.unref();
|
|
95
|
+
});
|
|
92
96
|
return Promise.race([
|
|
93
|
-
promise,
|
|
94
|
-
|
|
95
|
-
setTimeout(() => reject(new Error(`${operation} timed out after ${timeoutMs}ms`)), timeoutMs)
|
|
96
|
-
)
|
|
97
|
+
promise.finally(() => clearTimeout(timer)),
|
|
98
|
+
timeoutPromise
|
|
97
99
|
]);
|
|
98
100
|
}
|
|
99
101
|
|
|
@@ -333,10 +335,8 @@ function parseEmlx(filePath) {
|
|
|
333
335
|
// Includes both .emlx and .partial.emlx files (partial = not fully downloaded via IMAP)
|
|
334
336
|
async function findAllEmlxFiles() {
|
|
335
337
|
try {
|
|
336
|
-
//
|
|
337
|
-
|
|
338
|
-
const { stdout } = await execAsync(cmd, { encoding: "utf-8", maxBuffer: 50 * 1024 * 1024, timeout: 120000 });
|
|
339
|
-
return stdout.trim().split("\n").filter(f => f);
|
|
338
|
+
// "*.emlx" also matches "*.partial.emlx"
|
|
339
|
+
return safeFind(MAIL_DIR, { name: "*.emlx", type: "f" });
|
|
340
340
|
} catch (e) {
|
|
341
341
|
console.error("Error finding emlx files:", e.message);
|
|
342
342
|
return [];
|
|
@@ -358,9 +358,7 @@ async function findNewEmlxFiles(sinceTimestamp) {
|
|
|
358
358
|
|
|
359
359
|
// Use find with -mtime instead of mdfind for reliability
|
|
360
360
|
// find is more reliable than Spotlight which can have stale/incomplete indexes
|
|
361
|
-
const
|
|
362
|
-
const { stdout } = await execAsync(cmd, { encoding: "utf-8", maxBuffer: 50 * 1024 * 1024, timeout: 120000 });
|
|
363
|
-
const files = stdout.trim().split("\n").filter(f => f);
|
|
361
|
+
const files = safeFind(MAIL_DIR, { name: "*.emlx", type: "f", mtime: `-${daysAgo}` });
|
|
364
362
|
console.error(`find found ${files.length} new/modified emails in last ${daysAgo} days`);
|
|
365
363
|
return files;
|
|
366
364
|
} catch (e) {
|
|
@@ -718,10 +716,6 @@ function getCalendarEvents() {
|
|
|
718
716
|
const startTimestamp = macAbsoluteToUnixMs(row.start_date);
|
|
719
717
|
const attendees = attendeesMap.get(row.id) || [];
|
|
720
718
|
|
|
721
|
-
if (row.all_day === 1) {
|
|
722
|
-
console.error(`[Indexing] All-day event: "${row.summary}" at ${new Date(startTimestamp).toISOString()}`);
|
|
723
|
-
}
|
|
724
|
-
|
|
725
719
|
events.push({
|
|
726
720
|
dbId: row.id, // Stable database ID for deduplication
|
|
727
721
|
title: row.summary,
|
|
@@ -909,15 +903,20 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
909
903
|
if (DAYS_BACK) {
|
|
910
904
|
// DAYS_BACK is set - use it for mdfind filter (e.g., for testing or rebuild with constraints)
|
|
911
905
|
lastEmailIndexTime = Date.now() - (DAYS_BACK * 24 * 60 * 60 * 1000);
|
|
912
|
-
console.error(
|
|
906
|
+
console.error(`\n=== EMAIL INDEXING (DAYS_BACK=${DAYS_BACK}) ===`);
|
|
907
|
+
console.error(`Will search for emails modified since: ${new Date(lastEmailIndexTime).toISOString()}`);
|
|
913
908
|
} else if (forceFullScan) {
|
|
914
909
|
// Force full scan only when no DAYS_BACK constraint
|
|
915
910
|
lastEmailIndexTime = null;
|
|
911
|
+
console.error("\n=== EMAIL INDEXING (FULL SCAN) ===");
|
|
916
912
|
console.error("Force full scan requested - finding all emails");
|
|
917
913
|
} else if (meta.lastEmailIndexTime) {
|
|
918
914
|
lastEmailIndexTime = meta.lastEmailIndexTime - ONE_DAY_MS;
|
|
915
|
+
console.error(`\n=== EMAIL INDEXING (INCREMENTAL) ===`);
|
|
916
|
+
console.error(`Searching for emails modified since: ${new Date(lastEmailIndexTime).toISOString()}`);
|
|
919
917
|
} else {
|
|
920
918
|
lastEmailIndexTime = null;
|
|
919
|
+
console.error("\n=== EMAIL INDEXING (FIRST RUN - FULL SCAN) ===");
|
|
921
920
|
}
|
|
922
921
|
|
|
923
922
|
// Save the current time BEFORE we start - any emails arriving during indexing
|
|
@@ -926,6 +925,7 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
926
925
|
|
|
927
926
|
// Use fast incremental scan if we have a previous timestamp
|
|
928
927
|
const startTime = Date.now();
|
|
928
|
+
console.error(`Calling findNewEmlxFiles with timestamp: ${lastEmailIndexTime ? new Date(lastEmailIndexTime).toISOString() : 'null (full scan)'}`);
|
|
929
929
|
const newFiles = await findNewEmlxFiles(lastEmailIndexTime);
|
|
930
930
|
console.error(`Found ${newFiles.length} new/modified email files (${Date.now() - startTime}ms)`);
|
|
931
931
|
|
|
@@ -950,6 +950,11 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
950
950
|
let processed = 0;
|
|
951
951
|
let skippedCount = { parseNull: 0, shortSearchText: 0, duplicateMessageId: 0 };
|
|
952
952
|
|
|
953
|
+
// Progress tracking
|
|
954
|
+
const totalBatches = Math.ceil(toIndex.length / BATCH_SIZE);
|
|
955
|
+
const startProcessTime = Date.now();
|
|
956
|
+
console.error(`\nProcessing ${toIndex.length} emails in ${totalBatches} batches (batch size: ${BATCH_SIZE})...`);
|
|
957
|
+
|
|
953
958
|
for (let i = 0; i < toIndex.length; i += BATCH_SIZE) {
|
|
954
959
|
const batch = toIndex.slice(i, i + BATCH_SIZE);
|
|
955
960
|
|
|
@@ -1067,7 +1072,16 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
1067
1072
|
}
|
|
1068
1073
|
|
|
1069
1074
|
processed += batch.length;
|
|
1070
|
-
|
|
1075
|
+
|
|
1076
|
+
// Calculate and display progress with time estimates
|
|
1077
|
+
const currentBatch = Math.floor(i / BATCH_SIZE) + 1;
|
|
1078
|
+
const elapsedMs = Date.now() - startProcessTime;
|
|
1079
|
+
const avgTimePerBatch = elapsedMs / currentBatch;
|
|
1080
|
+
const remainingBatches = totalBatches - currentBatch;
|
|
1081
|
+
const estimatedRemainingMs = avgTimePerBatch * remainingBatches;
|
|
1082
|
+
const estimatedRemainingMin = Math.ceil(estimatedRemainingMs / 60000);
|
|
1083
|
+
|
|
1084
|
+
console.error(`Batch ${currentBatch}/${totalBatches} complete | Processed: ${processed}/${toIndex.length} emails | Est. remaining: ${estimatedRemainingMin}m`);
|
|
1071
1085
|
|
|
1072
1086
|
// Report progress after each batch
|
|
1073
1087
|
if (progressCallback) {
|
|
@@ -1083,7 +1097,8 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
1083
1097
|
// Save timestamp for next incremental scan
|
|
1084
1098
|
saveIndexMeta({ ...meta, lastEmailIndexTime: indexStartTime });
|
|
1085
1099
|
|
|
1086
|
-
// Log skip summary
|
|
1100
|
+
// Log skip summary with timing
|
|
1101
|
+
const totalProcessingTime = Date.now() - startProcessTime;
|
|
1087
1102
|
const totalSkipped = skippedCount.parseNull + skippedCount.shortSearchText + skippedCount.duplicateMessageId;
|
|
1088
1103
|
console.error(`\nEmail indexing summary:`);
|
|
1089
1104
|
console.error(` Files to index: ${toIndex.length}`);
|
|
@@ -1092,7 +1107,8 @@ export async function indexEmails(progressCallback = null, forceFullScan = false
|
|
|
1092
1107
|
console.error(` Skipped - short searchText: ${skippedCount.shortSearchText}`);
|
|
1093
1108
|
console.error(` Skipped - duplicate messageId: ${skippedCount.duplicateMessageId}`);
|
|
1094
1109
|
console.error(` Total skipped: ${totalSkipped}`);
|
|
1095
|
-
console.error(` Discrepancy: ${toIndex.length - processed - totalSkipped}
|
|
1110
|
+
console.error(` Discrepancy: ${toIndex.length - processed - totalSkipped}`);
|
|
1111
|
+
console.error(` Total time: ${Math.round(totalProcessingTime / 1000)}s (${Math.round(totalProcessingTime / 60000)}m)\n`);
|
|
1096
1112
|
|
|
1097
1113
|
// Return actual indexed count from database, not stale cache
|
|
1098
1114
|
const finalIndexedPaths = await getIndexedIdsWithRetry("emails", "filePath");
|
|
@@ -1129,6 +1145,12 @@ export async function indexMessages(forceFullScan = false) {
|
|
|
1129
1145
|
}
|
|
1130
1146
|
|
|
1131
1147
|
let processed = 0;
|
|
1148
|
+
|
|
1149
|
+
// Progress tracking
|
|
1150
|
+
const totalBatches = Math.ceil(toIndex.length / BATCH_SIZE);
|
|
1151
|
+
const startProcessTime = Date.now();
|
|
1152
|
+
console.error(`\nProcessing ${toIndex.length} messages in ${totalBatches} batches (batch size: ${BATCH_SIZE})...`);
|
|
1153
|
+
|
|
1132
1154
|
for (let i = 0; i < toIndex.length; i += BATCH_SIZE) {
|
|
1133
1155
|
const batch = toIndex.slice(i, i + BATCH_SIZE);
|
|
1134
1156
|
|
|
@@ -1221,7 +1243,16 @@ export async function indexMessages(forceFullScan = false) {
|
|
|
1221
1243
|
}
|
|
1222
1244
|
|
|
1223
1245
|
processed += batch.length;
|
|
1224
|
-
|
|
1246
|
+
|
|
1247
|
+
// Calculate and display progress with time estimates
|
|
1248
|
+
const currentBatch = Math.floor(i / BATCH_SIZE) + 1;
|
|
1249
|
+
const elapsedMs = Date.now() - startProcessTime;
|
|
1250
|
+
const avgTimePerBatch = elapsedMs / currentBatch;
|
|
1251
|
+
const remainingBatches = totalBatches - currentBatch;
|
|
1252
|
+
const estimatedRemainingMs = avgTimePerBatch * remainingBatches;
|
|
1253
|
+
const estimatedRemainingMin = Math.ceil(estimatedRemainingMs / 60000);
|
|
1254
|
+
|
|
1255
|
+
console.error(`Batch ${currentBatch}/${totalBatches} complete | Processed: ${processed}/${toIndex.length} messages | Est. remaining: ${estimatedRemainingMin}m`);
|
|
1225
1256
|
|
|
1226
1257
|
// Throttle to prevent thermal crashes
|
|
1227
1258
|
if (i + BATCH_SIZE < toIndex.length) {
|
package/lib/audit.js
CHANGED
|
@@ -13,8 +13,7 @@
|
|
|
13
13
|
|
|
14
14
|
import fs from "fs";
|
|
15
15
|
import path from "path";
|
|
16
|
-
import {
|
|
17
|
-
import { safeSqlite3Json } from "./shell.js";
|
|
16
|
+
import { safeSqlite3Json, safeFind } from "./shell.js";
|
|
18
17
|
import * as lancedb from "@lancedb/lancedb";
|
|
19
18
|
|
|
20
19
|
// ============================================================================
|
|
@@ -38,6 +37,16 @@ const DAYS_BACK = process.env.APPLE_TOOLS_INDEX_DAYS_BACK ?
|
|
|
38
37
|
// Exclude these folders from email indexing (matches indexer behavior)
|
|
39
38
|
const EXCLUDED_FOLDERS = ["Junk.mbox", "Saved Junk.mbox", "Trash.mbox", "Deleted Messages.mbox"];
|
|
40
39
|
|
|
40
|
+
function listEmailFiles() {
|
|
41
|
+
const options = { name: "*.emlx", type: "f" };
|
|
42
|
+
if (DAYS_BACK) {
|
|
43
|
+
options.mtime = `-${DAYS_BACK}`;
|
|
44
|
+
}
|
|
45
|
+
return safeFind(MAIL_DIR, options).filter(
|
|
46
|
+
p => !EXCLUDED_FOLDERS.some(folder => p.includes(`/${folder}/`))
|
|
47
|
+
);
|
|
48
|
+
}
|
|
49
|
+
|
|
41
50
|
let db = null;
|
|
42
51
|
let tables = {};
|
|
43
52
|
|
|
@@ -82,15 +91,7 @@ export function countRawEmails() {
|
|
|
82
91
|
if (!fs.existsSync(MAIL_DIR)) return 0;
|
|
83
92
|
|
|
84
93
|
try {
|
|
85
|
-
|
|
86
|
-
const exclusions = EXCLUDED_FOLDERS.map(folder => `-path "*/${folder}/*"`).join(" -o ");
|
|
87
|
-
|
|
88
|
-
// Add time filter if DAYS_BACK is set (matches indexer behavior)
|
|
89
|
-
const timeFilter = DAYS_BACK ? `-mtime -${DAYS_BACK}` : "";
|
|
90
|
-
const cmd = `find "${MAIL_DIR}" \\( -name "*.emlx" -o -name "*.partial.emlx" \\) ! \\( ${exclusions} \\) ${timeFilter} -type f 2>/dev/null | wc -l`;
|
|
91
|
-
|
|
92
|
-
const result = execSync(cmd, { encoding: "utf-8", timeout: 120000 }).trim();
|
|
93
|
-
return parseInt(result) || 0;
|
|
94
|
+
return listEmailFiles().length;
|
|
94
95
|
} catch (e) {
|
|
95
96
|
console.error("Error counting emails:", e.message);
|
|
96
97
|
return 0;
|
|
@@ -162,15 +163,7 @@ export function getRawEmailIds() {
|
|
|
162
163
|
if (!fs.existsSync(MAIL_DIR)) return new Set();
|
|
163
164
|
|
|
164
165
|
try {
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
// Add time filter if DAYS_BACK is set (matches indexer behavior)
|
|
168
|
-
const timeFilter = DAYS_BACK ? `-mtime -${DAYS_BACK}` : "";
|
|
169
|
-
const cmd = `find "${MAIL_DIR}" \\( -name "*.emlx" -o -name "*.partial.emlx" \\) ! \\( ${exclusions} \\) ${timeFilter} -type f 2>/dev/null`;
|
|
170
|
-
|
|
171
|
-
const result = execSync(cmd, { encoding: "utf-8", timeout: 120000, maxBuffer: 50 * 1024 * 1024 }).trim();
|
|
172
|
-
const paths = result.split("\n").filter(p => p);
|
|
173
|
-
return new Set(paths);
|
|
166
|
+
return new Set(listEmailFiles());
|
|
174
167
|
} catch (e) {
|
|
175
168
|
console.error("Error getting email IDs:", e.message);
|
|
176
169
|
return new Set();
|
package/lib/validators.js
CHANGED
|
@@ -137,8 +137,8 @@ export function validateMailboxName(mailbox) {
|
|
|
137
137
|
}
|
|
138
138
|
|
|
139
139
|
// Only allow alphanumeric, spaces, hyphens, underscores, and periods
|
|
140
|
-
//
|
|
141
|
-
if (!/^[a-zA-Z0-9
|
|
140
|
+
// Literal space only — \s would also allow newlines/tabs (injection risk)
|
|
141
|
+
if (!/^[a-zA-Z0-9._ -]+$/.test(mailbox)) {
|
|
142
142
|
return null;
|
|
143
143
|
}
|
|
144
144
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "apple-tools-mcp",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "MCP server for semantic search across Apple Mail, Messages, and Calendar",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.js",
|
|
@@ -11,12 +11,12 @@
|
|
|
11
11
|
"license": "MIT",
|
|
12
12
|
"repository": {
|
|
13
13
|
"type": "git",
|
|
14
|
-
"url": "git+https://github.com/sfls1397/
|
|
14
|
+
"url": "git+https://github.com/sfls1397/Apple-Tools-MCP.git"
|
|
15
15
|
},
|
|
16
16
|
"bugs": {
|
|
17
|
-
"url": "https://github.com/sfls1397/
|
|
17
|
+
"url": "https://github.com/sfls1397/Apple-Tools-MCP/issues"
|
|
18
18
|
},
|
|
19
|
-
"homepage": "https://github.com/sfls1397/
|
|
19
|
+
"homepage": "https://github.com/sfls1397/Apple-Tools-MCP#readme",
|
|
20
20
|
"keywords": [
|
|
21
21
|
"mcp",
|
|
22
22
|
"model-context-protocol",
|
|
@@ -34,6 +34,12 @@
|
|
|
34
34
|
"engines": {
|
|
35
35
|
"node": ">=18.0.0"
|
|
36
36
|
},
|
|
37
|
+
"os": [
|
|
38
|
+
"darwin"
|
|
39
|
+
],
|
|
40
|
+
"publishConfig": {
|
|
41
|
+
"access": "public"
|
|
42
|
+
},
|
|
37
43
|
"files": [
|
|
38
44
|
"index.js",
|
|
39
45
|
"indexer.js",
|
|
@@ -46,61 +52,28 @@
|
|
|
46
52
|
],
|
|
47
53
|
"scripts": {
|
|
48
54
|
"start": "node index.js",
|
|
49
|
-
"build-index": "
|
|
50
|
-
"audit": "
|
|
55
|
+
"build-index": "node -e \"import('./indexer.js').then(i=>i.rebuildIndex()).catch(e=>{console.error(e.message);process.exit(1)})\"",
|
|
56
|
+
"audit": "node scripts/audit-index.js",
|
|
51
57
|
"test": "vitest run",
|
|
52
58
|
"test:watch": "vitest",
|
|
53
|
-
"test:coverage": "vitest run --coverage",
|
|
54
59
|
"test:unit": "vitest run tests/unit",
|
|
55
60
|
"test:integration": "vitest run tests/integration",
|
|
56
|
-
"test:
|
|
57
|
-
"test:perf": "vitest run tests/performance",
|
|
58
|
-
"test:fuzz": "vitest run tests/fuzz",
|
|
59
|
-
"test:chaos": "vitest run tests/chaos",
|
|
60
|
-
"test:contract": "vitest run tests/contract",
|
|
61
|
-
"test:concurrency": "vitest run tests/concurrency",
|
|
62
|
-
"test:recovery": "vitest run tests/recovery",
|
|
63
|
-
"test:timezone": "vitest run tests/timezone",
|
|
64
|
-
"test:stress": "vitest run tests/stress",
|
|
65
|
-
"test:snapshot": "vitest run tests/snapshots",
|
|
66
|
-
"test:all": "vitest run",
|
|
61
|
+
"test:coverage": "vitest run --coverage",
|
|
67
62
|
"test:idx": "vitest run tests/indexing",
|
|
68
|
-
"test:idx:unit": "vitest run tests/indexing/unit",
|
|
69
|
-
"test:idx:integration": "vitest run tests/indexing/integration",
|
|
70
63
|
"test:idx:accuracy": "vitest run tests/indexing/accuracy",
|
|
71
|
-
"test:idx:perf": "vitest run tests/indexing/performance",
|
|
72
64
|
"test:idx:resource": "vitest run tests/indexing/resource",
|
|
73
|
-
"test:
|
|
74
|
-
"
|
|
75
|
-
"
|
|
76
|
-
"
|
|
77
|
-
"
|
|
78
|
-
"
|
|
79
|
-
"
|
|
80
|
-
"
|
|
81
|
-
"perf": "
|
|
82
|
-
"perf:
|
|
83
|
-
"perf:
|
|
84
|
-
"perf:
|
|
85
|
-
"perf:search": "vitest run tests/perf/search.perf.test.js --reporter=verbose",
|
|
86
|
-
"perf:tools": "vitest run tests/perf/tools.perf.test.js --reporter=verbose",
|
|
87
|
-
"perf:server": "vitest run tests/perf/mcp-server.perf.test.js --reporter=verbose",
|
|
88
|
-
"perf:embedding": "vitest run tests/perf/embedding.perf.test.js --reporter=verbose",
|
|
89
|
-
"perf:datasources": "vitest run tests/perf/datasources.perf.test.js --reporter=verbose",
|
|
90
|
-
"perf:memory": "vitest run tests/perf/memory.perf.test.js --reporter=verbose",
|
|
91
|
-
"perf:stress": "vitest run tests/perf/stress.perf.test.js --reporter=verbose --testTimeout=120000",
|
|
92
|
-
"perf:mail": "vitest run tests/perf/datasources.perf.test.js -t 'Email' --reporter=verbose",
|
|
93
|
-
"perf:messages": "vitest run tests/perf/datasources.perf.test.js -t 'Messages' --reporter=verbose",
|
|
94
|
-
"perf:calendar": "vitest run tests/perf/datasources.perf.test.js -t 'Calendar' --reporter=verbose",
|
|
95
|
-
"perf:contacts": "vitest run tests/perf/datasources.perf.test.js -t 'Contacts' --reporter=verbose",
|
|
96
|
-
"perf:quick": "vitest run tests/perf/tools.perf.test.js tests/perf/search.perf.test.js --reporter=verbose",
|
|
97
|
-
"perf:negative": "vitest run tests/perf/negative.perf.test.js --reporter=verbose",
|
|
98
|
-
"perf:edge-cases": "vitest run tests/perf/edge-cases.perf.test.js --reporter=verbose",
|
|
99
|
-
"perf:regression": "vitest run tests/perf/regression.perf.test.js --reporter=verbose",
|
|
100
|
-
"perf:lancedb": "vitest run tests/perf/lancedb.perf.test.js --reporter=verbose",
|
|
101
|
-
"perf:background": "vitest run tests/perf/background-indexing.perf.test.js --reporter=verbose",
|
|
102
|
-
"perf:dates": "vitest run tests/perf/date-parsing.perf.test.js --reporter=verbose",
|
|
103
|
-
"perf:full": "vitest run tests/perf --reporter=verbose --testTimeout=120000"
|
|
65
|
+
"test:security": "vitest run tests/indexing/security",
|
|
66
|
+
"perf": "vitest run tests/perf",
|
|
67
|
+
"perf:quick": "vitest run tests/perf/tools.perf.test.js",
|
|
68
|
+
"perf:embedding": "vitest run tests/perf/embedding.perf.test.js",
|
|
69
|
+
"perf:stress": "vitest run tests/perf/stress.perf.test.js",
|
|
70
|
+
"perf:indexing": "vitest run tests/perf/indexing.perf.test.js",
|
|
71
|
+
"perf:search": "vitest run tests/perf/search.perf.test.js",
|
|
72
|
+
"perf:negative": "vitest run tests/perf/negative.perf.test.js",
|
|
73
|
+
"perf:regression": "vitest run tests/perf/regression.perf.test.js",
|
|
74
|
+
"perf:lancedb": "vitest run tests/perf/lancedb.perf.test.js",
|
|
75
|
+
"perf:background": "vitest run tests/perf/background-indexing.perf.test.js",
|
|
76
|
+
"perf:dates": "vitest run tests/perf/date-parsing.perf.test.js"
|
|
104
77
|
},
|
|
105
78
|
"dependencies": {
|
|
106
79
|
"@lancedb/lancedb": "^0.22.3",
|
|
@@ -109,8 +82,8 @@
|
|
|
109
82
|
"chrono-node": "^2.9.0"
|
|
110
83
|
},
|
|
111
84
|
"devDependencies": {
|
|
112
|
-
"
|
|
113
|
-
"
|
|
114
|
-
"
|
|
85
|
+
"vitest": "^2.1.0",
|
|
86
|
+
"@vitest/coverage-v8": "^2.1.0",
|
|
87
|
+
"fast-check": "^3.22.0"
|
|
115
88
|
}
|
|
116
89
|
}
|
package/search.js
CHANGED
|
@@ -826,7 +826,7 @@ export async function getRecentEmailResults(limit = 30, daysBack = 7, unreadOnly
|
|
|
826
826
|
const subject = (r.subject || "").trim().toLowerCase();
|
|
827
827
|
// Index from field has two formats:
|
|
828
828
|
// 1. "Coinbase via Cloaked (Coinbase)" - just display name
|
|
829
|
-
// 2. "
|
|
829
|
+
// 2. "Jane Smith via Cloaked (Company)" <email@domain.com> - display name + email
|
|
830
830
|
// Extract just the display name from both formats
|
|
831
831
|
const fromRaw = (r.from || "").trim().toLowerCase();
|
|
832
832
|
const fromName = fromRaw.replace(/<[^>]+>$/, "").trim().replace(/^"|"$/g, "");
|