@iovdin/maik 1.0.0

package/.github/workflows/publish.yml

@@ -0,0 +1,27 @@
+name: Publish Package
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  id-token: write  # Required for OIDC
+  contents: read
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          registry-url: 'https://registry.npmjs.org'
+
+      # Ensure npm 11.5.1 or later is installed
+      - name: Update npm
+        run: npm install -g npm@latest
+      - run: npm i
+      - run: npm publish

package/README.md

@@ -0,0 +1,256 @@
+# Maik - CLI Email Client
+
+A command-line tool to download and index emails from an IMAP server.
+
+## Installation
+
+```bash
+npm install -g maik
+```
+
+## Configuration
+
+Create a configuration file at `~/.config/maik/config.txt`:
+
+> **Gmail users:** To get an app password, visit [https://myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords). You need to have 2-Step Verification enabled. Use the generated app password instead of your regular Gmail password.
+
+```
+# IMAP server settings
+host = imap.gmail.com
+user = your-email@example.com
+password = your-password
+
+# Output directory for downloaded emails
+output = ~/mail
+```
+
+## Usage
+
+### Download emails
+
+```bash
+# it might take a long time and a lot of space locally
+maik fetch
+
+```
+
+### Index emails
+
+After downloading emails, create a SQLite index for fast searching:
+
+```bash
+maik index
+```
+
+The index is stored in `~/<output>/index.sqlite` with the following schema:
+
+```sql
+CREATE TABLE emails (
+  uid INTEGER PRIMARY KEY,
+  date INTEGER,          -- Unix timestamp
+  sender TEXT,           -- Sender email address
+  sender_name TEXT,      -- Sender display name
+  "to" TEXT,             -- Recipients
+  subject TEXT,
+  size INTEGER,
+  message_id TEXT,
+  in_reply_to TEXT,
+  refs TEXT,             -- References header
+  cc TEXT,
+  bcc TEXT,
+  reply_to TEXT,
+  delivered_to TEXT,
+  attachments TEXT,      -- Attachments (filename:mimeType)
+  received_spf TEXT
+);
+
+```
+
+To query dates in human-readable format, use SQLite's datetime function:
+```sql
+SELECT uid, datetime(date, 'unixepoch') as date, sender, subject FROM emails;
+```
+
+## Email Storage
+
+Emails are stored in the format:
+```
+output_dir/
+  YYYY-MM/          # Year-Month folder
+    <uid>/          # Per-email folder
+      source.eml    # Raw email file
+```
+
+
+## crontab
+
+To download and index emails automatically, add the following to your crontab (`crontab -e`).
+Replace `your_username` with your actual macOS username and adjust the Node.js version path to match your environment (`node --version` to check):
+
+```crontab
+HOME=/Users/your_username
+PATH=/Users/your_username/.nvm/versions/node/v22.20.0/bin:/usr/local/bin:/usr/bin:/bin
+
+# Fetch new emails and rebuild the index every 5 minutes, logging output to ~/crontab.log
+*/5 * * * * maik fetch && maik index >> $HOME/crontab.log 2>&1
+```
+
+## Tune
+
+Add `maik` as a middleware in your [Tune](https://github.com/iovdin/tune) setup.
+
+Edit `~/.tune/default.ctx.js` and add `require("maik/middleware")` to your middleware array:
+
+```js
+const maik = require("maik/middleware")
+
+module.exports = [
+  // ... your other middleware ...
+  maik,
+  // ...
+]
+```
+
+The middleware allows Tune to read email bodies and attachments as files:
+
+```
+mail/YYYY-MM/<uid>/body.txt
+mail/YYYY-MM/<uid>/body.html
+mail/YYYY-MM/<uid>/<attachmentname.ext>
+```
+
+
+### Usage
+
+Use `mail.prompt` as a system prompt in your Tune chat. It automatically:
+- Loads the `rf` (read file) and `mistral_ocr` tools
+- Connects a `mail_query` tool wired to your `~/mail/index.sqlite` database
+- Injects the live email table schema
+- Reminds the model to always use `datetime(date, 'unixepoch')` for dates and to `LIMIT` queries
+
+```
+@rf
+@mistral_ocr
+
+@{ sqlite | curry filename=~/mail/index.sqlite $name=mail_query }
+You are mail assistant.
+
+<emails-table-schema>
+@{| proc sqlite filename=~/mail/index.sqlite text="PRAGMA table_info(emails);" }
+</emails-table-schema>
+
+You can read text of email from file
+mail/YYYY-MM/<uid>/body.txt
+or 
+mail/YYYY-MM/<uid>/body.html
+
+attachment are accessable by path
+mail/YYYY-MM/<uid>/<attachmentname.ext>
+
+
+Always transform date to datetime(date, 'unixepoch')
+...
+```
+
+here is a chat example
+
+```chat
+system: @@mail.prompt
+user:
+find emails from HomeNest (company I rent apartment from)
+
+assistant:
+
+tool_call: mail_query {"format":"table"}
+SELECT uid, datetime(date, 'unixepoch') as dt, sender, sender_name, subject 
+FROM emails 
+WHERE sender LIKE '%homenest%' 
+   OR sender_name LIKE '%homenest%' 
+   OR subject LIKE '%homenest%' 
+ORDER BY date DESC LIMIT 5;
+
+tool_result:
++--------+---------------------+---------------------------+-------------+--------------------------------------------+
+|  uid   |         dt          |          sender           | sender_name |                 subject                    |
++--------+---------------------+---------------------------+-------------+--------------------------------------------+
+| 285411 | 2026-05-18 14:22:10 | billing@homenestliving.com|             | Invoice period 1-6-2026 to 30-6-2026      |
+| 284992 | 2026-04-19 09:15:44 | billing@homenestliving.com|             | Invoice period 1-5-2026 to 31-5-2026      |
+| 284501 | 2026-04-05 11:02:18 | support@homenestliving.com|             | Maintenance notice                         |
+| 283876 | 2026-03-17 15:44:02 | billing@homenestliving.com|             | Invoice period 1-4-2026 to 30-4-2026      |
+| 282940 | 2026-02-12 10:28:33 | billing@homenestliving.com|             | Invoice period 1-3-2026 to 31-3-2026      |
++--------+---------------------+---------------------------+-------------+--------------------------------------------+
+
+assistant:
+Here are the latest emails from **HomeNest Living (billing@homenestliving.com)**:
+
+1. 2026-05-18 — Invoice period 1-6-2026 to 30-6-2026 (uid 285411)  
+2. 2026-04-19 — Invoice period 1-5-2026 to 31-5-2026 (uid 284992)  
+3. 2026-04-05 — Maintenance notice (uid 284501)  
+4. 2026-03-17 — Invoice period 1-4-2026 to 30-4-2026 (uid 283876)  
+5. 2026-02-12 — Invoice period 1-3-2026 to 31-3-2026 (uid 282940)  
+
+Would you like me to:
+- Open one of these?
+- Download the latest invoice attachment?
+- Show all emails from HomeNest?
+- Summarize billing history?
+
+user:
+can you read the latest one and parse the invoice
+
+assistant:
+
+tool_call: mail_query {"format":"table"}
+SELECT uid, datetime(date, 'unixepoch') as dt, subject, attachments 
+FROM emails 
+WHERE uid = 285411 LIMIT 1;
+
+tool_result:
++--------+---------------------+------------------------------------------+-------------------------------------------------------------------+
+|  uid   |         dt          |                 subject                  |                             attachments                            |
++--------+---------------------+------------------------------------------+-------------------------------------------------------------------+
+| 285411 | 2026-05-18 14:22:10 | Invoice period 1-6-2026 to 30-6-2026     | INV-2026-06-4587.pdf:application/pdf, INV-2026-06-4587.xml:application/xml |
++--------+---------------------+------------------------------------------+-------------------------------------------------------------------+
+
+assistant:
+
+tool_call: mistral_ocr {"filename":"mail/2026-05/285411/INV-2026-06-4587.pdf"}
+
+tool_result:
+
+HomeNest Living  
+<cut>  
+<cut>  
+Email: billing@homenestliving.com  
+Website: www.homenestliving.com  
+
+John Carter  
+<cut>  
+<cut>  
+United Kingdom  
+
+INVOICE  
+
+Property: Riverside Apartments, London  
+Invoice date: 18-05-2026  
+Subject: Invoice period 1-6-2026 to 30-6-2026  
+Due date: 05-06-2026  
+Invoice number: INV-2026-06-4587  
+Customer number: CUST-90421  
+
+The following amount is due:
+
+| Description      | Net (£) | VAT (%) | VAT (£) | Total (£) |
+|------------------|---------|---------|---------|-----------|
+| Rent             | 1,250.00| 0%      | 0.00    | 1,250.00  |
+| Service charges  | 150.00  | 0%      | 0.00    | 150.00    |
+| TOTAL            | 1,400.00|         | 0.00    | 1,400.00  |
+
+Please transfer the amount before the due date, referencing the invoice and customer number.
+
+IBAN: GB29NWBK60161331926819  
+Account name: Urban Living Group Ltd  
+BIC: NWBKGB2L  
+VAT number: GB123456789  
+Company number: 09876543
+```

package/cli.js

@@ -0,0 +1,68 @@
+#!/usr/bin/env node
+
+const { Command } = require('commander');
+const { 
+  loadConfig, 
+  mergeConfig, 
+  connectToImap, 
+  ensureDirectory, 
+  formatDate, 
+  getLatestUid,
+  downloadEmails,
+  parseEmails,
+  indexEmails
+} = require('./index.js');
+
+const program = new Command();
+
+program
+  .name('maik')
+  .description('CLI tool to download and index emails from IMAP server')
+  .version('1.0.0');
+
+// Fetch command
+program
+  .command('fetch')
+  .description('Download emails from IMAP server to local folder')
+  .option('-h, --host <host>', 'IMAP server hostname')
+  .option('-u, --user <user>', 'Email username')
+  .option('-p, --password <password>', 'Email password')
+  .option('-o, --output <directory>', 'Output directory for emails')
+  .option('-v, --verbose', 'Show detailed progress', false)
+  .option('--port <port>', 'IMAP server port', '993')
+  .option('--tls', 'Use TLS', true)
+  .option('--box <mailbox>', 'Mailbox to download', 'INBOX')
+  .option('--limit <number>', 'Limit number of emails to download', '0')
+  .action(async (options) => {
+    const config = await loadConfig();
+    const mergedOptions = mergeConfig(options, config);
+    await downloadEmails(mergedOptions);
+  });
+
+// Parse command
+program
+  .command('parse')
+  .description('Parse source.eml files and extract body.txt, body.html, and attachments')
+  .option('-o, --output <directory>', 'Mail folder directory')
+  .option('-v, --verbose', 'Show detailed progress', false)
+  .action(async (options) => {
+    const config = await loadConfig();
+    const mergedOptions = mergeConfig(options, config);
+    await parseEmails(mergedOptions);
+  });
+
+// Index command
+program
+  .command('index')
+  .description('Index .eml files into SQLite database')
+  .option('-o, --output <directory>', 'Mail folder directory')
+  .option('-v, --verbose', 'Show detailed progress', false)
+  .option('--clean', 'Remove existing index database and re-index everything', false)
+  .action(async (options) => {
+    const config = await loadConfig();
+    const mergedOptions = mergeConfig(options, config);
+    await indexEmails(mergedOptions);
+  });
+
+// Parse and run
+program.parse();

package/config.txt.example

@@ -0,0 +1,15 @@
+# Maik Email Client Configuration
+# Place this file at ~/.config/maik/config.txt
+
+# IMAP server settings
+host = imap.example.com
+user = your-email@example.com
+password = your-password
+
+# Output directory for downloaded emails
+output = ~/mail
+
+# Optional settings
+# port = 993
+# tls = true
+# box = INBOX

package/default.ctx.js

@@ -0,0 +1,82 @@
+const { loadConfig, getDb } =  require('./index.js')
+const path = require("path")
+const fs = require("fs")
+const PostalMime = require('postal-mime');
+let config
+
+module.exports = async function maik(name, args) {
+  if (!config) {
+    config = await loadConfig()
+  }
+
+  // should start with prefix
+  if (name.indexOf("mail/") !== 0) {
+    return
+  }
+
+  if (name === "mail/dir") {
+    return {
+      type: "text",
+      read: async () => config.output
+    }
+  }
+
+  const [_, month, uid, filename] = name.split("/")
+  const sourcePath = path.resolve(config.output, month, uid, "source.eml")
+
+  if (!fs.existsSync(sourcePath)) {
+    return
+  }
+
+  if (filename === "source.eml") {
+    return {
+      name,
+      type: "text",
+      source: "maik",
+      read: async() => fs.readFileSync(sourcePath, "utf8")
+    }
+  }
+
+  const content = fs.readFileSync(sourcePath, "utf8")
+
+  // Parse with PostalMime
+  const email = await PostalMime.parse(content);
+  let mimetype
+  let type
+  let value
+
+  if (filename === "body.txt") {
+    mimetype = "text/plain"
+    type = "text"
+    value = email.text
+  } else if (filename === "body.html"){ 
+    mimetype = "text/html"
+    type = "text"
+    value = email.html
+  } else {
+    const attachment = email.attachments.find(item => item.filename === filename)
+    if (attachment) {
+      mimetype = attachment.mimeType
+      value = Buffer.from(attachment.content)
+      if (mimetype.indexOf("text") === 0) {
+        type = "text"
+        value = value.toString("utf8")
+      } else if (mimetype.indexOf("image") === 0) {
+        type = "image"
+      } else {
+        type = "binary"
+      }
+    }
+  }
+
+  const result = {
+    name,
+    type,
+    mimetype,
+    source: "maik",
+    read: async () => value 
+  }
+  fs.writeFileSync("./image.jpg", await result.read())
+  // console.log(result, value)
+  return  result 
+}

package/index.js

@@ -0,0 +1,866 @@
+const { ImapFlow } = require('imapflow');
+const { mkdir, writeFile, readdir, stat, readFile, unlink } = require('fs/promises');
+const { existsSync, createReadStream } = require('fs');
+const path = require('path');
+const os = require('os');
+const Database = require('better-sqlite3');
+const PostalMime = require('postal-mime');
+
+function getConfigPath() {
+  return path.join(os.homedir(), '.config', 'maik', 'config.txt');
+}
+
+async function loadConfig() {
+  const configPath = getConfigPath();
+  const config = {};
+  
+  if (!existsSync(configPath)) {
+    return config;
+  }
+  
+  try {
+    const content = await readFile(configPath, 'utf-8');
+    const lines = content.split('\n');
+    
+    for (let line of lines) {
+      line = line.trim();
+      
+      // Skip comments and empty lines
+      if (!line || line.startsWith('#')) {
+        continue;
+      }
+      
+      const equalIndex = line.indexOf('=');
+      if (equalIndex > 0) {
+        const key = line.substring(0, equalIndex).trim().toLowerCase();
+        const value = line.substring(equalIndex + 1).trim();
+        
+        // Map config keys to options
+        switch (key) {
+          case 'host':
+            config.host = value;
+            break;
+          case 'user':
+          case 'username':
+            config.user = value;
+            break;
+          case 'password':
+          case 'pass':
+            config.password = value;
+            break;
+          case 'output':
+          case 'dir':
+          case 'directory':
+            config.output = value;
+            break;
+          case 'port':
+            config.port = value;
+            break;
+          case 'tls':
+            config.tls = value.toLowerCase() === 'true';
+            break;
+          case 'box':
+          case 'mailbox':
+            config.box = value;
+            break;
+        }
+      }
+    }
+  } catch (error) {
+    // Ignore errors, return empty config
+  }
+
+  if (config.output) {
+    config.output = expandHome(config.output)
+  }
+  
+  return config;
+}
+
+function expandHome(p) {
+  if (!p) return p;
+  // Only expand a leading ~ or ~/...
+  if (p === '~') return os.homedir();
+  if (p.startsWith('~/') || p.startsWith('~\\')) {
+    return path.join(os.homedir(), p.slice(2));
+  }
+  return p;
+}
+
+function mergeConfig(options, config) {
+  return {
+    host: options.host || config.host,
+    user: options.user || config.user,
+    password: options.password || config.password,
+    output: expandHome(options.output || config.output),
+    port: options.port || config.port || '993',
+    tls: options.tls !== undefined ? options.tls : (config.tls !== undefined ? config.tls : true),
+    box: options.box || config.box || 'INBOX',
+    limit: options.limit || '0',
+    verbose: options.verbose || false,
+    clean: options.clean || false
+  };
+}
+
+async function connectToImap(options) {
+  // Validate required options
+  if (!options.host) {
+    console.error('Error: host is required (set in config or pass via --host)');
+    process.exit(1);
+  }
+  if (!options.user) {
+    console.error('Error: user is required (set in config or pass via --user)');
+    process.exit(1);
+  }
+  if (!options.password) {
+    console.error('Error: password is required (set in config or pass via --password)');
+    process.exit(1);
+  }
+
+  const client = new ImapFlow({
+    host: options.host,
+    port: parseInt(options.port),
+    secure: options.tls,
+    auth: {
+      user: options.user,
+      pass: options.password
+    },
+    logger: {
+      debug: () => {},
+      info: () => {},
+      warn: (msg) => console.log('Warning:', msg),
+      error: (msg) => console.log('Error:', msg)
+    }
+  });
+
+  console.log(`Connecting to ${options.host}:${options.port}...`);
+  console.log(`User: ${options.user}`);
+  
+  try {
+    await client.connect();
+    console.log('Connected successfully!');
+    return client;
+  } catch (error) {
+    console.error('\nConnection failed!');
+    console.error('Error details:', error.message || error);
+    throw error;
+  }
+}
+
+async function ensureDirectory(dir) {
+  if (!existsSync(dir)) {
+    await mkdir(dir, { recursive: true });
+  }
+}
+
+function formatDate(date) {
+  const year = date.getFullYear();
+  const month = String(date.getMonth() + 1).padStart(2, '0');
+  return `${year}-${month}`;
+}
+
+async function getLatestUid(outputDir) {
+  if (!existsSync(outputDir)) {
+    return 0;
+  }
+  
+  let maxUid = 0;
+  
+  try {
+    const dateFolders = await readdir(outputDir);
+    
+    for (const folder of dateFolders) {
+      const folderPath = path.join(outputDir, folder);
+      const folderStats = await stat(folderPath);
+      
+      // Skip if not a directory
+      if (!folderStats.isDirectory()) {
+        continue;
+      }
+      
+      const files = await readdir(folderPath);
+      
+      for (const file of files) {
+        // Extract UID from filename (format: uid.eml)
+        const match = file.match(/^(\d+)\.eml$/);
+        if (match) {
+          const uid = parseInt(match[1]);
+          if (uid > maxUid) {
+            maxUid = uid;
+          }
+        }
+      }
+    }
+  } catch (error) {
+    // Ignore errors, return 0
+  }
+  
+  return maxUid;
+}
+
+async function getLatestUidNew(outputDir) {
+  if (!existsSync(outputDir)) {
+    return 0;
+  }
+  
+  let maxUid = 0;
+  
+  try {
+    const dateFolders = await readdir(outputDir);
+    
+    for (const folder of dateFolders) {
+      const folderPath = path.join(outputDir, folder);
+      const folderStats = await stat(folderPath);
+      
+      // Skip if not a directory
+      if (!folderStats.isDirectory()) {
+        continue;
+      }
+      
+      // Look for UID directories (not .eml files anymore)
+      const uidDirs = await readdir(folderPath);
+      
+      for (const uidDir of uidDirs) {
+        // Check if it's a numeric directory name (UID)
+        const uidNum = parseInt(uidDir);
+        if (!isNaN(uidNum) && uidNum > 0) {
+          const uidPath = path.join(folderPath, uidDir);
+          try {
+            const uidStats = await stat(uidPath);
+            if (uidStats.isDirectory() && uidNum > maxUid) {
+              maxUid = uidNum;
+            }
+          } catch {
+            // Ignore errors
+          }
+        }
+      }
+    }
+  } catch (error) {
+    // Ignore errors, return 0
+  }
+  
+  return maxUid;
+}
+
+async function saveEmailSource(source, uidDir, verbose) {
+  // Save original source only
+  const sourcePath = path.join(uidDir, 'source.eml');
+  await writeFile(sourcePath, source);
+  if (verbose) console.log(`  -> source.eml (${source.length} bytes)`);
+}
+
+async function extractEmailParts(uidDir, verbose) {
+  const sourcePath = path.join(uidDir, 'source.eml');
+  
+  if (!existsSync(sourcePath)) {
+    return null;
+  }
+  
+  // Read and parse the email with PostalMime
+  const source = await readFile(sourcePath);
+  const email = await PostalMime.parse(source);
+  
+  // Save text body
+  if (email.text) {
+    const textPath = path.join(uidDir, 'body.txt');
+    await writeFile(textPath, email.text);
+    if (verbose) console.log(`  -> body.txt (${email.text.length} chars)`);
+  }
+  
+  // Save html body
+  if (email.html) {
+    const htmlPath = path.join(uidDir, 'body.html');
+    await writeFile(htmlPath, email.html);
+    if (verbose) console.log(`  -> body.html (${email.html.length} chars)`);
+  }
+  
+  // Save attachments
+  let attachmentCount = 0;
+  if (email.attachments && email.attachments.length > 0) {
+    for (const att of email.attachments) {
+      const filename = att.filename || `attachment-${attachmentCount + 1}`;
+      const attPath = path.join(uidDir, filename);
+      
+      // Handle filename conflicts
+      let finalPath = attPath;
+      let counter = 1;
+      while (existsSync(finalPath)) {
+        const ext = path.extname(filename);
+        const base = path.basename(filename, ext);
+        finalPath = path.join(uidDir, `${base}-${counter}${ext}`);
+        counter++;
+      }
+      
+      await writeFile(finalPath, att.content);
+      attachmentCount++;
+      if (verbose) console.log(`  -> ${path.basename(finalPath)} (${att.mimeType}, ${att.content.length} bytes)`);
+    }
+  }
+  
+  return {
+    hasText: !!email.text,
+    hasHtml: !!email.html,
+    attachmentCount,
+    subject: email.subject || 'No Subject'
+  };
+}
+
+async function downloadEmails(options) {
+  // Validate output directory
+  if (!options.output) {
+    console.error('Error: output directory is required (set in config or pass via --output)');
+    process.exit(1);
+  }
+
+  const client = await connectToImap(options);
+  
+  try {
+    // Select mailbox
+    await client.mailboxOpen(options.box);
+    console.log(`Opened mailbox: ${options.box}`);
+    
+    // Get total message count
+    const status = await client.status(options.box, { messages: true });
+    const totalMessages = status.messages || 0;
+    console.log(`Total messages in ${options.box}: ${totalMessages}`);
+    
+    if (totalMessages === 0) {
+      console.log('No messages to download.');
+      return;
+    }
+    
+    // Scan for latest UID
+    console.log('Scanning for existing emails...');
+    const latestUid = await getLatestUidNew(options.output);
+    
+    if (latestUid > 0) {
+      console.log(`Latest downloaded UID: ${latestUid}`);
+    } else {
+      console.log('No existing emails found, will download all');
+    }
+    
+    const limit = parseInt(options.limit);
+    let processed = 0;
+    let saved = 0;
+    let lastValidDate = null;
+    const now = new Date();
+    
+    // Create output directory
+    await ensureDirectory(options.output);
+    
+    // Determine fetch range - only fetch UIDs greater than latestUid
+    const fetchRange = latestUid > 0 ? `${latestUid + 1}:*` : '1:*';
+    
+    // Fetch messages
+    for await (const message of client.fetch(fetchRange, { 
+      envelope: true, 
+      source: true,
+      internalDate: true
+    }, { 
+      uid: true
+    })) {
+      try {
+        processed++;
+        
+        // Get date from internal date or envelope
+        let emailDate = message.internalDate || new Date(message.envelope.date);
+        
+        if (isNaN(emailDate.getTime())) {
+          console.log(`Skipping message ${processed} - invalid date`);
+          continue;
+        }
+        
+        // Check if date is in the future
+        if (emailDate > now) {
+          if (lastValidDate) {
+            if (options.verbose) {
+              console.log(`  Date ${emailDate.toISOString()} is in the future, using ${lastValidDate.toISOString()}`);
+            }
+            emailDate = lastValidDate;
+          } else {
+            // If no last valid date, use current date
+            if (options.verbose) {
+              console.log(`  Date is in the future, using current date`);
+            }
+            emailDate = now;
+          }
+        } else {
+          // Update last valid date
+          lastValidDate = emailDate;
+        }
+        
+        const dateFolder = formatDate(emailDate);
+        const datePath = path.join(options.output, dateFolder);
+        
+        // Create date folder
+        await ensureDirectory(datePath);
+        
+        // Create UID directory
+        const uid = message.uid;
+        const uidDir = path.join(datePath, String(uid));
+        await ensureDirectory(uidDir);
+        
+        // Save source.eml only
+        const subject = message.envelope.subject || 'No Subject';
+        await saveEmailSource(message.source, uidDir, options.verbose);
+        
+        saved++;
+        console.log(`Saved [${saved}]: ${dateFolder}/${uid} - "${subject.substring(0, 50)}"`);
+        
+        // Check limit
+        if (limit > 0 && saved >= limit) {
+          console.log(`\nReached limit of ${limit} emails`);
+          break;
+        }
+        
+      } catch (error) {
+        console.error(`Error processing message:`, error);
+      }
+    }
+    
+    console.log(`\nDownload complete!`);
+    console.log(`Downloaded: ${saved} new messages`);
+    if (latestUid > 0 && saved === 0) {
+      console.log('No new emails to download (already up to date)');
+    }
+    
+  } finally {
+    await client.logout();
+    console.log('\nDisconnected from server.');
+  }
+}
+
+// Email parsing utilities using PostalMime
+async function parseEmailHeadersFast(filePath) {
+  // Read the email file
+  const emailContent = await readFile(filePath, 'utf-8');
+  
+  // Parse with PostalMime
+  const email = await PostalMime.parse(emailContent);
+  
+  // Extract sender info
+  const sender = email.from ? (typeof email.from === 'object' && 'address' in email.from ? email.from.address : '') : '';
+  const sender_name = email.from && typeof email.from === 'object' && 'name' in email.from ? email.from.name : '';
+  
+  // Extract recipients
+  const to = email.to 
+    ? (Array.isArray(email.to) 
+        ? email.to.map(addr => typeof addr === 'object' && 'address' in addr ? addr.address : '').join(', ')
+        : '')
+    : '';
+  
+  const cc = email.cc 
+    ? (Array.isArray(email.cc) 
+        ? email.cc.map(addr => typeof addr === 'object' && 'address' in addr ? addr.address : '').join(', ')
+        : '')
+    : '';
+  
+  const bcc = email.bcc 
+    ? (Array.isArray(email.bcc) 
+        ? email.bcc.map(addr => typeof addr === 'object' && 'address' in addr ? addr.address : '').join(', ')
+        : '')
+    : '';
+  
+  const reply_to = email.replyTo 
+    ? (Array.isArray(email.replyTo) 
+        ? email.replyTo.map(addr => typeof addr === 'object' && 'address' in addr ? addr.address : '').join(', ')
+        : (typeof email.replyTo === 'object' && 'address' in email.replyTo ? email.replyTo.address : ''))
+    : '';
+  
+  // Extract attachments (filename + mimeType)
+  const attachments = email.attachments 
+    ? email.attachments.map(att => `${att.filename || 'unnamed'}:${att.mimeType}`).join(', ')
+    : '';
+  
+  // Extract received-spf from headers
+  const received_spf_header = email.headers.find(h => h.key === 'received-spf');
+  const received_spf = received_spf_header ? received_spf_header.value : '';
+  
+  return {
+    date: email.date || '',
+    sender,
+    sender_name,
+    to,
+    subject: email.subject || '',
+    message_id: email.messageId || '',
+    in_reply_to: email.inReplyTo || '',
+    refs: email.references || '',
+    cc,
+    bcc,
+    reply_to,
+    delivered_to: email.deliveredTo || '',
+    attachments,
+    received_spf
+  };
+}
+
+function getDb(outputDir) {
+  const dbPath = path.join(outputDir, 'index.sqlite');
+  const db = new Database(dbPath);
+  db.pragma('journal_mode = WAL');
+  return db;
+}
+
+async function indexEmails(options) {
+  // Validate output directory
+  if (!options.output) {
+    console.error('Error: output directory is required (set in config or pass via --output)');
+    process.exit(1);
+  }
+
+  const outputDir = options.output;
+  const verbose = options.verbose;
+  const clean = options.clean;
+  
+  if (!existsSync(outputDir)) {
+    console.error(`Error: Directory '${outputDir}' does not exist`);
+    process.exit(1);
+  }
+  
+  const dbPath = path.join(outputDir, 'index.sqlite');
+  
+  // Handle --clean option: remove existing database
+  if (clean && existsSync(dbPath)) {
+    console.log('Clean mode: removing existing index database...');
+    
+    // Remove main database file
+    await unlink(dbPath);
+    
+    // Remove WAL and SHM files if they exist
+    const walPath = dbPath + '-wal';
+    const shmPath = dbPath + '-shm';
+    
+    if (existsSync(walPath)) {
+      await unlink(walPath);
+    }
+    if (existsSync(shmPath)) {
+      await unlink(shmPath);
+    }
+    
+    console.log('Database removed successfully.\n');
+  }
+  
+  const db = getDb(outputDir);
+  
+  // Create table if not exists
+  db.exec(`
+    CREATE TABLE IF NOT EXISTS emails (
+      uid INTEGER PRIMARY KEY,
+      date INTEGER,
+      sender TEXT,
+      sender_name TEXT,
+      "to" TEXT,
+      subject TEXT,
+      size INTEGER,
+      message_id TEXT,
+      in_reply_to TEXT,
+      refs TEXT,
+      cc TEXT,
+      bcc TEXT,
+      reply_to TEXT,
+      delivered_to TEXT,
+      attachments TEXT,
+      received_spf TEXT
+    )
+  `);
+  
+  // Create index on frequently queried columns
+  db.exec(`CREATE INDEX IF NOT EXISTS idx_emails_date ON emails(date)`);
+  db.exec(`CREATE INDEX IF NOT EXISTS idx_emails_sender ON emails(sender)`);
+  db.exec(`CREATE INDEX IF NOT EXISTS idx_emails_message_id ON emails(message_id)`);
+  db.exec(`CREATE INDEX IF NOT EXISTS idx_emails_in_reply_to ON emails(in_reply_to)`);
+  
+  // Get existing UIDs
+  const existingRows = db.prepare('SELECT uid FROM emails').all();
+  const existingUids = new Set(existingRows.map(row => row.uid));
+  
+  console.log(`Database: ${dbPath}`);
+  console.log(`Already indexed: ${existingUids.size} emails`);
+  
+  let totalFiles = 0;
+  let newFiles = 0;
+  let skippedFiles = 0;
+  let errorFiles = 0;
+  
+  // Prepare insert statement
+  const insertStmt = db.prepare(`
+    INSERT INTO emails (
+      uid, date, sender, sender_name, "to", subject, size,
+      message_id, in_reply_to, refs, cc, bcc, reply_to,
+      delivered_to, attachments, received_spf
+    ) VALUES (
+      @uid, @date, @sender, @sender_name, @to, @subject, @size,
+      @message_id, @in_reply_to, @refs, @cc, @bcc, @reply_to,
+      @delivered_to, @attachments, @received_spf
+    )
+  `);
+  
+  // Use transaction for batch inserts
+  const insertMany = db.transaction((emails) => {
+    for (const email of emails) {
+      insertStmt.run(email);
+    }
+  });
+  
+  // Collect emails to insert for batch processing
+  let batch = [];
+  
+  try {
+    const dateFolders = (await readdir(outputDir)).sort();
+    
+    for (const folder of dateFolders) {
+      const folderPath = path.join(outputDir, folder);
+      const folderStats = await stat(folderPath);
+      
+      // Skip if not a directory
+      if (!folderStats.isDirectory()) {
+        continue;
+      }
+      
+      // Look for UID directories
+      const uidDirs = (await readdir(folderPath)).sort();
+      
+      for (const uidDir of uidDirs) {
+        // Check if it's a numeric directory name (UID)
+        const uid = parseInt(uidDir);
+        if (isNaN(uid) || uid <= 0) {
+          continue;
+        }
+        
+        const uidPath = path.join(folderPath, uidDir);
+        let uidStats;
+        try {
+          uidStats = await stat(uidPath);
+        } catch {
+          continue;
+        }
+        
+        // Skip if not a directory
+        if (!uidStats.isDirectory()) {
+          continue;
+        }
+        
+        // Look for source.eml inside the UID directory
+        const sourcePath = path.join(uidPath, 'source.eml');
+        if (!existsSync(sourcePath)) {
+          if (verbose) {
+            console.log(`Skipping ${folder}/${uidDir} - no source.eml found`);
+          }
+          skippedFiles++;
+          continue;
+        }
+        
+        totalFiles++;
+        
+        // Check if already indexed
+        if (existingUids.has(uid)) {
+          if (verbose) {
+            console.log(`Skipping ${folder}/${uidDir} - already indexed`);
+          }
+          continue;
+        }
+        
+        // Parse the email file
+        try {
+          const fileStats = await stat(sourcePath);
+          const headers = await parseEmailHeadersFast(sourcePath);
+          
+          // Convert date string to Unix timestamp
+          let dateTimestamp = 0;
+          if (headers.date) {
+            const parsedDate = new Date(headers.date);
+            if (!isNaN(parsedDate.getTime())) {
+              dateTimestamp = Math.floor(parsedDate.getTime() / 1000);
+            }
+          }
+          
+          // Collect for batch insert
+          batch.push({
+            uid: uid,
+            date: dateTimestamp,
+            sender: headers.sender,
+            sender_name: headers.sender_name,
+            to: headers.to,
+            subject: headers.subject,
+            size: fileStats.size,
+            message_id: headers.message_id,
+            in_reply_to: headers.in_reply_to,
+            refs: headers.refs,
+            cc: headers.cc,
+            bcc: headers.bcc,
+            reply_to: headers.reply_to,
+            delivered_to: headers.delivered_to,
+            attachments: headers.attachments,
+            received_spf: headers.received_spf
+          });
+          
+          newFiles++;
+          
+          // Batch insert every 1000 emails
+          if (newFiles % 1000 === 0) {
+            console.log(`Parsed ${newFiles} new emails..., inserting`);
+            insertMany(batch);
+            batch = [];
+          }
+        } catch (error) {
+          console.error(`Error parsing ${folder}/${uidDir}/source.eml:`, error);
+          errorFiles++;
+        }
+      }
+    }
+    
+    // Insert remaining emails
+    if (batch.length > 0) {
+      insertMany(batch);
+    }
+    
+    // Get final count
+    console.log(`\nIndexing complete!`);
+    console.log(`Total .eml files found: ${totalFiles}`);
+    console.log(`New emails indexed: ${newFiles}`);
+    console.log(`Skipped (already indexed): ${totalFiles - newFiles - errorFiles}`);
+    if (errorFiles > 0) {
+      console.log(`Errors: ${errorFiles}`);
+    }
+    
+  } finally {
+    db.close();
+  }
+}
+
+async function parseEmails(options) {
+  // Validate output directory
+  if (!options.output) {
+    console.error('Error: output directory is required (set in config or pass via --output)');
+    process.exit(1);
+  }
+
+  const outputDir = options.output;
+  const verbose = options.verbose;
+  
+  if (!existsSync(outputDir)) {
+    console.error(`Error: Directory '${outputDir}' does not exist`);
+    process.exit(1);
+  }
+  
+  console.log(`Parsing emails in: ${outputDir}`);
+  
+  let totalDirs = 0;
+  let parsedDirs = 0;
+  let skippedDirs = 0;
+  let errorDirs = 0;
+  let totalAttachments = 0;
+  let totalText = 0;
+  let totalHtml = 0;
+  
+  try {
+    const dateFolders = (await readdir(outputDir)).sort();
+    
+    for (const folder of dateFolders) {
+      const folderPath = path.join(outputDir, folder);
+      const folderStats = await stat(folderPath);
+      
+      // Skip if not a directory
+      if (!folderStats.isDirectory()) {
+        continue;
+      }
+      
+      // Look for UID directories
+      const uidDirs = (await readdir(folderPath)).sort();
+      
+      for (const uidDir of uidDirs) {
+        // Check if it's a numeric directory name (UID)
+        const uid = parseInt(uidDir);
+        if (isNaN(uid) || uid <= 0) {
+          continue;
+        }
+        
+        const uidPath = path.join(folderPath, uidDir);
+        let uidStats;
+        try {
+          uidStats = await stat(uidPath);
+        } catch {
+          continue;
+        }
+        
+        // Skip if not a directory
+        if (!uidStats.isDirectory()) {
+          continue;
+        }
+        
+        // Check if source.eml exists
+        const sourcePath = path.join(uidPath, 'source.eml');
+        if (!existsSync(sourcePath)) {
+          if (verbose) {
+            console.log(`Skipping ${folder}/${uidDir} - no source.eml found`);
+          }
+          skippedDirs++;
+          continue;
+        }
+        
+        // Check if already parsed (has body.txt or body.html)
+        const bodyTxtPath = path.join(uidPath, 'body.txt');
+        const bodyHtmlPath = path.join(uidPath, 'body.html');
+        if (existsSync(bodyTxtPath) || existsSync(bodyHtmlPath)) {
+          if (verbose) {
+            console.log(`Skipping ${folder}/${uidDir} - already parsed`);
+          }
+          continue;
+        }
+        
+        totalDirs++;
+        
+        // Parse the email
+        try {
+          const result = await extractEmailParts(uidPath, verbose);
+          
+          if (result) {
+            parsedDirs++;
+            if (result.hasText) totalText++;
+            if (result.hasHtml) totalHtml++;
+            totalAttachments += result.attachmentCount;
+            
+            console.log(`Parsed [${parsedDirs}]: ${folder}/${uidDir} - "${result.subject.substring(0, 50)}"`);
+          }
+        } catch (error) {
+          console.error(`Error parsing ${folder}/${uidDir}:`, error);
+          errorDirs++;
+        }
+      }
+    }
+    
+    console.log(`\nParsing complete!`);
+    console.log(`Emails parsed: ${parsedDirs}`);
+    console.log(`  - with text body: ${totalText}`);
+    console.log(`  - with html body: ${totalHtml}`);
+    console.log(`  - total attachments: ${totalAttachments}`);
+    if (skippedDirs > 0) {
+      console.log(`Skipped (no source.eml): ${skippedDirs}`);
+    }
+    if (errorDirs > 0) {
+      console.log(`Errors: ${errorDirs}`);
+    }
+    
+  } catch (error) {
+    console.error('Error parsing emails:', error);
+    process.exit(1);
+  }
+}
+
+module.exports = {
+  loadConfig,
+  mergeConfig,
+  connectToImap,
+  ensureDirectory,
+  formatDate,
+  getLatestUid,
+  getLatestUidNew,
+  saveEmailSource,
+  extractEmailParts,
+  downloadEmails,
+  parseEmails,
+  indexEmails,
+  parseEmailHeadersFast,
+  getDb
+};

package/mail.prompt

@@ -0,0 +1,25 @@
+@rf 
+@mistral_ocr
+
+@{ sqlite | curry filename=~/mail/index.sqlite $name=mail_query }
+You are mail assistant. 
+
+<emails-table-schema>
+@{| proc sqlite filename=~/mail/index.sqlite text="PRAGMA table_info(emails);" }
+</emails-table-schema>
+
+Always transform date to datetime(date, 'unixepoch')
+
+You can read text of email from file
+mail/YYYY-MM/<uid>/body.txt
+or 
+mail/YYYY-MM/<uid>/body.html
+
+attachment are accessable by path
+mail/YYYY-MM/<uid>/<attachmentname.ext>
+
+original .eml files are located at
+@{mail/dir}/YYYY-MM/<uid>/source.eml
+
+
+the database is big, always add limit to query not to explode the context

package/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "@iovdin/maik",
+  "version": "1.0.0",
+  "description": "Command-line IMAP email client that downloads emails from your server and indexes them locally in SQLite for fast offline search and querying.",
+  "main": "index.js",
+  "bin": {
+    "maik": "cli.js"
+  },
+  "license": "MIT",
+  "author": "Ilya Ovdin <iovdin@gmail.com>",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/iovdin/maik.git"
+  },
+  "homepage": "https://github.com/iovdin/maik#readme",
+  "bugs": {
+    "url": "https://github.com/iovdin/maik/issues"
+  },
+  "keywords": [
+    "imap",
+    "email",
+    "cli",
+    "command-line",
+    "sqlite",
+    "offline",
+    "mail",
+    "index",
+    "search",
+    "download"
+  ],
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "type": "commonjs",
+  "exports": {
+    "./middleware": "./default.ctx.js"
+  },
+  "devDependencies": {
+    "@types/better-sqlite3": "^7.6.11"
+  },
+  "dependencies": {
+    "better-sqlite3": "^11.7.0",
+    "commander": "^14.0.3",
+    "imapflow": "^1.2.9",
+    "postal-mime": "^2.7.3"
+  }
+}