@nomadamas/mailcrawl 0.1.0 → 0.1.3

package/README.md

@@ -7,8 +7,8 @@ account as its mail transport, then maintains a local normalized archive with
 incremental synchronization, email-aware chunking, full-text search (FTS5),
 semantic vector search, and hybrid retrieval.
 
-> Status: concept scaffold. The architecture is documented before the first
-> implementation increment.
+> Status: first functional release candidate. Use the installation guide for
+> multilingual analyzer setup before indexing production mail.
 
 ## Product goal
 
@@ -26,20 +26,35 @@ The CLI owns email synchronization and indexes. Consumers such as AutoRAG,
 OpenClaw, MCP servers, Raycast, and custom scripts consume its stable JSON
 surface instead of opening the archive database.
 
-## Planned capabilities
+## Capabilities
 
 - Himalaya-backed IMAP/JMAP/Gmail/Microsoft Graph/Maildir access
 - Stable account/mailbox/message identity and cursor state
 - MIME normalization, HTML-to-text conversion, and quoted-reply handling
 - Email-aware, thread-aware chunking
 - Incremental archive, FTS5, and embedding updates
-- LanceDB vector storage with configurable local embedding providers
+- Local EmbeddingGemma vector storage with Transformers.js and ONNX Runtime
 - FTS, semantic, and hybrid search modes
 - JSON output, bounded diagnostics, `status`, `doctor`, and `repair`
 - No credential values in logs, diagnostics, or indexed metadata by default
 
-See [`docs/architecture.md`](docs/architecture.md) for the proposed design,
-data model, CLI contract, and implementation milestones.
+See [`docs/architecture.md`](docs/architecture.md) for the data model and CLI
+contract. Lexical analyzer changes invalidate language-specific FTS fields;
+run `mailcrawl sync` to rebuild them before multilingual search. Semantic model
+changes require `mailcrawl index` to create a new vector generation.
+
+When a lexical analyzer or its model changes, the stored analyzer fingerprint
+invalidates all language-specific FTS fields. Run a complete `mailcrawl sync`
+before multilingual search; the command re-analyzes existing messages and
+atomically records the new fingerprint. Embedding model changes are independent
+and require a new `mailcrawl index` generation.
+
+## Installation
+
+For the required Node setup, Kiwi model files, Go installation, Japanese and
+Chinese helper builds, environment variables, smoke tests, and license
+requirements, follow [`docs/multilingual-installation.md`](docs/multilingual-installation.md)
+before using multilingual indexing or search.
 
 ## Releasing
 
@@ -54,8 +69,12 @@ One-time setup:
    - Workflow filename: `release.yml`
    - Environment: `release`
 
-Then bump the version, push `main`, and publish a GitHub Release whose tag is `v<version>`. The workflow tests, builds, publishes with provenance, and attaches the tarball to the release.
+Then bump the version, push `main`, and publish a GitHub Release whose tag is
+`v<version>`. The workflow tests, builds, publishes with OIDC, and attaches the
+tarball to the release. Public npm provenance is unavailable while this source
+repository remains private.
 
 ## License
 
-MIT. This project is not yet a production release.
+MIT. See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) for runtime
+component licenses.

package/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,50 @@
+# Third-party notices
+
+mailcrawl is MIT-licensed. Runtime components retain their upstream licenses.
+
+The application source remains MIT. These notices are for bundled source and
+runtime dependencies; they do not relicense mailcrawl itself.
+
+## Kiwi / kiwi-nlp
+
+- Package: https://www.npmjs.com/package/kiwi-nlp
+- Source: https://github.com/bab2min/Kiwi
+- License: LGPL-2.1-or-later
+
+The Kiwi package is not modified by mailcrawl.
+
+## Japanese helper
+
+`tools/mailcrawl-ja` follows the Kagome helper design used by discrawl PR
+#180 and is built separately from the default Node package.
+Kagome does not currently provide an official Node binding.
+
+## Kagome and IPADIC
+
+- Kagome: https://github.com/ikawaha/kagome (MIT)
+- Kagome dictionary: https://github.com/ikawaha/kagome-dict
+  (IPADIC/ICOT terms apply)
+
+## GSE
+
+- Source: https://github.com/go-ego/gse
+- License: Apache-2.0
+
+`tools/mailcrawl-zh` uses the same persistent helper protocol as discrawl PR
+#180 and is built separately from the default Node package.
+The GSE repository mentions `gse-bind`, but no usable npm package was available
+at integration time, so this remains a separate Go helper.
+
+## EmbeddingGemma
+
+- Model: https://huggingface.co/onnx-community/embeddinggemma-300m-ONNX
+- Base model: Google EmbeddingGemma
+- Terms: https://ai.google.dev/gemma/terms
+
+The model is downloaded and executed locally through Transformers.js and ONNX
+Runtime. Model weights are not redistributed in the mailcrawl package.
+
+## Transformers.js
+
+- Source: https://github.com/huggingface/transformers.js
+- License: Apache-2.0

package/dist/archive.d.ts

@@ -2,27 +2,30 @@ import Database from "better-sqlite3";
 import type { ClassificationPolicy, Chunk, MailMessage, NormalizedMessage, SearchFilters, SearchHit, SyncReport } from "./types.js";
 export declare class Archive {
     readonly db: Database.Database;
+    private embedder?;
+    private lexical?;
+    private lexicalRebuildRequired;
     constructor(path?: string);
     close(): void;
     sync(messages: MailMessage[], policy?: ClassificationPolicy): Promise<SyncReport>;
-    searchBm25(query: string, filters?: SearchFilters, limit?: number): SearchHit[];
-    indexSemantic(): {
+    searchBm25(query: string, filters?: SearchFilters, limit?: number): Promise<SearchHit[]>;
+    indexSemantic(): Promise<{
         embedded: number;
         reused: number;
         archiveRevision: string;
-    };
-    indexSemanticGeneration(root: string): {
+    }>;
+    indexSemanticGeneration(root: string): Promise<{
         generation: string;
         embedded: number;
         reused: number;
-    };
+    }>;
     semanticGeneration(root: string): {
         generation: string;
         archiveRevision: string;
         vectorCount: number;
     };
-    searchSemantic(query: string, filters?: SearchFilters, limit?: number): SearchHit[];
-    searchHybrid(query: string, filters?: SearchFilters, limit?: number): SearchHit[];
+    searchSemantic(query: string, filters?: SearchFilters, limit?: number): Promise<SearchHit[]>;
+    searchHybrid(query: string, filters?: SearchFilters, limit?: number): Promise<SearchHit[]>;
     getMessage(messageId: string): NormalizedMessage | undefined;
     listAttachments(messageId?: string): Array<{
         attachmentId: string;
@@ -52,4 +55,6 @@ export declare class Archive {
     private removeMessage;
     private replaceMessageChunks;
     private revision;
+    private getEmbedder;
+    private searchLexicalTable;
 }

package/dist/archive.js

@@ -5,15 +5,30 @@ import { join } from "node:path";
 import { buildChunks } from "./chunk.js";
 import { normalizeMessage } from "./normalize.js";
 import { snippet } from "./util.js";
+import { createEmbedder, embeddingModelName } from "./embedding.js";
+import { createLexicalAnalyzers, languagesForText, lexicalFields, LEXICAL_ANALYZER_VERSION } from "./lexical.js";
 export class Archive {
     db;
+    embedder;
+    lexical;
+    lexicalRebuildRequired;
     constructor(path = ":memory:") {
         this.db = new Database(path);
         this.db.pragma("journal_mode = WAL");
         this.db.pragma("foreign_keys = ON");
         migrate(this.db);
+        this.lexicalRebuildRequired = this.db.prepare("SELECT version FROM lexical_index_meta WHERE id = 1").get() === undefined;
+        if (!this.lexicalRebuildRequired) {
+            const row = this.db.prepare("SELECT version FROM lexical_index_meta WHERE id = 1").get();
+            this.lexicalRebuildRequired = row.version !== LEXICAL_ANALYZER_VERSION;
+        }
+        if (this.lexicalRebuildRequired) {
+            for (const language of lexicalFields())
+                this.db.exec(`DELETE FROM chunks_fts_${language}`);
+        }
     }
     close() {
+        void this.lexical?.close();
         this.db.close();
     }
     async sync(messages, policy = {}) {
@@ -33,6 +48,25 @@ export class Archive {
         let updated = 0;
         let unchanged = 0;
         const touched = new Set();
+        const rebuildLexical = this.lexicalRebuildRequired;
+        const required = [...new Set(included.flatMap((message) => languagesForText(`${message.subject} ${message.text}`)))];
+        if (required.length)
+            this.lexical = await createLexicalAnalyzers(required);
+        const analyzedChunks = new Map();
+        for (const message of included) {
+            const oldHash = previous.get(message.providerKey);
+            if (!this.lexicalRebuildRequired && oldHash === message.normalizedHash)
+                continue;
+            for (const chunk of buildChunks(message)) {
+                const tokens = new Map();
+                for (const language of lexicalFields()) {
+                    tokens.set(language, languagesForText(chunk.text).includes(language)
+                        ? await this.lexical.tokenize(language, chunk.text)
+                        : "");
+                }
+                analyzedChunks.set(chunk.chunkId, tokens);
+            }
+        }
         const transaction = this.db.transaction((items) => {
             for (const message of items) {
                 const oldHash = previous.get(message.providerKey);
@@ -45,11 +79,15 @@ export class Archive {
                 if (oldHash !== message.normalizedHash)
                     touched.add(message.threadId);
                 this.upsertMessage(message);
-                if (oldHash !== message.normalizedHash)
-                    this.replaceMessageChunks(message);
+                if (rebuildLexical || oldHash !== message.normalizedHash)
+                    this.replaceMessageChunks(message, analyzedChunks);
             }
         });
         transaction(included);
+        this.db.prepare(`INSERT INTO lexical_index_meta(id, version, rebuilt_at)
+      VALUES (1, ?, CURRENT_TIMESTAMP)
+      ON CONFLICT(id) DO UPDATE SET version=excluded.version, rebuilt_at=excluded.rebuilt_at`).run(LEXICAL_ANALYZER_VERSION);
+        this.lexicalRebuildRequired = false;
         for (const message of excluded)
             this.removeMessage(message);
         const chunks = Number(this.db.prepare("SELECT COUNT(*) AS count FROM chunks").get().count);
@@ -62,43 +100,62 @@ export class Archive {
             excludedByReason: countExcluded(excluded, excludedCategories),
         };
     }
-    searchBm25(query, filters = {}, limit = 10) {
+    async searchBm25(query, filters = {}, limit = 10) {
         if (!query.trim())
             throw new Error("empty query");
-        const clauses = ["chunks_fts MATCH ?"];
-        const params = [literalFtsQuery(query)];
-        addFilters(clauses, params, filters);
-        const sql = `SELECT c.chunk_id, c.message_id, c.thread_id, c.account_id, c.mailbox,
-      m.subject, m.from_address, m.to_addresses, m.date, snippet(chunks_fts, 0, '[', ']', '…', 32) AS snippet,
-      bm25(chunks_fts, 8.0, 5.0, 2.0, 1.0, 1.0, 1.0, 0.5) AS score
-      FROM chunks_fts JOIN chunks c ON c.rowid = chunks_fts.rowid
-      JOIN messages m ON m.message_id = c.message_id
-      WHERE ${clauses.join(" AND ")} ORDER BY score LIMIT ?`;
-        params.push(limit);
-        return this.db.prepare(sql).all(...params).map((row) => hydrate(row, "bm25", query));
+        if (this.lexicalRebuildRequired && languagesForText(query).length) {
+            throw new Error("lexical indexes are stale; run sync before multilingual search");
+        }
+        const languages = languagesForText(query);
+        if (languages.length)
+            this.lexical ??= await createLexicalAnalyzers(languages);
+        const lists = [this.searchLexicalTable("chunks_fts", query, filters, limit * 2)];
+        for (const language of languagesForText(query)) {
+            lists.push(this.searchLexicalTable(`chunks_fts_${language}`, await this.lexical.tokenize(language, query), filters, limit * 2));
+        }
+        const merged = new Map();
+        const k = 60;
+        for (const list of lists) {
+            for (const [index, hit] of list.entries()) {
+                const rank = index + 1;
+                const prior = merged.get(hit.chunkId);
+                merged.set(hit.chunkId, prior ? { hit: prior.hit, score: prior.score + 1 / (k + rank) } : { hit, score: 1 / (k + rank) });
+            }
+        }
+        return [...merged.values()].sort((a, b) => b.score - a.score || a.hit.chunkId.localeCompare(b.hit.chunkId))
+            .slice(0, limit).map(({ hit, score }) => ({ ...hit, score }));
     }
-    indexSemantic() {
+    async indexSemantic() {
         const rows = this.db.prepare("SELECT chunk_id, text, content_hash FROM chunks ORDER BY chunk_id").all();
         let embedded = 0;
         let reused = 0;
         const upsert = this.db.prepare(`INSERT INTO semantic_vectors
-      (chunk_id, content_hash, vector) VALUES (?, ?, ?)
-      ON CONFLICT(chunk_id) DO UPDATE SET content_hash=excluded.content_hash, vector=excluded.vector`);
+      (chunk_id, content_hash, model, vector) VALUES (?, ?, ?, ?)
+      ON CONFLICT(chunk_id) DO UPDATE SET content_hash=excluded.content_hash, model=excluded.model, vector=excluded.vector`);
+        const embedder = await this.getEmbedder();
+        const pending = [];
         const transaction = this.db.transaction(() => {
             for (const row of rows) {
-                const old = this.db.prepare("SELECT content_hash FROM semantic_vectors WHERE chunk_id = ?").get(row.chunk_id);
-                if (old?.content_hash === row.content_hash) {
+                const old = this.db.prepare("SELECT content_hash, model FROM semantic_vectors WHERE chunk_id = ?").get(row.chunk_id);
+                if (old?.content_hash === row.content_hash && old.model === embeddingModelName()) {
                     reused++;
                     continue;
                 }
-                upsert.run(row.chunk_id, row.content_hash, JSON.stringify(embed(row.text)));
-                embedded++;
+                pending.push(row);
             }
         });
         transaction();
+        const vectors = await embedder.embedDocuments(pending.map((row) => row.text));
+        const write = this.db.transaction(() => {
+            for (const [index, row] of pending.entries()) {
+                upsert.run(row.chunk_id, row.content_hash, embeddingModelName(), JSON.stringify(vectors[index]));
+                embedded++;
+            }
+        });
+        write();
         return { embedded, reused, archiveRevision: this.revision() };
     }
-    indexSemanticGeneration(root) {
+    async indexSemanticGeneration(root) {
         const currentPath = join(root, "CURRENT");
         const generationRoot = join(root, "generations");
         mkdirSync(generationRoot, { recursive: true });
@@ -106,10 +163,10 @@ export class Archive {
         const staging = join(generationRoot, `.${generation}.staging`);
         mkdirSync(staging);
         try {
-            const report = this.indexSemantic();
+            const report = await this.indexSemantic();
             const vectors = this.db.prepare("SELECT chunk_id, content_hash, vector FROM semantic_vectors ORDER BY chunk_id").all();
             writeFileSync(join(staging, "manifest.json"), JSON.stringify({
-                archiveRevision: this.revision(), vectors, model: "local-hash-v1",
+                archiveRevision: this.revision(), vectors, model: embeddingModelName(),
             }));
             renameSync(staging, join(generationRoot, generation));
             const pointer = join(root, `.CURRENT.${process.pid}`);
@@ -127,10 +184,10 @@ export class Archive {
         const manifest = JSON.parse(readFileSync(join(root, "generations", generation, "manifest.json"), "utf8"));
         return { generation, archiveRevision: manifest.archiveRevision, vectorCount: manifest.vectors.length };
     }
-    searchSemantic(query, filters = {}, limit = 10) {
+    async searchSemantic(query, filters = {}, limit = 10) {
         if (!query.trim())
             throw new Error("empty query");
-        const queryVector = embed(query);
+        const queryVector = await (await this.getEmbedder()).embedQuery(query);
         const clauses = ["1 = 1"];
         const params = [];
         addFilters(clauses, params, filters);
@@ -148,18 +205,24 @@ export class Archive {
             snippet: snippet(row.text, query), score, mode: "semantic",
         }));
     }
-    searchHybrid(query, filters = {}, limit = 10) {
-        const lexical = this.searchBm25(query, filters, limit * 2);
-        const semantic = this.searchSemantic(query, filters, limit * 2);
+    async searchHybrid(query, filters = {}, limit = 10) {
+        const lexical = await this.searchBm25(query, filters, limit * 2);
+        const semantic = await this.searchSemantic(query, filters, limit * 2);
         const merged = new Map();
+        const k = 60;
         for (const [index, hit] of lexical.entries())
-            merged.set(hit.chunkId, { ...hit, score: 0.5 * (1 - index / Math.max(1, lexical.length)) });
+            merged.set(hit.chunkId, { hit: { ...hit, mode: "hybrid" }, score: 1 / (k + index + 1) });
         for (const [index, hit] of semantic.entries()) {
             const prior = merged.get(hit.chunkId);
-            const score = 0.5 * (1 - index / Math.max(1, semantic.length));
-            merged.set(hit.chunkId, prior ? { ...prior, score: prior.score + score, mode: "hybrid" } : { ...hit, score, mode: "hybrid" });
+            const score = 1 / (k + index + 1);
+            merged.set(hit.chunkId, prior
+                ? { hit: { ...prior.hit, mode: "hybrid" }, score: prior.score + score }
+                : { hit: { ...hit, mode: "hybrid" }, score });
         }
-        return [...merged.values()].sort((a, b) => b.score - a.score || a.chunkId.localeCompare(b.chunkId)).slice(0, limit);
+        return [...merged.values()]
+            .sort((a, b) => b.score - a.score || a.hit.chunkId.localeCompare(b.hit.chunkId))
+            .slice(0, limit)
+            .map(({ hit, score }) => ({ ...hit, score }));
     }
     getMessage(messageId) {
         const row = this.db.prepare("SELECT * FROM messages WHERE message_id = ?").get(messageId);
@@ -207,6 +270,8 @@ export class Archive {
         const rows = this.db.prepare("SELECT rowid, chunk_id, text, message_id FROM chunks").all();
         const rebuild = this.db.transaction(() => {
             this.db.exec("DELETE FROM chunks_fts");
+            for (const language of lexicalFields())
+                this.db.exec(`DELETE FROM chunks_fts_${language}`);
             for (const row of rows) {
                 const message = this.db.prepare("SELECT * FROM messages WHERE message_id = ?").get(row.message_id);
                 this.db.prepare(`INSERT INTO chunks_fts(rowid, subject, from_address, to_addresses, thread_subject,
@@ -215,6 +280,8 @@ export class Archive {
             }
         });
         rebuild();
+        this.db.prepare("DELETE FROM lexical_index_meta WHERE id = 1").run();
+        this.lexicalRebuildRequired = true;
         return { rows: rows.length, status: "repaired" };
     }
     upsertMessage(message) {
@@ -252,14 +319,18 @@ export class Archive {
         const rows = this.db.prepare("SELECT rowid FROM chunks WHERE message_id = ?").all(stored.message_id);
         for (const row of rows)
             this.db.prepare("DELETE FROM chunks_fts WHERE rowid = ?").run(row.rowid);
+        for (const language of lexicalFields())
+            this.db.prepare(`DELETE FROM chunks_fts_${language} WHERE chunk_id IN (SELECT chunk_id FROM chunks WHERE message_id = ?)`).run(stored.message_id);
         this.db.prepare("DELETE FROM embedding_queue WHERE chunk_id IN (SELECT chunk_id FROM chunks WHERE message_id = ?)").run(stored.message_id);
         this.db.prepare("DELETE FROM semantic_vectors WHERE chunk_id IN (SELECT chunk_id FROM chunks WHERE message_id = ?)").run(stored.message_id);
         this.db.prepare("DELETE FROM messages WHERE message_id = ?").run(stored.message_id);
     }
-    replaceMessageChunks(message) {
+    replaceMessageChunks(message, analyzedChunks) {
         const old = this.db.prepare("SELECT rowid, chunk_id FROM chunks WHERE message_id = ?").all(message.messageId);
         for (const row of old)
             this.db.prepare("DELETE FROM chunks_fts WHERE rowid = ?").run(row.rowid);
+        for (const language of lexicalFields())
+            this.db.prepare(`DELETE FROM chunks_fts_${language} WHERE chunk_id IN (SELECT chunk_id FROM chunks WHERE message_id = ?)`).run(message.messageId);
         this.db.prepare("DELETE FROM chunks WHERE message_id = ?").run(message.messageId);
         for (const chunk of buildChunks(message)) {
             const result = this.db.prepare(`INSERT INTO chunks
@@ -268,6 +339,10 @@ export class Archive {
             this.db.prepare(`INSERT INTO chunks_fts(rowid, subject, from_address, to_addresses, thread_subject,
         body_latest, body_quoted, forwarded_text, attachment_text)
         VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(result.lastInsertRowid, message.subject, message.from, message.to.join(" "), message.normalizedSubject, chunk.section === "latest" ? chunk.text : "", chunk.section === "quoted" ? chunk.text : "", "", chunk.section === "attachment" ? chunk.text : "");
+            for (const language of lexicalFields()) {
+                this.db.prepare(`INSERT INTO chunks_fts_${language}(chunk_id, text, subject, from_address, to_addresses)
+          VALUES (?, ?, ?, ?, ?)`).run(chunk.chunkId, analyzedChunks?.get(chunk.chunkId)?.get(language) ?? "", message.subject, message.from, message.to.join(" "));
+            }
             this.db.prepare(`INSERT INTO embedding_queue(chunk_id, content_hash, state, attempts)
         VALUES (?, ?, 'pending', 0) ON CONFLICT(chunk_id) DO UPDATE SET content_hash=excluded.content_hash, state='pending'`).run(chunk.chunkId, chunk.contentHash);
         }
@@ -276,6 +351,26 @@ export class Archive {
         const rows = this.db.prepare("SELECT chunk_id, content_hash FROM chunks ORDER BY chunk_id").all();
         return createHash("sha256").update(rows.map((row) => `${row.chunk_id}\0${row.content_hash}`).join("\0")).digest("hex");
     }
+    async getEmbedder() {
+        this.embedder ??= await createEmbedder();
+        return this.embedder;
+    }
+    searchLexicalTable(table, query, filters, limit) {
+        const clauses = [`${table} MATCH ?`];
+        const params = [literalFtsQuery(query)];
+        addFilters(clauses, params, filters);
+        const join = table === "chunks_fts"
+            ? `JOIN chunks c ON c.rowid = ${table}.rowid`
+            : `JOIN chunks c ON c.chunk_id = ${table}.chunk_id`;
+        const sql = `SELECT c.chunk_id, c.message_id, c.thread_id, c.account_id, c.mailbox,
+      m.subject, m.from_address, m.to_addresses, m.date,
+      bm25(${table}) AS score
+      FROM ${table} ${join}
+      JOIN messages m ON m.message_id = c.message_id
+      WHERE ${clauses.join(" AND ")} ORDER BY score LIMIT ?`;
+        params.push(limit);
+        return this.db.prepare(sql).all(...params).map((row) => hydrate(row, "bm25", query));
+    }
 }
 function migrate(db) {
     db.exec(`CREATE TABLE IF NOT EXISTS messages (
@@ -298,16 +393,30 @@ function migrate(db) {
     chunk_id TEXT PRIMARY KEY, content_hash TEXT NOT NULL, state TEXT NOT NULL, attempts INTEGER NOT NULL
   );
   CREATE TABLE IF NOT EXISTS semantic_vectors (
-    chunk_id TEXT PRIMARY KEY, content_hash TEXT NOT NULL, vector TEXT NOT NULL
+    chunk_id TEXT PRIMARY KEY, content_hash TEXT NOT NULL, model TEXT NOT NULL DEFAULT '', vector TEXT NOT NULL
   );
   CREATE TABLE IF NOT EXISTS attachments (
     attachment_id TEXT PRIMARY KEY, message_id TEXT NOT NULL REFERENCES messages(message_id) ON DELETE CASCADE,
     name TEXT NOT NULL, mime_type TEXT NOT NULL, size INTEGER, content_hash TEXT, extracted_text TEXT
   );
+  CREATE TABLE IF NOT EXISTS lexical_index_meta (
+    id INTEGER PRIMARY KEY CHECK (id = 1), version TEXT NOT NULL, rebuilt_at TEXT NOT NULL
+  );
   CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
     subject, from_address, to_addresses, thread_subject, body_latest,
-    body_quoted, forwarded_text, attachment_text
+    body_quoted, forwarded_text, attachment_text,
+    tokenize = 'unicode61'
   );`);
+    const vectorColumns = db.prepare("PRAGMA table_info(semantic_vectors)").all();
+    if (!vectorColumns.some((column) => column.name === "model"))
+        db.exec("ALTER TABLE semantic_vectors ADD COLUMN model TEXT NOT NULL DEFAULT ''");
+    for (const language of lexicalFields()) {
+        const table = `chunks_fts_${language}`;
+        db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS ${table} USING fts5(
+      chunk_id UNINDEXED, text, subject, from_address, to_addresses,
+      tokenize = 'unicode61'
+    );`);
+    }
     const columns = db.prepare("PRAGMA table_info(messages)").all();
     const existingColumns = new Set(columns.map((column) => column.name));
     for (const column of ["labels", "flags", "classifications"]) {

package/dist/cli/index.js

@@ -43,7 +43,7 @@ program
     const dataDir = command.parent.opts().dataDir;
     const archive = new Archive(`${dataDir}/archive.sqlite`);
     try {
-        output({ ...archive.indexSemanticGeneration(`${dataDir}/semantic`), embedder: "local-hash-v1" }, options.json);
+        output({ ...await archive.indexSemanticGeneration(`${dataDir}/semantic`), embedder: "onnx-community/embeddinggemma-300m-ONNX" }, options.json);
     }
     finally {
         archive.close();
@@ -67,10 +67,10 @@ for (const mode of ["bm25", "keyword", "semantic", "hybrid"]) {
         const archive = new Archive(`${dataDir}/archive.sqlite`);
         try {
             const result = mode === "bm25"
-                ? archive.searchBm25(query, filters(options), Number(options.limit))
+                ? await archive.searchBm25(query, filters(options), Number(options.limit))
                 : mode === "hybrid"
-                    ? archive.searchHybrid(query, filters(options), Number(options.limit))
-                    : archive.searchSemantic(query, filters(options), Number(options.limit));
+                    ? await archive.searchHybrid(query, filters(options), Number(options.limit))
+                    : await archive.searchSemantic(query, filters(options), Number(options.limit));
             output(result, options.json);
         }
         finally {
@@ -96,9 +96,9 @@ program
     try {
         const filter = filters(options);
         const limit = Number(options.limit);
-        const result = options.mode === "bm25" ? archive.searchBm25(query, filter, limit)
-            : options.mode === "semantic" ? archive.searchSemantic(query, filter, limit)
-                : options.mode === "hybrid" ? archive.searchHybrid(query, filter, limit)
+        const result = options.mode === "bm25" ? await archive.searchBm25(query, filter, limit)
+            : options.mode === "semantic" ? await archive.searchSemantic(query, filter, limit)
+                : options.mode === "hybrid" ? await archive.searchHybrid(query, filter, limit)
                     : (() => { throw new Error(`unsupported search mode: ${options.mode}`); })();
         output(result, options.json);
     }

package/dist/embedding.d.ts

@@ -0,0 +1,6 @@
+export interface Embedder {
+    embedDocuments(texts: string[]): Promise<number[][]>;
+    embedQuery(query: string): Promise<number[]>;
+}
+export declare function createEmbedder(): Promise<Embedder>;
+export declare function embeddingModelName(): string;

package/dist/embedding.js

@@ -0,0 +1,54 @@
+import { pipeline } from "@huggingface/transformers";
+const EMBEDDING_MODEL = "onnx-community/embeddinggemma-300m-ONNX";
+const QUERY_PREFIX = "task: search result | query: ";
+const DOCUMENT_PREFIX = "title: none | text: ";
+class EmbeddingGemma {
+    model;
+    constructor(model) {
+        this.model = model;
+    }
+    static async create() {
+        const model = await pipeline("feature-extraction", EMBEDDING_MODEL, {
+            dtype: "q8",
+            device: "cpu",
+        });
+        return new EmbeddingGemma(model);
+    }
+    async embedDocuments(texts) {
+        return this.embed(texts.map((text) => DOCUMENT_PREFIX + text));
+    }
+    async embedQuery(query) {
+        return (await this.embed([QUERY_PREFIX + query.trim()]))[0];
+    }
+    async embed(texts) {
+        const output = await this.model(texts, { pooling: "mean", normalize: true });
+        return output.tolist();
+    }
+}
+class TestEmbedder {
+    async embedDocuments(texts) {
+        return texts.map(hashVector);
+    }
+    async embedQuery(query) {
+        return hashVector(query);
+    }
+}
+export async function createEmbedder() {
+    if (process.env.MAILCRAWL_EMBEDDER === "mock" || process.env.NODE_ENV === "test")
+        return new TestEmbedder();
+    return EmbeddingGemma.create();
+}
+export function embeddingModelName() {
+    return EMBEDDING_MODEL;
+}
+function hashVector(text) {
+    const vector = new Array(128).fill(0);
+    for (const [index, term] of text.normalize("NFKC").toLocaleLowerCase().split(/\s+/u).entries()) {
+        let hash = 2166136261;
+        for (const char of term)
+            hash = Math.imul(hash ^ char.codePointAt(0), 16777619);
+        vector[Math.abs(hash + index) % vector.length] += 1;
+    }
+    const magnitude = Math.sqrt(vector.reduce((sum, value) => sum + value * value, 0)) || 1;
+    return vector.map((value) => value / magnitude);
+}

package/dist/lexical.d.ts

@@ -0,0 +1,10 @@
+import type { LexicalLanguage } from "./types.js";
+export declare const LEXICAL_ANALYZER_VERSION = "kiwi-nlp@0.23.x+kagome-ipa-search+gse-search+arabic-light-v1";
+export declare function lexicalFields(): LexicalLanguage[];
+export declare function languagesForText(text: string): LexicalLanguage[];
+export interface LexicalAnalyzers {
+    tokenize(language: LexicalLanguage, text: string): Promise<string>;
+    close(): Promise<void>;
+}
+export declare function createLexicalAnalyzers(required?: LexicalLanguage[]): Promise<LexicalAnalyzers>;
+export declare function tokenizeForLanguage(language: LexicalLanguage, text: string): string;

package/dist/lexical.js

@@ -0,0 +1,151 @@
+import { KiwiBuilder, Match } from "kiwi-nlp";
+import { spawn } from "node:child_process";
+import { createInterface } from "node:readline";
+import { createRequire } from "node:module";
+import { readFileSync, readdirSync } from "node:fs";
+export const LEXICAL_ANALYZER_VERSION = "kiwi-nlp@0.23.x+kagome-ipa-search+gse-search+arabic-light-v1";
+export function lexicalFields() {
+    return ["ko", "ja", "zh", "ar"];
+}
+export function languagesForText(text) {
+    const languages = [];
+    if (/[\uac00-\ud7a3]/u.test(text))
+        languages.push("ko");
+    if (/[\u3040-\u30ff]/u.test(text))
+        languages.push("ja");
+    if (/[\u4e00-\u9fff]/u.test(text))
+        languages.push("zh");
+    if (/[\u0600-\u06ff]/u.test(text))
+        languages.push("ar");
+    return languages;
+}
+export async function createLexicalAnalyzers(required = lexicalFields()) {
+    if (process.env.NODE_ENV === "test" || process.env.MAILCRAWL_LEXICAL_MODE === "mock") {
+        return { tokenize: async (language, text) => tokenizeMock(language, text), close: async () => undefined };
+    }
+    return new ConfiguredAnalyzers(required.includes("ko") ? await KiwiAnalyzer.create() : undefined, required.includes("ja") ? await HelperAnalyzer.create("ja", process.env.MAILCRAWL_JA_HELPER, "mailcrawl-ja") : undefined, required.includes("zh") ? await HelperAnalyzer.create("zh", process.env.MAILCRAWL_ZH_HELPER, "mailcrawl-zh") : undefined);
+}
+class ConfiguredAnalyzers {
+    korean;
+    japanese;
+    chinese;
+    constructor(korean, japanese, chinese) {
+        this.korean = korean;
+        this.japanese = japanese;
+        this.chinese = chinese;
+    }
+    tokenize(language, text) {
+        if (language === "ko" && this.korean)
+            return this.korean.tokenize(text);
+        if (language === "ja" && this.japanese)
+            return this.japanese.tokenize(text);
+        if (language === "zh" && this.chinese)
+            return this.chinese.tokenize(text);
+        return Promise.resolve(tokenizeArabic(text));
+    }
+    async close() {
+        await Promise.all([this.korean?.close(), this.japanese?.close(), this.chinese?.close()]);
+    }
+}
+class KiwiAnalyzer {
+    kiwi;
+    constructor(kiwi) {
+        this.kiwi = kiwi;
+    }
+    static async create() {
+        const wasmPath = process.env.MAILCRAWL_KIWI_WASM
+            ?? createRequire(import.meta.url).resolve("kiwi-nlp/dist/kiwi-wasm.wasm");
+        const modelDir = process.env.MAILCRAWL_KIWI_MODEL;
+        if (!wasmPath || !modelDir) {
+            throw new Error("Korean analyzer requires MAILCRAWL_KIWI_WASM and MAILCRAWL_KIWI_MODEL");
+        }
+        const builder = await KiwiBuilder.create(wasmPath);
+        const modelFiles = Object.fromEntries(readdirSync(modelDir, { withFileTypes: true })
+            .filter((entry) => entry.isFile())
+            .map((entry) => [entry.name, readFileSync(`${modelDir}/${entry.name}`)]));
+        if (!Object.keys(modelFiles).length)
+            throw new Error(`Korean Kiwi model directory is empty: ${modelDir}`);
+        return new KiwiAnalyzer(await builder.build({ modelFiles, modelType: "cong", loadDefaultDict: true, loadTypoDict: true }));
+    }
+    async tokenize(text) {
+        return this.kiwi.tokenize(text, Match.allWithNormalizing).map((token) => token.str).join(" ");
+    }
+    async close() { }
+}
+class HelperAnalyzer {
+    language;
+    process;
+    lines;
+    constructor(language, process, lines) {
+        this.language = language;
+        this.process = process;
+        this.lines = lines;
+    }
+    static async create(language, command, fallbackName) {
+        const resolved = command || fallbackName;
+        const process = spawn(resolved, [], { stdio: ["pipe", "pipe", "pipe"] });
+        const lines = createInterface({ input: process.stdout });
+        const analyzer = new HelperAnalyzer(language, process, lines);
+        const startup = await analyzer.readLine();
+        const response = JSON.parse(startup);
+        if (response.error)
+            throw new Error(`${fallbackName}: ${response.error}`);
+        if (!response.ready)
+            throw new Error(`${fallbackName} did not report ready`);
+        return analyzer;
+    }
+    async tokenize(text) {
+        return this.request(text);
+    }
+    async close() {
+        this.lines.close();
+        this.process.kill();
+    }
+    request(text) {
+        return new Promise((resolve, reject) => {
+            const onError = (error) => reject(error);
+            const onLine = (line) => {
+                try {
+                    const response = JSON.parse(line);
+                    this.lines.off("line", onLine);
+                    this.process.off("error", onError);
+                    if (response.error)
+                        reject(new Error(`${this.language} analyzer: ${response.error}`));
+                    else
+                        resolve(response.ready ? "ready" : response.tokens ?? "");
+                }
+                catch (error) {
+                    reject(error);
+                }
+            };
+            this.lines.on("line", onLine);
+            this.process.once("error", onError);
+            this.process.stdin.write(`${JSON.stringify({ text })}\n`);
+        });
+    }
+    readLine() {
+        return new Promise((resolve, reject) => {
+            const onLine = (line) => {
+                this.lines.off("line", onLine);
+                resolve(line);
+            };
+            this.lines.on("line", onLine);
+            this.process.once("error", reject);
+        });
+    }
+}
+function tokenizeMock(language, text) {
+    if (language === "ko")
+        return text.normalize("NFKC").toLocaleLowerCase().match(/[\p{L}\p{N}]+/gu)?.join(" ") ?? "";
+    if (language === "ja" || language === "zh")
+        return text.normalize("NFKC").match(/[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{L}\p{N}]+/gu)?.join(" ") ?? "";
+    return tokenizeArabic(text);
+}
+export function tokenizeForLanguage(language, text) {
+    return tokenizeMock(language, text);
+}
+function tokenizeArabic(text) {
+    return (text.normalize("NFKC").toLocaleLowerCase().match(/\p{Script=Arabic}+/gu) ?? [])
+        .flatMap((word) => [word, word.replace(/^(?:وال|فال|بال|كال|ال|لل|و|ف|ب|ك|ل)/u, "")])
+        .filter(Boolean).join(" ");
+}

package/dist/types.d.ts

@@ -71,6 +71,7 @@ export interface SearchHit {
     score: number;
     mode: SearchMode;
 }
+export type LexicalLanguage = "ko" | "ja" | "zh" | "ar";
 export interface SyncReport {
     added: number;
     updated: number;

package/docs/architecture.md

@@ -0,0 +1,223 @@
+# mailcrawl architecture
+
+## Decision
+
+mailcrawl is an independent CLI and local indexing service. AutoRAG should
+integrate with it through a thin process adapter, in the same way it
+integrates with `katok`, `discrawl`, and `qmd`.
+
+The boundary is intentional:
+
+```text
+Himalaya
+  -> mailcrawl sync/archive/chunk/FTS/vector
+  -> stable JSON CLI contract
+  -> AutoRAG datasource adapter
+  -> datasource authorization and agent retrieval
+```
+
+mailcrawl must remain useful without AutoRAG. AutoRAG must not read the
+mailcrawl database directly.
+
+## Responsibilities
+
+### mailcrawl owns
+
+- Provider synchronization through Himalaya
+- Account, mailbox, message, and thread identity
+- Raw/normalized message archive
+- MIME parsing and text normalization
+- Email-aware chunking
+- SQLite metadata and FTS5/BM25 index
+- Embedding queue and model identity
+- LanceDB vector index
+- FTS, semantic, and hybrid ranking
+- Schema migrations, repair, compaction, and health checks
+- Stable JSON output and bounded, redacted diagnostics
+
+### Consumers own
+
+- Authentication policy and trusted configuration handoff
+- Scheduling and polling
+- Access control and scope filtering
+- Mapping results into their own retrieval contracts
+- LLM answering and cross-source result merging
+
+## Storage layout
+
+The initial implementation should use two stores, with explicit
+reconciliation:
+
+```text
+<workspace>/
+  archive.sqlite
+    accounts
+    mailboxes
+    messages
+    threads
+    chunks
+    sync_cursors
+    embedding_queue
+    chunks_fts (FTS5)
+  vectors/
+    LanceDB tables keyed by chunk_id and embedding_model
+```
+
+SQLite is the source of truth for message/chunk metadata, synchronization
+state, and lexical search. LanceDB is a rebuildable semantic index. FTS must
+remain available when embeddings are unavailable or stale.
+
+## Synchronization model
+
+The first version uses a safe snapshot-diff algorithm:
+
+1. Ask Himalaya for bounded envelope pages.
+2. Normalize each envelope into a stable account/mailbox/message key.
+3. Compare provider identity plus envelope fingerprint with local state.
+4. Fetch full MIME content only for new or changed messages.
+5. Mark absent messages deleted only when the complete mailbox snapshot is
+   known; never infer deletion from a truncated page.
+6. Normalize and chunk changed messages.
+7. Transactionally update SQLite metadata and FTS5.
+8. Enqueue changed chunks for embedding.
+
+Later provider-specific cursors can improve scale without changing the public
+contract:
+
+- IMAP UIDVALIDITY + UID, with MODSEQ/QRESYNC where available
+- Gmail history ID
+- JMAP state token
+- Microsoft Graph delta link
+- Maildir file identity and mtime
+
+## Email-aware chunking
+
+Naive fixed-size slicing is not sufficient for email. The normalizer should
+separate:
+
+```text
+message
+  headers
+  latest authored text
+  quoted reply history
+  forwarded content
+  signature
+  attachments and extracted text
+```
+
+The first chunking policy should:
+
+- preserve subject/from/to/date metadata with every searchable chunk
+- prefer paragraph and reply-boundary cuts
+- remove repeated quoted history when safe
+- keep a stable `chunk_id` derived from message identity, section, and index
+- retain a link from every chunk to its source message and thread
+- version the normalization/chunking policy for controlled reindexing
+
+## Search model
+
+### FTS5/BM25
+
+SQLite FTS5 indexes subject, addresses, normalized body, thread subject, and
+attachment text. Field weighting should favor subject and sender matches.
+Search results return stable chunk and message identifiers.
+
+### Semantic
+
+Only normalized, non-empty chunks enter the embedding queue. Each vector row
+stores:
+
+```text
+chunk_id
+message_id
+thread_id
+embedding_model
+content_hash
+created_at
+```
+
+Changing the embedding model creates a new semantic generation. A failed
+embedding job must not make FTS unavailable.
+
+The default implementation uses Google's `EmbeddingGemma` ONNX model through
+Transformers.js and local ONNX Runtime CPU inference. Query and document
+prefixes follow the model's retrieval instructions. Model identity is written
+to semantic generation manifests, so changing it requires a new generation.
+
+### Hybrid
+
+Run lexical and semantic retrieval independently, normalize scores, dedupe by
+chunk/message policy, and merge with a deterministic weighted rank. If the
+vector index is missing or stale, hybrid falls back to FTS and reports a
+bounded warning in JSON diagnostics.
+
+## CLI contract
+
+All commands support `--json` where machine-readable output is useful.
+
+```text
+mailcrawl doctor
+mailcrawl status
+mailcrawl sync
+mailcrawl embed [--limit N]
+mailcrawl search --mode fts|semantic|hybrid [--limit N] [--mailbox NAME] QUERY
+mailcrawl message get MESSAGE_ID
+mailcrawl repair [--fts|--vectors|--all]
+```
+
+Example sync response:
+
+```json
+{
+  "added": 12,
+  "updated": 3,
+  "deleted": 1,
+  "unchanged": 4821,
+  "chunksAdded": 21,
+  "chunksDeleted": 6,
+  "embeddingBacklog": 21
+}
+```
+
+Example search hit:
+
+```json
+{
+  "chunkId": "msg-123:latest:0",
+  "messageId": "msg-123",
+  "threadId": "thread-42",
+  "score": 0.87,
+  "content": "계약 갱신 조건은...",
+  "subject": "Re: 2026 계약 갱신",
+  "mailbox": "INBOX"
+}
+```
+
+## Security and privacy
+
+- Himalaya remains responsible for credential resolution.
+- mailcrawl passes only configured account/backend/mailbox values.
+- Secrets never enter archive metadata or diagnostics.
+- Diagnostics are bounded and redact common email/path/secret patterns.
+- Local embeddings are the default recommendation.
+- Remote embedding providers require explicit configuration.
+- The archive is local and must be treated as sensitive data.
+
+## Implementation milestones
+
+1. Define schema, CLI JSON schemas, and fake Himalaya runner.
+2. Implement Himalaya doctor/account validation and snapshot sync.
+3. Implement MIME normalization and email-aware chunking.
+4. Implement SQLite metadata plus FTS5/BM25 search.
+5. Add the AutoRAG adapter and verify end-to-end lexical retrieval.
+6. Add embedding providers, queue draining, and LanceDB storage.
+7. Add semantic and hybrid retrieval with deterministic fallback.
+8. Add provider-native cursors, repair, benchmarks, and release packaging.
+
+## Non-goals for the first release
+
+- Sending, deleting, or mutating remote mail
+- Replacing Himalaya's mail client surface
+- Cloud-hosted indexing
+- Automatic remote embedding without opt-in
+- LLM-generated summaries as part of the storage engine

package/docs/classification-filtering.md

@@ -0,0 +1,48 @@
+# Classification filtering
+
+mailcrawl preserves provider classification metadata while applying an indexing
+policy at sync time. The default policy excludes `spam` and `promotions` from
+the local archive, FTS5/BM25, and semantic indexes.
+
+Classification values are normalized case-insensitively. Provider prefixes such
+as `CATEGORY_` and `LABEL_` are removed, so `CATEGORY_SPAM`, `spam`, and
+`label-spam` all map to `spam`. Values from `classifications`, `labels`, and
+`flags` are considered.
+
+## CLI policy
+
+The default is equivalent to:
+
+```bash
+mailcrawl sync --exclude-category spam --exclude-category promotions --json
+```
+
+Use `--include-category` to opt into a normally excluded category:
+
+```bash
+mailcrawl sync --include-category spam --json
+```
+
+Use one or more `--exclude-category` options to replace the default exclusion
+set. An empty exclusion set can be supplied through the library API:
+
+```ts
+await archive.sync(messages, { excludedCategories: [] });
+```
+
+Excluded messages are not treated as remote deletions. They are omitted from
+the local indexing source of truth for that sync and are reported separately:
+
+```json
+{
+  "excluded": 2,
+  "excludedByReason": {
+    "spam": 1,
+    "promotions": 1
+  }
+}
+```
+
+If a later sync changes a message's classification so that it is no longer
+excluded, the message is indexed normally. This makes classification changes
+safe for incremental sync.

package/docs/multilingual-installation.md

@@ -0,0 +1,176 @@
+# Multilingual analyzer installation
+
+This guide covers the runtime components used by mailcrawl's lexical search.
+Korean runs through the `kiwi-nlp` WebAssembly binding. Japanese and Chinese
+use persistent Go helpers because Kagome has no official Node binding and the
+GSE project does not currently publish a usable `gse-bind` npm package. English
+uses SQLite's built-in Unicode tokenizer, and Arabic uses mailcrawl's
+in-process light stemmer.
+
+Semantic search uses Google's EmbeddingGemma through Transformers.js and ONNX
+Runtime on the local machine. The model is downloaded to the local
+Transformers cache on first use and is not a hosted service.
+
+## 1. Install Node.js dependencies
+
+Use Node.js 24 or newer:
+
+```bash
+node --version
+npm install
+npm run build
+```
+
+The `kiwi-nlp` package is installed by `npm install`. Its WASM binary is
+resolved automatically from the installed package.
+
+## 2. Configure the Korean Kiwi model
+
+The npm package contains the Kiwi WASM engine, but the model files are
+distributed separately. Set `MAILCRAWL_KIWI_MODEL` to a directory containing
+the matching files from the same Kiwi release:
+
+```bash
+export MAILCRAWL_KIWI_MODEL="$HOME/.local/share/mailcrawl/kiwi-model"
+```
+
+For Kiwi `v0.23.x`, point to the model variant directory, usually
+`cong/base`. The directory must contain the model assets for that release.
+
+```text
+combiningRule.txt
+default.dict
+extract.mdl
+multi.dict
+cong.mdl
+sj.morph
+typo.dict
+```
+
+To use a different WASM file:
+
+```bash
+export MAILCRAWL_KIWI_WASM="/path/to/kiwi-wasm.wasm"
+```
+
+Older Kiwi releases may use `sj.knlm` and `skipbigram.mdl` instead. Keep the
+WASM and model files from compatible Kiwi releases. A missing model
+configuration is an error for production Korean indexing; mailcrawl does not
+silently substitute its test tokenizer.
+
+## 3. Install Go
+
+Go is required only for Japanese and Chinese lexical helpers. Install Go 1.26
+or newer, then verify it:
+
+```bash
+go version
+```
+
+If you do not need Japanese or Chinese search, you can omit this step and
+leave the corresponding helper unset.
+
+## 4. Build the Japanese Kagome helper
+
+From the mailcrawl repository:
+
+```bash
+mkdir -p "$HOME/.local/bin"
+(cd tools/mailcrawl-ja && go build -trimpath -o "$HOME/.local/bin/mailcrawl-ja" .)
+export MAILCRAWL_JA_HELPER="$HOME/.local/bin/mailcrawl-ja"
+```
+
+The helper embeds Kagome's IPADIC dictionary and stays alive as a persistent
+JSON-lines process. It reports a `ready` response at startup and accepts
+`{"text":"..."}` requests.
+
+Smoke test:
+
+```bash
+printf '{"text":"契約更新を確認"}\n' | "$MAILCRAWL_JA_HELPER"
+```
+
+The output should contain Japanese lexical terms such as `契約` and `更新`.
+
+## 5. Build the Chinese GSE helper
+
+Build and configure the Chinese helper:
+
+```bash
+(cd tools/mailcrawl-zh && go build -trimpath -o "$HOME/.local/bin/mailcrawl-zh" .)
+export MAILCRAWL_ZH_HELPER="$HOME/.local/bin/mailcrawl-zh"
+```
+
+Smoke test:
+
+```bash
+printf '{"text":"合同更新确认"}\n' | "$MAILCRAWL_ZH_HELPER"
+```
+
+The output should contain Chinese lexical terms such as `合同` and `更新`.
+
+## 6. Persist the environment
+
+Put the exports in the shell startup file used to run mailcrawl, or provide
+them through the service manager that launches it:
+
+```bash
+export MAILCRAWL_KIWI_MODEL="$HOME/.local/share/mailcrawl/kiwi-model"
+export MAILCRAWL_JA_HELPER="$HOME/.local/bin/mailcrawl-ja"
+export MAILCRAWL_ZH_HELPER="$HOME/.local/bin/mailcrawl-zh"
+```
+
+The helper paths may instead be plain executable names when they are on
+`PATH`.
+
+## 7. Verify mailcrawl
+
+Run the project checks:
+
+```bash
+npm run typecheck
+npm test
+npm run build
+```
+
+Then synchronize and search:
+
+```bash
+mailcrawl sync --json
+mailcrawl search --mode bm25 --json "계약 갱신"
+mailcrawl search --mode bm25 --json "契約 更新"
+mailcrawl search --mode bm25 --json "合同 更新"
+mailcrawl search --mode bm25 --json "كتاب"
+mailcrawl search --mode bm25 --json "contract renewal"
+mailcrawl index --json
+mailcrawl search --mode semantic --json "contract renewal"
+```
+
+## 8. Lexical rebuild policy
+
+Language-specific FTS fields are derived artifacts. Their version fingerprint
+includes the Kiwi, Kagome, GSE, and Arabic analyzer versions. On a fingerprint
+change, mailcrawl clears those fields and marks them stale. The next complete
+`mailcrawl sync` re-analyzes changed and existing messages from the source and
+commits the new fingerprint atomically. Multilingual search fails clearly until
+that sync completes; the default Unicode FTS remains independent.
+
+Semantic vectors are separate derived artifacts. Changing the embedding model
+or its preprocessing requires a new semantic generation; run `mailcrawl index`
+after synchronization. Existing lexical fields remain valid when only the
+embedding model changes.
+
+## Packaging and licenses
+
+mailcrawl remains MIT-licensed. `kiwi-nlp` and Kiwi remain
+LGPL-2.1-or-later components. Kagome is MIT-licensed, its IPADIC dictionary
+has separate IPADIC/ICOT terms, and GSE is Apache-2.0. See
+[`THIRD_PARTY_NOTICES.md`](../THIRD_PARTY_NOTICES.md) before redistributing
+the application or helper binaries.
+
+## 9. Semantic model
+
+Semantic indexing uses `onnx-community/embeddinggemma-300m-ONNX` with
+Transformers.js and ONNX Runtime on CPU. The first `mailcrawl index` downloads
+the model to the local Transformers cache; later runs reuse that cache. The
+model is not sent to a remote embedding service.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nomadamas/mailcrawl",
-  "version": "0.1.0",
+  "version": "0.1.3",
   "description": "Local Himalaya-backed incremental email indexing and hybrid search CLI",
   "license": "MIT",
   "type": "module",
@@ -10,6 +10,8 @@
   "files": [
     "dist",
     "skills",
+    "docs",
+    "THIRD_PARTY_NOTICES.md",
     "README.md",
     "LICENSE"
   ],
@@ -35,8 +37,10 @@
     "commit:check": "git diff --check"
   },
   "dependencies": {
+    "@huggingface/transformers": "^4.2.0",
     "better-sqlite3": "^12.4.1",
     "commander": "^14.0.0",
+    "kiwi-nlp": "^0.23.0",
     "mailparser": "^3.9.0",
     "mime-types": "^3.0.1"
   },

package/skills/mailcrawl/SKILL.md

@@ -27,6 +27,15 @@ Required runtime dependencies are Node.js 24+, the package dependencies, and
 the Himalaya CLI for live synchronization. Fixture synchronization needs no
 mail account.
 
+For Korean lexical search, provide a Kiwi model directory through
+`MAILCRAWL_KIWI_MODEL`; the WASM binary is bundled by `kiwi-nlp`. Kagome has
+no official Node binding, and GSE's referenced `gse-bind` is not available as
+a usable npm package. Therefore Japanese and Chinese lexical search use the
+repository helpers `tools/mailcrawl-ja` and `tools/mailcrawl-zh`; set
+`MAILCRAWL_JA_HELPER` / `MAILCRAWL_ZH_HELPER` when they are not on PATH.
+Production multilingual search fails clearly when its analyzer is not
+configured; it never silently uses the test tokenizer.
+
 ## Data and credential boundaries
 
 The archive lives under `.mailcrawl` by default. Set `--data-dir` or