mixgram 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/.mixgram/index.db +0 -0
  2. package/.mixgram/test-esm.db +0 -0
  3. package/.mixgram/test-index.db +0 -0
  4. package/.mixgram/test-stage1.db +0 -0
  5. package/README.md +335 -0
  6. package/bin/mixgram.js +309 -0
  7. package/index.js +9 -0
  8. package/package.json +51 -0
  9. package/projects/castlebravo/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
  10. package/projects/test/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
  11. package/spec/fixtures/realistic-content.js +97 -0
  12. package/spec/reporter.js +127 -0
  13. package/spec/run-tests-runner.js +40 -0
  14. package/spec/run-tests.js +160 -0
  15. package/spec/scenarios/scenarios.js +520 -0
  16. package/spec/sdd.md +1002 -0
  17. package/spec/tdd.md +1237 -0
  18. package/src/config.js +56 -0
  19. package/src/core/documents/documents.js +207 -0
  20. package/src/core/embeddings/embedder.js +58 -0
  21. package/src/core/embeddings/queue.js +118 -0
  22. package/src/core/embeddings/vectorStore.js +127 -0
  23. package/src/core/embeddings/worker.js +27 -0
  24. package/src/core/indexing/indexer.js +131 -0
  25. package/src/core/indexing/parser.js +78 -0
  26. package/src/core/indexing/reindex.js +101 -0
  27. package/src/core/prompts/prompts.js +15 -0
  28. package/src/core/search/search.js +223 -0
  29. package/src/core/sessions/sessions.js +27 -0
  30. package/src/db/migrations/001_initial.sql +86 -0
  31. package/src/db/sqlite.js +39 -0
  32. package/src/fs/paths.js +79 -0
  33. package/src/fs/watcher.js +49 -0
  34. package/src/mcp/cli-adapter.js +118 -0
  35. package/src/mcp/server.js +47 -0
  36. package/src/mcp/tool-registry.js +138 -0
  37. package/src/mcp/tools.js +159 -0
  38. package/src/utils/hash.js +6 -0
  39. package/src/utils/ids.js +22 -0
  40. package/src/utils/markdown.js +49 -0
package/spec/sdd.md ADDED
@@ -0,0 +1,1002 @@
1
+ # SDD
2
+
3
+ ## Mixgram
4
+
5
+ ### A Node.js project memory system compatible with Engram MCP, using Markdown as the source of truth, SQLite as the text index, and optional asynchronous local embeddings
6
+
7
+ **Status:** Final
8
+ **Target language:** Node.js
9
+ **Compatibility target:** drop-in MCP replacement for Engram
10
+ **Source of truth:** Markdown files
11
+ **Text index:** SQLite + FTS5
12
+ **Semantic index:** optional, local, asynchronous
13
+ **Technical references:** Engram, SearchMix, SeekMix
14
+
15
+ ---
16
+
17
+ # 1. Purpose
18
+
19
+ This document defines a memory system for software projects and agents that preserves **Engram-compatible MCP interfaces**, while replacing Engram’s storage model with a more transparent architecture:
20
+
21
+ * **Markdown files are the canonical memory store**
22
+ * **SQLite is a derived and rebuildable index**
23
+ * **Embeddings are optional**
24
+ * **When enabled, embeddings run asynchronously in the background**
25
+
26
+ The goal is to preserve Engram’s external MCP tool contract while improving visibility, Git compatibility, manual editing, and long-term maintainability.
27
+
28
+ ---
29
+
30
+ # 2. Core goals
31
+
32
+ ## 2.1 Functional goals
33
+
34
+ * Replace Engram at the MCP interface level
35
+ * Preserve the same tool names and external semantics
36
+ * Store durable memory as Markdown files
37
+ * Maintain a rebuildable SQLite text index
38
+ * Reindex new and modified files automatically
39
+ * Support fast full-text retrieval
40
+ * Support optional local embeddings
41
+ * Continue working fully when embeddings are disabled
42
+
43
+ ## 2.2 Non-functional goals
44
+
45
+ * Human-readable memory
46
+ * Git-friendly workflows
47
+ * Minimal external dependencies
48
+ * Rebuildable derived state
49
+ * Graceful degradation when semantic indexing is unavailable
50
+ * Strong support for manual editing
51
+ * Clear knowledge reuse across projects
52
+
53
+ ---
54
+
55
+ # 3. Design principles
56
+
57
+ ## 3.1 Markdown is canonical
58
+
59
+ All durable, meaningful memory should live as Markdown files.
60
+
61
+ ## 3.2 SQLite is derived
62
+
63
+ SQLite is not the canonical store. It is a rebuildable search index and retrieval layer.
64
+
65
+ ## 3.3 MCP compatibility first
66
+
67
+ The system must expose the same MCP tools as Engram so it can replace it in existing integrations.
68
+
69
+ ## 3.4 Full-text is mandatory, embeddings are optional
70
+
71
+ The system must work correctly with SQLite FTS5 alone. Semantic retrieval is an enhancement, not a requirement.
72
+
73
+ ## 3.5 Embeddings must be asynchronous
74
+
75
+ If embeddings are enabled, they must be computed in the background. Saving or updating memory must never block on embedding generation.
76
+
77
+ ## 3.6 Human edits win
78
+
79
+ If a Markdown file is edited manually, the system must treat the file as authoritative and reindex it.
80
+
81
+ ## 3.7 Knowledge should support multiple scopes
82
+
83
+ Memory should support both:
84
+
85
+ * **project-local knowledge**
86
+ * **cross-project reusable knowledge**
87
+
88
+ ---
89
+
90
+ # 4. Technical references
91
+
92
+ ## 4.1 Engram
93
+
94
+ Engram is the MCP compatibility reference. Its tool names, argument structure, and overall memory workflow define the external contract to preserve.
95
+
96
+ ## 4.2 SearchMix
97
+
98
+ SearchMix is the reference for:
99
+
100
+ * Markdown-first document indexing
101
+ * SQLite FTS5
102
+ * BM25 ranking
103
+ * incremental reindexing
104
+ * document/path-centered indexing
105
+ * snippets and structured text retrieval
106
+
107
+ **Adaptation to Mixgram.** This design takes the same ideas—Markdown parsing, FTS5, BM25, mtime-based incremental reindex, and snippets—but applies them to a **chunk-based** model with **frontmatter**, **scope/type/project**, and an **external file watcher** that triggers indexing. SearchMix has no built-in watch; Mixgram does.
108
+
109
+ ## 4.3 SeekMix
110
+
111
+ SeekMix is the reference for:
112
+
113
+ * optional local embeddings
114
+ * local vector storage
115
+ * configurable semantic thresholds
116
+ * provider abstraction for embedding generation
117
+
118
+ In this design, SeekMix is used as the semantic layer reference, not as a semantic cache model.
119
+
120
+ ---
121
+
122
+ # 5. Architecture overview
123
+
124
+ ```text
125
+ MCP Client / Agent
126
+ |
127
+ v
128
+ MCP Server (Node.js)
129
+ |
130
+ +----------------------------------+
131
+ | |
132
+ v v
133
+ Markdown Memory Store SQLite Index
134
+ (source of truth) (derived state)
135
+ | |
136
+ | +-- documents
137
+ | +-- chunks
138
+ | +-- FTS5
139
+ | +-- sessions
140
+ | +-- prompts
141
+ | +-- tags
142
+ | +-- embedding_jobs
143
+ | +-- vector metadata
144
+ |
145
+ v
146
+ File Watcher / Incremental Reindexer
147
+ |
148
+ v
149
+ Optional Local Embedding Worker
150
+ ```
151
+
152
+ ---
153
+
154
+ # 6. Directory conventions
155
+
156
+ This is the part I would definitely adjust based on what you said.
157
+
158
+ A **date-first filename convention should not be the default**. Date is useful only for memories where chronology matters. For architecture, reusable knowledge, decisions, patterns, and reference notes, the more important dimensions are:
159
+
160
+ * scope
161
+ * topic
162
+ * type
163
+ * project relevance
164
+
165
+ ## 6.1 Recommended root structure
166
+
167
+ ```text
168
+ workspace/
169
+ home/
170
+ memory/
171
+ architecture/
172
+ decisions/
173
+ patterns/
174
+ reference/
175
+ learnings/
176
+ discoveries/
177
+ prompts/
178
+ projects/
179
+ project-a/
180
+ memory/
181
+ architecture/
182
+ decisions/
183
+ bugs/
184
+ sessions/
185
+ learnings/
186
+ discoveries/
187
+ generated/
188
+ project-b/
189
+ memory/
190
+ architecture/
191
+ decisions/
192
+ bugs/
193
+ sessions/
194
+ learnings/
195
+ discoveries/
196
+ generated/
197
+ .mixgram/
198
+ index.db
199
+ jobs/
200
+ logs/
201
+ cache/
202
+ ```
203
+
204
+ ## 6.2 Meaning of `home/`
205
+
206
+ `home/memory/` stores **cross-project knowledge**.
207
+
208
+ This includes:
209
+
210
+ * reusable architecture notes
211
+ * patterns
212
+ * general discoveries
213
+ * general learnings
214
+ * prompts or workflows relevant across projects
215
+ * implementation techniques that should be searchable from any project context
216
+
217
+ This lets the system reuse knowledge globally instead of duplicating it inside each project.
218
+
219
+ ## 6.3 Meaning of `projects/<name>/memory/`
220
+
221
+ This stores **project-scoped knowledge**.
222
+
223
+ This includes:
224
+
225
+ * project decisions
226
+ * project-specific bugs
227
+ * session summaries
228
+ * feature notes
229
+ * implementation details tied to that project
230
+ * generated notes that should remain local
231
+
232
+ ## 6.4 Scope model
233
+
234
+ The system should support at least these scopes:
235
+
236
+ * `home`
237
+ * `project`
238
+ * `session`
239
+ * `generated`
240
+
241
+ This scope must be represented both in:
242
+
243
+ * Markdown frontmatter
244
+ * SQLite metadata
245
+
246
+ ---
247
+
248
+ # 7. Document classification model
249
+
250
+ Instead of relying primarily on dates in filenames, the system should classify memory by **type + scope + topic**.
251
+
252
+ ## 7.1 Recommended document types
253
+
254
+ * `architecture`
255
+ * `decision`
256
+ * `bug`
257
+ * `learning`
258
+ * `discovery`
259
+ * `pattern`
260
+ * `reference`
261
+ * `session_summary`
262
+ * `prompt`
263
+ * `generated_note`
264
+
265
+ ## 7.2 Recommended directory mapping
266
+
267
+ ```text
268
+ memory/
269
+ architecture/
270
+ decisions/
271
+ bugs/
272
+ learnings/
273
+ discoveries/
274
+ patterns/
275
+ reference/
276
+ sessions/
277
+ prompts/
278
+ generated/
279
+ ```
280
+
281
+ This gives both humans and agents a predictable structure.
282
+
283
+ ---
284
+
285
+ # 8. File naming convention
286
+
287
+ I would recommend moving away from date-first naming except when time is central.
288
+
289
+ ## 8.1 Default naming format
290
+
291
+ ```text
292
+ <type>--<slug>.md
293
+ ```
294
+
295
+ Examples:
296
+
297
+ ```text
298
+ decision--use-sqlite-derived-index.md
299
+ architecture--memory-layer-layout.md
300
+ pattern--incremental-reindexing.md
301
+ reference--sqlite-fts-ranking-notes.md
302
+ ```
303
+
304
+ ## 8.2 When date should be included
305
+
306
+ Only include time in the filename when chronology is important, for example:
307
+
308
+ * session summaries
309
+ * incident investigations
310
+ * temporary discoveries
311
+ * logs or research trails
312
+ * time-sensitive notes
313
+
314
+ Examples:
315
+
316
+ ```text
317
+ session-summary--2026-03-08--agent-debugging.md
318
+ bug--2026-03-08--timeline-ordering-regression.md
319
+ discovery--2026-03-08--redis-vs-sqlite-observation.md
320
+ ```
321
+
322
+ ## 8.3 Topic key is more important than the filename
323
+
324
+ The true identity of evolving knowledge should be driven primarily by:
325
+
326
+ * `id`
327
+ * `topic_key`
328
+ * `scope`
329
+ * `project`
330
+
331
+ The filename is a stable human convenience, not the core identity mechanism.
332
+
333
+ ---
334
+
335
+ # 9. Markdown frontmatter
336
+
337
+ Each memory document should include structured frontmatter.
338
+
339
+ ```md
340
+ ---
341
+ id: obs_01HXYZ123ABC
342
+ title: Use SQLite as derived index
343
+ type: decision
344
+ scope: project
345
+ project: castlebravo
346
+ topic_key: architecture/sqlite-derived-index
347
+ session_id: sess_01HXYZ456DEF
348
+ tool_name: mem_save
349
+ created_at: 2026-03-08T18:00:00Z
350
+ updated_at: 2026-03-08T18:00:00Z
351
+ revision_count: 1
352
+ duplicate_count: 0
353
+ deleted: false
354
+ tags:
355
+ - sqlite
356
+ - indexing
357
+ - architecture
358
+ embedding_status: pending
359
+ ---
360
+
361
+ # Summary
362
+
363
+ SQLite will be used as a derived index over Markdown documents.
364
+ ```
365
+
366
+ ## 9.1 Rules
367
+
368
+ * `project` may be omitted for `home` scope
369
+ * `scope: home` means globally reusable knowledge
370
+ * `scope: project` means project-local knowledge
371
+ * `scope: session` is intended for chronological notes
372
+ * `scope: generated` is for machine-generated notes
373
+
374
+ ---
375
+
376
+ # 10. Cross-project knowledge behavior
377
+
378
+ This is an important addition.
379
+
380
+ ## 10.1 Home knowledge must be searchable everywhere
381
+
382
+ When searching inside a project, the system should be able to retrieve:
383
+
384
+ * project-local memory
385
+ * home/global memory
386
+
387
+ This means project search should support:
388
+
389
+ * `local-only`
390
+ * `home-only`
391
+ * `merged`
392
+
393
+ The recommended default is **merged**, with local results slightly boosted.
394
+
395
+ ## 10.2 Ranking recommendation
396
+
397
+ If the same topic exists in both `home` and project scope:
398
+
399
+ * prefer project-specific content when the query is project-contextual
400
+ * still allow home results as fallback or supporting context
401
+
402
+ Suggested bias:
403
+
404
+ * `project` scope gets a ranking bonus in project searches
405
+ * `home` scope remains fully visible
406
+
407
+ ## 10.3 Reuse rule
408
+
409
+ If knowledge is broadly reusable, it should be saved under `home/memory/` instead of duplicating it into multiple projects.
410
+
411
+ ---
412
+
413
+ # 11. SQLite data model
414
+
415
+ ## 11.1 `documents`
416
+
417
+ ```sql
418
+ CREATE TABLE documents (
419
+ id TEXT PRIMARY KEY,
420
+ path TEXT NOT NULL UNIQUE,
421
+ title TEXT,
422
+ type TEXT NOT NULL,
423
+ scope TEXT NOT NULL,
424
+ project TEXT,
425
+ topic_key TEXT,
426
+ session_id TEXT,
427
+ tool_name TEXT,
428
+ content_hash TEXT NOT NULL,
429
+ file_mtime_ms INTEGER,
430
+ created_at TEXT NOT NULL,
431
+ updated_at TEXT NOT NULL,
432
+ indexed_at TEXT,
433
+ deleted_at TEXT,
434
+ revision_count INTEGER NOT NULL DEFAULT 1,
435
+ duplicate_count INTEGER NOT NULL DEFAULT 0,
436
+ embedding_status TEXT NOT NULL DEFAULT 'disabled'
437
+ );
438
+ ```
439
+
440
+ ## 11.2 `document_chunks`
441
+
442
+ ```sql
443
+ CREATE TABLE document_chunks (
444
+ id TEXT PRIMARY KEY,
445
+ document_id TEXT NOT NULL,
446
+ chunk_index INTEGER NOT NULL,
447
+ heading_path TEXT,
448
+ heading_level INTEGER,
449
+ content TEXT NOT NULL,
450
+ token_count INTEGER,
451
+ content_hash TEXT NOT NULL,
452
+ created_at TEXT NOT NULL,
453
+ updated_at TEXT NOT NULL,
454
+ FOREIGN KEY(document_id) REFERENCES documents(id)
455
+ );
456
+ ```
457
+
458
+ ## 11.3 `document_tags`
459
+
460
+ ```sql
461
+ CREATE TABLE document_tags (
462
+ document_id TEXT NOT NULL,
463
+ tag TEXT NOT NULL,
464
+ PRIMARY KEY (document_id, tag),
465
+ FOREIGN KEY(document_id) REFERENCES documents(id)
466
+ );
467
+ ```
468
+
469
+ ## 11.4 `sessions`
470
+
471
+ ```sql
472
+ CREATE TABLE sessions (
473
+ id TEXT PRIMARY KEY,
474
+ project TEXT NOT NULL,
475
+ directory TEXT,
476
+ started_at TEXT NOT NULL,
477
+ ended_at TEXT,
478
+ summary TEXT
479
+ );
480
+ ```
481
+
482
+ ## 11.5 `prompts`
483
+
484
+ ```sql
485
+ CREATE TABLE prompts (
486
+ id TEXT PRIMARY KEY,
487
+ session_id TEXT NOT NULL,
488
+ project TEXT,
489
+ content TEXT NOT NULL,
490
+ created_at TEXT NOT NULL,
491
+ FOREIGN KEY(session_id) REFERENCES sessions(id)
492
+ );
493
+ ```
494
+
495
+ ## 11.6 `embedding_jobs`
496
+
497
+ ```sql
498
+ CREATE TABLE embedding_jobs (
499
+ id TEXT PRIMARY KEY,
500
+ chunk_id TEXT NOT NULL,
501
+ model_name TEXT NOT NULL,
502
+ status TEXT NOT NULL,
503
+ attempts INTEGER NOT NULL DEFAULT 0,
504
+ last_error TEXT,
505
+ created_at TEXT NOT NULL,
506
+ updated_at TEXT NOT NULL,
507
+ FOREIGN KEY(chunk_id) REFERENCES document_chunks(id)
508
+ );
509
+ ```
510
+
511
+ ## 11.7 FTS5
512
+
513
+ ```sql
514
+ CREATE VIRTUAL TABLE document_chunks_fts USING fts5(
515
+ chunk_id UNINDEXED,
516
+ document_id UNINDEXED,
517
+ title,
518
+ topic_key,
519
+ type,
520
+ scope,
521
+ project,
522
+ heading_path,
523
+ body,
524
+ tokenize = 'unicode61 remove_diacritics 2'
525
+ );
526
+ ```
527
+
528
+ **Column roles.** The FTS5 table stores two kinds of columns. **UNINDEXED** columns (`chunk_id`, `document_id`) are kept for joins and identity but are not full-text indexed. All other columns (`title`, `topic_key`, `type`, `scope`, `project`, `heading_path`, `body`) are **indexed** and participate in `MATCH` and BM25 ranking.
529
+
530
+ **Normalization.** Indexable text should be normalized so that search is case- and accent-insensitive. The tokenizer `unicode61 remove_diacritics 2` contributes to this. Any application-side normalization (e.g. NFD, remove diacritics, lowercase) for title or heading_path should match the same rules so that query terms align with indexed content.
531
+
532
+ **BM25.** Ranking uses FTS5’s BM25 with configurable weights (see §18 `indexing.ftsWeights`: title, topicKey, heading, body) so that matches in title and topic_key rank higher than body-only matches.
533
+
534
+ ---
535
+
536
+ ## 11.8 Vector cache and vec tables (optional, when semantic indexing enabled)
537
+
538
+ When embeddings are enabled, the same SQLite DB (or a dedicated DB) holds per-model cache and vector tables. Suffix is derived from the sanitized embedding model name (e.g. `Xenova_multilingual_e5_large`).
539
+
540
+ **Metadata table** (links chunk to vector row):
541
+
542
+ ```sql
543
+ CREATE TABLE cache_<suffix> (
544
+ id INTEGER PRIMARY KEY,
545
+ chunk_id TEXT NOT NULL UNIQUE,
546
+ document_id TEXT NOT NULL,
547
+ content_hash TEXT NOT NULL,
548
+ embedding_model TEXT NOT NULL,
549
+ created_at TEXT NOT NULL,
550
+ FOREIGN KEY(chunk_id) REFERENCES document_chunks(id),
551
+ FOREIGN KEY(document_id) REFERENCES documents(id)
552
+ );
553
+ ```
554
+
555
+ **Vector table** (sqlite-vec virtual table; rowid must align with `cache_<suffix>.id`):
556
+
557
+ ```sql
558
+ -- After sqliteVec.load(db):
559
+ CREATE VIRTUAL TABLE vec_<suffix> USING vec0(
560
+ embedding float[1024],
561
+ distance_metric=cosine
562
+ );
563
+ ```
564
+
565
+ **Insert:** Insert into `cache_<suffix>`, then insert into `vec_<suffix>` with `rowid = cache_<suffix>.id` and `embedding = Float32Array(embedding)`. **Search:** KNN via `WHERE embedding MATCH ? AND k = N`, then join `vec_<suffix>.rowid = cache_<suffix>.id`. See §13.7.4 and §13.7.5.
566
+
567
+ ---
568
+
569
+ # 12. Text indexing
570
+
571
+ ## 12.1 Immediate consistency
572
+
573
+ Every create or update operation must synchronously update:
574
+
575
+ * documents
576
+ * chunks
577
+ * tags
578
+ * FTS
579
+
580
+ ## 12.2 Incremental reindexing
581
+
582
+ The watcher/indexer must:
583
+
584
+ * detect new files
585
+ * detect modified files
586
+ * skip unchanged files
587
+ * reindex only what changed
588
+
589
+ ## 12.3 Chunking strategy
590
+
591
+ Chunking should be semantic:
592
+
593
+ * split by headings when possible
594
+ * preserve heading hierarchy
595
+ * avoid tiny fragments
596
+ * keep chunk size configurable
597
+
598
+ ## 12.4 Weighted retrieval
599
+
600
+ Search should favor:
601
+
602
+ * title
603
+ * topic key
604
+ * heading path
605
+ * body
606
+
607
+ This follows the SearchMix-style principle of field-aware ranking.
608
+
609
+ ## 12.5 Indexing pipeline (SearchMix-inspired)
610
+
611
+ The indexer follows a single pipeline from input to FTS5 and metadata writes.
612
+
613
+ * **Input.** The pipeline accepts a path to a `.md` file, a directory, or in-memory content (e.g. from `mem_save`).
614
+ * **Type detection.** The system distinguishes file vs directory. For directories, it recurses and selects `*.md` / `*.markdown` with optional exclude patterns.
615
+ * **Read and encoding.** File content is read; encoding is detected (e.g. jschardet) and decoded to UTF-8 (e.g. iconv-lite) so that non-UTF-8 Markdown is supported.
616
+ * **Markdown parsing.** Content is parsed with a unified/remark-style pipeline to an AST. The implementation extracts frontmatter (id, title, type, scope, project, topic_key, etc.) and body structure (headings h1–h6, paragraphs, optional code blocks), and builds a section tree (id, parentId, childrenIds) and a flat sections index with positions for snippet boundaries.
617
+ * **Chunking.** Chunks are derived from the section tree and headings (see §12.3). Each chunk gets heading_path, heading_level, content, and optional token_count and content_hash.
618
+ * **Write.** The pipeline updates `documents`, `document_chunks`, `document_tags`, and the FTS5 table, and stores `file_mtime_ms` (or equivalent) for mtime-based skip on the next run.
619
+
620
+ This pipeline is invoked by (a) the file watcher when a file is added or changed, (b) explicit reindex APIs, and (c) `mem_save` / `mem_update` for in-memory content.
621
+
622
+ ## 12.6 Watch and incremental reindex (difference from SearchMix)
623
+
624
+ SearchMix has no built-in file watch; reindexing happens only when the caller invokes an add-document API, and mtime is used inside that call to skip unchanged files.
625
+
626
+ Mixgram includes a **file watcher** (e.g. chokidar or `fs.watch`) that observes configured roots for new, changed, and deleted Markdown files and calls the indexer (add/update/remove). When the indexer runs—whether from the watcher or from an explicit reindex—it uses **mtime** (with optional tolerance, e.g. 1s) to skip unchanged files and reindex only modified or new ones. Options such as skip-when-unchanged (e.g. `checkModified` / `skipExisting`) can be supported: skip reindex when the stored mtime matches and the document is already in the index.
627
+
628
+ ## 12.7 Search resolution and snippets
629
+
630
+ * **Query execution.** Search is performed with SQLite FTS5 (`WHERE document_chunks_fts MATCH ?`). The query string is **normalized** using the same rules as indexed text (e.g. NFD, remove diacritics, lowercase), while respecting FTS5 operators (AND, OR, NOT, quoted phrases).
631
+ * **Field prefixes.** If the design supports field-scoped search (e.g. `title:`, `body:`), these are mapped to the corresponding FTS5 columns in the normalized query.
632
+ * **Ranking.** Results are ordered by BM25 with the configured weights; an optional `minScore` threshold may filter weak matches.
633
+ * **Filters.** Optional scope, project, or tag filters (e.g. via `documents` / `document_tags`) are applied in SQL (JOIN or subquery).
634
+ * **Snippets.** For each result, snippets are extracted by finding query terms (after stripping operators) in the normalized title, heading_path, and body. The section/chunk structure and stored positions are used to trim context (snippet length configurable) and attach the snippet to the correct chunk/section. Snippet objects include range, document/chunk id, and optional hierarchy (e.g. heading path).
635
+
636
+ ## 12.8 Libraries for indexing and search (reference)
637
+
638
+ The following libraries are used for indexing and search. This is a reference list; actual dependencies may be chosen to match the project’s constraints (e.g. ESM vs CJS).
639
+
640
+ | Purpose | Library / stack | Role in Mixgram |
641
+ |---------|------------------|-----------------|
642
+ | SQLite + FTS5 | better-sqlite3 | Database and FTS5 index; BM25 ranking. |
643
+ | Markdown AST | unified, remark-parse | Parse Markdown to AST. |
644
+ | AST traversal | unist-util-visit | Walk AST to extract headings, paragraphs, sections, positions. |
645
+ | Language detection | franc (optional) | Optional tag or metadata for document language. |
646
+ | File discovery | glob (or similar) | List `*.md` / `*.markdown` in directories; optional recursion and exclude (e.g. micromatch). |
647
+ | Encoding | jschardet, iconv-lite | Detect file encoding and decode to UTF-8. |
648
+ | File watch | chokidar or fs.watch | Detect filesystem changes and trigger reindex (Mixgram-specific). |
649
+
650
+ ---
651
+
652
+ # 13. Optional semantic layer
653
+
654
+ ## 13.1 Rule
655
+
656
+ Embeddings are optional.
657
+
658
+ The system must fully function when embeddings are disabled.
659
+
660
+ ## 13.2 When disabled
661
+
662
+ Everything still works:
663
+
664
+ * `mem_save`
665
+ * `mem_update`
666
+ * `mem_delete`
667
+ * `mem_search`
668
+ * `mem_context`
669
+ * `mem_timeline`
670
+ * `mem_get_observation`
671
+ * session tools
672
+ * prompt tools
673
+
674
+ ## 13.3 When enabled
675
+
676
+ If embeddings are enabled:
677
+
678
+ * chunks are queued for semantic indexing
679
+ * a local worker processes them asynchronously
680
+ * vector results may improve retrieval quality
681
+ * write operations still return immediately after text indexing
682
+
683
+ ## 13.4 Why asynchronous
684
+
685
+ Embedding generation is expensive. It must run behind the main flow and must never block save/update operations.
686
+
687
+ ## 13.5 Status values
688
+
689
+ Each chunk can be:
690
+
691
+ * `disabled`
692
+ * `pending`
693
+ * `processing`
694
+ * `completed`
695
+ * `failed`
696
+ * `stale`
697
+
698
+ ## 13.6 Invalidation
699
+
700
+ Embeddings become `stale` when:
701
+
702
+ * chunk content changes
703
+ * embedding model changes
704
+ * chunking strategy changes
705
+
706
+ ## 13.7 Technical implementation (default local semantic indexing)
707
+
708
+ When semantic indexing is enabled, Mixgram uses the same stack as SeekMix’s default local setup. The following is the technical reference for replication and implementation.
709
+
710
+ ### 13.7.1 Architecture summary
711
+
712
+ | Component | Technology / model |
713
+ |------------------|---------------------|
714
+ | Embeddings | Produced locally by **Hugging Face Transformers.js** (`@huggingface/transformers`), no API keys. |
715
+ | Storage | **SQLite** (`better-sqlite3`) with the **sqlite-vec** extension for vector similarity search. |
716
+ | Matching | Cosine similarity over normalized embeddings; configurable threshold (default `0.87`). |
717
+
718
+ ### 13.7.2 Default embedding model
719
+
720
+ | Property | Value |
721
+ |-------------|--------|
722
+ | **Provider** | HuggingfaceProvider (local) |
723
+ | **Model** | `Xenova/multilingual-e5-large` |
724
+ | **Dimensions**| 1024 (from model `hidden_size`) |
725
+ | **Library** | `@huggingface/transformers` |
726
+ | **Pipeline** | `feature-extraction` |
727
+ | **Quantization** | `dtype: 'q8'` (optional, for smaller/faster load) |
728
+
729
+ The **Xenova/** prefix indicates the model is used in ONNX form via Transformers.js; the first run downloads it from Hugging Face, then it runs fully on-device.
730
+
731
+ ### 13.7.3 How embeddings are produced (local)
732
+
733
+ * **One-time init:** `pipeline('feature-extraction', 'Xenova/multilingual-e5-large', { dtype: 'q8' })` (model download on first run).
734
+ * **Per-text embedding:** Mean pooling over token embeddings (no CLS-only), then L2-normalize. Cosine similarity is then equivalent to dot product.
735
+ * **Output:** Array of 1024 floats; same format is stored and used for search.
736
+
737
+ Example pattern:
738
+
739
+ ```javascript
740
+ const output = await extractor(text, { pooling: 'mean', normalize: true });
741
+ const embedding = output.tolist(); // 1024-dim, L2-normalized
742
+ ```
743
+
744
+ ### 13.7.4 Vector storage (sqlite-vec)
745
+
746
+ * **DB:** Single SQLite file (e.g. same as main index or a dedicated DB). One DB can host multiple logical caches per embedding model via different table name suffixes (e.g. sanitized model name: `Xenova_multilingual_e5_large`).
747
+
748
+ * **Tables per model:**
749
+
750
+ 1. **Metadata table** `cache_<suffix>`: `id` (PK), `chunk_id` (unique), `document_id`, `content_hash`, `embedding_model`, `created_at`, etc. Links chunk identity and metadata to vector rows.
751
+
752
+ 2. **Vector table** `vec_<suffix>` (sqlite-vec virtual table): `vec0(embedding float[1024] distance_metric=cosine)`. Row count matches cache; **rowid** of the vector table equals **id** of the cache table for joins.
753
+
754
+ * **Insert (indexing):** Insert row into `cache_<suffix>` → get `id` → insert into `vec_<suffix>` with `rowid = id` and `embedding = Float32Array(embedding)`.
755
+
756
+ * **Extension:** `sqliteVec.load(db)` once per connection; vectors stored as float32.
757
+
758
+ ### 13.7.5 Search flow (KNN + threshold)
759
+
760
+ 1. **Embed** the incoming query with the same model and options (mean pooling + normalize).
761
+ 2. **KNN query** (sqlite-vec), e.g.:
762
+ ```sql
763
+ SELECT rowid, distance
764
+ FROM vec_<suffix>
765
+ WHERE embedding MATCH ?
766
+ AND k = N
767
+ ORDER BY distance;
768
+ ```
769
+ For tag/scope filtering, `k` is increased and results are filtered in application code.
770
+ 3. **Join** with `cache_<suffix>` on `cache.id = vec.rowid` to get chunk_id, document_id, etc., then join to `document_chunks` / `documents` for full metadata and snippets.
771
+ 4. **Similarity threshold:** sqlite-vec returns **cosine distance** (0 = identical). Convert to similarity as `similarity = 1 - distance`. Accept a hit only if `similarity >= similarityThreshold` (default `0.87`, from config `embeddings.similarityThreshold`).
772
+ 5. **Optional:** TTL, tag/scope filters (e.g. require all requested tags or scope).
773
+
774
+ ### 13.7.6 Dependencies for replication
775
+
776
+ ```json
777
+ {
778
+ "@huggingface/transformers": "^3.4.2",
779
+ "better-sqlite3": "^11.10.0",
780
+ "sqlite-vec": "^0.1.6"
781
+ }
782
+ ```
783
+
784
+ Node must support native addons for `better-sqlite3` and `sqlite-vec`. First run downloads `Xenova/multilingual-e5-large` from Hugging Face (no API key for the model).
785
+
786
+ ### 13.7.7 Summary table
787
+
788
+ | Component | Technology / model |
789
+ |-----------------|---------------------|
790
+ | Embedding model | Xenova/multilingual-e5-large (1024 dims) |
791
+ | Embedding lib | @huggingface/transformers, feature-extraction pipeline |
792
+ | Pooling/norm | mean pooling, normalize: true |
793
+ | DB | SQLite (better-sqlite3) |
794
+ | Vector index | sqlite-vec vec0, cosine |
795
+ | Match rule | cosine similarity ≥ 0.87 (configurable via `embeddings.similarityThreshold`) |
796
+
797
+ ---
798
+
799
+ ## 13.8 Search score blending (full-text vs semantic)
800
+
801
+ Retrieval can combine full-text (FTS5/BM25) and semantic (vector) results. The blend is controlled by configuration.
802
+
803
+ ### 13.8.1 When semantic indexing is disabled
804
+
805
+ * **Effective weights:** 100% full-text, 0% semantic.
806
+ * **Behavior:** All search (e.g. `mem_search`, `mem_context`) uses only FTS5 and BM25. Config keys such as `search.semanticWeight` are ignored; no vector queries are run.
807
+
808
+ ### 13.8.2 When semantic indexing is enabled
809
+
810
+ * **Effective weights:** Configurable via `search.ftsWeight` and `search.semanticWeight` (see §18).
811
+ * **Example:** `ftsWeight: 0.7`, `semanticWeight: 0.3` → 70% full-text, 30% semantic.
812
+ * **Normalization:** Weights should be normalized so that `ftsWeight + semanticWeight === 1.0`. If only one source is available (e.g. no vectors for a chunk), the available scores are used and the blend applies to the set of results that have both FTS and semantic scores where applicable.
813
+ * **Combination:** Implementation may use normalized FTS score (e.g. BM25 normalized to a 0–1 scale) and semantic similarity (already 0–1); combined score = `ftsWeight * normFtsScore + semanticWeight * similarity`. Ranking and filtering (e.g. `minScore`) apply to this combined score.
814
+
815
+ ### 13.8.3 Configuration rule
816
+
817
+ * If `embeddings.enabled === false` → treat as 100% full-text (semantic weight 0).
818
+ * If `embeddings.enabled === true` → use `search.ftsWeight` and `search.semanticWeight`; default suggestion: 70% FTS, 30% semantic (e.g. `0.7` and `0.3`).
819
+
820
+ ---
821
+
822
+ # 14. MCP compatibility requirements
823
+
824
+ The system must expose the same Engram-compatible tools.
825
+
826
+ ## 14.1 Required tools
827
+
828
+ * `mem_save`
829
+ * `mem_update`
830
+ * `mem_delete`
831
+ * `mem_suggest_topic_key`
832
+ * `mem_search`
833
+ * `mem_context`
834
+ * `mem_timeline`
835
+ * `mem_get_observation`
836
+ * `mem_session_summary`
837
+ * `mem_session_start`
838
+ * `mem_session_end`
839
+ * `mem_save_prompt`
840
+ * `mem_stats`
841
+
842
+ ## 14.2 Compatibility rule
843
+
844
+ This system does not need to match Engram internally. It must match it **at the MCP tool contract level** closely enough to replace it in practice.
845
+
846
+ ---
847
+
848
+ # 15. Identity, upsert, and deduplication
849
+
850
+ ## 15.1 Resolution order
851
+
852
+ 1. If `id` is provided, update that exact document
853
+ 2. If `topic_key` exists within the relevant scope, update that document
854
+ 3. If recent normalized content matches, increment `duplicate_count`
855
+ 4. Otherwise create a new Markdown document
856
+
857
+ ## 15.2 Scope-aware topic resolution
858
+
859
+ Topic matching must respect scope:
860
+
861
+ * `home` topic keys should update global documents
862
+ * `project` topic keys should update local documents
863
+ * project tools may optionally consult `home` for reuse suggestions, but should not overwrite home documents accidentally
864
+
865
+ ---
866
+
867
+ # 16. Search behavior with `home` and `project`
868
+
869
+ ## 16.1 Search modes
870
+
871
+ * `project-only`
872
+ * `home-only`
873
+ * `merged`
874
+
875
+ ## 16.2 Recommended default
876
+
877
+ For project workflows, use `merged`.
878
+
879
+ That gives the agent:
880
+
881
+ * local project context
882
+ * shared global knowledge
883
+
884
+ ## 16.3 Ranking policy
885
+
886
+ Default ranking should slightly prefer:
887
+
888
+ 1. exact `topic_key`
889
+ 2. same project
890
+ 3. same scope
891
+ 4. home/global fallback
892
+ 5. when semantic indexing is enabled: combined score from full-text and semantic (see §13.8); when disabled: full-text only (100% FTS).
893
+
894
+ ---
895
+
896
+ # 17. Suggested internal Node.js structure
897
+
898
+ ```text
899
+ src/
900
+ mcp/
901
+ server.ts
902
+ tools/
903
+ core/
904
+ documents/
905
+ indexing/
906
+ search/
907
+ sessions/
908
+ prompts/
909
+ embeddings/
910
+ db/
911
+ sqlite.ts
912
+ migrations/
913
+ fts/
914
+ vec/
915
+ fs/
916
+ watcher.ts
917
+ paths.ts
918
+ utils/
919
+ hash.ts
920
+ markdown.ts
921
+ yaml.ts
922
+ ids.ts
923
+ ```
924
+
925
+ ---
926
+
927
+ # 18. Suggested config
928
+
929
+ ```json
930
+ {
931
+ "homeMemoryRoot": "./home/memory",
932
+ "projectsRoot": "./projects",
933
+ "sqlitePath": "./.Mixgram/index.db",
934
+ "watch": true,
935
+ "indexing": {
936
+ "chunkSize": 1200,
937
+ "chunkOverlap": 120,
938
+ "reindexOnStartup": true,
939
+ "ftsWeights": {
940
+ "title": 10.0,
941
+ "topicKey": 8.0,
942
+ "heading": 5.0,
943
+ "body": 1.0
944
+ }
945
+ },
946
+ "embeddings": {
947
+ "enabled": false,
948
+ "mode": "optional",
949
+ "provider": "local-huggingface",
950
+ "vectorStore": "sqlite-vec",
951
+ "model": "default-seekmix-reference",
952
+ "workerConcurrency": 2,
953
+ "maxRetries": 3,
954
+ "queueOnWrite": true,
955
+ "similarityThreshold": 0.87
956
+ },
957
+ "search": {
958
+ "mode": "fts-only",
959
+ "defaultLimit": 10,
960
+ "ftsWeight": 0.7,
961
+ "semanticWeight": 0.3,
962
+ "defaultScopeMode": "merged"
963
+ }
964
+ }
965
+ ```
966
+
967
+ **Search weights.** When `embeddings.enabled` is `false`, search uses **100% full-text** (FTS5/BM25); `search.semanticWeight` is ignored. When `embeddings.enabled` is `true`, `search.ftsWeight` and `search.semanticWeight` define the blend (e.g. `0.7` and `0.3` → 70% full-text, 30% semantic). These two values should be normalized (e.g. sum to 1.0).
968
+
969
+ ---
970
+
971
+ The system is acceptable when:
972
+
973
+ 1. It can replace Engram at the MCP interface level
974
+ 2. Saving memory creates or updates Markdown and immediately updates the text index
975
+ 3. Manual Markdown edits trigger incremental reindexing
976
+ 4. Deleting SQLite and running full reindex reconstructs the index from files
977
+ 5. The system works fully with embeddings disabled
978
+ 6. When embeddings are enabled, they run asynchronously and never block save/update flows
979
+ 7. `home` knowledge is reusable across projects
980
+ 8. Project searches can retrieve both local and home knowledge
981
+ 9. File organization is understandable to humans without opening SQLite
982
+ 10. Git diffs remain meaningful for durable knowledge
983
+
984
+ ---
985
+
986
+ # 20. Executive summary
987
+
988
+ This design keeps **Engram’s MCP interface**, but replaces opaque database-first memory with a more durable and inspectable structure:
989
+
990
+ * **Markdown is the system of record**
991
+ * **SQLite is a rebuildable retrieval layer**
992
+ * **SearchMix inspires the text indexing model**
993
+ * **SeekMix inspires the optional semantic layer**
994
+ * **Embeddings are optional and asynchronous**
995
+ * **Global reusable knowledge lives under `home/`**
996
+ * **Project-specific knowledge lives under each project’s `memory/` directory**
997
+
998
+ And the key structural decision is this:
999
+
1000
+ > **Store durable knowledge by scope and type, not by date. Use dates only when time is actually part of the meaning.**
1001
+
1002
+ If you want, next I can turn this into a **clean Markdown file artifact** named something like `SDD-Mixgram-final-en.md`, or I can produce the **black-box TDD in English** that matches this SDD.