mixgram 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.mixgram/index.db +0 -0
- package/.mixgram/test-esm.db +0 -0
- package/.mixgram/test-index.db +0 -0
- package/.mixgram/test-stage1.db +0 -0
- package/README.md +335 -0
- package/bin/mixgram.js +309 -0
- package/index.js +9 -0
- package/package.json +51 -0
- package/projects/castlebravo/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
- package/projects/test/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
- package/spec/fixtures/realistic-content.js +97 -0
- package/spec/reporter.js +127 -0
- package/spec/run-tests-runner.js +40 -0
- package/spec/run-tests.js +160 -0
- package/spec/scenarios/scenarios.js +520 -0
- package/spec/sdd.md +1002 -0
- package/spec/tdd.md +1237 -0
- package/src/config.js +56 -0
- package/src/core/documents/documents.js +207 -0
- package/src/core/embeddings/embedder.js +58 -0
- package/src/core/embeddings/queue.js +118 -0
- package/src/core/embeddings/vectorStore.js +127 -0
- package/src/core/embeddings/worker.js +27 -0
- package/src/core/indexing/indexer.js +131 -0
- package/src/core/indexing/parser.js +78 -0
- package/src/core/indexing/reindex.js +101 -0
- package/src/core/prompts/prompts.js +15 -0
- package/src/core/search/search.js +223 -0
- package/src/core/sessions/sessions.js +27 -0
- package/src/db/migrations/001_initial.sql +86 -0
- package/src/db/sqlite.js +39 -0
- package/src/fs/paths.js +79 -0
- package/src/fs/watcher.js +49 -0
- package/src/mcp/cli-adapter.js +118 -0
- package/src/mcp/server.js +47 -0
- package/src/mcp/tool-registry.js +138 -0
- package/src/mcp/tools.js +159 -0
- package/src/utils/hash.js +6 -0
- package/src/utils/ids.js +22 -0
- package/src/utils/markdown.js +49 -0
package/spec/sdd.md
ADDED
|
@@ -0,0 +1,1002 @@
|
|
|
1
|
+
# SDD
|
|
2
|
+
|
|
3
|
+
## Mixgram
|
|
4
|
+
|
|
5
|
+
### A Node.js project memory system compatible with Engram MCP, using Markdown as the source of truth, SQLite as the text index, and optional asynchronous local embeddings
|
|
6
|
+
|
|
7
|
+
**Status:** Final
|
|
8
|
+
**Target language:** Node.js
|
|
9
|
+
**Compatibility target:** drop-in MCP replacement for Engram
|
|
10
|
+
**Source of truth:** Markdown files
|
|
11
|
+
**Text index:** SQLite + FTS5
|
|
12
|
+
**Semantic index:** optional, local, asynchronous
|
|
13
|
+
**Technical references:** Engram, SearchMix, SeekMix
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# 1. Purpose
|
|
18
|
+
|
|
19
|
+
This document defines a memory system for software projects and agents that preserves **Engram-compatible MCP interfaces**, while replacing Engram’s storage model with a more transparent architecture:
|
|
20
|
+
|
|
21
|
+
* **Markdown files are the canonical memory store**
|
|
22
|
+
* **SQLite is a derived and rebuildable index**
|
|
23
|
+
* **Embeddings are optional**
|
|
24
|
+
* **When enabled, embeddings run asynchronously in the background**
|
|
25
|
+
|
|
26
|
+
The goal is to preserve Engram’s external MCP tool contract while improving visibility, Git compatibility, manual editing, and long-term maintainability.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
# 2. Core goals
|
|
31
|
+
|
|
32
|
+
## 2.1 Functional goals
|
|
33
|
+
|
|
34
|
+
* Replace Engram at the MCP interface level
|
|
35
|
+
* Preserve the same tool names and external semantics
|
|
36
|
+
* Store durable memory as Markdown files
|
|
37
|
+
* Maintain a rebuildable SQLite text index
|
|
38
|
+
* Reindex new and modified files automatically
|
|
39
|
+
* Support fast full-text retrieval
|
|
40
|
+
* Support optional local embeddings
|
|
41
|
+
* Continue working fully when embeddings are disabled
|
|
42
|
+
|
|
43
|
+
## 2.2 Non-functional goals
|
|
44
|
+
|
|
45
|
+
* Human-readable memory
|
|
46
|
+
* Git-friendly workflows
|
|
47
|
+
* Minimal external dependencies
|
|
48
|
+
* Rebuildable derived state
|
|
49
|
+
* Graceful degradation when semantic indexing is unavailable
|
|
50
|
+
* Strong support for manual editing
|
|
51
|
+
* Clear knowledge reuse across projects
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
# 3. Design principles
|
|
56
|
+
|
|
57
|
+
## 3.1 Markdown is canonical
|
|
58
|
+
|
|
59
|
+
All durable, meaningful memory should live as Markdown files.
|
|
60
|
+
|
|
61
|
+
## 3.2 SQLite is derived
|
|
62
|
+
|
|
63
|
+
SQLite is not the canonical store. It is a rebuildable search index and retrieval layer.
|
|
64
|
+
|
|
65
|
+
## 3.3 MCP compatibility first
|
|
66
|
+
|
|
67
|
+
The system must expose the same MCP tools as Engram so it can replace it in existing integrations.
|
|
68
|
+
|
|
69
|
+
## 3.4 Full-text is mandatory, embeddings are optional
|
|
70
|
+
|
|
71
|
+
The system must work correctly with SQLite FTS5 alone. Semantic retrieval is an enhancement, not a requirement.
|
|
72
|
+
|
|
73
|
+
## 3.5 Embeddings must be asynchronous
|
|
74
|
+
|
|
75
|
+
If embeddings are enabled, they must be computed in the background. Saving or updating memory must never block on embedding generation.
|
|
76
|
+
|
|
77
|
+
## 3.6 Human edits win
|
|
78
|
+
|
|
79
|
+
If a Markdown file is edited manually, the system must treat the file as authoritative and reindex it.
|
|
80
|
+
|
|
81
|
+
## 3.7 Knowledge should support multiple scopes
|
|
82
|
+
|
|
83
|
+
Memory should support both:
|
|
84
|
+
|
|
85
|
+
* **project-local knowledge**
|
|
86
|
+
* **cross-project reusable knowledge**
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
# 4. Technical references
|
|
91
|
+
|
|
92
|
+
## 4.1 Engram
|
|
93
|
+
|
|
94
|
+
Engram is the MCP compatibility reference. Its tool names, argument structure, and overall memory workflow define the external contract to preserve.
|
|
95
|
+
|
|
96
|
+
## 4.2 SearchMix
|
|
97
|
+
|
|
98
|
+
SearchMix is the reference for:
|
|
99
|
+
|
|
100
|
+
* Markdown-first document indexing
|
|
101
|
+
* SQLite FTS5
|
|
102
|
+
* BM25 ranking
|
|
103
|
+
* incremental reindexing
|
|
104
|
+
* document/path-centered indexing
|
|
105
|
+
* snippets and structured text retrieval
|
|
106
|
+
|
|
107
|
+
**Adaptation to Mixgram.** This design takes the same ideas—Markdown parsing, FTS5, BM25, mtime-based incremental reindex, and snippets—but applies them to a **chunk-based** model with **frontmatter**, **scope/type/project**, and an **external file watcher** that triggers indexing. SearchMix has no built-in watch; Mixgram does.
|
|
108
|
+
|
|
109
|
+
## 4.3 SeekMix
|
|
110
|
+
|
|
111
|
+
SeekMix is the reference for:
|
|
112
|
+
|
|
113
|
+
* optional local embeddings
|
|
114
|
+
* local vector storage
|
|
115
|
+
* configurable semantic thresholds
|
|
116
|
+
* provider abstraction for embedding generation
|
|
117
|
+
|
|
118
|
+
In this design, SeekMix is used as the semantic layer reference, not as a semantic cache model.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
# 5. Architecture overview
|
|
123
|
+
|
|
124
|
+
```text
|
|
125
|
+
MCP Client / Agent
|
|
126
|
+
|
|
|
127
|
+
v
|
|
128
|
+
MCP Server (Node.js)
|
|
129
|
+
|
|
|
130
|
+
+----------------------------------+
|
|
131
|
+
| |
|
|
132
|
+
v v
|
|
133
|
+
Markdown Memory Store SQLite Index
|
|
134
|
+
(source of truth) (derived state)
|
|
135
|
+
| |
|
|
136
|
+
| +-- documents
|
|
137
|
+
| +-- chunks
|
|
138
|
+
| +-- FTS5
|
|
139
|
+
| +-- sessions
|
|
140
|
+
| +-- prompts
|
|
141
|
+
| +-- tags
|
|
142
|
+
| +-- embedding_jobs
|
|
143
|
+
| +-- vector metadata
|
|
144
|
+
|
|
|
145
|
+
v
|
|
146
|
+
File Watcher / Incremental Reindexer
|
|
147
|
+
|
|
|
148
|
+
v
|
|
149
|
+
Optional Local Embedding Worker
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
# 6. Directory conventions
|
|
155
|
+
|
|
156
|
+
This is the part I would definitely adjust based on what you said.
|
|
157
|
+
|
|
158
|
+
A **date-first filename convention should not be the default**. Date is useful only for memories where chronology matters. For architecture, reusable knowledge, decisions, patterns, and reference notes, the more important dimensions are:
|
|
159
|
+
|
|
160
|
+
* scope
|
|
161
|
+
* topic
|
|
162
|
+
* type
|
|
163
|
+
* project relevance
|
|
164
|
+
|
|
165
|
+
## 6.1 Recommended root structure
|
|
166
|
+
|
|
167
|
+
```text
|
|
168
|
+
workspace/
|
|
169
|
+
home/
|
|
170
|
+
memory/
|
|
171
|
+
architecture/
|
|
172
|
+
decisions/
|
|
173
|
+
patterns/
|
|
174
|
+
reference/
|
|
175
|
+
learnings/
|
|
176
|
+
discoveries/
|
|
177
|
+
prompts/
|
|
178
|
+
projects/
|
|
179
|
+
project-a/
|
|
180
|
+
memory/
|
|
181
|
+
architecture/
|
|
182
|
+
decisions/
|
|
183
|
+
bugs/
|
|
184
|
+
sessions/
|
|
185
|
+
learnings/
|
|
186
|
+
discoveries/
|
|
187
|
+
generated/
|
|
188
|
+
project-b/
|
|
189
|
+
memory/
|
|
190
|
+
architecture/
|
|
191
|
+
decisions/
|
|
192
|
+
bugs/
|
|
193
|
+
sessions/
|
|
194
|
+
learnings/
|
|
195
|
+
discoveries/
|
|
196
|
+
generated/
|
|
197
|
+
.mixgram/
|
|
198
|
+
index.db
|
|
199
|
+
jobs/
|
|
200
|
+
logs/
|
|
201
|
+
cache/
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
## 6.2 Meaning of `home/`
|
|
205
|
+
|
|
206
|
+
`home/memory/` stores **cross-project knowledge**.
|
|
207
|
+
|
|
208
|
+
This includes:
|
|
209
|
+
|
|
210
|
+
* reusable architecture notes
|
|
211
|
+
* patterns
|
|
212
|
+
* general discoveries
|
|
213
|
+
* general learnings
|
|
214
|
+
* prompts or workflows relevant across projects
|
|
215
|
+
* implementation techniques that should be searchable from any project context
|
|
216
|
+
|
|
217
|
+
This lets the system reuse knowledge globally instead of duplicating it inside each project.
|
|
218
|
+
|
|
219
|
+
## 6.3 Meaning of `projects/<name>/memory/`
|
|
220
|
+
|
|
221
|
+
This stores **project-scoped knowledge**.
|
|
222
|
+
|
|
223
|
+
This includes:
|
|
224
|
+
|
|
225
|
+
* project decisions
|
|
226
|
+
* project-specific bugs
|
|
227
|
+
* session summaries
|
|
228
|
+
* feature notes
|
|
229
|
+
* implementation details tied to that project
|
|
230
|
+
* generated notes that should remain local
|
|
231
|
+
|
|
232
|
+
## 6.4 Scope model
|
|
233
|
+
|
|
234
|
+
The system should support at least these scopes:
|
|
235
|
+
|
|
236
|
+
* `home`
|
|
237
|
+
* `project`
|
|
238
|
+
* `session`
|
|
239
|
+
* `generated`
|
|
240
|
+
|
|
241
|
+
This scope must be represented both in:
|
|
242
|
+
|
|
243
|
+
* Markdown frontmatter
|
|
244
|
+
* SQLite metadata
|
|
245
|
+
|
|
246
|
+
---
|
|
247
|
+
|
|
248
|
+
# 7. Document classification model
|
|
249
|
+
|
|
250
|
+
Instead of relying primarily on dates in filenames, the system should classify memory by **type + scope + topic**.
|
|
251
|
+
|
|
252
|
+
## 7.1 Recommended document types
|
|
253
|
+
|
|
254
|
+
* `architecture`
|
|
255
|
+
* `decision`
|
|
256
|
+
* `bug`
|
|
257
|
+
* `learning`
|
|
258
|
+
* `discovery`
|
|
259
|
+
* `pattern`
|
|
260
|
+
* `reference`
|
|
261
|
+
* `session_summary`
|
|
262
|
+
* `prompt`
|
|
263
|
+
* `generated_note`
|
|
264
|
+
|
|
265
|
+
## 7.2 Recommended directory mapping
|
|
266
|
+
|
|
267
|
+
```text
|
|
268
|
+
memory/
|
|
269
|
+
architecture/
|
|
270
|
+
decisions/
|
|
271
|
+
bugs/
|
|
272
|
+
learnings/
|
|
273
|
+
discoveries/
|
|
274
|
+
patterns/
|
|
275
|
+
reference/
|
|
276
|
+
sessions/
|
|
277
|
+
prompts/
|
|
278
|
+
generated/
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
This gives both humans and agents a predictable structure.
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
# 8. File naming convention
|
|
286
|
+
|
|
287
|
+
I would recommend moving away from date-first naming except when time is central.
|
|
288
|
+
|
|
289
|
+
## 8.1 Default naming format
|
|
290
|
+
|
|
291
|
+
```text
|
|
292
|
+
<type>--<slug>.md
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
Examples:
|
|
296
|
+
|
|
297
|
+
```text
|
|
298
|
+
decision--use-sqlite-derived-index.md
|
|
299
|
+
architecture--memory-layer-layout.md
|
|
300
|
+
pattern--incremental-reindexing.md
|
|
301
|
+
reference--sqlite-fts-ranking-notes.md
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
## 8.2 When date should be included
|
|
305
|
+
|
|
306
|
+
Only include time in the filename when chronology is important, for example:
|
|
307
|
+
|
|
308
|
+
* session summaries
|
|
309
|
+
* incident investigations
|
|
310
|
+
* temporary discoveries
|
|
311
|
+
* logs or research trails
|
|
312
|
+
* time-sensitive notes
|
|
313
|
+
|
|
314
|
+
Examples:
|
|
315
|
+
|
|
316
|
+
```text
|
|
317
|
+
session-summary--2026-03-08--agent-debugging.md
|
|
318
|
+
bug--2026-03-08--timeline-ordering-regression.md
|
|
319
|
+
discovery--2026-03-08--redis-vs-sqlite-observation.md
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
## 8.3 Topic key is more important than the filename
|
|
323
|
+
|
|
324
|
+
The true identity of evolving knowledge should be driven primarily by:
|
|
325
|
+
|
|
326
|
+
* `id`
|
|
327
|
+
* `topic_key`
|
|
328
|
+
* `scope`
|
|
329
|
+
* `project`
|
|
330
|
+
|
|
331
|
+
The filename is a stable human convenience, not the core identity mechanism.
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
# 9. Markdown frontmatter
|
|
336
|
+
|
|
337
|
+
Each memory document should include structured frontmatter.
|
|
338
|
+
|
|
339
|
+
```md
|
|
340
|
+
---
|
|
341
|
+
id: obs_01HXYZ123ABC
|
|
342
|
+
title: Use SQLite as derived index
|
|
343
|
+
type: decision
|
|
344
|
+
scope: project
|
|
345
|
+
project: castlebravo
|
|
346
|
+
topic_key: architecture/sqlite-derived-index
|
|
347
|
+
session_id: sess_01HXYZ456DEF
|
|
348
|
+
tool_name: mem_save
|
|
349
|
+
created_at: 2026-03-08T18:00:00Z
|
|
350
|
+
updated_at: 2026-03-08T18:00:00Z
|
|
351
|
+
revision_count: 1
|
|
352
|
+
duplicate_count: 0
|
|
353
|
+
deleted: false
|
|
354
|
+
tags:
|
|
355
|
+
- sqlite
|
|
356
|
+
- indexing
|
|
357
|
+
- architecture
|
|
358
|
+
embedding_status: pending
|
|
359
|
+
---
|
|
360
|
+
|
|
361
|
+
# Summary
|
|
362
|
+
|
|
363
|
+
SQLite will be used as a derived index over Markdown documents.
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
## 9.1 Rules
|
|
367
|
+
|
|
368
|
+
* `project` may be omitted for `home` scope
|
|
369
|
+
* `scope: home` means globally reusable knowledge
|
|
370
|
+
* `scope: project` means project-local knowledge
|
|
371
|
+
* `scope: session` is intended for chronological notes
|
|
372
|
+
* `scope: generated` is for machine-generated notes
|
|
373
|
+
|
|
374
|
+
---
|
|
375
|
+
|
|
376
|
+
# 10. Cross-project knowledge behavior
|
|
377
|
+
|
|
378
|
+
This is an important addition.
|
|
379
|
+
|
|
380
|
+
## 10.1 Home knowledge must be searchable everywhere
|
|
381
|
+
|
|
382
|
+
When searching inside a project, the system should be able to retrieve:
|
|
383
|
+
|
|
384
|
+
* project-local memory
|
|
385
|
+
* home/global memory
|
|
386
|
+
|
|
387
|
+
This means project search should support:
|
|
388
|
+
|
|
389
|
+
* `local-only`
|
|
390
|
+
* `home-only`
|
|
391
|
+
* `merged`
|
|
392
|
+
|
|
393
|
+
The recommended default is **merged**, with local results slightly boosted.
|
|
394
|
+
|
|
395
|
+
## 10.2 Ranking recommendation
|
|
396
|
+
|
|
397
|
+
If the same topic exists in both `home` and project scope:
|
|
398
|
+
|
|
399
|
+
* prefer project-specific content when the query is project-contextual
|
|
400
|
+
* still allow home results as fallback or supporting context
|
|
401
|
+
|
|
402
|
+
Suggested bias:
|
|
403
|
+
|
|
404
|
+
* `project` scope gets a ranking bonus in project searches
|
|
405
|
+
* `home` scope remains fully visible
|
|
406
|
+
|
|
407
|
+
## 10.3 Reuse rule
|
|
408
|
+
|
|
409
|
+
If knowledge is broadly reusable, it should be saved under `home/memory/` instead of duplicating it into multiple projects.
|
|
410
|
+
|
|
411
|
+
---
|
|
412
|
+
|
|
413
|
+
# 11. SQLite data model
|
|
414
|
+
|
|
415
|
+
## 11.1 `documents`
|
|
416
|
+
|
|
417
|
+
```sql
|
|
418
|
+
CREATE TABLE documents (
|
|
419
|
+
id TEXT PRIMARY KEY,
|
|
420
|
+
path TEXT NOT NULL UNIQUE,
|
|
421
|
+
title TEXT,
|
|
422
|
+
type TEXT NOT NULL,
|
|
423
|
+
scope TEXT NOT NULL,
|
|
424
|
+
project TEXT,
|
|
425
|
+
topic_key TEXT,
|
|
426
|
+
session_id TEXT,
|
|
427
|
+
tool_name TEXT,
|
|
428
|
+
content_hash TEXT NOT NULL,
|
|
429
|
+
file_mtime_ms INTEGER,
|
|
430
|
+
created_at TEXT NOT NULL,
|
|
431
|
+
updated_at TEXT NOT NULL,
|
|
432
|
+
indexed_at TEXT,
|
|
433
|
+
deleted_at TEXT,
|
|
434
|
+
revision_count INTEGER NOT NULL DEFAULT 1,
|
|
435
|
+
duplicate_count INTEGER NOT NULL DEFAULT 0,
|
|
436
|
+
embedding_status TEXT NOT NULL DEFAULT 'disabled'
|
|
437
|
+
);
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
## 11.2 `document_chunks`
|
|
441
|
+
|
|
442
|
+
```sql
|
|
443
|
+
CREATE TABLE document_chunks (
|
|
444
|
+
id TEXT PRIMARY KEY,
|
|
445
|
+
document_id TEXT NOT NULL,
|
|
446
|
+
chunk_index INTEGER NOT NULL,
|
|
447
|
+
heading_path TEXT,
|
|
448
|
+
heading_level INTEGER,
|
|
449
|
+
content TEXT NOT NULL,
|
|
450
|
+
token_count INTEGER,
|
|
451
|
+
content_hash TEXT NOT NULL,
|
|
452
|
+
created_at TEXT NOT NULL,
|
|
453
|
+
updated_at TEXT NOT NULL,
|
|
454
|
+
FOREIGN KEY(document_id) REFERENCES documents(id)
|
|
455
|
+
);
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
## 11.3 `document_tags`
|
|
459
|
+
|
|
460
|
+
```sql
|
|
461
|
+
CREATE TABLE document_tags (
|
|
462
|
+
document_id TEXT NOT NULL,
|
|
463
|
+
tag TEXT NOT NULL,
|
|
464
|
+
PRIMARY KEY (document_id, tag),
|
|
465
|
+
FOREIGN KEY(document_id) REFERENCES documents(id)
|
|
466
|
+
);
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
## 11.4 `sessions`
|
|
470
|
+
|
|
471
|
+
```sql
|
|
472
|
+
CREATE TABLE sessions (
|
|
473
|
+
id TEXT PRIMARY KEY,
|
|
474
|
+
project TEXT NOT NULL,
|
|
475
|
+
directory TEXT,
|
|
476
|
+
started_at TEXT NOT NULL,
|
|
477
|
+
ended_at TEXT,
|
|
478
|
+
summary TEXT
|
|
479
|
+
);
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
## 11.5 `prompts`
|
|
483
|
+
|
|
484
|
+
```sql
|
|
485
|
+
CREATE TABLE prompts (
|
|
486
|
+
id TEXT PRIMARY KEY,
|
|
487
|
+
session_id TEXT NOT NULL,
|
|
488
|
+
project TEXT,
|
|
489
|
+
content TEXT NOT NULL,
|
|
490
|
+
created_at TEXT NOT NULL,
|
|
491
|
+
FOREIGN KEY(session_id) REFERENCES sessions(id)
|
|
492
|
+
);
|
|
493
|
+
```
|
|
494
|
+
|
|
495
|
+
## 11.6 `embedding_jobs`
|
|
496
|
+
|
|
497
|
+
```sql
|
|
498
|
+
CREATE TABLE embedding_jobs (
|
|
499
|
+
id TEXT PRIMARY KEY,
|
|
500
|
+
chunk_id TEXT NOT NULL,
|
|
501
|
+
model_name TEXT NOT NULL,
|
|
502
|
+
status TEXT NOT NULL,
|
|
503
|
+
attempts INTEGER NOT NULL DEFAULT 0,
|
|
504
|
+
last_error TEXT,
|
|
505
|
+
created_at TEXT NOT NULL,
|
|
506
|
+
updated_at TEXT NOT NULL,
|
|
507
|
+
FOREIGN KEY(chunk_id) REFERENCES document_chunks(id)
|
|
508
|
+
);
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
## 11.7 FTS5
|
|
512
|
+
|
|
513
|
+
```sql
|
|
514
|
+
CREATE VIRTUAL TABLE document_chunks_fts USING fts5(
|
|
515
|
+
chunk_id UNINDEXED,
|
|
516
|
+
document_id UNINDEXED,
|
|
517
|
+
title,
|
|
518
|
+
topic_key,
|
|
519
|
+
type,
|
|
520
|
+
scope,
|
|
521
|
+
project,
|
|
522
|
+
heading_path,
|
|
523
|
+
body,
|
|
524
|
+
tokenize = 'unicode61 remove_diacritics 2'
|
|
525
|
+
);
|
|
526
|
+
```
|
|
527
|
+
|
|
528
|
+
**Column roles.** The FTS5 table stores two kinds of columns. **UNINDEXED** columns (`chunk_id`, `document_id`) are kept for joins and identity but are not full-text indexed. All other columns (`title`, `topic_key`, `type`, `scope`, `project`, `heading_path`, `body`) are **indexed** and participate in `MATCH` and BM25 ranking.
|
|
529
|
+
|
|
530
|
+
**Normalization.** Indexable text should be normalized so that search is case- and accent-insensitive. The tokenizer `unicode61 remove_diacritics 2` contributes to this. Any application-side normalization (e.g. NFD, remove diacritics, lowercase) for title or heading_path should match the same rules so that query terms align with indexed content.
|
|
531
|
+
|
|
532
|
+
**BM25.** Ranking uses FTS5’s BM25 with configurable weights (see §18 `indexing.ftsWeights`: title, topicKey, heading, body) so that matches in title and topic_key rank higher than body-only matches.
|
|
533
|
+
|
|
534
|
+
---
|
|
535
|
+
|
|
536
|
+
## 11.8 Vector cache and vec tables (optional, when semantic indexing enabled)
|
|
537
|
+
|
|
538
|
+
When embeddings are enabled, the same SQLite DB (or a dedicated DB) holds per-model cache and vector tables. Suffix is derived from the sanitized embedding model name (e.g. `Xenova_multilingual_e5_large`).
|
|
539
|
+
|
|
540
|
+
**Metadata table** (links chunk to vector row):
|
|
541
|
+
|
|
542
|
+
```sql
|
|
543
|
+
CREATE TABLE cache_<suffix> (
|
|
544
|
+
id INTEGER PRIMARY KEY,
|
|
545
|
+
chunk_id TEXT NOT NULL UNIQUE,
|
|
546
|
+
document_id TEXT NOT NULL,
|
|
547
|
+
content_hash TEXT NOT NULL,
|
|
548
|
+
embedding_model TEXT NOT NULL,
|
|
549
|
+
created_at TEXT NOT NULL,
|
|
550
|
+
FOREIGN KEY(chunk_id) REFERENCES document_chunks(id),
|
|
551
|
+
FOREIGN KEY(document_id) REFERENCES documents(id)
|
|
552
|
+
);
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
**Vector table** (sqlite-vec virtual table; rowid must align with `cache_<suffix>.id`):
|
|
556
|
+
|
|
557
|
+
```sql
|
|
558
|
+
-- After sqliteVec.load(db):
|
|
559
|
+
CREATE VIRTUAL TABLE vec_<suffix> USING vec0(
|
|
560
|
+
embedding float[1024],
|
|
561
|
+
distance_metric=cosine
|
|
562
|
+
);
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
**Insert:** Insert into `cache_<suffix>`, then insert into `vec_<suffix>` with `rowid = cache_<suffix>.id` and `embedding = Float32Array(embedding)`. **Search:** KNN via `WHERE embedding MATCH ? AND k = N`, then join `vec_<suffix>.rowid = cache_<suffix>.id`. See §13.7.4 and §13.7.5.
|
|
566
|
+
|
|
567
|
+
---
|
|
568
|
+
|
|
569
|
+
# 12. Text indexing
|
|
570
|
+
|
|
571
|
+
## 12.1 Immediate consistency
|
|
572
|
+
|
|
573
|
+
Every create or update operation must synchronously update:
|
|
574
|
+
|
|
575
|
+
* documents
|
|
576
|
+
* chunks
|
|
577
|
+
* tags
|
|
578
|
+
* FTS
|
|
579
|
+
|
|
580
|
+
## 12.2 Incremental reindexing
|
|
581
|
+
|
|
582
|
+
The watcher/indexer must:
|
|
583
|
+
|
|
584
|
+
* detect new files
|
|
585
|
+
* detect modified files
|
|
586
|
+
* skip unchanged files
|
|
587
|
+
* reindex only what changed
|
|
588
|
+
|
|
589
|
+
## 12.3 Chunking strategy
|
|
590
|
+
|
|
591
|
+
Chunking should be semantic:
|
|
592
|
+
|
|
593
|
+
* split by headings when possible
|
|
594
|
+
* preserve heading hierarchy
|
|
595
|
+
* avoid tiny fragments
|
|
596
|
+
* keep chunk size configurable
|
|
597
|
+
|
|
598
|
+
## 12.4 Weighted retrieval
|
|
599
|
+
|
|
600
|
+
Search should favor:
|
|
601
|
+
|
|
602
|
+
* title
|
|
603
|
+
* topic key
|
|
604
|
+
* heading path
|
|
605
|
+
* body
|
|
606
|
+
|
|
607
|
+
This follows the SearchMix-style principle of field-aware ranking.
|
|
608
|
+
|
|
609
|
+
## 12.5 Indexing pipeline (SearchMix-inspired)
|
|
610
|
+
|
|
611
|
+
The indexer follows a single pipeline from input to FTS5 and metadata writes.
|
|
612
|
+
|
|
613
|
+
* **Input.** The pipeline accepts a path to a `.md` file, a directory, or in-memory content (e.g. from `mem_save`).
|
|
614
|
+
* **Type detection.** The system distinguishes file vs directory. For directories, it recurses and selects `*.md` / `*.markdown` with optional exclude patterns.
|
|
615
|
+
* **Read and encoding.** File content is read; encoding is detected (e.g. jschardet) and decoded to UTF-8 (e.g. iconv-lite) so that non-UTF-8 Markdown is supported.
|
|
616
|
+
* **Markdown parsing.** Content is parsed with a unified/remark-style pipeline to an AST. The implementation extracts frontmatter (id, title, type, scope, project, topic_key, etc.) and body structure (headings h1–h6, paragraphs, optional code blocks), and builds a section tree (id, parentId, childrenIds) and a flat sections index with positions for snippet boundaries.
|
|
617
|
+
* **Chunking.** Chunks are derived from the section tree and headings (see §12.3). Each chunk gets heading_path, heading_level, content, and optional token_count and content_hash.
|
|
618
|
+
* **Write.** The pipeline updates `documents`, `document_chunks`, `document_tags`, and the FTS5 table, and stores `file_mtime_ms` (or equivalent) for mtime-based skip on the next run.
|
|
619
|
+
|
|
620
|
+
This pipeline is invoked by (a) the file watcher when a file is added or changed, (b) explicit reindex APIs, and (c) `mem_save` / `mem_update` for in-memory content.
|
|
621
|
+
|
|
622
|
+
## 12.6 Watch and incremental reindex (difference from SearchMix)
|
|
623
|
+
|
|
624
|
+
SearchMix has no built-in file watch; reindexing happens only when the caller invokes an add-document API, and mtime is used inside that call to skip unchanged files.
|
|
625
|
+
|
|
626
|
+
Mixgram includes a **file watcher** (e.g. chokidar or `fs.watch`) that observes configured roots for new, changed, and deleted Markdown files and calls the indexer (add/update/remove). When the indexer runs—whether from the watcher or from an explicit reindex—it uses **mtime** (with optional tolerance, e.g. 1s) to skip unchanged files and reindex only modified or new ones. Options such as skip-when-unchanged (e.g. `checkModified` / `skipExisting`) can be supported: skip reindex when the stored mtime matches and the document is already in the index.
|
|
627
|
+
|
|
628
|
+
## 12.7 Search resolution and snippets
|
|
629
|
+
|
|
630
|
+
* **Query execution.** Search is performed with SQLite FTS5 (`WHERE document_chunks_fts MATCH ?`). The query string is **normalized** using the same rules as indexed text (e.g. NFD, remove diacritics, lowercase), while respecting FTS5 operators (AND, OR, NOT, quoted phrases).
|
|
631
|
+
* **Field prefixes.** If the design supports field-scoped search (e.g. `title:`, `body:`), these are mapped to the corresponding FTS5 columns in the normalized query.
|
|
632
|
+
* **Ranking.** Results are ordered by BM25 with the configured weights; an optional `minScore` threshold may filter weak matches.
|
|
633
|
+
* **Filters.** Optional scope, project, or tag filters (e.g. via `documents` / `document_tags`) are applied in SQL (JOIN or subquery).
|
|
634
|
+
* **Snippets.** For each result, snippets are extracted by finding query terms (after stripping operators) in the normalized title, heading_path, and body. The section/chunk structure and stored positions are used to trim context (snippet length configurable) and attach the snippet to the correct chunk/section. Snippet objects include range, document/chunk id, and optional hierarchy (e.g. heading path).
|
|
635
|
+
|
|
636
|
+
## 12.8 Libraries for indexing and search (reference)
|
|
637
|
+
|
|
638
|
+
The following libraries are used for indexing and search. This is a reference list; actual dependencies may be chosen to match the project’s constraints (e.g. ESM vs CJS).
|
|
639
|
+
|
|
640
|
+
| Purpose | Library / stack | Role in Mixgram |
|
|
641
|
+
|---------|------------------|-----------------|
|
|
642
|
+
| SQLite + FTS5 | better-sqlite3 | Database and FTS5 index; BM25 ranking. |
|
|
643
|
+
| Markdown AST | unified, remark-parse | Parse Markdown to AST. |
|
|
644
|
+
| AST traversal | unist-util-visit | Walk AST to extract headings, paragraphs, sections, positions. |
|
|
645
|
+
| Language detection | franc (optional) | Optional tag or metadata for document language. |
|
|
646
|
+
| File discovery | glob (or similar) | List `*.md` / `*.markdown` in directories; optional recursion and exclude (e.g. micromatch). |
|
|
647
|
+
| Encoding | jschardet, iconv-lite | Detect file encoding and decode to UTF-8. |
|
|
648
|
+
| File watch | chokidar or fs.watch | Detect filesystem changes and trigger reindex (Mixgram-specific). |
|
|
649
|
+
|
|
650
|
+
---
|
|
651
|
+
|
|
652
|
+
# 13. Optional semantic layer
|
|
653
|
+
|
|
654
|
+
## 13.1 Rule
|
|
655
|
+
|
|
656
|
+
Embeddings are optional.
|
|
657
|
+
|
|
658
|
+
The system must fully function when embeddings are disabled.
|
|
659
|
+
|
|
660
|
+
## 13.2 When disabled
|
|
661
|
+
|
|
662
|
+
Everything still works:
|
|
663
|
+
|
|
664
|
+
* `mem_save`
|
|
665
|
+
* `mem_update`
|
|
666
|
+
* `mem_delete`
|
|
667
|
+
* `mem_search`
|
|
668
|
+
* `mem_context`
|
|
669
|
+
* `mem_timeline`
|
|
670
|
+
* `mem_get_observation`
|
|
671
|
+
* session tools
|
|
672
|
+
* prompt tools
|
|
673
|
+
|
|
674
|
+
## 13.3 When enabled
|
|
675
|
+
|
|
676
|
+
If embeddings are enabled:
|
|
677
|
+
|
|
678
|
+
* chunks are queued for semantic indexing
|
|
679
|
+
* a local worker processes them asynchronously
|
|
680
|
+
* vector results may improve retrieval quality
|
|
681
|
+
* write operations still return immediately after text indexing
|
|
682
|
+
|
|
683
|
+
## 13.4 Why asynchronous
|
|
684
|
+
|
|
685
|
+
Embedding generation is expensive. It must run behind the main flow and must never block save/update operations.
|
|
686
|
+
|
|
687
|
+
## 13.5 Status values
|
|
688
|
+
|
|
689
|
+
Each chunk can be:
|
|
690
|
+
|
|
691
|
+
* `disabled`
|
|
692
|
+
* `pending`
|
|
693
|
+
* `processing`
|
|
694
|
+
* `completed`
|
|
695
|
+
* `failed`
|
|
696
|
+
* `stale`
|
|
697
|
+
|
|
698
|
+
## 13.6 Invalidation
|
|
699
|
+
|
|
700
|
+
Embeddings become `stale` when:
|
|
701
|
+
|
|
702
|
+
* chunk content changes
|
|
703
|
+
* embedding model changes
|
|
704
|
+
* chunking strategy changes
|
|
705
|
+
|
|
706
|
+
## 13.7 Technical implementation (default local semantic indexing)
|
|
707
|
+
|
|
708
|
+
When semantic indexing is enabled, Mixgram uses the same stack as SeekMix’s default local setup. The following is the technical reference for replication and implementation.
|
|
709
|
+
|
|
710
|
+
### 13.7.1 Architecture summary
|
|
711
|
+
|
|
712
|
+
| Component | Technology / model |
|
|
713
|
+
|------------------|---------------------|
|
|
714
|
+
| Embeddings | Produced locally by **Hugging Face Transformers.js** (`@huggingface/transformers`), no API keys. |
|
|
715
|
+
| Storage | **SQLite** (`better-sqlite3`) with the **sqlite-vec** extension for vector similarity search. |
|
|
716
|
+
| Matching | Cosine similarity over normalized embeddings; configurable threshold (default `0.87`). |
|
|
717
|
+
|
|
718
|
+
### 13.7.2 Default embedding model
|
|
719
|
+
|
|
720
|
+
| Property | Value |
|
|
721
|
+
|-------------|--------|
|
|
722
|
+
| **Provider** | HuggingfaceProvider (local) |
|
|
723
|
+
| **Model** | `Xenova/multilingual-e5-large` |
|
|
724
|
+
| **Dimensions**| 1024 (from model `hidden_size`) |
|
|
725
|
+
| **Library** | `@huggingface/transformers` |
|
|
726
|
+
| **Pipeline** | `feature-extraction` |
|
|
727
|
+
| **Quantization** | `dtype: 'q8'` (optional, for smaller/faster load) |
|
|
728
|
+
|
|
729
|
+
The **Xenova/** prefix indicates the model is used in ONNX form via Transformers.js; the first run downloads it from Hugging Face, then it runs fully on-device.
|
|
730
|
+
|
|
731
|
+
### 13.7.3 How embeddings are produced (local)
|
|
732
|
+
|
|
733
|
+
* **One-time init:** `pipeline('feature-extraction', 'Xenova/multilingual-e5-large', { dtype: 'q8' })` (model download on first run).
|
|
734
|
+
* **Per-text embedding:** Mean pooling over token embeddings (no CLS-only), then L2-normalize. Cosine similarity is then equivalent to dot product.
|
|
735
|
+
* **Output:** Array of 1024 floats; same format is stored and used for search.
|
|
736
|
+
|
|
737
|
+
Example pattern:
|
|
738
|
+
|
|
739
|
+
```javascript
|
|
740
|
+
const output = await extractor(text, { pooling: 'mean', normalize: true });
|
|
741
|
+
const embedding = output.tolist(); // 1024-dim, L2-normalized
|
|
742
|
+
```
|
|
743
|
+
|
|
744
|
+
### 13.7.4 Vector storage (sqlite-vec)
|
|
745
|
+
|
|
746
|
+
* **DB:** Single SQLite file (e.g. same as main index or a dedicated DB). One DB can host multiple logical caches per embedding model via different table name suffixes (e.g. sanitized model name: `Xenova_multilingual_e5_large`).
|
|
747
|
+
|
|
748
|
+
* **Tables per model:**
|
|
749
|
+
|
|
750
|
+
1. **Metadata table** `cache_<suffix>`: `id` (PK), `chunk_id` (unique), `document_id`, `content_hash`, `embedding_model`, `created_at`, etc. Links chunk identity and metadata to vector rows.
|
|
751
|
+
|
|
752
|
+
2. **Vector table** `vec_<suffix>` (sqlite-vec virtual table): `vec0(embedding float[1024] distance_metric=cosine)`. Row count matches cache; **rowid** of the vector table equals **id** of the cache table for joins.
|
|
753
|
+
|
|
754
|
+
* **Insert (indexing):** Insert row into `cache_<suffix>` → get `id` → insert into `vec_<suffix>` with `rowid = id` and `embedding = Float32Array(embedding)`.
|
|
755
|
+
|
|
756
|
+
* **Extension:** `sqliteVec.load(db)` once per connection; vectors stored as float32.
|
|
757
|
+
|
|
758
|
+
### 13.7.5 Search flow (KNN + threshold)
|
|
759
|
+
|
|
760
|
+
1. **Embed** the incoming query with the same model and options (mean pooling + normalize).
|
|
761
|
+
2. **KNN query** (sqlite-vec), e.g.:
|
|
762
|
+
```sql
|
|
763
|
+
SELECT rowid, distance
|
|
764
|
+
FROM vec_<suffix>
|
|
765
|
+
WHERE embedding MATCH ?
|
|
766
|
+
AND k = N
|
|
767
|
+
ORDER BY distance;
|
|
768
|
+
```
|
|
769
|
+
For tag/scope filtering, `k` is increased and results are filtered in application code.
|
|
770
|
+
3. **Join** with `cache_<suffix>` on `cache.id = vec.rowid` to get chunk_id, document_id, etc., then join to `document_chunks` / `documents` for full metadata and snippets.
|
|
771
|
+
4. **Similarity threshold:** sqlite-vec returns **cosine distance** (0 = identical). Convert to similarity as `similarity = 1 - distance`. Accept a hit only if `similarity >= similarityThreshold` (default `0.87`, from config `embeddings.similarityThreshold`).
|
|
772
|
+
5. **Optional:** TTL, tag/scope filters (e.g. require all requested tags or scope).
|
|
773
|
+
|
|
774
|
+
### 13.7.6 Dependencies for replication
|
|
775
|
+
|
|
776
|
+
```json
|
|
777
|
+
{
|
|
778
|
+
"@huggingface/transformers": "^3.4.2",
|
|
779
|
+
"better-sqlite3": "^11.10.0",
|
|
780
|
+
"sqlite-vec": "^0.1.6"
|
|
781
|
+
}
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
Node must support native addons for `better-sqlite3` and `sqlite-vec`. First run downloads `Xenova/multilingual-e5-large` from Hugging Face (no API key for the model).
|
|
785
|
+
|
|
786
|
+
### 13.7.7 Summary table
|
|
787
|
+
|
|
788
|
+
| Component | Technology / model |
|
|
789
|
+
|-----------------|---------------------|
|
|
790
|
+
| Embedding model | Xenova/multilingual-e5-large (1024 dims) |
|
|
791
|
+
| Embedding lib | @huggingface/transformers, feature-extraction pipeline |
|
|
792
|
+
| Pooling/norm | mean pooling, normalize: true |
|
|
793
|
+
| DB | SQLite (better-sqlite3) |
|
|
794
|
+
| Vector index | sqlite-vec vec0, cosine |
|
|
795
|
+
| Match rule | cosine similarity ≥ 0.87 (configurable via `embeddings.similarityThreshold`) |
|
|
796
|
+
|
|
797
|
+
---
|
|
798
|
+
|
|
799
|
+
## 13.8 Search score blending (full-text vs semantic)
|
|
800
|
+
|
|
801
|
+
Retrieval can combine full-text (FTS5/BM25) and semantic (vector) results. The blend is controlled by configuration.
|
|
802
|
+
|
|
803
|
+
### 13.8.1 When semantic indexing is disabled
|
|
804
|
+
|
|
805
|
+
* **Effective weights:** 100% full-text, 0% semantic.
|
|
806
|
+
* **Behavior:** All search (e.g. `mem_search`, `mem_context`) uses only FTS5 and BM25. Config keys such as `search.semanticWeight` are ignored; no vector queries are run.
|
|
807
|
+
|
|
808
|
+
### 13.8.2 When semantic indexing is enabled
|
|
809
|
+
|
|
810
|
+
* **Effective weights:** Configurable via `search.ftsWeight` and `search.semanticWeight` (see §18).
|
|
811
|
+
* **Example:** `ftsWeight: 0.7`, `semanticWeight: 0.3` → 70% full-text, 30% semantic.
|
|
812
|
+
* **Normalization:** Weights should be normalized so that `ftsWeight + semanticWeight === 1.0`. If only one source is available (e.g. no vectors for a chunk), the available scores are used and the blend applies to the set of results that have both FTS and semantic scores where applicable.
|
|
813
|
+
* **Combination:** Implementation may use normalized FTS score (e.g. BM25 normalized to a 0–1 scale) and semantic similarity (already 0–1); combined score = `ftsWeight * normFtsScore + semanticWeight * similarity`. Ranking and filtering (e.g. `minScore`) apply to this combined score.
|
|
814
|
+
|
|
815
|
+
### 13.8.3 Configuration rule
|
|
816
|
+
|
|
817
|
+
* If `embeddings.enabled === false` → treat as 100% full-text (semantic weight 0).
|
|
818
|
+
* If `embeddings.enabled === true` → use `search.ftsWeight` and `search.semanticWeight`; default suggestion: 70% FTS, 30% semantic (e.g. `0.7` and `0.3`).
|
|
819
|
+
|
|
820
|
+
---
|
|
821
|
+
|
|
822
|
+
# 14. MCP compatibility requirements
|
|
823
|
+
|
|
824
|
+
The system must expose the same Engram-compatible tools.
|
|
825
|
+
|
|
826
|
+
## 14.1 Required tools
|
|
827
|
+
|
|
828
|
+
* `mem_save`
|
|
829
|
+
* `mem_update`
|
|
830
|
+
* `mem_delete`
|
|
831
|
+
* `mem_suggest_topic_key`
|
|
832
|
+
* `mem_search`
|
|
833
|
+
* `mem_context`
|
|
834
|
+
* `mem_timeline`
|
|
835
|
+
* `mem_get_observation`
|
|
836
|
+
* `mem_session_summary`
|
|
837
|
+
* `mem_session_start`
|
|
838
|
+
* `mem_session_end`
|
|
839
|
+
* `mem_save_prompt`
|
|
840
|
+
* `mem_stats`
|
|
841
|
+
|
|
842
|
+
## 14.2 Compatibility rule
|
|
843
|
+
|
|
844
|
+
This system does not need to match Engram internally. It must match it **at the MCP tool contract level** closely enough to replace it in practice.
|
|
845
|
+
|
|
846
|
+
---
|
|
847
|
+
|
|
848
|
+
# 15. Identity, upsert, and deduplication
|
|
849
|
+
|
|
850
|
+
## 15.1 Resolution order
|
|
851
|
+
|
|
852
|
+
1. If `id` is provided, update that exact document
|
|
853
|
+
2. If `topic_key` exists within the relevant scope, update that document
|
|
854
|
+
3. If recent normalized content matches, increment `duplicate_count`
|
|
855
|
+
4. Otherwise create a new Markdown document
|
|
856
|
+
|
|
857
|
+
## 15.2 Scope-aware topic resolution
|
|
858
|
+
|
|
859
|
+
Topic matching must respect scope:
|
|
860
|
+
|
|
861
|
+
* `home` topic keys should update global documents
|
|
862
|
+
* `project` topic keys should update local documents
|
|
863
|
+
* project tools may optionally consult `home` for reuse suggestions, but should not overwrite home documents accidentally
|
|
864
|
+
|
|
865
|
+
---
|
|
866
|
+
|
|
867
|
+
# 16. Search behavior with `home` and `project`
|
|
868
|
+
|
|
869
|
+
## 16.1 Search modes
|
|
870
|
+
|
|
871
|
+
* `project-only`
|
|
872
|
+
* `home-only`
|
|
873
|
+
* `merged`
|
|
874
|
+
|
|
875
|
+
## 16.2 Recommended default
|
|
876
|
+
|
|
877
|
+
For project workflows, use `merged`.
|
|
878
|
+
|
|
879
|
+
That gives the agent:
|
|
880
|
+
|
|
881
|
+
* local project context
|
|
882
|
+
* shared global knowledge
|
|
883
|
+
|
|
884
|
+
## 16.3 Ranking policy
|
|
885
|
+
|
|
886
|
+
Default ranking should slightly prefer:
|
|
887
|
+
|
|
888
|
+
1. exact `topic_key`
|
|
889
|
+
2. same project
|
|
890
|
+
3. same scope
|
|
891
|
+
4. home/global fallback
|
|
892
|
+
5. when semantic indexing is enabled: combined score from full-text and semantic (see §13.8); when disabled: full-text only (100% FTS).
|
|
893
|
+
|
|
894
|
+
---
|
|
895
|
+
|
|
896
|
+
# 17. Suggested internal Node.js structure
|
|
897
|
+
|
|
898
|
+
```text
|
|
899
|
+
src/
|
|
900
|
+
mcp/
|
|
901
|
+
server.ts
|
|
902
|
+
tools/
|
|
903
|
+
core/
|
|
904
|
+
documents/
|
|
905
|
+
indexing/
|
|
906
|
+
search/
|
|
907
|
+
sessions/
|
|
908
|
+
prompts/
|
|
909
|
+
embeddings/
|
|
910
|
+
db/
|
|
911
|
+
sqlite.ts
|
|
912
|
+
migrations/
|
|
913
|
+
fts/
|
|
914
|
+
vec/
|
|
915
|
+
fs/
|
|
916
|
+
watcher.ts
|
|
917
|
+
paths.ts
|
|
918
|
+
utils/
|
|
919
|
+
hash.ts
|
|
920
|
+
markdown.ts
|
|
921
|
+
yaml.ts
|
|
922
|
+
ids.ts
|
|
923
|
+
```
|
|
924
|
+
|
|
925
|
+
---
|
|
926
|
+
|
|
927
|
+
# 18. Suggested config
|
|
928
|
+
|
|
929
|
+
```json
|
|
930
|
+
{
|
|
931
|
+
"homeMemoryRoot": "./home/memory",
|
|
932
|
+
"projectsRoot": "./projects",
|
|
933
|
+
"sqlitePath": "./.Mixgram/index.db",
|
|
934
|
+
"watch": true,
|
|
935
|
+
"indexing": {
|
|
936
|
+
"chunkSize": 1200,
|
|
937
|
+
"chunkOverlap": 120,
|
|
938
|
+
"reindexOnStartup": true,
|
|
939
|
+
"ftsWeights": {
|
|
940
|
+
"title": 10.0,
|
|
941
|
+
"topicKey": 8.0,
|
|
942
|
+
"heading": 5.0,
|
|
943
|
+
"body": 1.0
|
|
944
|
+
}
|
|
945
|
+
},
|
|
946
|
+
"embeddings": {
|
|
947
|
+
"enabled": false,
|
|
948
|
+
"mode": "optional",
|
|
949
|
+
"provider": "local-huggingface",
|
|
950
|
+
"vectorStore": "sqlite-vec",
|
|
951
|
+
"model": "default-seekmix-reference",
|
|
952
|
+
"workerConcurrency": 2,
|
|
953
|
+
"maxRetries": 3,
|
|
954
|
+
"queueOnWrite": true,
|
|
955
|
+
"similarityThreshold": 0.87
|
|
956
|
+
},
|
|
957
|
+
"search": {
|
|
958
|
+
"mode": "fts-only",
|
|
959
|
+
"defaultLimit": 10,
|
|
960
|
+
"ftsWeight": 0.7,
|
|
961
|
+
"semanticWeight": 0.3,
|
|
962
|
+
"defaultScopeMode": "merged"
|
|
963
|
+
}
|
|
964
|
+
}
|
|
965
|
+
```
|
|
966
|
+
|
|
967
|
+
**Search weights.** When `embeddings.enabled` is `false`, search uses **100% full-text** (FTS5/BM25); `search.semanticWeight` is ignored. When `embeddings.enabled` is `true`, `search.ftsWeight` and `search.semanticWeight` define the blend (e.g. `0.7` and `0.3` → 70% full-text, 30% semantic). These two values should be normalized (e.g. sum to 1.0).
|
|
968
|
+
|
|
969
|
+
---
|
|
970
|
+
|
|
971
|
+
The system is acceptable when:
|
|
972
|
+
|
|
973
|
+
1. It can replace Engram at the MCP interface level
|
|
974
|
+
2. Saving memory creates or updates Markdown and immediately updates the text index
|
|
975
|
+
3. Manual Markdown edits trigger incremental reindexing
|
|
976
|
+
4. Deleting SQLite and running full reindex reconstructs the index from files
|
|
977
|
+
5. The system works fully with embeddings disabled
|
|
978
|
+
6. When embeddings are enabled, they run asynchronously and never block save/update flows
|
|
979
|
+
7. `home` knowledge is reusable across projects
|
|
980
|
+
8. Project searches can retrieve both local and home knowledge
|
|
981
|
+
9. File organization is understandable to humans without opening SQLite
|
|
982
|
+
10. Git diffs remain meaningful for durable knowledge
|
|
983
|
+
|
|
984
|
+
---
|
|
985
|
+
|
|
986
|
+
# 20. Executive summary
|
|
987
|
+
|
|
988
|
+
This design keeps **Engram’s MCP interface**, but replaces opaque database-first memory with a more durable and inspectable structure:
|
|
989
|
+
|
|
990
|
+
* **Markdown is the system of record**
|
|
991
|
+
* **SQLite is a rebuildable retrieval layer**
|
|
992
|
+
* **SearchMix inspires the text indexing model**
|
|
993
|
+
* **SeekMix inspires the optional semantic layer**
|
|
994
|
+
* **Embeddings are optional and asynchronous**
|
|
995
|
+
* **Global reusable knowledge lives under `home/`**
|
|
996
|
+
* **Project-specific knowledge lives under each project’s `memory/` directory**
|
|
997
|
+
|
|
998
|
+
And the key structural decision is this:
|
|
999
|
+
|
|
1000
|
+
> **Store durable knowledge by scope and type, not by date. Use dates only when time is actually part of the meaning.**
|
|
1001
|
+
|
|
1002
|
+
If you want, next I can turn this into a **clean Markdown file artifact** named something like `SDD-Mixgram-final-en.md`, or I can produce the **black-box TDD in English** that matches this SDD.
|