mixgram 1.3.7 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +68 -345
  2. package/bin/mixgram.js +51 -160
  3. package/examples/embedding-demo.js +22 -142
  4. package/package.json +19 -19
  5. package/spec/bench-indexing.js +23 -211
  6. package/spec/bench-search.js +26 -212
  7. package/spec/run-tests.js +1 -214
  8. package/spec/scenarios/scenarios.js +277 -607
  9. package/spec/sdd.md +14 -988
  10. package/spec/tdd.md +15 -1232
  11. package/spec/test-large-document.js +20 -91
  12. package/src/config.js +57 -48
  13. package/src/core/documents/documents.js +62 -224
  14. package/src/core/embeddings/embedder.js +14 -54
  15. package/src/core/embeddings/queue.js +33 -106
  16. package/src/core/embeddings/vectorStore.js +30 -111
  17. package/src/core/embeddings/worker-process.js +23 -74
  18. package/src/core/indexing/indexer.js +51 -237
  19. package/src/core/indexing/parser.js +13 -114
  20. package/src/core/indexing/reindex.js +28 -137
  21. package/src/core/search/search.js +64 -158
  22. package/src/db/migrations/001_initial.sql +14 -63
  23. package/src/db/sqlite.js +19 -29
  24. package/src/fs/paths.js +11 -101
  25. package/src/fs/watcher.js +15 -48
  26. package/src/mcp/cli-adapter.js +32 -110
  27. package/src/mcp/server.js +64 -141
  28. package/src/mcp/tool-registry.js +21 -132
  29. package/src/mcp/tools.js +27 -199
  30. package/src/utils/markdown.js +15 -82
  31. package/spec/fixtures/realistic-content.js +0 -106
  32. package/spec/reporter.js +0 -127
  33. package/spec/run-tests-runner.js +0 -40
  34. package/src/core/embeddings/worker.js +0 -27
  35. package/src/core/prompts/prompts.js +0 -15
  36. package/src/core/sessions/sessions.js +0 -27
  37. package/src/utils/ids.js +0 -27
package/README.md CHANGED
@@ -1,406 +1,129 @@
1
- # Mixgram
1
+ # Mixgram 2
2
2
 
3
- MCP memory server compatible with [Engram](https://github.com/gentleman-programming/engram): **Markdown as source of truth**, SQLite FTS5 for full-text search. Optional local embeddings when you need hybrid search.
3
+ General-purpose memory for notes, documentation, research and agent knowledge. Markdown files are the source of truth; SQLite FTS5 and optional local embeddings provide search. Collections map names to folders, and tags classify documents across collections.
4
4
 
5
- - **Human-readable memory** — All durable memory is stored as Markdown files.
6
- - **Git-friendly** — Edit files by hand and reindex; the index is derived and rebuildable.
7
- - **Engram-compatible** — Same MCP tool names and semantics; drop-in replacement.
5
+ Version 2 uses a new API and fresh storage under `~/.mixgram/v2/`. It does not migrate version 1 data or expose Engram compatibility tools. Do not point it at an existing version 1 memory folder; legacy SQLite databases are rejected.
8
6
 
9
- ### Advantages over Engram
10
-
11
- - **Visible, editable docs** — As you advance with agentic development you see the actual documentation files on disk. You can open them, read them, and edit them manually; the agent’s memory is not hidden in a black box.
12
- - **Versioned** — Memory lives in Markdown under your repo (e.g. `docs/<type>/` by default for project scope) or in a shared home (e.g. `home/memory/` for cross-project). Commit, branch, and diff as with any other docs.
13
- - **No semantic deps required** — Core experience is text-only (FTS5). No need to install embeddings or vector libs unless you opt in.
14
-
15
- ---
16
-
17
- ## Installation
18
-
19
- **Global install (recommended for MCP clients):**
7
+ ## Install and run
20
8
 
21
9
  ```bash
22
10
  npm install -g mixgram
11
+ mixgram mcp
23
12
  ```
24
13
 
25
- Then use the `mixgram` command in your client config (see below). No extra dependencies for core (text-only) memory; embeddings are optional.
26
-
27
- ---
28
-
29
- ## CLI
30
-
31
- | Command | Description |
32
- |--------|-------------|
33
- | `mixgram mcp [options]` | Run the MCP server (stdio). This is what your client runs. |
34
- | `mixgram setup cursor` | Add Mixgram to Cursor’s MCP config. |
35
- | `mixgram setup gemini-cli` | Add Mixgram to Gemini CLI settings. |
36
- | `mixgram setup codex` | Add Mixgram to Codex config. |
37
- | `mixgram help` | Show usage. |
38
- | `mixgram help <tool>` | Show options for a specific MCP tool. |
39
- | `mixgram <tool> [--key value ...]` | Run any MCP tool from the command line (see below). |
40
-
41
- **MCP tools from the CLI** — Every MCP tool is automatically available as a subcommand. No extra wiring is needed: if a tool is added to the internal registry, it is exposed both to MCP clients and to the CLI. Use `mixgram help` to list tools and `mixgram help <tool>` (or `mixgram <tool> --help`) to see a tool’s options.
42
-
43
- ### Command-line examples
44
-
45
- ```bash
46
- # Stats and health
47
- mixgram mem_stats
48
- mixgram mem_stats --embeddings # enable embeddings for this run (or use env/config, see below)
49
-
50
- # Save a note
51
- mixgram mem_save --title "My note" --content "Search" --type decision --project my-app
52
- mixgram mem_save --title "Other note" --content "Model Context Protocol" --type architecture
53
-
54
- # Search (default: merged scope, limit 10)
55
- mixgram mem_search --query "index"
56
- mixgram mem_search --query "index" --limit 5
57
- mixgram mem_search --query "indexing" --scope-mode project-only --project my-app
58
-
59
- # OR-style query: use FTS5 keyword OR (and optional quoted phrases)
60
- mixgram mem_search --query 'mcp OR "model context protocol"' --limit 10
61
- mixgram mem_search --query 'cursor OR "model context protocol"' --limit 5
62
-
63
- # Reindex from disk (e.g. after editing Markdown by hand)
64
- mixgram mem_reindex --full
65
-
66
- # Context (recent or by query)
67
- mixgram mem_context --query "indexing decisions" --project my-app --limit 5
68
- mixgram mem_context --limit 5
69
- ```
70
-
71
- **Example: index and find by embedding (not FTS)** — Save a note with one wording, then search with different words that mean the same. FTS finds nothing; semantic search returns the doc:
72
-
73
- ```bash
74
- # 1. Index a document (content about storing memory)
75
- mixgram mem_save --title "Where we store memory" --content "We keep agent memory in Markdown under docs/." --type decision --project demo --embeddings
76
-
77
- # 2. Search by meaning: query has no literal overlap with the text
78
- mixgram mem_search --query "where is the agent memory stored" --project demo --embeddings --limit 5
79
- ```
80
-
81
- The first run embeds the document (worker runs in-process and processes the queue after save; the first time can take a few seconds while the embedding model loads). The second run uses the embedding worker to turn the query into a vector and returns the document by similarity. You can also use `npm run example:embedding` for a self-contained demo in a temp dir.
82
-
83
- **Query syntax (FTS5):** Search uses SQLite FTS5. You can use the operators `OR`, `AND`, `NOT` and quoted phrases. For an OR-style search (e.g. “mcp” or “model context protocol”), use the word `OR` in the query, not the pipe character: `--query 'mcp OR "model context protocol"'`. Unquoted terms are ANDed by default; use `OR` explicitly to match any of several terms.
84
-
85
- Arguments are passed as `--key value`. Booleans use `--flag` or `--no-flag`. Arrays use repeated flags: `--tags a --tags b`.
86
-
87
- **Enabling embeddings from the CLI:** For the MCP server use `mixgram mcp --embeddings`. For one-off tool runs (e.g. `mixgram mem_stats`, `mixgram mem_search`) you can pass `--embeddings` after the tool name, set the env var `MIXGRAM_EMBEDDINGS_ENABLED=1` (or `true`), or enable in `.mixgram/config.json` with `"embeddings": { "enabled": true }`. Example: `mixgram mem_search --query "something" --embeddings --limit 5`.
88
-
89
- **`mcp` options** (and env / config file):
90
-
91
- | Option | Env | Description |
92
- |--------|-----|-------------|
93
- | `--config <path>` | `MIXGRAM_CONFIG` | Config file (default: `./.mixgram/config.json` or `~/.mixgram/config.json`). |
94
- | `--embeddings` | `MIXGRAM_EMBEDDINGS_ENABLED` | Enable optional semantic (hybrid) search. |
95
- | `--watch` | `MIXGRAM_WATCH` | Watch files and reindex on change. |
96
- | `--home <path>` | `MIXGRAM_HOME` | Home memory root (cross-project). |
97
- | `--project-memory <path>` | `MIXGRAM_PROJECT_MEMORY` | Project memory root (default: `./docs`, relative to repo). |
98
- | `--projects <path>` | `MIXGRAM_PROJECTS` | Projects root (legacy). |
99
- | `--sqlite-path <path>` | `MIXGRAM_SQLITE_PATH` | SQLite index path. |
100
-
101
- **Example — Cursor**
102
-
103
- Create `.cursor/mcp.json` in your repo (recommended, and required for Cursor Cloud Agents) or add the same config to `~/.cursor/mcp.json` (or run `mixgram setup cursor`):
14
+ Client configuration:
104
15
 
105
16
  ```json
106
17
  {
107
18
  "mcpServers": {
108
- "mixgram": {
109
- "command": "mixgram",
110
- "args": ["mcp", "--project-memory", "${workspaceFolder}/docs"]
111
- }
19
+ "mixgram": { "command": "mixgram", "args": ["mcp"] }
112
20
  }
113
21
  }
114
22
  ```
115
23
 
116
- With embeddings and custom paths:
117
-
118
- ```json
119
- "mixgram": {
120
- "command": "mixgram",
121
- "args": ["mcp", "--project-memory", "${workspaceFolder}/docs", "--embeddings", "--home", "/data/memory"]
122
- }
123
- ```
24
+ `mixgram setup cursor`, `mixgram setup gemini-cli` and `mixgram setup codex` register that command in the corresponding client config. `mixgram help` lists commands; `mixgram help mem_save` describes a tool's arguments.
124
25
 
125
- Restart Cursor after changing config.
26
+ ## Six tools
126
27
 
127
- ---
28
+ | Tool | Arguments | Result |
29
+ |---|---|---|
30
+ | `mem_save` | Optional `id`, `collection`, `title`, `content`, `type`, `key`, `tags` | `id`, `path`, `created`, `saved`, `indexed` |
31
+ | `mem_search` | Optional `query`, `collection`, `type`, `tags`, `limit`, `offset` | `results`, `mode`, `degraded`, optional `reason` |
32
+ | `mem_get` | `id` | Metadata fields and separate `content` |
33
+ | `mem_delete` | `id`, optional `hardDelete` | Deletion and indexing status |
34
+ | `mem_stats` | None | Document counts, index location and embedding job status |
35
+ | `mem_reindex` | Optional `full` | `scanned`, `indexed`, `skipped`, `removed`, `errors` |
128
36
 
129
- ## Quick start
37
+ All tools are available through MCP and the CLI with the same validation. Invalid or unknown arguments produce errors. CLI failures return a nonzero exit code. MCP failures set `isError`. Tool responses contain JSON; protocol validation failures may contain SDK error text.
130
38
 
131
- Run the MCP server (stdio):
39
+ ### Saving and updating
132
40
 
133
41
  ```bash
134
- mixgram mcp
135
- ```
136
-
137
- From the repo without global install: `npx mixgram mcp` or `npm start`. Configure your MCP client with `command: "mixgram"`, `args: ["mcp"]` when Mixgram is installed globally.
138
-
139
- ---
140
-
141
- ## Configuration
142
-
143
- When you run `mixgram mcp`, config is merged from (lowest to highest priority):
144
-
145
- 1. **Defaults** (paths relative to current directory)
146
- 2. **Config file** — `./.mixgram/config.json` (project) or `~/.mixgram/config.json` (user), or `--config /path`
147
- 3. **Environment** — `MIXGRAM_*` (see table above)
148
- 4. **CLI args** — `--embeddings`, `--home`, etc.
149
-
150
- Example **`.mixgram/config.json`** (project or home):
151
-
152
- ```json
153
- {
154
- "homeMemoryRoot": "~/.mixgram/docs",
155
- "projectMemoryRoot": "./docs",
156
- "sqlitePath": "~/.mixgram/index.db",
157
- "embeddings": {
158
- "enabled": true
159
- }
160
- }
42
+ mixgram mem_save --title "Sourdough notes" --content "Feed the starter daily." --type recipe --tags baking
43
+ mixgram mem_save --collection research --key climate/method --title "Method" --content "Study notes"
44
+ mixgram mem_save --id DOCUMENT_ID --title "Revised title"
45
+ mixgram mem_save --id DOCUMENT_ID --content '' --no-tags
161
46
  ```
162
47
 
163
- Paths in the config file are relative to the config file’s directory (project) or to the current directory when using `~/.mixgram/config.json`. **Project memory** is resolved relative to the current working directory (repo) when you use a global config, so docs stay in the repo. The project memory folder name is configurable: set `projectMemoryRoot` to e.g. `./specs` to get `specs/architecture/`, `specs/decisions/`, etc. Defaults:
164
-
165
- | Option | Default | Description |
166
- |--------|---------|-------------|
167
- | `homeMemoryRoot` | `~/.mixgram/docs` | Cross-project memory (Markdown files). |
168
- | `projectMemoryRoot` | `./docs` | Project memory in the current repo (e.g. `docs/architecture/`, `docs/decisions/`). You can use another folder name, e.g. `./specs` for `specs/architecture/`. |
169
- | `sqlitePath` | `~/.mixgram/index.db` | SQLite index (FTS5 + optional vectors). |
170
- | `watch` | `true` | Watch files and reindex on change. |
171
- | `indexing.reindexOnStartup` | `true` | Full reindex when server starts. |
172
- | `embeddings.enabled` | `false` | Enable async local embeddings and hybrid search. |
173
- | `search.defaultScopeMode` | `'merged'` | `'project-only'` \| `'home-only'` \| `'merged'`. |
174
- | `search.ftsWeight` / `search.semanticWeight` | `0.7` / `0.3` | Weights when embeddings are enabled. |
175
-
176
- Example minimal override:
177
-
178
- ```js
179
- import { run } from 'mixgram/src/mcp/server.js';
180
-
181
- await run({
182
- homeMemoryRoot: '/data/memory',
183
- projectMemoryRoot: '/path/to/repo/docs',
184
- sqlitePath: '/data/.mixgram/index.db',
185
- watch: true
186
- });
187
- ```
188
-
189
- ---
48
+ With `id`, update an existing live document; an unknown ID is an error. Without `id`, an explicit `key` upserts within the selected collection. Without either, create a new document even if its title matches another note. No key is automatically inferred from a title.
190
49
 
191
- ## Text-only memory
50
+ Omitted fields are preserved on updates. `content: ""` and `tags: []` clear those fields; `key: null` removes a key. CLI equivalents are `--content ''`, `--no-tags` and `--no-key`. Repeat `--tags value` to supply multiple tags. Defaults for new documents are empty title/content/tags, type `note`, and no key. Types are arbitrary nonempty strings.
192
51
 
193
- Core flow: save Markdown-backed documents, search with FTS5, get context.
52
+ New files use `<title-slug>-<id>.md`; updating a title retains the path. Collection changes are made by moving the file to another configured collection folder and reindexing. Updating an ID with a different collection is rejected.
194
53
 
195
- ### Save (create or update by topic)
196
-
197
- ```json
198
- // mem_save — create or update by topic_key
199
- {
200
- "title": "Use SQLite as derived index",
201
- "type": "decision",
202
- "scope": "project",
203
- "project": "my-app",
204
- "topic_key": "architecture/sqlite-derived-index",
205
- "content": "SQLite will be the derived index over Markdown documents."
206
- }
207
- ```
54
+ Prompts and conversation summaries are ordinary documents. For example, use type `prompt` or `summary` and tag `conversation:123` to group a conversation.
208
55
 
209
- - **scope** `project` → file under `<repo>/docs/<type>/...` (e.g. `docs/architecture/`, `docs/decisions/`).
210
- - **scope** `home` → file under `home/memory/...` (cross-project).
211
- - Same `topic_key` + scope + project → **update** existing doc; otherwise create.
56
+ ### Search and listing
212
57
 
213
- ### Search
214
-
215
- ```json
216
- // mem_search
217
- {
218
- "query": "derived SQLite index",
219
- "scope_mode": "merged",
220
- "project": "my-app",
221
- "limit": 10
222
- }
223
- ```
224
-
225
- **scope_mode:**
226
-
227
- - `project-only` — only project memory.
228
- - `home-only` — only home memory.
229
- - `merged` — both; project results ranked first when `project` is set.
230
-
231
- Response includes `documentId`, `chunkId`, `title`, `topicKey`, `snippet`, `score`, etc.
232
-
233
- ### Context (for prompts)
234
-
235
- ```json
236
- // mem_context — recent or query-based
237
- {
238
- "query": "indexing decisions",
239
- "project": "my-app",
240
- "limit": 5
241
- }
58
+ ```bash
59
+ mixgram mem_search --query 'climate OR temperature' --collection research
60
+ mixgram mem_search --tags baking --type recipe --limit 5 --offset 0
61
+ mixgram mem_get --id DOCUMENT_ID
242
62
  ```
243
63
 
244
- Omit `query` or use `*` to get **recent** context only.
64
+ Without `query` (or with blank text), list recent documents by update time. With a query, use FTS5 syntax: quoted phrases and uppercase `AND`, `OR`, `NOT` are supported. Invalid FTS expressions return an error. Omit `collection` to search all configured collections. Every supplied tag must match. `limit` defaults to 10 and may not exceed `search.maxLimit` (100 by default); `offset` defaults to zero.
245
65
 
246
- ### Get / update / delete
66
+ Results contain `id`, title, collection, type, key, tags, timestamps and a bounded snippet. Ranked results also contain a score. Read full content using `mem_get`.
247
67
 
248
- - **mem_get_observation** `{ "id": "<document-id>" }` — full document content.
249
- - **mem_update** `{ "id": "<id>", "title": "...", "content": "..." }` — update by id.
250
- - **mem_delete** `{ "id": "<id>", "hardDelete": false }` — soft or hard delete.
68
+ When embeddings are enabled, ranks are fused using `weight / (fusionConstant + rank)`, then deduplicated and paginated. Text and semantic searches share collection/type/tag filters. Semantic search uses exact cosine distance over eligible vectors, so filtering cannot lose matches behind a global candidate limit. This favors correctness for a local memory corpus; vector retrieval and rank fusion scale with the matching corpus size.
251
69
 
252
- ### Helpers
70
+ Embeddings are optional at runtime. Model and native vector packages remain installed dependencies. The model runs in a child process; saves queue work and do not wait for model inference. Keep `mixgram mcp --embeddings` running to process queued documents, including saves made by one-off CLI commands. A one-off command does not wait for its entire embedding queue to drain. If the worker or vector store fails, search returns textual results with `degraded: true` and a reason. A healthy worker with no completed vectors yet may return only text matches in hybrid mode.
253
71
 
254
- - **mem_suggest_topic_key** — suggest a stable `topic_key` from title/content/type.
255
- - **mem_stats** — counts (documents, sessions, prompts, `embeddings_enabled`).
72
+ ### Persistence and rebuilding
256
73
 
257
- ### Example: one round-trip
74
+ Frontmatter stores `id`, `title`, `type`, `key`, `tags`, `created_at`, `updated_at` and nullable `deleted_at`. Dates use UTC ISO 8601 strings. The body preserves its content, including empty content. Collection is derived from the configured root containing the file.
258
75
 
259
- ```text
260
- 1. mem_save → { success: true, id: "<10-char id>", path: "...", created: true }
261
- 2. mem_search({ query: "SQLite", project: "my-app" }) → { results: [ { title, snippet, ... } ] }
262
- 3. mem_get_observation({ id: "<10-char id>" }) → { title, content, type, scope, project, ... }
263
- ```
264
-
265
- ---
266
-
267
- ## Reindex, watch, and sessions
268
-
269
- ### Reindex from disk
270
-
271
- Useful after cloning a repo or editing Markdown by hand:
272
-
273
- ```json
274
- // mem_reindex
275
- { "full": true }
76
+ ```bash
77
+ mixgram mem_reindex
78
+ mixgram mem_reindex --full
79
+ mixgram mem_delete --id DOCUMENT_ID
80
+ mixgram mem_delete --id DOCUMENT_ID --hardDelete
276
81
  ```
277
82
 
278
- - `full: true` — rebuild entire index from `home/**/*.md` and `docs/**/*.md` (project memory in repo).
279
- - `full: false` (default) — incremental (by mtime).
280
-
281
- Manual edit example:
282
-
283
- 1. Edit a `.md` file under `home/memory` or `docs/<type>/` in your repo.
284
- 2. Call `mem_reindex({ full: true })` (or run with `watch: true` so changes are picked up).
285
- 3. `mem_search` and `mem_get_observation` reflect the new content.
83
+ Save writes a temporary file and renames it before updating SQLite. If indexing fails afterwards, the response explicitly reports `saved: true`, `indexed: false`, and an error; the file is durable and reindexing repairs the index. Soft deletion persists `deleted_at` in Markdown and survives rebuilding; hard deletion removes the file, including a previously soft-deleted file.
286
84
 
287
- ### Watcher
85
+ Incremental reindexing compares file content hashes. Full reindexing reconstructs the text index from the configured folders. Both report errors per file. Metadata edited in Markdown takes precedence over SQLite. Plain Markdown without frontmatter is indexed using its first heading, file timestamps and a path-derived ID; to retain identity across moves, give the file an explicit `id` in frontmatter. Duplicate IDs, conflicting live keys and invalid metadata are reported as errors.
288
86
 
289
- Start the server with `watch: true` so that add/change/delete under the memory paths trigger reindex automatically.
87
+ The watcher handles Markdown additions, changes and removals. Files outside configured roots, including symlinks pointing outside, are rejected. If the OS cannot provide native watches, Chokidar's `CHOKIDAR_USEPOLLING=1` environment setting selects polling.
290
88
 
291
- ### Sessions and prompts
89
+ ## Configuration
292
90
 
293
- - **mem_session_start** `{ "project": "my-app" }` → `{ session_id }`
294
- - **mem_session_end** `{ "session_id": "..." }`
295
- - **mem_session_summary** `{ "session_id": "...", "content": "...", "title": "..." }` — persists a summary as a memory document.
296
- - **mem_save_prompt** `{ "session_id": "...", "content": "..." }` — store a prompt for the session.
297
- - **mem_timeline** `{ "observation_id": "..." }` — before/focus/after in the same session.
91
+ Load order: defaults, config file, environment, CLI flags. Nested options merge; an explicitly supplied `collections` map replaces the default collection map. Relative config paths resolve from the config file's directory.
298
92
 
299
- ### Scope examples
93
+ Default config locations are `./.mixgram/v2/config.json`, then `~/.mixgram/v2/config.json`. Select another file with `--config` or `MIXGRAM_CONFIG`. Version 1 config files are not loaded automatically.
300
94
 
301
95
  ```json
302
- // Only project memory
303
- mem_search({ "query": "reindex", "scope_mode": "project-only", "project": "my-app" })
304
-
305
- // Only home (cross-project)
306
- mem_search({ "query": "reindex", "scope_mode": "home-only" })
307
-
308
- // Both; project first
309
- mem_search({ "query": "reindex", "scope_mode": "merged", "project": "my-app" })
310
- ```
311
-
312
- ---
313
-
314
- ## Optional semantic layer
315
-
316
- When `embeddings.enabled` is `true` and the optional embedding stack is available:
317
-
318
- - **Saves** are durable and searchable by text immediately; embeddings are queued and processed in the background.
319
- - **Search** becomes **hybrid**: FTS + vector similarity, blended with `search.ftsWeight` and `search.semanticWeight` (e.g. 0.7 and 0.3).
320
- - A **worker** runs in a separate process and processes the embedding queue (e.g. every 2 seconds). Query embeddings for hybrid search are also computed in that process via IPC, so the main MCP server never loads `@huggingface/transformers` and is not affected by native/ONNX crashes in the embedding stack.
321
-
322
- ### Enable embeddings
323
-
324
- 1. Ensure optional deps are installed if your setup uses them (e.g. `@huggingface/transformers`, `sqlite-vec`).
325
- 2. Configure:
326
-
327
- ```js
328
- await run({
329
- embeddings: {
330
- enabled: true,
331
- similarityThreshold: 0.80,
332
- maxRetries: 3
96
+ {
97
+ "collections": {
98
+ "general": "./general",
99
+ "research": "./research"
333
100
  },
334
- search: {
335
- ftsWeight: 0.7,
336
- semanticWeight: 0.3
337
- }
338
- });
339
- ```
340
-
341
- ### Behaviour
342
-
343
- - **mem_save** / **mem_update** return as soon as the Markdown is written and the text index is updated; they do **not** wait for embeddings.
344
- - **mem_search** uses FTS only until vectors exist for the matched chunks; then it uses the hybrid blend.
345
- - If the embedding provider or sqlite-vec is unavailable, save and search still work (text-only).
346
-
347
- ### Example: hybrid search
348
-
349
- ```json
350
- // Same tool; behaviour depends on config and whether vectors exist
351
- mem_search({
352
- "query": "persistir índices derivados",
353
- "scope_mode": "merged",
354
- "project": "my-app",
355
- "limit": 5
356
- })
357
- ```
358
-
359
- With embeddings enabled and vectors ready, results combine literal matches (FTS) and semantic similarity (e.g. paraphrases).
360
-
361
- ### Example: indexing and search (embedding effectiveness)
362
-
363
- To see the difference between **FTS-only** and **semantic** search in one run, run the demo script (uses a temp dir; first run may download the embedding model):
364
-
365
- ```bash
366
- node examples/embedding-demo.js
367
- # or
368
- npm run example:embedding
101
+ "defaultCollection": "general",
102
+ "sqlitePath": "./index.db",
103
+ "watch": true,
104
+ "embeddings": { "enabled": false }
105
+ }
369
106
  ```
370
107
 
371
- The script:
108
+ Defaults: collection `general` at `~/.mixgram/v2/general`, index `~/.mixgram/v2/index.db`, watching and incremental startup synchronization enabled. Collection roots must not overlap, including through symlinks. `defaultCollection` must name a configured collection. Use a separate SQLite path for each distinct set of collections.
372
109
 
373
- 1. **Indexes** three short documents (e.g. “memoria en Markdown”, “índice derivado”, “persistencia de sesión”).
374
- 2. **Processes the embedding queue** so each document gets a vector.
375
- 3. **Runs FTS search** for a query that does *not* appear literally in the text (e.g. “dónde se almacena la memoria del agente”) — FTS returns nothing.
376
- 4. **Runs semantic (vector) search** for the same query — returns the relevant document by meaning.
110
+ Environment: `MIXGRAM_CONFIG`, `MIXGRAM_SQLITE_PATH`, `MIXGRAM_EMBEDDINGS_ENABLED`, `MIXGRAM_WATCH`. Boolean values accept `1`, `0`, `true`, `false`. CLI overrides: `--config`, `--sqlite-path`, `--embeddings` / `--no-embeddings`, `--watch` / `--no-watch`.
377
111
 
378
- Example output:
112
+ Operational defaults live in `src/config.js`. Search weights default to 0.7 text / 0.3 semantic with fusion constant 60. The default embedding model is `Xenova/multilingual-e5-large`, 1024 dimensions, q8. Model output dimensions must match configuration. Embedding jobs retry up to three times; abandoned processing jobs become eligible after the configured five-minute lease (`embeddings.jobLeaseMs`). No silent configuration fallback is applied to invalid inputs.
379
113
 
380
- ```text
381
- --- 3. FTS search (text only) ---
382
- Query: "where is the agent memory stored"
383
- (FTS finds no literal match; these words are not in the documents.)
384
-
385
- --- 4. Semantic search (embeddings) ---
386
- Query: "where is the agent memory stored"
387
- - [Session persistence] similarity=0.856
388
- Persist the session context across agent restarts. The important state...
389
- ```
114
+ ## Development
390
115
 
391
- So: **FTS** only matches when the query words appear in the document; **embeddings** can retrieve documents that share the same meaning without sharing the same words.
392
-
393
- ---
394
-
395
- ## Tests
116
+ Node.js with ESM; package manager `pnpm@11.3.0`. No build step is required.
396
117
 
397
118
  ```bash
398
119
  npm test
120
+ npm run test:large
121
+ npm run quality
122
+ npm run bench:indexing
123
+ npm run bench:search
399
124
  ```
400
125
 
401
- Runs black-box scenario tests for text-only memory, reindex/sessions/scope, and semantic fallback and error handling. Use `npm run test:visual` to see a narrative per scenario (inputs, outputs, snippets, and checks) for human review.
402
-
403
- ---
126
+ Tests use temporary folders and SQLite databases. Semantic ranking tests use real sqlite-vec distance calculations with deterministic vectors, without downloading a model. Optional model-backed runs: `npm run example:embedding`, `npm run bench:embed`, `npm run bench:search:embed`; these may download the model on first run.
404
127
 
405
128
  ## License
406
129