mixgram 1.3.6 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -345
- package/bin/mixgram.js +51 -160
- package/examples/embedding-demo.js +22 -142
- package/package.json +19 -19
- package/spec/bench-indexing.js +23 -211
- package/spec/bench-search.js +26 -212
- package/spec/run-tests.js +1 -214
- package/spec/scenarios/scenarios.js +277 -607
- package/spec/sdd.md +14 -988
- package/spec/tdd.md +15 -1232
- package/spec/test-large-document.js +20 -91
- package/src/config.js +57 -48
- package/src/core/documents/documents.js +62 -224
- package/src/core/embeddings/embedder.js +14 -54
- package/src/core/embeddings/queue.js +33 -106
- package/src/core/embeddings/vectorStore.js +30 -111
- package/src/core/embeddings/worker-process.js +23 -74
- package/src/core/indexing/indexer.js +51 -237
- package/src/core/indexing/parser.js +13 -114
- package/src/core/indexing/reindex.js +28 -137
- package/src/core/search/search.js +64 -158
- package/src/db/migrations/001_initial.sql +14 -63
- package/src/db/sqlite.js +19 -29
- package/src/fs/paths.js +11 -101
- package/src/fs/watcher.js +15 -48
- package/src/mcp/cli-adapter.js +32 -110
- package/src/mcp/server.js +64 -141
- package/src/mcp/tool-registry.js +21 -132
- package/src/mcp/tools.js +27 -199
- package/src/utils/markdown.js +15 -82
- package/spec/fixtures/realistic-content.js +0 -106
- package/spec/reporter.js +0 -127
- package/spec/run-tests-runner.js +0 -40
- package/src/core/embeddings/worker.js +0 -27
- package/src/core/prompts/prompts.js +0 -15
- package/src/core/sessions/sessions.js +0 -27
- package/src/utils/ids.js +0 -27
package/README.md
CHANGED
|
@@ -1,406 +1,129 @@
|
|
|
1
|
-
# Mixgram
|
|
1
|
+
# Mixgram 2
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
General-purpose memory for notes, documentation, research and agent knowledge. Markdown files are the source of truth; SQLite FTS5 and optional local embeddings provide search. Collections map names to folders, and tags classify documents across collections.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- **Git-friendly** — Edit files by hand and reindex; the index is derived and rebuildable.
|
|
7
|
-
- **Engram-compatible** — Same MCP tool names and semantics; drop-in replacement.
|
|
5
|
+
Version 2 uses a new API and fresh storage under `~/.mixgram/v2/`. It does not migrate version 1 data or expose Engram compatibility tools. Do not point it at an existing version 1 memory folder; legacy SQLite databases are rejected.
|
|
8
6
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
- **Visible, editable docs** — As you advance with agentic development you see the actual documentation files on disk. You can open them, read them, and edit them manually; the agent’s memory is not hidden in a black box.
|
|
12
|
-
- **Versioned** — Memory lives in Markdown under your repo (e.g. `docs/<type>/` by default for project scope) or in a shared home (e.g. `home/memory/` for cross-project). Commit, branch, and diff as with any other docs.
|
|
13
|
-
- **No semantic deps required** — Core experience is text-only (FTS5). No need to install embeddings or vector libs unless you opt in.
|
|
14
|
-
|
|
15
|
-
---
|
|
16
|
-
|
|
17
|
-
## Installation
|
|
18
|
-
|
|
19
|
-
**Global install (recommended for MCP clients):**
|
|
7
|
+
## Install and run
|
|
20
8
|
|
|
21
9
|
```bash
|
|
22
10
|
npm install -g mixgram
|
|
11
|
+
mixgram mcp
|
|
23
12
|
```
|
|
24
13
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
---
|
|
28
|
-
|
|
29
|
-
## CLI
|
|
30
|
-
|
|
31
|
-
| Command | Description |
|
|
32
|
-
|--------|-------------|
|
|
33
|
-
| `mixgram mcp [options]` | Run the MCP server (stdio). This is what your client runs. |
|
|
34
|
-
| `mixgram setup cursor` | Add Mixgram to Cursor’s MCP config. |
|
|
35
|
-
| `mixgram setup gemini-cli` | Add Mixgram to Gemini CLI settings. |
|
|
36
|
-
| `mixgram setup codex` | Add Mixgram to Codex config. |
|
|
37
|
-
| `mixgram help` | Show usage. |
|
|
38
|
-
| `mixgram help <tool>` | Show options for a specific MCP tool. |
|
|
39
|
-
| `mixgram <tool> [--key value ...]` | Run any MCP tool from the command line (see below). |
|
|
40
|
-
|
|
41
|
-
**MCP tools from the CLI** — Every MCP tool is automatically available as a subcommand. No extra wiring is needed: if a tool is added to the internal registry, it is exposed both to MCP clients and to the CLI. Use `mixgram help` to list tools and `mixgram help <tool>` (or `mixgram <tool> --help`) to see a tool’s options.
|
|
42
|
-
|
|
43
|
-
### Command-line examples
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
# Stats and health
|
|
47
|
-
mixgram mem_stats
|
|
48
|
-
mixgram mem_stats --embeddings # enable embeddings for this run (or use env/config, see below)
|
|
49
|
-
|
|
50
|
-
# Save a note
|
|
51
|
-
mixgram mem_save --title "My note" --content "Search" --type decision --project my-app
|
|
52
|
-
mixgram mem_save --title "Other note" --content "Model Context Protocol" --type architecture
|
|
53
|
-
|
|
54
|
-
# Search (default: merged scope, limit 10)
|
|
55
|
-
mixgram mem_search --query "index"
|
|
56
|
-
mixgram mem_search --query "index" --limit 5
|
|
57
|
-
mixgram mem_search --query "indexing" --scope-mode project-only --project my-app
|
|
58
|
-
|
|
59
|
-
# OR-style query: use FTS5 keyword OR (and optional quoted phrases)
|
|
60
|
-
mixgram mem_search --query 'mcp OR "model context protocol"' --limit 10
|
|
61
|
-
mixgram mem_search --query 'cursor OR "model context protocol"' --limit 5
|
|
62
|
-
|
|
63
|
-
# Reindex from disk (e.g. after editing Markdown by hand)
|
|
64
|
-
mixgram mem_reindex --full
|
|
65
|
-
|
|
66
|
-
# Context (recent or by query)
|
|
67
|
-
mixgram mem_context --query "indexing decisions" --project my-app --limit 5
|
|
68
|
-
mixgram mem_context --limit 5
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
**Example: index and find by embedding (not FTS)** — Save a note with one wording, then search with different words that mean the same. FTS finds nothing; semantic search returns the doc:
|
|
72
|
-
|
|
73
|
-
```bash
|
|
74
|
-
# 1. Index a document (content about storing memory)
|
|
75
|
-
mixgram mem_save --title "Where we store memory" --content "We keep agent memory in Markdown under docs/." --type decision --project demo --embeddings
|
|
76
|
-
|
|
77
|
-
# 2. Search by meaning: query has no literal overlap with the text
|
|
78
|
-
mixgram mem_search --query "where is the agent memory stored" --project demo --embeddings --limit 5
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
The first run embeds the document (worker runs in-process and processes the queue after save; the first time can take a few seconds while the embedding model loads). The second run uses the embedding worker to turn the query into a vector and returns the document by similarity. You can also use `npm run example:embedding` for a self-contained demo in a temp dir.
|
|
82
|
-
|
|
83
|
-
**Query syntax (FTS5):** Search uses SQLite FTS5. You can use the operators `OR`, `AND`, `NOT` and quoted phrases. For an OR-style search (e.g. “mcp” or “model context protocol”), use the word `OR` in the query, not the pipe character: `--query 'mcp OR "model context protocol"'`. Unquoted terms are ANDed by default; use `OR` explicitly to match any of several terms.
|
|
84
|
-
|
|
85
|
-
Arguments are passed as `--key value`. Booleans use `--flag` or `--no-flag`. Arrays use repeated flags: `--tags a --tags b`.
|
|
86
|
-
|
|
87
|
-
**Enabling embeddings from the CLI:** For the MCP server use `mixgram mcp --embeddings`. For one-off tool runs (e.g. `mixgram mem_stats`, `mixgram mem_search`) you can pass `--embeddings` after the tool name, set the env var `MIXGRAM_EMBEDDINGS_ENABLED=1` (or `true`), or enable in `.mixgram/config.json` with `"embeddings": { "enabled": true }`. Example: `mixgram mem_search --query "something" --embeddings --limit 5`.
|
|
88
|
-
|
|
89
|
-
**`mcp` options** (and env / config file):
|
|
90
|
-
|
|
91
|
-
| Option | Env | Description |
|
|
92
|
-
|--------|-----|-------------|
|
|
93
|
-
| `--config <path>` | `MIXGRAM_CONFIG` | Config file (default: `./.mixgram/config.json` or `~/.mixgram/config.json`). |
|
|
94
|
-
| `--embeddings` | `MIXGRAM_EMBEDDINGS_ENABLED` | Enable optional semantic (hybrid) search. |
|
|
95
|
-
| `--watch` | `MIXGRAM_WATCH` | Watch files and reindex on change. |
|
|
96
|
-
| `--home <path>` | `MIXGRAM_HOME` | Home memory root (cross-project). |
|
|
97
|
-
| `--project-memory <path>` | `MIXGRAM_PROJECT_MEMORY` | Project memory root (default: `./docs`, relative to repo). |
|
|
98
|
-
| `--projects <path>` | `MIXGRAM_PROJECTS` | Projects root (legacy). |
|
|
99
|
-
| `--sqlite-path <path>` | `MIXGRAM_SQLITE_PATH` | SQLite index path. |
|
|
100
|
-
|
|
101
|
-
**Example — Cursor**
|
|
102
|
-
|
|
103
|
-
Create `.cursor/mcp.json` in your repo (recommended, and required for Cursor Cloud Agents) or add the same config to `~/.cursor/mcp.json` (or run `mixgram setup cursor`):
|
|
14
|
+
Client configuration:
|
|
104
15
|
|
|
105
16
|
```json
|
|
106
17
|
{
|
|
107
18
|
"mcpServers": {
|
|
108
|
-
"mixgram": {
|
|
109
|
-
"command": "mixgram",
|
|
110
|
-
"args": ["mcp", "--project-memory", "${workspaceFolder}/docs"]
|
|
111
|
-
}
|
|
19
|
+
"mixgram": { "command": "mixgram", "args": ["mcp"] }
|
|
112
20
|
}
|
|
113
21
|
}
|
|
114
22
|
```
|
|
115
23
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
```json
|
|
119
|
-
"mixgram": {
|
|
120
|
-
"command": "mixgram",
|
|
121
|
-
"args": ["mcp", "--project-memory", "${workspaceFolder}/docs", "--embeddings", "--home", "/data/memory"]
|
|
122
|
-
}
|
|
123
|
-
```
|
|
24
|
+
`mixgram setup cursor`, `mixgram setup gemini-cli` and `mixgram setup codex` register that command in the corresponding client config. `mixgram help` lists commands; `mixgram help mem_save` describes a tool's arguments.
|
|
124
25
|
|
|
125
|
-
|
|
26
|
+
## Six tools
|
|
126
27
|
|
|
127
|
-
|
|
28
|
+
| Tool | Arguments | Result |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| `mem_save` | Optional `id`, `collection`, `title`, `content`, `type`, `key`, `tags` | `id`, `path`, `created`, `saved`, `indexed` |
|
|
31
|
+
| `mem_search` | Optional `query`, `collection`, `type`, `tags`, `limit`, `offset` | `results`, `mode`, `degraded`, optional `reason` |
|
|
32
|
+
| `mem_get` | `id` | Metadata fields and separate `content` |
|
|
33
|
+
| `mem_delete` | `id`, optional `hardDelete` | Deletion and indexing status |
|
|
34
|
+
| `mem_stats` | None | Document counts, index location and embedding job status |
|
|
35
|
+
| `mem_reindex` | Optional `full` | `scanned`, `indexed`, `skipped`, `removed`, `errors` |
|
|
128
36
|
|
|
129
|
-
|
|
37
|
+
All tools are available through MCP and the CLI with the same validation. Invalid or unknown arguments produce errors. CLI failures return a nonzero exit code. MCP failures set `isError`. Tool responses contain JSON; protocol validation failures may contain SDK error text.
|
|
130
38
|
|
|
131
|
-
|
|
39
|
+
### Saving and updating
|
|
132
40
|
|
|
133
41
|
```bash
|
|
134
|
-
mixgram
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
---
|
|
140
|
-
|
|
141
|
-
## Configuration
|
|
142
|
-
|
|
143
|
-
When you run `mixgram mcp`, config is merged from (lowest to highest priority):
|
|
144
|
-
|
|
145
|
-
1. **Defaults** (paths relative to current directory)
|
|
146
|
-
2. **Config file** — `./.mixgram/config.json` (project) or `~/.mixgram/config.json` (user), or `--config /path`
|
|
147
|
-
3. **Environment** — `MIXGRAM_*` (see table above)
|
|
148
|
-
4. **CLI args** — `--embeddings`, `--home`, etc.
|
|
149
|
-
|
|
150
|
-
Example **`.mixgram/config.json`** (project or home):
|
|
151
|
-
|
|
152
|
-
```json
|
|
153
|
-
{
|
|
154
|
-
"homeMemoryRoot": "~/.mixgram/docs",
|
|
155
|
-
"projectMemoryRoot": "./docs",
|
|
156
|
-
"sqlitePath": "~/.mixgram/index.db",
|
|
157
|
-
"embeddings": {
|
|
158
|
-
"enabled": true
|
|
159
|
-
}
|
|
160
|
-
}
|
|
42
|
+
mixgram mem_save --title "Sourdough notes" --content "Feed the starter daily." --type recipe --tags baking
|
|
43
|
+
mixgram mem_save --collection research --key climate/method --title "Method" --content "Study notes"
|
|
44
|
+
mixgram mem_save --id DOCUMENT_ID --title "Revised title"
|
|
45
|
+
mixgram mem_save --id DOCUMENT_ID --content '' --no-tags
|
|
161
46
|
```
|
|
162
47
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
| Option | Default | Description |
|
|
166
|
-
|--------|---------|-------------|
|
|
167
|
-
| `homeMemoryRoot` | `~/.mixgram/docs` | Cross-project memory (Markdown files). |
|
|
168
|
-
| `projectMemoryRoot` | `./docs` | Project memory in the current repo (e.g. `docs/architecture/`, `docs/decisions/`). You can use another folder name, e.g. `./specs` for `specs/architecture/`. |
|
|
169
|
-
| `sqlitePath` | `~/.mixgram/index.db` | SQLite index (FTS5 + optional vectors). |
|
|
170
|
-
| `watch` | `true` | Watch files and reindex on change. |
|
|
171
|
-
| `indexing.reindexOnStartup` | `true` | Full reindex when server starts. |
|
|
172
|
-
| `embeddings.enabled` | `false` | Enable async local embeddings and hybrid search. |
|
|
173
|
-
| `search.defaultScopeMode` | `'merged'` | `'project-only'` \| `'home-only'` \| `'merged'`. |
|
|
174
|
-
| `search.ftsWeight` / `search.semanticWeight` | `0.7` / `0.3` | Weights when embeddings are enabled. |
|
|
175
|
-
|
|
176
|
-
Example minimal override:
|
|
177
|
-
|
|
178
|
-
```js
|
|
179
|
-
import { run } from 'mixgram/src/mcp/server.js';
|
|
180
|
-
|
|
181
|
-
await run({
|
|
182
|
-
homeMemoryRoot: '/data/memory',
|
|
183
|
-
projectMemoryRoot: '/path/to/repo/docs',
|
|
184
|
-
sqlitePath: '/data/.mixgram/index.db',
|
|
185
|
-
watch: true
|
|
186
|
-
});
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
---
|
|
48
|
+
With `id`, update an existing live document; an unknown ID is an error. Without `id`, an explicit `key` upserts within the selected collection. Without either, create a new document even if its title matches another note. No key is automatically inferred from a title.
|
|
190
49
|
|
|
191
|
-
|
|
50
|
+
Omitted fields are preserved on updates. `content: ""` and `tags: []` clear those fields; `key: null` removes a key. CLI equivalents are `--content ''`, `--no-tags` and `--no-key`. Repeat `--tags value` to supply multiple tags. Defaults for new documents are empty title/content/tags, type `note`, and no key. Types are arbitrary nonempty strings.
|
|
192
51
|
|
|
193
|
-
|
|
52
|
+
New files use `<title-slug>-<id>.md`; updating a title retains the path. Collection changes are made by moving the file to another configured collection folder and reindexing. Updating an ID with a different collection is rejected.
|
|
194
53
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
```json
|
|
198
|
-
// mem_save — create or update by topic_key
|
|
199
|
-
{
|
|
200
|
-
"title": "Use SQLite as derived index",
|
|
201
|
-
"type": "decision",
|
|
202
|
-
"scope": "project",
|
|
203
|
-
"project": "my-app",
|
|
204
|
-
"topic_key": "architecture/sqlite-derived-index",
|
|
205
|
-
"content": "SQLite will be the derived index over Markdown documents."
|
|
206
|
-
}
|
|
207
|
-
```
|
|
54
|
+
Prompts and conversation summaries are ordinary documents. For example, use type `prompt` or `summary` and tag `conversation:123` to group a conversation.
|
|
208
55
|
|
|
209
|
-
|
|
210
|
-
- **scope** `home` → file under `home/memory/...` (cross-project).
|
|
211
|
-
- Same `topic_key` + scope + project → **update** existing doc; otherwise create.
|
|
56
|
+
### Search and listing
|
|
212
57
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
{
|
|
218
|
-
"query": "derived SQLite index",
|
|
219
|
-
"scope_mode": "merged",
|
|
220
|
-
"project": "my-app",
|
|
221
|
-
"limit": 10
|
|
222
|
-
}
|
|
223
|
-
```
|
|
224
|
-
|
|
225
|
-
**scope_mode:**
|
|
226
|
-
|
|
227
|
-
- `project-only` — only project memory.
|
|
228
|
-
- `home-only` — only home memory.
|
|
229
|
-
- `merged` — both; project results ranked first when `project` is set.
|
|
230
|
-
|
|
231
|
-
Response includes `documentId`, `chunkId`, `title`, `topicKey`, `snippet`, `score`, etc.
|
|
232
|
-
|
|
233
|
-
### Context (for prompts)
|
|
234
|
-
|
|
235
|
-
```json
|
|
236
|
-
// mem_context — recent or query-based
|
|
237
|
-
{
|
|
238
|
-
"query": "indexing decisions",
|
|
239
|
-
"project": "my-app",
|
|
240
|
-
"limit": 5
|
|
241
|
-
}
|
|
58
|
+
```bash
|
|
59
|
+
mixgram mem_search --query 'climate OR temperature' --collection research
|
|
60
|
+
mixgram mem_search --tags baking --type recipe --limit 5 --offset 0
|
|
61
|
+
mixgram mem_get --id DOCUMENT_ID
|
|
242
62
|
```
|
|
243
63
|
|
|
244
|
-
|
|
64
|
+
Without `query` (or with blank text), list recent documents by update time. With a query, use FTS5 syntax: quoted phrases and uppercase `AND`, `OR`, `NOT` are supported. Invalid FTS expressions return an error. Omit `collection` to search all configured collections. Every supplied tag must match. `limit` defaults to 10 and may not exceed `search.maxLimit` (100 by default); `offset` defaults to zero.
|
|
245
65
|
|
|
246
|
-
|
|
66
|
+
Results contain `id`, title, collection, type, key, tags, timestamps and a bounded snippet. Ranked results also contain a score. Read full content using `mem_get`.
|
|
247
67
|
|
|
248
|
-
|
|
249
|
-
- **mem_update** `{ "id": "<id>", "title": "...", "content": "..." }` — update by id.
|
|
250
|
-
- **mem_delete** `{ "id": "<id>", "hardDelete": false }` — soft or hard delete.
|
|
68
|
+
When embeddings are enabled, ranks are fused using `weight / (fusionConstant + rank)`, then deduplicated and paginated. Text and semantic searches share collection/type/tag filters. Semantic search uses exact cosine distance over eligible vectors, so filtering cannot lose matches behind a global candidate limit. This favors correctness for a local memory corpus; vector retrieval and rank fusion scale with the matching corpus size.
|
|
251
69
|
|
|
252
|
-
|
|
70
|
+
Embeddings are optional at runtime. Model and native vector packages remain installed dependencies. The model runs in a child process; saves queue work and do not wait for model inference. Keep `mixgram mcp --embeddings` running to process queued documents, including saves made by one-off CLI commands. A one-off command does not wait for its entire embedding queue to drain. If the worker or vector store fails, search returns textual results with `degraded: true` and a reason. A healthy worker with no completed vectors yet may return only text matches in hybrid mode.
|
|
253
71
|
|
|
254
|
-
|
|
255
|
-
- **mem_stats** — counts (documents, sessions, prompts, `embeddings_enabled`).
|
|
72
|
+
### Persistence and rebuilding
|
|
256
73
|
|
|
257
|
-
|
|
74
|
+
Frontmatter stores `id`, `title`, `type`, `key`, `tags`, `created_at`, `updated_at` and nullable `deleted_at`. Dates use UTC ISO 8601 strings. The body preserves its content, including empty content. Collection is derived from the configured root containing the file.
|
|
258
75
|
|
|
259
|
-
```
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
---
|
|
266
|
-
|
|
267
|
-
## Reindex, watch, and sessions
|
|
268
|
-
|
|
269
|
-
### Reindex from disk
|
|
270
|
-
|
|
271
|
-
Useful after cloning a repo or editing Markdown by hand:
|
|
272
|
-
|
|
273
|
-
```json
|
|
274
|
-
// mem_reindex
|
|
275
|
-
{ "full": true }
|
|
76
|
+
```bash
|
|
77
|
+
mixgram mem_reindex
|
|
78
|
+
mixgram mem_reindex --full
|
|
79
|
+
mixgram mem_delete --id DOCUMENT_ID
|
|
80
|
+
mixgram mem_delete --id DOCUMENT_ID --hardDelete
|
|
276
81
|
```
|
|
277
82
|
|
|
278
|
-
|
|
279
|
-
- `full: false` (default) — incremental (by mtime).
|
|
280
|
-
|
|
281
|
-
Manual edit example:
|
|
282
|
-
|
|
283
|
-
1. Edit a `.md` file under `home/memory` or `docs/<type>/` in your repo.
|
|
284
|
-
2. Call `mem_reindex({ full: true })` (or run with `watch: true` so changes are picked up).
|
|
285
|
-
3. `mem_search` and `mem_get_observation` reflect the new content.
|
|
83
|
+
Save writes a temporary file and renames it before updating SQLite. If indexing fails afterwards, the response explicitly reports `saved: true`, `indexed: false`, and an error; the file is durable and reindexing repairs the index. Soft deletion persists `deleted_at` in Markdown and survives rebuilding; hard deletion removes the file, including a previously soft-deleted file.
|
|
286
84
|
|
|
287
|
-
|
|
85
|
+
Incremental reindexing compares file content hashes. Full reindexing reconstructs the text index from the configured folders. Both report errors per file. Metadata edited in Markdown takes precedence over SQLite. Plain Markdown without frontmatter is indexed using its first heading, file timestamps and a path-derived ID; to retain identity across moves, give the file an explicit `id` in frontmatter. Duplicate IDs, conflicting live keys and invalid metadata are reported as errors.
|
|
288
86
|
|
|
289
|
-
|
|
87
|
+
The watcher handles Markdown additions, changes and removals. Files outside configured roots, including symlinks pointing outside, are rejected. If the OS cannot provide native watches, Chokidar's `CHOKIDAR_USEPOLLING=1` environment setting selects polling.
|
|
290
88
|
|
|
291
|
-
|
|
89
|
+
## Configuration
|
|
292
90
|
|
|
293
|
-
|
|
294
|
-
- **mem_session_end** `{ "session_id": "..." }`
|
|
295
|
-
- **mem_session_summary** `{ "session_id": "...", "content": "...", "title": "..." }` — persists a summary as a memory document.
|
|
296
|
-
- **mem_save_prompt** `{ "session_id": "...", "content": "..." }` — store a prompt for the session.
|
|
297
|
-
- **mem_timeline** `{ "observation_id": "..." }` — before/focus/after in the same session.
|
|
91
|
+
Load order: defaults, config file, environment, CLI flags. Nested options merge; an explicitly supplied `collections` map replaces the default collection map. Relative config paths resolve from the config file's directory.
|
|
298
92
|
|
|
299
|
-
|
|
93
|
+
Default config locations are `./.mixgram/v2/config.json`, then `~/.mixgram/v2/config.json`. Select another file with `--config` or `MIXGRAM_CONFIG`. Version 1 config files are not loaded automatically.
|
|
300
94
|
|
|
301
95
|
```json
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
mem_search({ "query": "reindex", "scope_mode": "home-only" })
|
|
307
|
-
|
|
308
|
-
// Both; project first
|
|
309
|
-
mem_search({ "query": "reindex", "scope_mode": "merged", "project": "my-app" })
|
|
310
|
-
```
|
|
311
|
-
|
|
312
|
-
---
|
|
313
|
-
|
|
314
|
-
## Optional semantic layer
|
|
315
|
-
|
|
316
|
-
When `embeddings.enabled` is `true` and the optional embedding stack is available:
|
|
317
|
-
|
|
318
|
-
- **Saves** are durable and searchable by text immediately; embeddings are queued and processed in the background.
|
|
319
|
-
- **Search** becomes **hybrid**: FTS + vector similarity, blended with `search.ftsWeight` and `search.semanticWeight` (e.g. 0.7 and 0.3).
|
|
320
|
-
- A **worker** runs in a separate process and processes the embedding queue (e.g. every 2 seconds). Query embeddings for hybrid search are also computed in that process via IPC, so the main MCP server never loads `@huggingface/transformers` and is not affected by native/ONNX crashes in the embedding stack.
|
|
321
|
-
|
|
322
|
-
### Enable embeddings
|
|
323
|
-
|
|
324
|
-
1. Ensure optional deps are installed if your setup uses them (e.g. `@huggingface/transformers`, `sqlite-vec`).
|
|
325
|
-
2. Configure:
|
|
326
|
-
|
|
327
|
-
```js
|
|
328
|
-
await run({
|
|
329
|
-
embeddings: {
|
|
330
|
-
enabled: true,
|
|
331
|
-
similarityThreshold: 0.80,
|
|
332
|
-
maxRetries: 3
|
|
96
|
+
{
|
|
97
|
+
"collections": {
|
|
98
|
+
"general": "./general",
|
|
99
|
+
"research": "./research"
|
|
333
100
|
},
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
}
|
|
338
|
-
}
|
|
339
|
-
```
|
|
340
|
-
|
|
341
|
-
### Behaviour
|
|
342
|
-
|
|
343
|
-
- **mem_save** / **mem_update** return as soon as the Markdown is written and the text index is updated; they do **not** wait for embeddings.
|
|
344
|
-
- **mem_search** uses FTS only until vectors exist for the matched chunks; then it uses the hybrid blend.
|
|
345
|
-
- If the embedding provider or sqlite-vec is unavailable, save and search still work (text-only).
|
|
346
|
-
|
|
347
|
-
### Example: hybrid search
|
|
348
|
-
|
|
349
|
-
```json
|
|
350
|
-
// Same tool; behaviour depends on config and whether vectors exist
|
|
351
|
-
mem_search({
|
|
352
|
-
"query": "persistir índices derivados",
|
|
353
|
-
"scope_mode": "merged",
|
|
354
|
-
"project": "my-app",
|
|
355
|
-
"limit": 5
|
|
356
|
-
})
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
With embeddings enabled and vectors ready, results combine literal matches (FTS) and semantic similarity (e.g. paraphrases).
|
|
360
|
-
|
|
361
|
-
### Example: indexing and search (embedding effectiveness)
|
|
362
|
-
|
|
363
|
-
To see the difference between **FTS-only** and **semantic** search in one run, run the demo script (uses a temp dir; first run may download the embedding model):
|
|
364
|
-
|
|
365
|
-
```bash
|
|
366
|
-
node examples/embedding-demo.js
|
|
367
|
-
# or
|
|
368
|
-
npm run example:embedding
|
|
101
|
+
"defaultCollection": "general",
|
|
102
|
+
"sqlitePath": "./index.db",
|
|
103
|
+
"watch": true,
|
|
104
|
+
"embeddings": { "enabled": false }
|
|
105
|
+
}
|
|
369
106
|
```
|
|
370
107
|
|
|
371
|
-
|
|
108
|
+
Defaults: collection `general` at `~/.mixgram/v2/general`, index `~/.mixgram/v2/index.db`, watching and incremental startup synchronization enabled. Collection roots must not overlap, including through symlinks. `defaultCollection` must name a configured collection. Use a separate SQLite path for each distinct set of collections.
|
|
372
109
|
|
|
373
|
-
1
|
|
374
|
-
2. **Processes the embedding queue** so each document gets a vector.
|
|
375
|
-
3. **Runs FTS search** for a query that does *not* appear literally in the text (e.g. “dónde se almacena la memoria del agente”) — FTS returns nothing.
|
|
376
|
-
4. **Runs semantic (vector) search** for the same query — returns the relevant document by meaning.
|
|
110
|
+
Environment: `MIXGRAM_CONFIG`, `MIXGRAM_SQLITE_PATH`, `MIXGRAM_EMBEDDINGS_ENABLED`, `MIXGRAM_WATCH`. Boolean values accept `1`, `0`, `true`, `false`. CLI overrides: `--config`, `--sqlite-path`, `--embeddings` / `--no-embeddings`, `--watch` / `--no-watch`.
|
|
377
111
|
|
|
378
|
-
|
|
112
|
+
Operational defaults live in `src/config.js`. Search weights default to 0.7 text / 0.3 semantic with fusion constant 60. The default embedding model is `Xenova/multilingual-e5-large`, 1024 dimensions, q8. Model output dimensions must match configuration. Embedding jobs retry up to three times; abandoned processing jobs become eligible after the configured five-minute lease (`embeddings.jobLeaseMs`). No silent configuration fallback is applied to invalid inputs.
|
|
379
113
|
|
|
380
|
-
|
|
381
|
-
--- 3. FTS search (text only) ---
|
|
382
|
-
Query: "where is the agent memory stored"
|
|
383
|
-
(FTS finds no literal match; these words are not in the documents.)
|
|
384
|
-
|
|
385
|
-
--- 4. Semantic search (embeddings) ---
|
|
386
|
-
Query: "where is the agent memory stored"
|
|
387
|
-
- [Session persistence] similarity=0.856
|
|
388
|
-
Persist the session context across agent restarts. The important state...
|
|
389
|
-
```
|
|
114
|
+
## Development
|
|
390
115
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
---
|
|
394
|
-
|
|
395
|
-
## Tests
|
|
116
|
+
Node.js with ESM; package manager `pnpm@11.3.0`. No build step is required.
|
|
396
117
|
|
|
397
118
|
```bash
|
|
398
119
|
npm test
|
|
120
|
+
npm run test:large
|
|
121
|
+
npm run quality
|
|
122
|
+
npm run bench:indexing
|
|
123
|
+
npm run bench:search
|
|
399
124
|
```
|
|
400
125
|
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
---
|
|
126
|
+
Tests use temporary folders and SQLite databases. Semantic ranking tests use real sqlite-vec distance calculations with deterministic vectors, without downloading a model. Optional model-backed runs: `npm run example:embedding`, `npm run bench:embed`, `npm run bench:search:embed`; these may download the model on first run.
|
|
404
127
|
|
|
405
128
|
## License
|
|
406
129
|
|