mixgram 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.mixgram/index.db +0 -0
- package/.mixgram/test-esm.db +0 -0
- package/.mixgram/test-index.db +0 -0
- package/.mixgram/test-stage1.db +0 -0
- package/README.md +335 -0
- package/bin/mixgram.js +309 -0
- package/index.js +9 -0
- package/package.json +51 -0
- package/projects/castlebravo/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
- package/projects/test/memory/decisions/decision--architecture/sqlite-derived-index.md +17 -0
- package/spec/fixtures/realistic-content.js +97 -0
- package/spec/reporter.js +127 -0
- package/spec/run-tests-runner.js +40 -0
- package/spec/run-tests.js +160 -0
- package/spec/scenarios/scenarios.js +520 -0
- package/spec/sdd.md +1002 -0
- package/spec/tdd.md +1237 -0
- package/src/config.js +56 -0
- package/src/core/documents/documents.js +207 -0
- package/src/core/embeddings/embedder.js +58 -0
- package/src/core/embeddings/queue.js +118 -0
- package/src/core/embeddings/vectorStore.js +127 -0
- package/src/core/embeddings/worker.js +27 -0
- package/src/core/indexing/indexer.js +131 -0
- package/src/core/indexing/parser.js +78 -0
- package/src/core/indexing/reindex.js +101 -0
- package/src/core/prompts/prompts.js +15 -0
- package/src/core/search/search.js +223 -0
- package/src/core/sessions/sessions.js +27 -0
- package/src/db/migrations/001_initial.sql +86 -0
- package/src/db/sqlite.js +39 -0
- package/src/fs/paths.js +79 -0
- package/src/fs/watcher.js +49 -0
- package/src/mcp/cli-adapter.js +118 -0
- package/src/mcp/server.js +47 -0
- package/src/mcp/tool-registry.js +138 -0
- package/src/mcp/tools.js +159 -0
- package/src/utils/hash.js +6 -0
- package/src/utils/ids.js +22 -0
- package/src/utils/markdown.js +49 -0
package/spec/tdd.md
ADDED
|
@@ -0,0 +1,1237 @@
|
|
|
1
|
+
# TDD
|
|
2
|
+
|
|
3
|
+
## Mixgram
|
|
4
|
+
|
|
5
|
+
### Black-box Test Design for an Engram-compatible MCP memory system with Markdown source of truth, SQLite text indexing, and optional asynchronous local embeddings
|
|
6
|
+
|
|
7
|
+
**Status:** Final
|
|
8
|
+
**Test approach:** Black-box
|
|
9
|
+
**System under test:** Mixgram
|
|
10
|
+
**Primary compatibility target:** Engram MCP tool contract
|
|
11
|
+
**Source of truth:** Markdown files
|
|
12
|
+
**Text retrieval:** SQLite + FTS5
|
|
13
|
+
**Semantic retrieval:** Optional local embeddings, asynchronous
|
|
14
|
+
**Language:** English
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# 1. Purpose
|
|
19
|
+
|
|
20
|
+
This document defines the **black-box test design** for Mixgram.
|
|
21
|
+
|
|
22
|
+
The goal is to verify the system from the outside, based on:
|
|
23
|
+
|
|
24
|
+
* MCP tool behavior
|
|
25
|
+
* file-system side effects
|
|
26
|
+
* search and retrieval behavior
|
|
27
|
+
* indexing behavior
|
|
28
|
+
* scope behavior
|
|
29
|
+
* optional semantic behavior
|
|
30
|
+
* failure and degradation behavior
|
|
31
|
+
|
|
32
|
+
These tests intentionally avoid coupling to internal implementation details such as repository classes, SQL statements, queue internals, parser internals, or watcher internals.
|
|
33
|
+
|
|
34
|
+
The system passes these tests if it behaves correctly from the perspective of:
|
|
35
|
+
|
|
36
|
+
* an MCP client
|
|
37
|
+
* a user editing Markdown files
|
|
38
|
+
* a retrieval workflow consuming the stored knowledge
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
# 2. Black-box scope
|
|
43
|
+
|
|
44
|
+
## 2.1 Included
|
|
45
|
+
|
|
46
|
+
* MCP-compatible tool behavior
|
|
47
|
+
* Markdown persistence behavior
|
|
48
|
+
* full-text retrieval behavior
|
|
49
|
+
* scope handling
|
|
50
|
+
* `home` versus `project` knowledge behavior
|
|
51
|
+
* deduplication and update semantics
|
|
52
|
+
* session behavior
|
|
53
|
+
* prompt behavior
|
|
54
|
+
* rebuild and reindex behavior
|
|
55
|
+
* embeddings disabled behavior
|
|
56
|
+
* embeddings enabled asynchronous behavior
|
|
57
|
+
* graceful degradation when semantic indexing is unavailable
|
|
58
|
+
* configurable full-text vs semantic search weight when embeddings are enabled
|
|
59
|
+
|
|
60
|
+
## 2.2 Excluded
|
|
61
|
+
|
|
62
|
+
* internal method calls
|
|
63
|
+
* internal database schema correctness beyond observable outcomes
|
|
64
|
+
* queue implementation details
|
|
65
|
+
* private ranking formulas
|
|
66
|
+
* exact chunking internals unless visible in output behavior
|
|
67
|
+
* internal file watcher mechanism
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
# 3. Test strategy
|
|
72
|
+
|
|
73
|
+
## 3.1 Test philosophy
|
|
74
|
+
|
|
75
|
+
The system is tested as a product, not as a collection of modules.
|
|
76
|
+
|
|
77
|
+
Each test defines:
|
|
78
|
+
|
|
79
|
+
* initial conditions
|
|
80
|
+
* external action
|
|
81
|
+
* observable outcome
|
|
82
|
+
|
|
83
|
+
## 3.2 Observable interfaces
|
|
84
|
+
|
|
85
|
+
The black-box surfaces are:
|
|
86
|
+
|
|
87
|
+
* MCP tools
|
|
88
|
+
* created or updated Markdown files
|
|
89
|
+
* search results
|
|
90
|
+
* returned observation data
|
|
91
|
+
* stats output
|
|
92
|
+
* reindex behavior
|
|
93
|
+
* behavior when semantic indexing is off or delayed
|
|
94
|
+
|
|
95
|
+
## 3.3 Pass/fail rule
|
|
96
|
+
|
|
97
|
+
A test passes if the externally visible behavior matches the specified result, regardless of how the system achieves it.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
# 4. Test environments
|
|
102
|
+
|
|
103
|
+
At minimum, tests should run in these environments.
|
|
104
|
+
|
|
105
|
+
## 4.1 Environment A: text-only mode
|
|
106
|
+
|
|
107
|
+
* embeddings disabled
|
|
108
|
+
* watcher enabled
|
|
109
|
+
* SQLite empty at start
|
|
110
|
+
|
|
111
|
+
## 4.2 Environment B: semantic optional mode
|
|
112
|
+
|
|
113
|
+
* embeddings enabled
|
|
114
|
+
* local embedding worker available
|
|
115
|
+
* asynchronous queue enabled
|
|
116
|
+
|
|
117
|
+
## 4.3 Environment C: semantic unavailable mode
|
|
118
|
+
|
|
119
|
+
* embeddings enabled in config
|
|
120
|
+
* embedding worker unavailable or provider failing
|
|
121
|
+
|
|
122
|
+
This environment is critical because the system must still work.
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
# 5. Core test data model
|
|
127
|
+
|
|
128
|
+
The following kinds of documents should exist in fixtures:
|
|
129
|
+
|
|
130
|
+
* `home` architecture note
|
|
131
|
+
* `home` pattern note
|
|
132
|
+
* `project` decision note
|
|
133
|
+
* `project` bug note
|
|
134
|
+
* `project` session summary
|
|
135
|
+
* `project` learning note
|
|
136
|
+
* a duplicate or near-duplicate note
|
|
137
|
+
* a note that is later manually edited
|
|
138
|
+
* a note with a shared topic between `home` and `project`
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
# 6. Functional areas under test
|
|
143
|
+
|
|
144
|
+
1. Markdown persistence
|
|
145
|
+
2. MCP compatibility
|
|
146
|
+
3. Topic-key upsert behavior
|
|
147
|
+
4. Scope-aware retrieval
|
|
148
|
+
5. Full-text search
|
|
149
|
+
6. Snippet retrieval
|
|
150
|
+
7. Session lifecycle
|
|
151
|
+
8. Prompt storage
|
|
152
|
+
9. Reindexing and rebuild
|
|
153
|
+
10. Embeddings disabled mode
|
|
154
|
+
11. Embeddings enabled asynchronous mode
|
|
155
|
+
12. Failure tolerance
|
|
156
|
+
13. Stats and observability
|
|
157
|
+
14. Configurable search weight (full-text vs semantic)
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
# 7. Test cases
|
|
162
|
+
|
|
163
|
+
## 7.1 `mem_save` creates a new project-scoped document
|
|
164
|
+
|
|
165
|
+
**ID:** BB-001
|
|
166
|
+
**Preconditions:**
|
|
167
|
+
|
|
168
|
+
* system is running
|
|
169
|
+
* project `castlebravo` exists
|
|
170
|
+
* no prior document with topic key `architecture/sqlite-derived-index`
|
|
171
|
+
|
|
172
|
+
**Action:**
|
|
173
|
+
Call `mem_save` with:
|
|
174
|
+
|
|
175
|
+
* title: `Use SQLite as derived index`
|
|
176
|
+
* type: `decision`
|
|
177
|
+
* scope: `project`
|
|
178
|
+
* project: `castlebravo`
|
|
179
|
+
* topic_key: `architecture/sqlite-derived-index`
|
|
180
|
+
* content: meaningful body text
|
|
181
|
+
|
|
182
|
+
**Expected result:**
|
|
183
|
+
|
|
184
|
+
* the tool returns success
|
|
185
|
+
* one Markdown file is created under the project memory tree
|
|
186
|
+
* the file contains frontmatter with matching metadata
|
|
187
|
+
* the content is searchable immediately through text search
|
|
188
|
+
* the result is retrievable through `mem_get_observation`
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## 7.2 `mem_save` creates a new home-scoped document
|
|
193
|
+
|
|
194
|
+
**ID:** BB-002
|
|
195
|
+
**Preconditions:**
|
|
196
|
+
|
|
197
|
+
* no prior `home` document with the same topic key
|
|
198
|
+
|
|
199
|
+
**Action:**
|
|
200
|
+
Call `mem_save` with:
|
|
201
|
+
|
|
202
|
+
* scope: `home`
|
|
203
|
+
* no project
|
|
204
|
+
* type: `pattern`
|
|
205
|
+
* topic_key: `pattern/incremental-reindexing`
|
|
206
|
+
|
|
207
|
+
**Expected result:**
|
|
208
|
+
|
|
209
|
+
* a Markdown file is created under `home/memory/patterns/`
|
|
210
|
+
* the document is searchable from a global context
|
|
211
|
+
* the document is also retrievable in merged project search mode
|
|
212
|
+
|
|
213
|
+
---
|
|
214
|
+
|
|
215
|
+
## 7.3 `mem_save` with existing topic key updates instead of creating a second document
|
|
216
|
+
|
|
217
|
+
**ID:** BB-003
|
|
218
|
+
**Preconditions:**
|
|
219
|
+
|
|
220
|
+
* an existing project document has topic key `architecture/sqlite-derived-index`
|
|
221
|
+
|
|
222
|
+
**Action:**
|
|
223
|
+
Call `mem_save` again with the same:
|
|
224
|
+
|
|
225
|
+
* scope
|
|
226
|
+
* project
|
|
227
|
+
* topic key
|
|
228
|
+
|
|
229
|
+
but updated content
|
|
230
|
+
|
|
231
|
+
**Expected result:**
|
|
232
|
+
|
|
233
|
+
* no second Markdown document is created for the same logical topic
|
|
234
|
+
* the original document content is updated
|
|
235
|
+
* `revision_count` is increased
|
|
236
|
+
* search reflects the updated content
|
|
237
|
+
* `mem_get_observation` returns the updated version
|
|
238
|
+
|
|
239
|
+
---
|
|
240
|
+
|
|
241
|
+
## 7.4 `mem_update` updates document by ID
|
|
242
|
+
|
|
243
|
+
**ID:** BB-004
|
|
244
|
+
**Preconditions:**
|
|
245
|
+
|
|
246
|
+
* an existing document is available
|
|
247
|
+
* its ID is known
|
|
248
|
+
|
|
249
|
+
**Action:**
|
|
250
|
+
Call `mem_update` with:
|
|
251
|
+
|
|
252
|
+
* id
|
|
253
|
+
* updated title
|
|
254
|
+
* updated content
|
|
255
|
+
|
|
256
|
+
**Expected result:**
|
|
257
|
+
|
|
258
|
+
* the same Markdown file is updated
|
|
259
|
+
* `updated_at` changes
|
|
260
|
+
* `revision_count` increases
|
|
261
|
+
* old terms are no longer dominant in search
|
|
262
|
+
* new terms become searchable
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## 7.5 `mem_delete` soft delete hides document from normal search
|
|
267
|
+
|
|
268
|
+
**ID:** BB-005
|
|
269
|
+
**Preconditions:**
|
|
270
|
+
|
|
271
|
+
* an existing searchable document is available
|
|
272
|
+
|
|
273
|
+
**Action:**
|
|
274
|
+
Call `mem_delete` with:
|
|
275
|
+
|
|
276
|
+
* id
|
|
277
|
+
* hardDelete: false
|
|
278
|
+
|
|
279
|
+
**Expected result:**
|
|
280
|
+
|
|
281
|
+
* the Markdown file remains present unless the product explicitly rewrites soft-deleted files elsewhere
|
|
282
|
+
* the document is marked deleted in observable metadata
|
|
283
|
+
* the document is excluded from normal `mem_search`
|
|
284
|
+
* direct retrieval behavior follows product policy, but deleted state must be visible if retrieved
|
|
285
|
+
|
|
286
|
+
---
|
|
287
|
+
|
|
288
|
+
## 7.6 `mem_delete` hard delete removes retrievability
|
|
289
|
+
|
|
290
|
+
**ID:** BB-006
|
|
291
|
+
**Preconditions:**
|
|
292
|
+
|
|
293
|
+
* an existing document is available
|
|
294
|
+
|
|
295
|
+
**Action:**
|
|
296
|
+
Call `mem_delete` with:
|
|
297
|
+
|
|
298
|
+
* id
|
|
299
|
+
* hardDelete: true
|
|
300
|
+
|
|
301
|
+
**Expected result:**
|
|
302
|
+
|
|
303
|
+
* the Markdown file is removed
|
|
304
|
+
* the document no longer appears in search
|
|
305
|
+
* `mem_get_observation` no longer returns it as an active observation
|
|
306
|
+
|
|
307
|
+
---
|
|
308
|
+
|
|
309
|
+
## 7.7 `mem_suggest_topic_key` returns a stable category-aligned topic key
|
|
310
|
+
|
|
311
|
+
**ID:** BB-007
|
|
312
|
+
**Preconditions:**
|
|
313
|
+
|
|
314
|
+
* system running
|
|
315
|
+
|
|
316
|
+
**Action:**
|
|
317
|
+
Call `mem_suggest_topic_key` with:
|
|
318
|
+
|
|
319
|
+
* title indicating a bug
|
|
320
|
+
* content describing timeline ordering problems
|
|
321
|
+
|
|
322
|
+
**Expected result:**
|
|
323
|
+
|
|
324
|
+
* the returned topic key is non-empty
|
|
325
|
+
* it follows a stable category structure
|
|
326
|
+
* it reflects the document type semantically
|
|
327
|
+
* repeated calls with equivalent input produce compatible results
|
|
328
|
+
|
|
329
|
+
---
|
|
330
|
+
|
|
331
|
+
## 7.8 `mem_search` returns relevant full-text results when embeddings are disabled
|
|
332
|
+
|
|
333
|
+
**ID:** BB-008
|
|
334
|
+
**Environment:** text-only mode
|
|
335
|
+
|
|
336
|
+
**Preconditions:**
|
|
337
|
+
|
|
338
|
+
* multiple documents exist
|
|
339
|
+
* at least one contains a unique phrase such as `derived SQLite index`
|
|
340
|
+
|
|
341
|
+
**Action:**
|
|
342
|
+
Call `mem_search` with a text query targeting that phrase
|
|
343
|
+
|
|
344
|
+
**Expected result:**
|
|
345
|
+
|
|
346
|
+
* relevant result is returned
|
|
347
|
+
* result includes snippet-like context
|
|
348
|
+
* result includes enough metadata to identify the document
|
|
349
|
+
* system works with no semantic layer active
|
|
350
|
+
* search ranking is determined solely by full-text (FTS/BM25); no semantic contribution (effective 100% full-text)
|
|
351
|
+
|
|
352
|
+
---
|
|
353
|
+
|
|
354
|
+
## 7.9 `mem_search` supports merged project + home retrieval
|
|
355
|
+
|
|
356
|
+
**ID:** BB-009
|
|
357
|
+
**Preconditions:**
|
|
358
|
+
|
|
359
|
+
* project note exists
|
|
360
|
+
* home note exists
|
|
361
|
+
* both are relevant to the query
|
|
362
|
+
|
|
363
|
+
**Action:**
|
|
364
|
+
Run a project-context search in merged mode
|
|
365
|
+
|
|
366
|
+
**Expected result:**
|
|
367
|
+
|
|
368
|
+
* both project and home results may appear
|
|
369
|
+
* project-specific result ranks above home result when both are similarly relevant
|
|
370
|
+
* home result remains available as fallback/global knowledge
|
|
371
|
+
|
|
372
|
+
---
|
|
373
|
+
|
|
374
|
+
## 7.10 `mem_search` supports project-only retrieval
|
|
375
|
+
|
|
376
|
+
**ID:** BB-010
|
|
377
|
+
**Preconditions:**
|
|
378
|
+
|
|
379
|
+
* both home and project documents exist
|
|
380
|
+
|
|
381
|
+
**Action:**
|
|
382
|
+
Run search restricted to project scope only
|
|
383
|
+
|
|
384
|
+
**Expected result:**
|
|
385
|
+
|
|
386
|
+
* only project-scoped results are returned
|
|
387
|
+
* no home documents appear
|
|
388
|
+
|
|
389
|
+
---
|
|
390
|
+
|
|
391
|
+
## 7.11 `mem_search` supports home-only retrieval
|
|
392
|
+
|
|
393
|
+
**ID:** BB-011
|
|
394
|
+
**Preconditions:**
|
|
395
|
+
|
|
396
|
+
* both home and project documents exist
|
|
397
|
+
|
|
398
|
+
**Action:**
|
|
399
|
+
Run search restricted to home scope only
|
|
400
|
+
|
|
401
|
+
**Expected result:**
|
|
402
|
+
|
|
403
|
+
* only home-scoped results are returned
|
|
404
|
+
|
|
405
|
+
---
|
|
406
|
+
|
|
407
|
+
## 7.12 `mem_search` returns snippets rather than only whole-document blobs
|
|
408
|
+
|
|
409
|
+
**ID:** BB-012
|
|
410
|
+
**Preconditions:**
|
|
411
|
+
|
|
412
|
+
* a long document exists with multiple sections
|
|
413
|
+
* query targets a specific section
|
|
414
|
+
|
|
415
|
+
**Action:**
|
|
416
|
+
Call `mem_search`
|
|
417
|
+
|
|
418
|
+
**Expected result:**
|
|
419
|
+
|
|
420
|
+
* result includes a focused excerpt or snippet
|
|
421
|
+
* snippet is relevant to the matched section
|
|
422
|
+
* heading or section context is visible if supported
|
|
423
|
+
|
|
424
|
+
---
|
|
425
|
+
|
|
426
|
+
## 7.13 `mem_get_observation` returns full document content
|
|
427
|
+
|
|
428
|
+
**ID:** BB-013
|
|
429
|
+
**Preconditions:**
|
|
430
|
+
|
|
431
|
+
* a document exists
|
|
432
|
+
|
|
433
|
+
**Action:**
|
|
434
|
+
Call `mem_get_observation` with its ID
|
|
435
|
+
|
|
436
|
+
**Expected result:**
|
|
437
|
+
|
|
438
|
+
* full content is returned
|
|
439
|
+
* metadata includes document identity information
|
|
440
|
+
* metadata includes scope
|
|
441
|
+
* metadata includes project when applicable
|
|
442
|
+
* metadata includes topic key when present
|
|
443
|
+
|
|
444
|
+
---
|
|
445
|
+
|
|
446
|
+
## 7.14 `mem_context` returns usable contextual memory for a project
|
|
447
|
+
|
|
448
|
+
**ID:** BB-014
|
|
449
|
+
**Preconditions:**
|
|
450
|
+
|
|
451
|
+
* project contains multiple relevant notes, such as decision, learning, session summary
|
|
452
|
+
|
|
453
|
+
**Action:**
|
|
454
|
+
Call `mem_context` for that project
|
|
455
|
+
|
|
456
|
+
**Expected result:**
|
|
457
|
+
|
|
458
|
+
* response contains condensed useful context
|
|
459
|
+
* response is not empty
|
|
460
|
+
* context reflects project-local knowledge
|
|
461
|
+
* merged mode may also include relevant home knowledge
|
|
462
|
+
|
|
463
|
+
---
|
|
464
|
+
|
|
465
|
+
## 7.15 `mem_timeline` returns surrounding session observations
|
|
466
|
+
|
|
467
|
+
**ID:** BB-015
|
|
468
|
+
**Preconditions:**
|
|
469
|
+
|
|
470
|
+
* multiple observations exist in the same session
|
|
471
|
+
* one target observation is in the middle
|
|
472
|
+
|
|
473
|
+
**Action:**
|
|
474
|
+
Call `mem_timeline` using the middle observation ID
|
|
475
|
+
|
|
476
|
+
**Expected result:**
|
|
477
|
+
|
|
478
|
+
* response contains before/focus/after structure or equivalent observable ordering
|
|
479
|
+
* the focus observation is clearly identifiable
|
|
480
|
+
* neighboring observations are chronologically meaningful
|
|
481
|
+
|
|
482
|
+
---
|
|
483
|
+
|
|
484
|
+
## 7.16 `mem_session_start` creates a session
|
|
485
|
+
|
|
486
|
+
**ID:** BB-016
|
|
487
|
+
**Preconditions:**
|
|
488
|
+
|
|
489
|
+
* no session with the given ID exists
|
|
490
|
+
|
|
491
|
+
**Action:**
|
|
492
|
+
Call `mem_session_start`
|
|
493
|
+
|
|
494
|
+
**Expected result:**
|
|
495
|
+
|
|
496
|
+
* success is returned
|
|
497
|
+
* the session becomes available for subsequent memory operations
|
|
498
|
+
* later observations can attach to this session
|
|
499
|
+
|
|
500
|
+
---
|
|
501
|
+
|
|
502
|
+
## 7.17 `mem_session_end` closes a session
|
|
503
|
+
|
|
504
|
+
**ID:** BB-017
|
|
505
|
+
**Preconditions:**
|
|
506
|
+
|
|
507
|
+
* an active session exists
|
|
508
|
+
|
|
509
|
+
**Action:**
|
|
510
|
+
Call `mem_session_end`
|
|
511
|
+
|
|
512
|
+
**Expected result:**
|
|
513
|
+
|
|
514
|
+
* success is returned
|
|
515
|
+
* session metadata reflects closure
|
|
516
|
+
* later timeline/session retrieval reflects ended state if exposed
|
|
517
|
+
|
|
518
|
+
---
|
|
519
|
+
|
|
520
|
+
## 7.18 `mem_session_summary` persists a session summary as durable memory
|
|
521
|
+
|
|
522
|
+
**ID:** BB-018
|
|
523
|
+
**Preconditions:**
|
|
524
|
+
|
|
525
|
+
* session exists
|
|
526
|
+
|
|
527
|
+
**Action:**
|
|
528
|
+
Call `mem_session_summary` with summary content
|
|
529
|
+
|
|
530
|
+
**Expected result:**
|
|
531
|
+
|
|
532
|
+
* a session summary note is stored as durable memory
|
|
533
|
+
* it is searchable
|
|
534
|
+
* it is associated with the relevant session/project
|
|
535
|
+
|
|
536
|
+
---
|
|
537
|
+
|
|
538
|
+
## 7.19 `mem_save_prompt` stores prompt data without requiring Markdown persistence by default
|
|
539
|
+
|
|
540
|
+
**ID:** BB-019
|
|
541
|
+
**Preconditions:**
|
|
542
|
+
|
|
543
|
+
* session exists
|
|
544
|
+
|
|
545
|
+
**Action:**
|
|
546
|
+
Call `mem_save_prompt`
|
|
547
|
+
|
|
548
|
+
**Expected result:**
|
|
549
|
+
|
|
550
|
+
* success is returned
|
|
551
|
+
* prompt contributes to prompt statistics
|
|
552
|
+
* prompt storage behavior matches product policy
|
|
553
|
+
* core memory behavior is unaffected
|
|
554
|
+
|
|
555
|
+
---
|
|
556
|
+
|
|
557
|
+
## 7.20 `mem_stats` reflects stored memory
|
|
558
|
+
|
|
559
|
+
**ID:** BB-020
|
|
560
|
+
**Preconditions:**
|
|
561
|
+
|
|
562
|
+
* system contains documents, sessions, and prompts
|
|
563
|
+
|
|
564
|
+
**Action:**
|
|
565
|
+
Call `mem_stats`
|
|
566
|
+
|
|
567
|
+
**Expected result:**
|
|
568
|
+
|
|
569
|
+
* counts are non-zero where expected
|
|
570
|
+
* reported totals are consistent with prior actions
|
|
571
|
+
* stats do not require embeddings to be enabled
|
|
572
|
+
|
|
573
|
+
---
|
|
574
|
+
|
|
575
|
+
# 8. Manual Markdown edit test cases
|
|
576
|
+
|
|
577
|
+
## 8.1 Manual edit updates search results after reindex
|
|
578
|
+
|
|
579
|
+
**ID:** BB-021
|
|
580
|
+
**Preconditions:**
|
|
581
|
+
|
|
582
|
+
* a document exists
|
|
583
|
+
* current content is searchable by phrase A
|
|
584
|
+
|
|
585
|
+
**Action:**
|
|
586
|
+
Manually edit the Markdown file:
|
|
587
|
+
|
|
588
|
+
* remove phrase A
|
|
589
|
+
* add phrase B
|
|
590
|
+
|
|
591
|
+
Wait for automatic reindex or trigger official reindex command
|
|
592
|
+
|
|
593
|
+
**Expected result:**
|
|
594
|
+
|
|
595
|
+
* phrase A is no longer the primary match
|
|
596
|
+
* phrase B becomes searchable
|
|
597
|
+
* `mem_get_observation` reflects edited content
|
|
598
|
+
|
|
599
|
+
---
|
|
600
|
+
|
|
601
|
+
## 8.2 Manual creation of a valid Markdown file becomes searchable
|
|
602
|
+
|
|
603
|
+
**ID:** BB-022
|
|
604
|
+
**Preconditions:**
|
|
605
|
+
|
|
606
|
+
* none
|
|
607
|
+
|
|
608
|
+
**Action:**
|
|
609
|
+
Manually place a valid Markdown memory file in a watched directory
|
|
610
|
+
|
|
611
|
+
**Expected result:**
|
|
612
|
+
|
|
613
|
+
* file becomes indexed
|
|
614
|
+
* file appears in search
|
|
615
|
+
* metadata is retrievable through observation access if identity is supported
|
|
616
|
+
|
|
617
|
+
---
|
|
618
|
+
|
|
619
|
+
## 8.3 Manual deletion removes search visibility
|
|
620
|
+
|
|
621
|
+
**ID:** BB-023
|
|
622
|
+
**Preconditions:**
|
|
623
|
+
|
|
624
|
+
* a valid indexed Markdown file exists
|
|
625
|
+
|
|
626
|
+
**Action:**
|
|
627
|
+
Delete the file manually and allow reindexing
|
|
628
|
+
|
|
629
|
+
**Expected result:**
|
|
630
|
+
|
|
631
|
+
* document no longer appears in normal search
|
|
632
|
+
* retrieval reflects deletion according to product policy
|
|
633
|
+
|
|
634
|
+
---
|
|
635
|
+
|
|
636
|
+
# 9. Rebuild and recovery test cases
|
|
637
|
+
|
|
638
|
+
## 9.1 Full reindex rebuilds index from Markdown files
|
|
639
|
+
|
|
640
|
+
**ID:** BB-024
|
|
641
|
+
**Preconditions:**
|
|
642
|
+
|
|
643
|
+
* multiple valid Markdown files exist
|
|
644
|
+
* SQLite index exists and is then removed
|
|
645
|
+
|
|
646
|
+
**Action:**
|
|
647
|
+
Run the official full reindex command
|
|
648
|
+
|
|
649
|
+
**Expected result:**
|
|
650
|
+
|
|
651
|
+
* the text index is rebuilt successfully
|
|
652
|
+
* all valid Markdown notes become searchable again
|
|
653
|
+
* no loss of canonical memory occurs
|
|
654
|
+
|
|
655
|
+
---
|
|
656
|
+
|
|
657
|
+
## 9.2 Unchanged files are not duplicated after reindex
|
|
658
|
+
|
|
659
|
+
**ID:** BB-025
|
|
660
|
+
**Preconditions:**
|
|
661
|
+
|
|
662
|
+
* indexed corpus exists
|
|
663
|
+
|
|
664
|
+
**Action:**
|
|
665
|
+
Run normal incremental reindex without changing files
|
|
666
|
+
|
|
667
|
+
**Expected result:**
|
|
668
|
+
|
|
669
|
+
* no duplicate documents appear
|
|
670
|
+
* search result counts remain stable
|
|
671
|
+
* stats remain logically consistent
|
|
672
|
+
|
|
673
|
+
---
|
|
674
|
+
|
|
675
|
+
# 10. Home versus project behavior
|
|
676
|
+
|
|
677
|
+
## 10.1 Home knowledge is reusable in project search
|
|
678
|
+
|
|
679
|
+
**ID:** BB-026
|
|
680
|
+
**Preconditions:**
|
|
681
|
+
|
|
682
|
+
* a home-scoped reference note exists
|
|
683
|
+
* project does not yet contain equivalent local knowledge
|
|
684
|
+
|
|
685
|
+
**Action:**
|
|
686
|
+
Search from project context in merged mode
|
|
687
|
+
|
|
688
|
+
**Expected result:**
|
|
689
|
+
|
|
690
|
+
* home note is returned
|
|
691
|
+
* it is visible as reusable supporting knowledge
|
|
692
|
+
|
|
693
|
+
---
|
|
694
|
+
|
|
695
|
+
## 10.2 Project knowledge outranks home knowledge in project context
|
|
696
|
+
|
|
697
|
+
**ID:** BB-027
|
|
698
|
+
**Preconditions:**
|
|
699
|
+
|
|
700
|
+
* both home and project notes cover similar topic
|
|
701
|
+
* project note is specifically tailored to the active project
|
|
702
|
+
|
|
703
|
+
**Action:**
|
|
704
|
+
Search in merged mode from the project context
|
|
705
|
+
|
|
706
|
+
**Expected result:**
|
|
707
|
+
|
|
708
|
+
* project-specific note ranks higher than home note
|
|
709
|
+
* home note may still appear
|
|
710
|
+
|
|
711
|
+
---
|
|
712
|
+
|
|
713
|
+
## 10.3 Saving project-scoped memory does not overwrite home-scoped memory accidentally
|
|
714
|
+
|
|
715
|
+
**ID:** BB-028
|
|
716
|
+
**Preconditions:**
|
|
717
|
+
|
|
718
|
+
* home document exists with topic key `pattern/incremental-reindexing`
|
|
719
|
+
|
|
720
|
+
**Action:**
|
|
721
|
+
Call `mem_save` for a project-scoped note with a similar topic key
|
|
722
|
+
|
|
723
|
+
**Expected result:**
|
|
724
|
+
|
|
725
|
+
* the project note is saved within project scope
|
|
726
|
+
* home note remains intact
|
|
727
|
+
* scope boundaries are respected
|
|
728
|
+
|
|
729
|
+
---
|
|
730
|
+
|
|
731
|
+
# 11. Deduplication behavior
|
|
732
|
+
|
|
733
|
+
## 11.1 Repeated save of identical content does not create uncontrolled duplicates
|
|
734
|
+
|
|
735
|
+
**ID:** BB-029
|
|
736
|
+
**Preconditions:**
|
|
737
|
+
|
|
738
|
+
* a document already exists
|
|
739
|
+
|
|
740
|
+
**Action:**
|
|
741
|
+
Call `mem_save` again with effectively identical content and same logical identity
|
|
742
|
+
|
|
743
|
+
**Expected result:**
|
|
744
|
+
|
|
745
|
+
* duplicate proliferation does not occur
|
|
746
|
+
* either the same document is reused or duplicate handling is reflected in metadata
|
|
747
|
+
* search remains clean and usable
|
|
748
|
+
|
|
749
|
+
---
|
|
750
|
+
|
|
751
|
+
## 11.2 Near-duplicate content with different topic should remain distinguishable
|
|
752
|
+
|
|
753
|
+
**ID:** BB-030
|
|
754
|
+
**Preconditions:**
|
|
755
|
+
|
|
756
|
+
* one document exists
|
|
757
|
+
|
|
758
|
+
**Action:**
|
|
759
|
+
Create a second document with similar wording but a different logical topic
|
|
760
|
+
|
|
761
|
+
**Expected result:**
|
|
762
|
+
|
|
763
|
+
* both are preserved if they are semantically distinct enough
|
|
764
|
+
* retrieval does not collapse legitimate separate notes incorrectly
|
|
765
|
+
|
|
766
|
+
---
|
|
767
|
+
|
|
768
|
+
# 12. Embeddings disabled behavior
|
|
769
|
+
|
|
770
|
+
## 12.1 System fully works with embeddings disabled
|
|
771
|
+
|
|
772
|
+
**ID:** BB-031
|
|
773
|
+
**Environment:** text-only mode
|
|
774
|
+
|
|
775
|
+
**Preconditions:**
|
|
776
|
+
|
|
777
|
+
* embeddings disabled in config
|
|
778
|
+
|
|
779
|
+
**Action:**
|
|
780
|
+
Perform normal sequence:
|
|
781
|
+
|
|
782
|
+
* save
|
|
783
|
+
* update
|
|
784
|
+
* search
|
|
785
|
+
* context
|
|
786
|
+
* get observation
|
|
787
|
+
* stats
|
|
788
|
+
|
|
789
|
+
**Expected result:**
|
|
790
|
+
|
|
791
|
+
* all operations succeed
|
|
792
|
+
* search remains useful
|
|
793
|
+
* no semantic subsystem is required
|
|
794
|
+
|
|
795
|
+
---
|
|
796
|
+
|
|
797
|
+
## 12.2 `mem_stats` reflects disabled semantic mode appropriately
|
|
798
|
+
|
|
799
|
+
**ID:** BB-032
|
|
800
|
+
**Environment:** text-only mode
|
|
801
|
+
|
|
802
|
+
**Preconditions:**
|
|
803
|
+
|
|
804
|
+
* embeddings disabled
|
|
805
|
+
|
|
806
|
+
**Action:**
|
|
807
|
+
Call `mem_stats`
|
|
808
|
+
|
|
809
|
+
**Expected result:**
|
|
810
|
+
|
|
811
|
+
* stats remain valid
|
|
812
|
+
* semantic counters, if present, reflect disabled or zeroed state consistently
|
|
813
|
+
|
|
814
|
+
---
|
|
815
|
+
|
|
816
|
+
# 13. Embeddings enabled asynchronous behavior
|
|
817
|
+
|
|
818
|
+
## 13.1 Save returns before semantic indexing completes
|
|
819
|
+
|
|
820
|
+
**ID:** BB-033
|
|
821
|
+
**Environment:** semantic optional mode
|
|
822
|
+
|
|
823
|
+
**Preconditions:**
|
|
824
|
+
|
|
825
|
+
* embeddings enabled
|
|
826
|
+
* worker active
|
|
827
|
+
* save content large enough to require noticeable background work
|
|
828
|
+
|
|
829
|
+
**Action:**
|
|
830
|
+
Call `mem_save`
|
|
831
|
+
|
|
832
|
+
**Expected result:**
|
|
833
|
+
|
|
834
|
+
* tool returns success promptly after durable save and text indexing
|
|
835
|
+
* document is searchable immediately via text search
|
|
836
|
+
* semantic enrichment may appear later
|
|
837
|
+
* operation does not wait for background embedding completion
|
|
838
|
+
|
|
839
|
+
---
|
|
840
|
+
|
|
841
|
+
## 13.2 Search works before semantic indexing completes
|
|
842
|
+
|
|
843
|
+
**ID:** BB-034
|
|
844
|
+
**Environment:** semantic optional mode
|
|
845
|
+
|
|
846
|
+
**Preconditions:**
|
|
847
|
+
|
|
848
|
+
* recently saved content is still pending semantic indexing
|
|
849
|
+
|
|
850
|
+
**Action:**
|
|
851
|
+
Run `mem_search`
|
|
852
|
+
|
|
853
|
+
**Expected result:**
|
|
854
|
+
|
|
855
|
+
* relevant text results are already available
|
|
856
|
+
* absence of finished embeddings does not block retrieval
|
|
857
|
+
|
|
858
|
+
---
|
|
859
|
+
|
|
860
|
+
## 13.3 Semantic quality may improve after background completion
|
|
861
|
+
|
|
862
|
+
**ID:** BB-035
|
|
863
|
+
**Environment:** semantic optional mode
|
|
864
|
+
|
|
865
|
+
**Preconditions:**
|
|
866
|
+
|
|
867
|
+
* embeddings enabled
|
|
868
|
+
* query uses paraphrased wording not strongly matching the literal text
|
|
869
|
+
* background indexing completes after some delay
|
|
870
|
+
|
|
871
|
+
**Action:**
|
|
872
|
+
Run search before and after semantic completion
|
|
873
|
+
|
|
874
|
+
**Expected result:**
|
|
875
|
+
|
|
876
|
+
* initial search may rely mostly on text
|
|
877
|
+
* later search may improve recall or ranking
|
|
878
|
+
* no regression in basic functionality occurs
|
|
879
|
+
|
|
880
|
+
---
|
|
881
|
+
|
|
882
|
+
## 13.4 Updating a document invalidates prior semantic state without breaking text retrieval
|
|
883
|
+
|
|
884
|
+
**ID:** BB-036
|
|
885
|
+
**Environment:** semantic optional mode
|
|
886
|
+
|
|
887
|
+
**Preconditions:**
|
|
888
|
+
|
|
889
|
+
* document has already been semantically indexed
|
|
890
|
+
|
|
891
|
+
**Action:**
|
|
892
|
+
Update the document content
|
|
893
|
+
|
|
894
|
+
**Expected result:**
|
|
895
|
+
|
|
896
|
+
* text retrieval reflects the new content immediately
|
|
897
|
+
* semantic state is refreshed eventually
|
|
898
|
+
* no stale semantic result should permanently dominate after update
|
|
899
|
+
|
|
900
|
+
---
|
|
901
|
+
|
|
902
|
+
## 13.5 Search blend respects configured full-text vs semantic weight when embeddings enabled
|
|
903
|
+
|
|
904
|
+
**ID:** BB-036b
|
|
905
|
+
**Environment:** semantic optional mode
|
|
906
|
+
|
|
907
|
+
**Preconditions:**
|
|
908
|
+
|
|
909
|
+
* embeddings enabled
|
|
910
|
+
* search config has configurable `ftsWeight` and `semanticWeight` (e.g. 0.7 and 0.3)
|
|
911
|
+
* corpus includes documents that match a query both by literal text and by semantic similarity (e.g. paraphrased)
|
|
912
|
+
|
|
913
|
+
**Action:**
|
|
914
|
+
Run `mem_search` with a query that has both lexical and semantic matches
|
|
915
|
+
|
|
916
|
+
**Expected result:**
|
|
917
|
+
|
|
918
|
+
* results reflect both full-text and semantic relevance when both are available
|
|
919
|
+
* changing the configured weights (e.g. toward more semantic or more full-text) can change the relative ranking of results for the same query
|
|
920
|
+
* when semantic indexing is disabled (or unavailable), effective behavior is 100% full-text (see BB-008, BB-031)
|
|
921
|
+
|
|
922
|
+
---
|
|
923
|
+
|
|
924
|
+
# 14. Semantic failure tolerance
|
|
925
|
+
|
|
926
|
+
## 14.1 System remains functional when semantic provider is unavailable
|
|
927
|
+
|
|
928
|
+
**ID:** BB-037
|
|
929
|
+
**Environment:** semantic unavailable mode
|
|
930
|
+
|
|
931
|
+
**Preconditions:**
|
|
932
|
+
|
|
933
|
+
* embeddings enabled in config
|
|
934
|
+
* provider fails or worker is absent
|
|
935
|
+
|
|
936
|
+
**Action:**
|
|
937
|
+
Perform save and search operations
|
|
938
|
+
|
|
939
|
+
**Expected result:**
|
|
940
|
+
|
|
941
|
+
* save still succeeds
|
|
942
|
+
* text search still works
|
|
943
|
+
* system does not become unusable because semantic indexing failed
|
|
944
|
+
|
|
945
|
+
---
|
|
946
|
+
|
|
947
|
+
## 14.2 Semantic job failures do not corrupt durable memory
|
|
948
|
+
|
|
949
|
+
**ID:** BB-038
|
|
950
|
+
**Environment:** semantic unavailable mode
|
|
951
|
+
|
|
952
|
+
**Preconditions:**
|
|
953
|
+
|
|
954
|
+
* provider fails repeatedly
|
|
955
|
+
|
|
956
|
+
**Action:**
|
|
957
|
+
Save multiple notes
|
|
958
|
+
|
|
959
|
+
**Expected result:**
|
|
960
|
+
|
|
961
|
+
* Markdown files remain correct
|
|
962
|
+
* text retrieval remains correct
|
|
963
|
+
* failures are isolated to semantic enrichment only
|
|
964
|
+
|
|
965
|
+
---
|
|
966
|
+
|
|
967
|
+
# 15. Error handling tests
|
|
968
|
+
|
|
969
|
+
## 15.1 Invalid save request is rejected cleanly
|
|
970
|
+
|
|
971
|
+
**ID:** BB-039
|
|
972
|
+
**Preconditions:**
|
|
973
|
+
|
|
974
|
+
* system running
|
|
975
|
+
|
|
976
|
+
**Action:**
|
|
977
|
+
Call `mem_save` with missing required content or invalid arguments
|
|
978
|
+
|
|
979
|
+
**Expected result:**
|
|
980
|
+
|
|
981
|
+
* tool returns a clear error
|
|
982
|
+
* no malformed durable note is created
|
|
983
|
+
|
|
984
|
+
---
|
|
985
|
+
|
|
986
|
+
## 15.2 Invalid update ID is rejected cleanly
|
|
987
|
+
|
|
988
|
+
**ID:** BB-040
|
|
989
|
+
**Preconditions:**
|
|
990
|
+
|
|
991
|
+
* non-existent document ID
|
|
992
|
+
|
|
993
|
+
**Action:**
|
|
994
|
+
Call `mem_update`
|
|
995
|
+
|
|
996
|
+
**Expected result:**
|
|
997
|
+
|
|
998
|
+
* clear failure response
|
|
999
|
+
* no unrelated file is modified
|
|
1000
|
+
|
|
1001
|
+
---
|
|
1002
|
+
|
|
1003
|
+
## 15.3 Invalid observation ID in retrieval returns controlled failure
|
|
1004
|
+
|
|
1005
|
+
**ID:** BB-041
|
|
1006
|
+
**Preconditions:**
|
|
1007
|
+
|
|
1008
|
+
* non-existent observation ID
|
|
1009
|
+
|
|
1010
|
+
**Action:**
|
|
1011
|
+
Call `mem_get_observation`
|
|
1012
|
+
|
|
1013
|
+
**Expected result:**
|
|
1014
|
+
|
|
1015
|
+
* controlled error or not-found result
|
|
1016
|
+
* no ambiguous success response
|
|
1017
|
+
|
|
1018
|
+
---
|
|
1019
|
+
|
|
1020
|
+
# 16. Acceptance matrix
|
|
1021
|
+
|
|
1022
|
+
The product is acceptable only if all of the following are true:
|
|
1023
|
+
|
|
1024
|
+
1. It behaves as an Engram-compatible MCP replacement at the tool level.
|
|
1025
|
+
2. Markdown is the durable source of truth.
|
|
1026
|
+
3. Text retrieval works independently of embeddings.
|
|
1027
|
+
4. Manual file edits are respected after reindex.
|
|
1028
|
+
5. Home knowledge can be reused across projects.
|
|
1029
|
+
6. Project-local knowledge remains distinct from home knowledge.
|
|
1030
|
+
7. Topic-based updates do not create uncontrolled duplication.
|
|
1031
|
+
8. Rebuild from Markdown files is possible after index loss.
|
|
1032
|
+
9. Embeddings, when enabled, run asynchronously in the background.
|
|
1033
|
+
10. Embedding failure does not break core memory operations.
|
|
1034
|
+
11. When embeddings are disabled, search uses 100% full-text; when enabled, search blend respects configured ftsWeight and semanticWeight.
|
|
1035
|
+
|
|
1036
|
+
---
|
|
1037
|
+
|
|
1038
|
+
# 17. Suggested execution order
|
|
1039
|
+
|
|
1040
|
+
Recommended order for implementation testing:
|
|
1041
|
+
|
|
1042
|
+
## Phase 1
|
|
1043
|
+
|
|
1044
|
+
* BB-001 through BB-013
|
|
1045
|
+
|
|
1046
|
+
## Phase 2
|
|
1047
|
+
|
|
1048
|
+
* BB-014 through BB-025
|
|
1049
|
+
|
|
1050
|
+
## Phase 3
|
|
1051
|
+
|
|
1052
|
+
* BB-026 through BB-032
|
|
1053
|
+
|
|
1054
|
+
## Phase 4
|
|
1055
|
+
|
|
1056
|
+
* BB-033 through BB-036, BB-036b (search blend), BB-037 through BB-041
|
|
1057
|
+
|
|
1058
|
+
## Phase 5 (benchmarks)
|
|
1059
|
+
|
|
1060
|
+
* BM-001 through BM-006 as needed; baseline establishment and regression checks
|
|
1061
|
+
|
|
1062
|
+
---
|
|
1063
|
+
|
|
1064
|
+
# 18. Executive summary
|
|
1065
|
+
|
|
1066
|
+
This TDD verifies Mixgram as a **black-box MCP memory system** with these guarantees:
|
|
1067
|
+
|
|
1068
|
+
* durable Markdown memory
|
|
1069
|
+
* rebuildable SQLite retrieval
|
|
1070
|
+
* project and home knowledge scopes
|
|
1071
|
+
* clean topic-based update behavior
|
|
1072
|
+
* full functionality with embeddings disabled
|
|
1073
|
+
* asynchronous semantic indexing when enabled
|
|
1074
|
+
* graceful fallback when semantic indexing fails
|
|
1075
|
+
* configurable full-text vs semantic search weight when embeddings are enabled (e.g. 100% FTS when disabled; 70% FTS / 30% semantic when enabled)
|
|
1076
|
+
* **performance benchmarks** (section 19) for tool latency and reindex duration, with baseline and regression targets
|
|
1077
|
+
|
|
1078
|
+
The key testing rule is simple:
|
|
1079
|
+
|
|
1080
|
+
> The system must always remain useful and correct in text-only mode. Semantic indexing may improve retrieval, but it must never become a dependency for core behavior.
|
|
1081
|
+
|
|
1082
|
+
---
|
|
1083
|
+
|
|
1084
|
+
# 19. Benchmarks
|
|
1085
|
+
|
|
1086
|
+
Benchmarks define **performance targets** observable from outside the system (MCP client timing, reindex duration). They are not unit-level; they measure end-to-end tool behavior and indexing under defined conditions.
|
|
1087
|
+
|
|
1088
|
+
## 19.1 Scope
|
|
1089
|
+
|
|
1090
|
+
* Tool response latency (e.g. `mem_save`, `mem_search`, `mem_get_observation`, `mem_context`)
|
|
1091
|
+
* Full reindex duration from Markdown corpus to searchable state
|
|
1092
|
+
* Optional: sustained throughput for repeated saves or searches
|
|
1093
|
+
|
|
1094
|
+
All timings are measured from the client’s perspective (e.g. time from tool call to tool response). Environment (text-only vs semantic, corpus size) must be stated for each benchmark.
|
|
1095
|
+
|
|
1096
|
+
## 19.2 Environment assumptions
|
|
1097
|
+
|
|
1098
|
+
* **Benchmark environment:** Same machine class for baseline comparisons; cold vs warm index may be specified.
|
|
1099
|
+
* **Corpus size:** Small (e.g. < 100 docs), medium (e.g. 500–2K), or large (e.g. 10K+) as needed per benchmark.
|
|
1100
|
+
* **Embeddings:** Benchmarks may be defined for text-only mode only, or for semantic-optional mode with explicit “before/after embedding” semantics.
|
|
1101
|
+
|
|
1102
|
+
## 19.3 Benchmark cases
|
|
1103
|
+
|
|
1104
|
+
### 19.3.1 `mem_save` latency (single document)
|
|
1105
|
+
|
|
1106
|
+
**ID:** BM-001
|
|
1107
|
+
**Environment:** Text-only mode; small or medium corpus.
|
|
1108
|
+
|
|
1109
|
+
**Preconditions:**
|
|
1110
|
+
|
|
1111
|
+
* System running; index already built for the given corpus size.
|
|
1112
|
+
|
|
1113
|
+
**Action:**
|
|
1114
|
+
|
|
1115
|
+
* Call `mem_save` once with a typical document (e.g. 1–3 KB body).
|
|
1116
|
+
* Measure time from request start to tool response (success).
|
|
1117
|
+
|
|
1118
|
+
**Target:**
|
|
1119
|
+
|
|
1120
|
+
* Response returns within a defined upper bound (e.g. p95 < 500 ms for small/medium corpus on reference hardware).
|
|
1121
|
+
* Exact threshold to be set per project/reference machine; regression is relative to baseline.
|
|
1122
|
+
|
|
1123
|
+
---
|
|
1124
|
+
|
|
1125
|
+
### 19.3.2 `mem_search` latency
|
|
1126
|
+
|
|
1127
|
+
**ID:** BM-002
|
|
1128
|
+
**Environment:** Text-only mode; corpus size stated (e.g. medium).
|
|
1129
|
+
|
|
1130
|
+
**Preconditions:**
|
|
1131
|
+
|
|
1132
|
+
* Index contains a known number of documents.
|
|
1133
|
+
* Query is a short phrase that matches at least one document.
|
|
1134
|
+
|
|
1135
|
+
**Action:**
|
|
1136
|
+
|
|
1137
|
+
* Call `mem_search` with that query.
|
|
1138
|
+
* Measure time from request start to tool response.
|
|
1139
|
+
|
|
1140
|
+
**Target:**
|
|
1141
|
+
|
|
1142
|
+
* p95 latency under a defined cap (e.g. < 200 ms for medium corpus on reference hardware).
|
|
1143
|
+
* Baseline to be established and used for regression checks.
|
|
1144
|
+
|
|
1145
|
+
---
|
|
1146
|
+
|
|
1147
|
+
### 19.3.3 `mem_context` latency
|
|
1148
|
+
|
|
1149
|
+
**ID:** BM-003
|
|
1150
|
+
**Environment:** Text-only mode; project with multiple relevant notes.
|
|
1151
|
+
|
|
1152
|
+
**Preconditions:**
|
|
1153
|
+
|
|
1154
|
+
* Project has a defined number of notes (e.g. 20–50).
|
|
1155
|
+
|
|
1156
|
+
**Action:**
|
|
1157
|
+
|
|
1158
|
+
* Call `mem_context` for that project.
|
|
1159
|
+
* Measure time from request start to tool response.
|
|
1160
|
+
|
|
1161
|
+
**Target:**
|
|
1162
|
+
|
|
1163
|
+
* p95 latency under a defined cap (e.g. < 300 ms on reference hardware).
|
|
1164
|
+
* Used for regression comparison.
|
|
1165
|
+
|
|
1166
|
+
---
|
|
1167
|
+
|
|
1168
|
+
### 19.3.4 Full reindex duration
|
|
1169
|
+
|
|
1170
|
+
**ID:** BM-004
|
|
1171
|
+
**Environment:** Text-only mode.
|
|
1172
|
+
|
|
1173
|
+
**Preconditions:**
|
|
1174
|
+
|
|
1175
|
+
* A set of valid Markdown memory files exists on disk (count and total size stated).
|
|
1176
|
+
* SQLite index is missing or cleared.
|
|
1177
|
+
|
|
1178
|
+
**Action:**
|
|
1179
|
+
|
|
1180
|
+
* Run the official full reindex command.
|
|
1181
|
+
* Measure time from start until reindex is complete and search is usable.
|
|
1182
|
+
|
|
1183
|
+
**Target:**
|
|
1184
|
+
|
|
1185
|
+
* Total time under a defined upper bound for the given corpus (e.g. < N seconds for 1K documents on reference hardware).
|
|
1186
|
+
* Serves as scalability and regression benchmark.
|
|
1187
|
+
|
|
1188
|
+
---
|
|
1189
|
+
|
|
1190
|
+
### 19.3.5 `mem_search` latency with embeddings enabled (optional)
|
|
1191
|
+
|
|
1192
|
+
**ID:** BM-005
|
|
1193
|
+
**Environment:** Semantic optional mode; embeddings available and indexed for the corpus.
|
|
1194
|
+
|
|
1195
|
+
**Preconditions:**
|
|
1196
|
+
|
|
1197
|
+
* Corpus is fully text- and semantic-indexed.
|
|
1198
|
+
* Query has both lexical and semantic matches.
|
|
1199
|
+
|
|
1200
|
+
**Action:**
|
|
1201
|
+
|
|
1202
|
+
* Call `mem_search` with blended full-text/semantic config.
|
|
1203
|
+
* Measure time from request start to tool response.
|
|
1204
|
+
|
|
1205
|
+
**Target:**
|
|
1206
|
+
|
|
1207
|
+
* p95 latency under a defined cap (e.g. < 400 ms for medium corpus), or no more than a defined multiple of text-only search (e.g. < 2× BM-002).
|
|
1208
|
+
* Ensures semantic path does not make search unusably slow.
|
|
1209
|
+
|
|
1210
|
+
---
|
|
1211
|
+
|
|
1212
|
+
### 19.3.6 Throughput: repeated `mem_save` (optional)
|
|
1213
|
+
|
|
1214
|
+
**ID:** BM-006
|
|
1215
|
+
**Environment:** Text-only mode; small or medium corpus.
|
|
1216
|
+
|
|
1217
|
+
**Preconditions:**
|
|
1218
|
+
|
|
1219
|
+
* System running; initial corpus size stated.
|
|
1220
|
+
|
|
1221
|
+
**Action:**
|
|
1222
|
+
|
|
1223
|
+
* Call `mem_save` repeatedly (e.g. 50–100 times) with distinct topic keys and typical content.
|
|
1224
|
+
* Measure total time and success count.
|
|
1225
|
+
|
|
1226
|
+
**Target:**
|
|
1227
|
+
|
|
1228
|
+
* No systematic degradation (e.g. p95 per call stays within 2× of BM-001).
|
|
1229
|
+
* No failures attributable to resource exhaustion.
|
|
1230
|
+
|
|
1231
|
+
---
|
|
1232
|
+
|
|
1233
|
+
## 19.4 Reporting and regression
|
|
1234
|
+
|
|
1235
|
+
* Benchmark results should be reported with: environment (text-only vs semantic), corpus size, and metric (e.g. p95 latency, total reindex time).
|
|
1236
|
+
* CI or release gates may compare current run to a stored baseline and fail on regression (e.g. p95 > 1.5× baseline).
|
|
1237
|
+
* Exact thresholds and reference hardware should be documented in the project (e.g. in a `docs/benchmarks.md` or CI config) and updated when the reference environment changes.
|