nanomem 0.6.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (62) hide show
  1. nanomem-0.6.3/CHANGELOG.md +1299 -0
  2. nanomem-0.6.3/COMMERCIAL-LICENSE.md +85 -0
  3. nanomem-0.6.3/COMPETITIVE_POSITION.md +1637 -0
  4. nanomem-0.6.3/LICENSE +661 -0
  5. nanomem-0.6.3/LICENSE.preview-v1.0.md +57 -0
  6. nanomem-0.6.3/MANIFEST.in +30 -0
  7. nanomem-0.6.3/MULTIHOP_REASONING_AND_TOPOLOGY_GUIDE.md +256 -0
  8. nanomem-0.6.3/PKG-INFO +286 -0
  9. nanomem-0.6.3/README.md +262 -0
  10. nanomem-0.6.3/SERVICES_AND_API_SPECIFICATION.md +428 -0
  11. nanomem-0.6.3/USER_MANUAL.md +361 -0
  12. nanomem-0.6.3/USER_MANUAL_DEVELOPER.md +833 -0
  13. nanomem-0.6.3/USER_MANUAL_PERSONAL.md +277 -0
  14. nanomem-0.6.3/nanomem/__init__.py +126 -0
  15. nanomem-0.6.3/nanomem/arena.py +2590 -0
  16. nanomem-0.6.3/nanomem/assets/write_classifier.npz +0 -0
  17. nanomem-0.6.3/nanomem/classifier.py +677 -0
  18. nanomem-0.6.3/nanomem/cli.py +483 -0
  19. nanomem-0.6.3/nanomem/container.py +1592 -0
  20. nanomem-0.6.3/nanomem/crypto.py +325 -0
  21. nanomem-0.6.3/nanomem/embed.py +169 -0
  22. nanomem-0.6.3/nanomem/engine.py +3397 -0
  23. nanomem-0.6.3/nanomem/entities.py +1516 -0
  24. nanomem-0.6.3/nanomem/errors.py +106 -0
  25. nanomem-0.6.3/nanomem/legacy_v2.py +283 -0
  26. nanomem-0.6.3/nanomem/mcp.py +281 -0
  27. nanomem-0.6.3/nanomem/proxy.py +390 -0
  28. nanomem-0.6.3/nanomem/routing.py +630 -0
  29. nanomem-0.6.3/nanomem/screen.py +405 -0
  30. nanomem-0.6.3/nanomem/users.py +124 -0
  31. nanomem-0.6.3/nanomem/vault.py +1767 -0
  32. nanomem-0.6.3/nanomem.egg-info/SOURCES.txt +59 -0
  33. nanomem-0.6.3/pyproject.toml +73 -0
  34. nanomem-0.6.3/setup.cfg +4 -0
  35. nanomem-0.6.3/tests/conftest.py +60 -0
  36. nanomem-0.6.3/tests/data/adjacent_attributes.npz +0 -0
  37. nanomem-0.6.3/tests/data/adjacent_attributes_source.py +121 -0
  38. nanomem-0.6.3/tests/data/fixture_vocab.txt +479 -0
  39. nanomem-0.6.3/tests/data/fixture_vocab_source.py +165 -0
  40. nanomem-0.6.3/tests/test_adjacent_attributes.py +182 -0
  41. nanomem-0.6.3/tests/test_arena_cache.py +946 -0
  42. nanomem-0.6.3/tests/test_arena_offsets.py +674 -0
  43. nanomem-0.6.3/tests/test_arena_residency.py +769 -0
  44. nanomem-0.6.3/tests/test_container.py +168 -0
  45. nanomem-0.6.3/tests/test_contract.py +408 -0
  46. nanomem-0.6.3/tests/test_crypto.py +287 -0
  47. nanomem-0.6.3/tests/test_embed_dim.py +114 -0
  48. nanomem-0.6.3/tests/test_entities.py +485 -0
  49. nanomem-0.6.3/tests/test_llm_routing.py +132 -0
  50. nanomem-0.6.3/tests/test_mcp.py +139 -0
  51. nanomem-0.6.3/tests/test_migration.py +182 -0
  52. nanomem-0.6.3/tests/test_ranking_exactness.py +243 -0
  53. nanomem-0.6.3/tests/test_reopen_and_concurrency.py +384 -0
  54. nanomem-0.6.3/tests/test_round3_regressions.py +428 -0
  55. nanomem-0.6.3/tests/test_round4_regressions.py +404 -0
  56. nanomem-0.6.3/tests/test_round5_temporal.py +351 -0
  57. nanomem-0.6.3/tests/test_routing.py +498 -0
  58. nanomem-0.6.3/tests/test_scale.py +268 -0
  59. nanomem-0.6.3/tests/test_screen.py +745 -0
  60. nanomem-0.6.3/tests/test_temporal_api.py +402 -0
  61. nanomem-0.6.3/tests/test_vault.py +303 -0
  62. nanomem-0.6.3/tests/test_write_gate.py +99 -0
@@ -0,0 +1,1299 @@
1
+ # Changelog
2
+
3
+ Paths of the form `scratch/refound/…` are relative to the repository root. Every
4
+ number below is from one of those files.
5
+
6
+ ---
7
+
8
+ ## 0.6.3 — engine 3.3.0 (unchanged)
9
+
10
+ **The macOS bundles carried 139 MB of a model nanomem never opened.**
11
+ `nanomem/assets/model.bin` is a genuine GGUF of `nomic-embed-text-v1.5` (its
12
+ header reads `general.architecture = nomic-bert`), downloaded in some earlier
13
+ session. The only code that ever touched it called `os.path.exists` and
14
+ `os.path.getsize` to set `has_fused_weights` and `fused_size_mb`, which nothing
15
+ read — while the class docstring advertised "bundled offline neural weights".
16
+
17
+ It could not have been used. GGUF is llama.cpp's format; reading it means a C++
18
+ dependency or writing BERT inference plus GGUF dequantisation in numpy, which
19
+ would be slower than the daemon it replaces and would cost the
20
+ only-dependency-is-numpy property that is the point of the project. It is also
21
+ redundant — `ollama pull nomic-embed-text` is the same model, and that is what
22
+ nanomem talks to.
23
+
24
+ Removed from both bundles, along with `FUSED_MODEL_PATH` and the two dead
25
+ attributes. One archived copy kept at `scratch/refound/assets/` with a note;
26
+ there were four (two bundles, two snapshots, 556 MB).
27
+
28
+ **The macOS bundle goes 152 MB → 10 MB.**
29
+
30
+ And the offline fallback is now described honestly. The README said "usable, but
31
+ not the real model", which understates it: it hashes words to sine frequencies
32
+ and bumps dimensions on character trigrams, so it scores shared WORDS AND
33
+ CHARACTERS rather than agreement.
34
+
35
+ | pair | cosine |
36
+ | --- | --- |
37
+ | "the server is up" / "the server is down" — opposites | 0.783 |
38
+ | "my dog is black" / "my car is black" — unrelated | 0.740 |
39
+ | "I drive a car" / "I own an automobile" — same meaning | 0.286 |
40
+
41
+ Opposites score near-identical and synonyms score unrelated. It is fuzzy string
42
+ matching that keeps the pipeline running offline, and anything measured with it
43
+ is measuring string overlap. Both the numbers and that sentence are in the
44
+ docstring and the README now, because "falls back to a deterministic hash
45
+ encoder" invites a reader to assume it degrades gracefully, and it does not.
46
+
47
+ No behaviour changes. 509 tests.
48
+
49
+ ---
50
+
51
+ ## 0.6.2 — engine 3.3.0 (unchanged)
52
+
53
+ **`embed_model=` was a parameter you could pass, not a model you could use.**
54
+ `EmbeddingProvider.dim` was the constant `768`, and `Vault` sizes its engine
55
+ from that number, so naming any other model built a 768-d vault that then
56
+ rejected every write:
57
+
58
+ ```
59
+ Vault(embed_model="all-minilm") # a real 384-d model
60
+ -> ValueError: embedding has 384 dims, vault has 768 on the first add()
61
+ ```
62
+
63
+ The container was never the limitation — it stores whatever width it is given,
64
+ and an existing file's header width already won over the requested one. Only
65
+ the constant was.
66
+
67
+ `dim` is now probed once from the model itself, lazily, with a single short
68
+ request, and cached from the first real batch if one happens first. Any width
69
+ works: `all-minilm` (384), `mxbai-embed-large` (1024), OpenAI
70
+ `text-embedding-3-small` (1536). Pass `EmbeddingProvider(dim=…)` to skip the
71
+ probe for an air-gapped install or a lookup-table provider. The offline hash
72
+ encoder keeps its own fixed 768 — that width is a property of *that encoder*,
73
+ not of the store.
74
+
75
+ Verified end to end against a real 384-d model: sized, written, searched at
76
+ cosine 1.0 on identical text, closed and reopened. 502 → 509 tests.
77
+
78
+ ---
79
+
80
+ ## 0.6.1 — engine 3.3.0 (unchanged)
81
+
82
+ **LLM provider routing was decided by substring matching, and it misrouted three
83
+ real cases.** The test was:
84
+
85
+ ```python
86
+ is_openai_compat = (base.endswith("/v1") or "/chat" in base
87
+ or "1234" in base or "8000" in base)
88
+ ```
89
+
90
+ Those port numbers were matched against the *whole URL*, not the parsed port:
91
+
92
+ * **Anthropic silently 404'd.** `https://api.anthropic.com/v1` ends `/v1`, so it
93
+ was POSTed to `/v1/chat/completions`. Claude's API is `/v1/messages`, with a
94
+ top-level `system`, a required `max_tokens`, and `x-api-key` rather than a
95
+ bearer token. Callers saw `[Model not found]` for a model that exists.
96
+ * a host named `web8000.internal` on port 11434 was treated as OpenAI-compatible
97
+ * **llama.cpp on :8080** was sent to Ollama's `/api/generate`, because "8080"
98
+ does not contain "8000"
99
+
100
+ `_llm_endpoint()` now resolves on the parsed host and port, most-explicit-first,
101
+ and returns the request *shape* as well as the URL. Anthropic is a first-class
102
+ flavour. An explicit `/chat/completions` path is used verbatim, which is how an
103
+ Azure deployment URL with its `api-version` query is supported.
104
+
105
+ Verified on the wire against a capturing server, not just by URL: the Anthropic
106
+ request carries `x-api-key`, `anthropic-version`, `max_tokens` and a top-level
107
+ `system`, and carries no bearer token; the OpenAI request carries
108
+ `Authorization: Bearer` and a system+user message pair. 16 provider URLs route
109
+ correctly, 443 -> 502 tests.
110
+
111
+ | routes to | providers |
112
+ | --- | --- |
113
+ | Ollama native | `localhost:11434` (default) |
114
+ | OpenAI-compatible | OpenAI, Groq, Together, Mistral, DeepSeek, OpenRouter, Fireworks, LM Studio, vLLM, llama.cpp |
115
+ | Anthropic native | `api.anthropic.com`, or any path ending `/messages` |
116
+
117
+ Also: `volatility()` and `staleness()` are now on `Vault`, not only on
118
+ `VaultEngine`. 0.6.0 shipped `history` and `changes` as `Vault` wrappers and left
119
+ these two reachable only through `Vault.engine`, so the README had to tell people
120
+ to reach past the public object for the headline capability.
121
+
122
+ ---
123
+
124
+ ## 0.6.0 — engine 3.3.0, container format 3 (unchanged), arena cache format 3 (unchanged)
125
+
126
+ **The revision layer stops being something only the ranker can reach.** Every
127
+ personal-memory query already assembled a *revision group* — the set of records
128
+ that are competing statements of one fact — used it to decide which member to
129
+ surface, and then discarded it. Three calls return it instead.
130
+
131
+ Nothing existing changes. `search()` with no `as_of` is bitwise identical to
132
+ 0.5.0 across 520 recorded query results, and the entire 0.5.0 suite passes
133
+ untouched — 443 tests before, 467 after, 0 failures
134
+ (`scratch/refound/temporal_g1_results.json`; the gate is stated in
135
+ `scratch/refound/design/temporal_api_spec.md`).
136
+
137
+ **`history(query)`** — every value a fact has held, oldest first, the current one
138
+ last, each with `timestamp`, `revision` and `superseded`. No boost is applied, so
139
+ `cosine` is the raw similarity. A fact that never changed has a one-element
140
+ history, which is an answer rather than an empty result.
141
+
142
+ It deliberately does **not** apply the ranker's relevance floor. That floor asks
143
+ "is this member a plausible answer to THIS QUESTION", which is the right question
144
+ for ranking and the wrong one for an audit surface: measured on four restatements
145
+ at cosine 0.80 / 0.77 / 0.74 / 0.71, `GROUP_COS_DELTA = 0.06` drops the fourth,
146
+ and the chain then reports the **third** as current with `superseded=False` while
147
+ a newer value exists (`scratch/refound/temporal_g1_results.json`). The cost of the opt-out is the opposite error — a record
148
+ carrying the tag without being a restatement can appear — which is the safer
149
+ direction, because the entry arrives with its own text and timestamp and nothing
150
+ is hidden.
151
+
152
+ **`search(..., as_of=<unix ts>)`** — the answer as the vault stood at that
153
+ moment. It forces an exhaustive scan and turns the PCA screen off: masking a
154
+ routed or screened shortlist would be wrong, because a newer record can crowd an
155
+ older one out of selection before the mask is applied, and that older record is
156
+ exactly what an as-of query is asking for. Proven against physically truncated
157
+ vaults — the admitted row set is exactly `{rows : ts <= t}` over 705 checks with
158
+ no tolerance, and 5,670 comparisons give **0 id differences, 0 order differences
159
+ and 0 timestamp leaks** (`scratch/refound/temporal_as_of_results.json`).
160
+
161
+ **`changes(since, until)`** — what was written in a window, with no query vector
162
+ and no embedding call, read off the resident timestamp column. The interval is
163
+ half-open on `since` so that `changes(t)` reports exactly what `search(as_of=t)`
164
+ could not see.
165
+
166
+ The CLI gains `nanomem history`, `nanomem changes --since …`, and `--as-of` on
167
+ `nanomem search`.
168
+
169
+ **`volatility()`** — how often each fact actually changes, measured from the log
170
+ alone: `n_revisions`, the intervals between them, `median_interval`, and how long
171
+ the current value has stood unconfirmed. No model, no query, no embedding call —
172
+ differenced timestamps off the resident `ts`/`group_id` columns. This is the
173
+ capability no vector store can represent, because none of them keeps a revision
174
+ history to difference. Records with no entity share the empty group key and are
175
+ excluded; pooling them would report one enormous fake fact.
176
+
177
+ **`staleness()`** returns the same rows plus a modelled `p_superseded` — **and
178
+ suppresses it by default**. That is the pre-registered consequence of a gate the
179
+ model did not clear, not caution. Held-out last-interval forecast:
180
+
181
+ | corpus | ECE (bar ≤ 0.15) | Brier | constant-rate baseline |
182
+ |---|---|---|---|
183
+ | uniform intervals | 0.0771 | 0.1916 | 0.1909 — **ties/wins** |
184
+ | exponential intervals | 0.0593 | 0.1704 | 0.1779 — model wins |
185
+
186
+ Calibration passes on both. The second clause fails: against a single
187
+ corpus-wide rate, the per-fact rate wins only on the corpus whose intervals were
188
+ generated to match its own memoryless assumption. Elsewhere it is a tie, and a
189
+ tie means the per-fact rate earned nothing. `assume_memoryless=True` turns it on
190
+ for a caller whose domain justifies it. `scratch/refound/staleness_calibration.json`
191
+
192
+ An earlier run of that gate used `obs.mean()` as the "constant baseline" —
193
+ estimated from the test outcomes, an in-sample oracle with access to the answers
194
+ (its ECE was exactly 0.0000, which is the tell). Both runs are in the results
195
+ file; the fair baseline is the one quoted.
196
+
197
+ **A personal-memory search is 47.4% faster, and every result is identical.**
198
+ Profiling at 70,000 rows found `_resolve_revisions` was **50.1% of the search**,
199
+ and ~90% of *that* was a single line in `_cosine_window`:
200
+
201
+ ```python
202
+ order = cand[np.argsort(-cos[cand], kind="stable")[:int(self.window_max)]]
203
+ ```
204
+
205
+ A full stable argsort over every candidate, to keep **four** of them. Replaced
206
+ with `_top_k_stable`: partition to locate the k-th largest value (O(N)), keep
207
+ everything at or above it — a superset of the top k including every boundary tie
208
+ — then stable-sort only that.
209
+
210
+ | at 70,000 rows | before | after | |
211
+ |---|---|---|---|
212
+ | `_resolve_revisions` p50 | 1.917 ms | **0.194 ms** | −89.9% |
213
+ | `search` p50 | 3.899 ms | **2.050 ms** | −47.4% |
214
+
215
+ Exact, not approximate: `_top_k_stable` is identical to the expression it
216
+ replaces element-for-element across **21,200 fuzz cases** — including
217
+ all-identical, two-value and three-value tie profiles and arrays of 5k–80k — plus
218
+ the 520-query recorded baseline, unchanged. `argpartition`'s own tie order is
219
+ unspecified and is never relied on; it is read for a *value*, never to choose
220
+ between equal ones. `scratch/refound/window_topk_results.json`
221
+
222
+ Two corrections to earlier notes in this project:
223
+
224
+ * This is **not** the "O(top_k) rewrite of `_resolve_revisions`" those notes
225
+ described. No group formation changes and no candidate pool is narrowed, so
226
+ that rewrite's accuracy risk does not apply.
227
+ * Those notes also claimed the fix would "re-enable the PCA screen for
228
+ personal-memory workloads". **That is wrong.** The screen is gated by
229
+ `_ranking_is_inert`, which asks whether the entity/temporal layer can fire —
230
+ a pruned row cannot be brought back by a boost — so on a personal corpus the
231
+ screen stays off however fast revision resolution becomes. Unrelated.
232
+
233
+ **The relevance floor was retuned, and the retune was refused** — but the sweep
234
+ produced a measured recommendation. `group_floor_sim` stays at 0.0. On records
235
+ whose entity the CALLER declares, 0.45 is worth **+19.0 points** of top-1 on
236
+ revision chains whose newest statement is phrased furthest from the question
237
+ (canonical +1.0, sibling probes +0.7). On the 3-persona chat set, where the
238
+ tagger infers the entity instead, the same 0.45 costs **−13.9 points**; the two
239
+ values that cost nothing there (0.65, 0.70) buy +2.0 and −1.0, i.e. nothing.
240
+
241
+ No single threshold serves both, because the floor is compensating for *tagger
242
+ precision* and a caller that declares its own schema has no imprecision to
243
+ compensate for. So it stays a knob, now documented in `_apply_group_floor` with
244
+ both numbers. This independently reproduces the engine's own recorded finding
245
+ that three earlier floor fixes each cost more than they gained — using an arm
246
+ its authors did not have. `scratch/refound/floor_retune_results.json`,
247
+ `floor_chatcheck_results.json`
248
+
249
+ An earlier note in `temporal_drift_results.json` dismissed this sweep as
250
+ pointless because a tagged group forms in 0 of 100 drifting chains. That was
251
+ true of *that* arm only: the floor defect reproduces with **perfect** tags, so
252
+ it is independent of tagger recall, and the schema-aware path is exactly where
253
+ it bites. The correction is recorded in both files.
254
+
255
+ **Two mechanisms were built, measured and refused.** Both were attempts to make
256
+ fact-grouping work without a vocabulary, which is what bounds every temporal
257
+ feature here:
258
+
259
+ * *Vocabulary-free grouping signals.* Five candidates scored on drifting
260
+ phrasing. Against **sibling** attributes — the discrimination that matters,
261
+ since home vs office address must never merge — direct cosine reaches AUC
262
+ 0.456, second-order profile 0.512, neighbour-set Jaccard 0.426 and value-shape
263
+ matching 0.517. All chance. Only the query-relative signal discriminates
264
+ (0.880), and that is what the relevance floor already computes. Siblings share
265
+ value shapes by construction, so shape can never separate them.
266
+ `scratch/refound/grouping_signal_results.json`
267
+ * *Write-time anaphora* (change marker + value shape + recency). Attached 41.7%
268
+ of narrative restatements at 78.7% precision, 8.9% wrong — inside the wrong-rate
269
+ bar, well under the 70% attach bar. 105 of 180 restatements carry no specific
270
+ value shape at all, and the confusions are exactly the siblings
271
+ (`primary_email→backup_email` 7, `home_address→office_address` 6): recency does
272
+ not break sibling ties. `scratch/refound/write_time_anchor_results.json`
273
+
274
+ The conclusion both share is worth stating plainly: **a revision cannot be told
275
+ from a sibling attribute without either a vocabulary or the query.** That is a
276
+ property of the problem. It is why the tagger is lexical, and it means every
277
+ temporal capability here is bounded by tagger recall — 70/100 chains on canonical
278
+ phrasing, 0/100 on drifting.
279
+
280
+ **Cost**: +0.21% on p50 at 20,000 rows over 5 alternating paired runs, against a
281
+ pre-registered bar of +1.0% (`scratch/refound/temporal_cost_results.json`). The
282
+ first unrepeated pair read +1.88% and was not acted on in either direction; the
283
+ stdev across pairs is 1.11%, which is what that single sample was measuring.
284
+
285
+ **One gate did not survive contact.** G2 was written as "`search(as_of=t)` equals
286
+ `search()` on a vault holding only the rows at or before t, 0 mismatches", and
287
+ read as bitwise float equality that is not satisfiable by any correct
288
+ implementation: the two vaults hold different row counts, so the score matmul has
289
+ a different shape and float32 accumulation is not associative (max observed delta
290
+ 1.49e-08, inside one ulp). That was discovered *after* measuring. The episode is
291
+ recorded in the results file rather than erased, and the replacement is strictly
292
+ stronger on the property in question — it proves the admitted row set exactly,
293
+ with no float comparison in it at all.
294
+
295
+ **Two defects found, neither introduced here, neither fixed here.**
296
+
297
+ * `search("where do I work")` on a real vault holding three `employer`
298
+ statements returns the **middle** one and ranks the current employer **third**.
299
+ The relevance floor drops the newest revision from the group before the
300
+ revision lead runs, because it is phrased further from the question (cosine
301
+ 0.5317 against the group best's 0.6348, gap 0.1031, floor 0.06). Reproduced
302
+ identically on 0.5.0. The knob that fixes it, `group_floor_sim`, ships off on
303
+ measured evidence that three such fixes each cost more than they gained, so
304
+ retuning it needs its own benchmark run — and the temporal benchmark reports
305
+ 100.0% on `current` questions while getting this one wrong, so the coverage
306
+ gap should be closed first. `scratch/refound/finding_floor_drops_current_value.json`
307
+ * The five distribution copies of the package are at **0.3.0 / engine 3.0.3**,
308
+ four releases behind, and live under untracked trees. They still answer "what
309
+ was my original address?" with the current value, peak at ~818 MB on a 71k
310
+ ingest, and gate writes at 0.60.
311
+ `scratch/refound/finding_stale_distribution_copies.json`
312
+
313
+ ---
314
+
315
+ ## 0.5.0 — engine 3.2.0, container format 3 (unchanged), arena cache format 3
316
+
317
+ **The disk axis 0.4.0 gave up is back, and the biggest loss in the product
318
+ pipeline was never in the engine at all.** A cached 71,433-document vault goes
319
+ **406.15 MiB → 153.05 MiB**, below sqlite-vec's 258.0 and 4.4 MiB above a vault
320
+ with no sidecar, with the O(1) reopen kept and every score bitwise identical.
321
+ Separately, the write gate's decision point moves **0.60 → 0.05**, worth
322
+ **+13.3 pt of end-to-end top-1 [CI +8.7, +18.3]** on a 300-question held-out
323
+ split — 15.2 points of the chat pipeline's loss were being destroyed at write
324
+ time, before retrieval ran at all.
325
+
326
+ Two mechanisms are integrated. **Four are not, including one that beat the exact
327
+ cosine ceiling** — they are listed at the bottom with the criterion each missed,
328
+ because a measured negative is the more useful half of this release.
329
+
330
+ `format_version` is still 3 and no vault file changes: a vault written by 0.3.x
331
+ or 0.4.0 opens unchanged, and the vault this release writes is byte-identical to
332
+ one written with every new flag off.
333
+
334
+ ### Changed — the `.arena` sidecar keeps OFFSETS, not a copy (`arena_cache_vectors`, `arena_cache_records`)
335
+
336
+ Measured by `scratch/refound/bench_sidecar_size.py` →
337
+ `sidecar_size_results.json`; independently re-measured on the final tree in
338
+ `scratch/refound/quality_summary.json`.
339
+
340
+ 0.4.0's sidecar *was* the resident fp32 arena, so it duplicated the vault: 209.28
341
+ MiB of vectors and 43.81 MiB of record text beside a 148.7 MiB vault. It no
342
+ longer copies either. The block table the sidecar already stored is enough to
343
+ derive, for every block, where its fp16 vector run and its record section live
344
+ **inside the vault**, so those two sections cost zero new bytes:
345
+
346
+ | at 71,433 rows | sidecar | vault + sidecar | reopen | p50 (screen `pca`) |
347
+ |---|---|---|---|---|
348
+ | 0.4.0 `cache` + records `cache` | 257.48 MiB | 406.15 MiB | 0.000187 s | baseline |
349
+ | **0.5.0 default** (`offsets_ram` / `vault`) | **4.38 MiB** | **153.05 MiB** | 0.000232 s | x0.9976 paired |
350
+ | `arena_cache="off"` | 0.00 | 148.66 MiB | 0.174820 s | x0.9913 |
351
+
352
+ **Nothing an answer depends on changed.** 36 of 36 arm × corpus × screen
353
+ comparisons share one SHA-256 over the concatenated fp32 score vectors of 500
354
+ queries against every row, and 0 of 500 top-10 lists differ, at 71,433 / 10,000
355
+ / 1,190 documents with the screen on and off.
356
+
357
+ **Two new flags**, both on `VaultEngine`:
358
+
359
+ * `arena_cache_vectors` — `"offsets_ram"` (default) maps the vault and upcasts
360
+ fp16 → fp32 **once, on the first vector read**; `"offsets"` gathers from the
361
+ mapping on every query; `"cache"` is 0.4.0's fp32 sidecar exactly.
362
+ * `arena_cache_records` — `"vault"` (default) reads each block's record section
363
+ from the vault; `"cache"` copies them into the sidecar as 0.4.0 did.
364
+
365
+ **It is a three-cornered trade, not a free win.** `"cache"` keeps the p50 and a
366
+ 43.5 MB `phys_footprint` and pays 406 MiB of disk. `"offsets"` has the best disk
367
+ *and* the lowest memory of any arm (227 MB peak, 57 MB phys) and pays **x1.73**
368
+ (screen on) / **x2.98** (screen off) on p50, because an exact cosine wants fp32
369
+ and the vault stores fp16, so a scan would convert 54.9M values per query.
370
+ `"offsets_ram"` keeps the disk and the p50 and pays the `phys_footprint`:
371
+ **270.5 MB against 43.5 MB**, because an upcast array is dirty anonymous memory
372
+ where a mapped sidecar is clean, evictable, file-backed pages. That cost was
373
+ **not** priced by the decision rule that chose the default; it is stated here and
374
+ `arena_cache_vectors="cache"` is the escape hatch.
375
+
376
+ **The cache format goes 2 → 3.** Every existing `.arena` is refused once and
377
+ rebuilt — one slow open, once, and the same again if the flags are changed.
378
+
379
+ ### Added — `VaultShrankError`, and a crash mode that the default has and `"cache"` does not
380
+
381
+ Reading through a mapping means a vault truncated **out of band, under a live
382
+ engine** is a `SIGBUS` — an uncatchable process kill, exit 138 — where 0.4.0's
383
+ copying layout could not notice at all. Every read through the mapping now
384
+ checks one `os.fstat` against the last byte the block table can address and
385
+ raises `nanomem.errors.VaultShrankError` instead. Cost: **0.486 µs per call**,
386
+ about 0.4% of a query that materialises ten records.
387
+
388
+ It is a guard, not a guarantee: a truncation landing between the check and the
389
+ page touch still faults. nanomem's own torn-tail recovery does **not** trip it
390
+ (it only removes bytes past the last valid block, which the check does not
391
+ address), and that case is a test rather than an assertion.
392
+
393
+ ### Changed — the write gate now keeps 57% more turns (`DEPLOYMENT_THRESHOLD_FULL` = 0.05)
394
+
395
+ Measured by `prime_4d_unified_engine_2026_09_13/write_policy.py` →
396
+ `scratch/refound/write_policy_results.json`, pre-registered and hash-verified
397
+ before any number, tuned on a 120-question dev split, scored **once** on a
398
+ disjoint 300-question test split. Reproduced end to end through the shipped
399
+ classifier in `scratch/refound/quality_summary.json`.
400
+
401
+ The trainer chose 0.60 by leave-one-persona-out **accuracy/F1**, which prices a
402
+ false positive and a false negative the same. Deployment does not: a refused
403
+ fact is never written and no retrieval quality can recover it, while a kept
404
+ noise turn costs ~1.9 kB and no measurable query time. At 0.60 the gate ran at
405
+ **100.0% precision / 64.6% recall** and made **135 of 420 questions (32.1%)
406
+ unanswerable before retrieval ran**.
407
+
408
+ | on the 300-question test split | shipped 0.60 | **0.05** | store everything |
409
+ |---|---|---|---|
410
+ | end-to-end top-1 | 35.7% | **49.0%** | 49.7% |
411
+ | delta vs 0.60 (paired bootstrap) | — | **+13.3 [+8.7, +18.3]** | +14.0 [+8.3, +19.7] |
412
+ | persona-clustered CI | — | **[+9.7, +16.7]** | [+9.7, +18.3] |
413
+ | gate recall / precision | 64.5 / 100.0 | **94.0 / 92.9** | 100.0 / 35.6 |
414
+ | documents per 10 personas | 271 | **425** | 1,180 |
415
+ | vault bytes | 0.492 MiB | **0.764 MiB** | 2.014 MiB |
416
+ | questions left unanswerable | 100 | **16** | 0 |
417
+
418
+ Storing everything buys 0.7 pt more for 2.8x the rows and 5.6x the added bytes,
419
+ so this is the efficient point rather than the extreme one: **0.0205 MiB per
420
+ point of accuracy against 0.1087**. The gate's own F1 is also better here
421
+ (0.935) than at 0.60 (0.784). Both curves are flat from 0.10 to 0.02 — **0.05 is
422
+ a region, not a tuned constant.**
423
+
424
+ Not changed, deliberately: the **surface** head still decides at 0.60 (the study
425
+ handed a real embedding to every turn, so it measured the full head and only the
426
+ full head), and the **rule layers** are untouched — with the learned head fully
427
+ off they still drop 52 of 1,180 turns, 14 of them answers, which is a second
428
+ loss worth ~3.4 pt that needs its own study.
429
+
430
+ `WriteClassifier(threshold=0.60)` restores 0.4.0's gate exactly, and the trained
431
+ value is still readable as `inspect()["model_info"]["threshold_trained_full"]`.
432
+
433
+ ### Measured and NOT shipped
434
+
435
+ Every one of these was pre-registered before it was measured, and each is
436
+ reported against the criterion it was registered against rather than a criterion
437
+ chosen afterwards.
438
+
439
+ 1. **BM25 + dense fusion — the first mechanism in this project to beat the exact
440
+ cosine ceiling, and it is not in the build.** A numpy+stdlib inverted index
441
+ (`hybrid_results.json`) lifts evidence recall@10 from **71.2% → 77.4%,
442
+ +6.2 pt [+4.0, +8.6]**, still positive after Bonferroni over 7 fusion
443
+ families, because BM25's top-200 holds 3.0 pt of gold the cosine never
444
+ returns at any depth. It failed the bar it registered: recall**@4**, where
445
+ the pre-committed arm scores +2.60 [-0.20, +5.20] and no arm survives
446
+ multiplicity correction. It also failed the registered latency cost gate
447
+ (**+1.0611 ms** against +1.0 ms) and the index is **50.39 MiB = 32.9%** of
448
+ the vault this release now writes, against a registered 25% limit. And the
449
+ pipeline this engine actually serves reads a **top-1 to top-3 window**
450
+ (`chat.py`, `server.py`), not a top-10, so the gain sits outside the window.
451
+ Not shipped, not behind a flag, and it should not be the default. What would
452
+ change that: a caller that consumes a top-10, a re-measured single-query
453
+ fusion path against the same +1.0 ms bar, and a cost bar re-registered
454
+ against 153.05 MiB.
455
+ 2. **Learned attribute selection** (`selection_results.json`). The ensemble
456
+ clears its bar — +11.0 pt [+6.7, +15.7] overall, +10.7 [+3.8, +18.5] on the
457
+ questions that never name their attribute — but leave-one-ATTRIBUTE-out
458
+ collapses the two components that carry that gain to 3.7% and 1.9% selection
459
+ accuracy, i.e. the gain is per-attribute supervision transferring through a
460
+ shared question-template table, not language understanding. The
461
+ attribute-agnostic arm that survives LOAO clears the overall bar (+5.3
462
+ [+1.7, +9.3]) and does **nothing** on the head-dropped questions the study
463
+ exists for (-1.6 [-6.2, +3.1]). It replaces `entities.query_intents`
464
+ wholesale and was never measured against the temporal set that hook was tuned
465
+ for. Not shipped; the measurement that would decide it is named in
466
+ COMPETITIVE_POSITION.md.
467
+ 3. **Write-confidence down-weighting at retrieval time** — dead. Its dev argmax
468
+ is weight 0, i.e. the mechanism switched off; every strictly positive weight
469
+ scores at or below store-everything on dev and on test.
470
+ 4. **A two-tier confidence shard** — passes its bar (+8.0 [+3.7, +12.3]) and is
471
+ Pareto-dominated on every axis: lower accuracy than either arm above, the
472
+ largest footprint and the highest p50. The mechanism is measured, and it is a
473
+ general warning: splitting one vault into two costs **-7.5 pt on dev even
474
+ when both shards are always searched and merged by raw score**, because
475
+ nanomem's entity boosts are per-vault statistics. A confidence shard is not a
476
+ free index split.
477
+
478
+ ### Tests
479
+
480
+ **443 passed** (408 at 0.4.0): 24 for the offset layout and its attacks, 6 for
481
+ the truncation guard, 5 pinning the write gate's decision point — nothing in the
482
+ suite touched the classifier before this release, so that constant was a single
483
+ unpinned float.
484
+
485
+ ### Known gaps in this release
486
+
487
+ * The p50 gate that chose the sidecar default was first reported against a
488
+ substituted criterion after failing as written. It was then re-measured once
489
+ on a quiet machine under a rule fixed in advance — including the branch that
490
+ would have reverted the default — and passes as written (baseline 1.2055 ms
491
+ against a 1.2643 ms bar; only `"offsets"` fails, at 2.0857 ms). Both readings
492
+ and the superseded verdict are in `sidecar_size_results.json`. The margin is
493
+ thin in both directions, ~4-5%, so that gate separates `"offsets"` from
494
+ everything else and resolves nothing finer.
495
+ * `phys_footprint` 43.5 → 270.5 MB was not in the decision rule that chose the
496
+ default and is not being added to it after the fact.
497
+ * The write-gate study is one 14-persona synthetic fixture. The classifier's own
498
+ held-out generalisation set is quarantined and was not opened, so what 0.05
499
+ does to its published 90.5% held-out accuracy is **unmeasured**; whoever
500
+ publishes should re-run `train_write_classifier.py --heldout` at the new
501
+ threshold.
502
+ * No cold-page-cache number exists anywhere: `purge` needs root. Every reopen,
503
+ p50 and footprint figure here is warm.
504
+ * Landmark tables (`m > 0`) are exercised by a unit test only; no benchmark
505
+ corpus produces one. `residency="int8"` and `"float16_mmap"` still get no
506
+ cache, and encrypted vaults still get no sidecar and no mapping.
507
+
508
+ ---
509
+
510
+ ## 0.4.0 — engine 3.1.0, container format 3 (unchanged)
511
+
512
+ **The one axis nanomem lost worst is now a win, and it was paid for on disk.**
513
+ Reopening a 71,433-document vault went **0.1634 s → 0.000193 s (846x)**, against
514
+ sqlite-vec's 0.0014 s — from **115x behind the winner to 7.3x ahead of it**. The
515
+ same vault now occupies **406.2 MiB instead of 148.7 MiB**, which loses an axis
516
+ nanomem used to lead. Both numbers are below; neither is a rounding error and
517
+ neither is optional reading.
518
+
519
+ Two mechanisms are integrated. Both are exactness-preserving, and that is
520
+ measured, not asserted. One is ON by default (`arena_cache`), one is OFF
521
+ (`screen`). `format_version` is still 3: a vault written by 0.3.x opens
522
+ unchanged, and a vault written by this release is **byte-identical** to one
523
+ written with both features off.
524
+
525
+ ### Added — `arena_cache="map" | "copy" | "verify" | "off"` (default `"map"`)
526
+
527
+ The resident fp32 arena is kept in a `<vault>.arena` sidecar laid out so every
528
+ array is a page-aligned section `mmap` can hand to numpy with no copy and no
529
+ parse. An open becomes a stat, a 4 KiB header read, a bind and an `mmap`
530
+ instead of a replay of every block.
531
+
532
+ Measured by `scratch/refound/bench_reopen.py` → `reopen_results.json`,
533
+ regenerated against this build at machine load 2.1. Reopen is the median of 25
534
+ opens in one warm interpreter — the protocol `competitors_standard_results.json`
535
+ used, and the agreement is checked rather than assumed: a cache-less open here
536
+ reads 0.1634 / 0.0226 / 0.0027 s against that file's 0.1609 / 0.0218 / 0.0027.
537
+
538
+ | rows | scan open | mapped open | speedup |
539
+ |---|---|---|---|
540
+ | 1,190 | 0.002657 s | 0.000186 s | 14.3x |
541
+ | 10,000 | 0.022559 s | 0.000206 s | 109.5x |
542
+ | 71,433 | 0.163351 s | **0.000193 s** | **846.4x** |
543
+
544
+ It also changes what a serving process's resident bytes *are*. At 71,433 rows
545
+ `phys_footprint` is **9 MB against 288 MB** and peak `ru_maxrss` **258.2 MB
546
+ against 287.5 MB**, because the vectors become clean file-backed pages the
547
+ kernel may evict rather than dirty anonymous ones. Steady-state latency does not
548
+ move: the paired duel, arms alternated cycle by cycle on the same vault, reads
549
+ mapped/scanned at **0.9983**.
550
+
551
+ **Nothing an answer depends on changed.** Same build, cache on against cache
552
+ off, 500 queries scored against every row, at all three corpus sizes: the
553
+ SHA-256 of the concatenated fp32 score vectors is **identical** and **0 of 500**
554
+ top-10 lists change, for `map`, `copy` and `verify` alike. All four modes
555
+ produce one digest (`post_fix_check`).
556
+
557
+ **What it costs, all four things.**
558
+
559
+ 1. **Disk, and this is a trade rather than an oversight.** 257.5 MiB beside a
560
+ 148.7 MiB vault at 71,433 rows = **406.2 MiB**, against sqlite-vec's 258.0
561
+ (1.57x) and nanomem's own previous 148.7, which was the smallest of every arm
562
+ measured. The sidecar is 1.73x the vault because the vault stores **fp16**
563
+ and the sidecar **is** the fp32 arena — storing fp16 there would halve it and
564
+ put the O(rows) upcast back into the open, which is the thing being deleted.
565
+ `residency="float16"` gets an fp16 sidecar for the same reason.
566
+ `arena_cache="off"` writes nothing.
567
+ 2. **The first query**, which now pays the page faults the open skipped:
568
+ 0.0053 s → 0.0107 s at 71,433. Time to first answer is therefore
569
+ **0.1687 s → 0.0109 s, 15.4x** — the honest secondary number, and it is
570
+ reported next to the 846x rather than instead of it.
571
+ 3. **A fresh process on a SMALL vault is slower**: 0.00764 s against 0.00321 s
572
+ at 1,190 rows. It wins from ~10,000 rows up (0.00475 s against 0.02299 s).
573
+ `ARENA_CACHE_MIN_ROWS` stays at 256 and was deliberately **not** re-tuned:
574
+ the crossover is bracketed only by those two measured sizes, and picking a
575
+ floor inside that bracket after seeing the result would be choosing a
576
+ criterion to pass it. Small vault, short-lived processes: pass
577
+ `arena_cache="off"`.
578
+ 4. **The open that writes it** costs 0.3051 s against a 0.1659 s plain open,
579
+ of which 0.1315 s is the write — 1.84x, paid once. The first *write* after a
580
+ cached open pays for the lazy tables the open skipped: 0.0474 s against
581
+ 0.0124 s.
582
+
583
+ **On integrity, which is where the first version of this work was wrong.**
584
+ Two claims made earlier in development are **withdrawn**, in the source, the
585
+ tests and the results file:
586
+
587
+ * *"`verify` is safe."* It is not. The cache's `content_sha256` is an **unkeyed**
588
+ digest stored **inside the header of the file it authenticates**, behind a
589
+ CRC32. A ~10-line forgery updates both, and `verify` then serves the planted
590
+ row at **cosine 1.0** (`cache_integrity.cache_tamper.forged_digest`, all three
591
+ corpora; `tests/test_arena_cache.py::test_verify_mode_is_hijacked_by_a_forged_content_digest`).
592
+ * *"The sidecar is a strictly weaker, unauthenticated path to the same answers."*
593
+ False in the only configuration a sidecar can exist in. A cache is refused
594
+ outright for an encrypted vault, so every vault that has one is **plaintext**,
595
+ and a plaintext block trailer is an unkeyed SHA-256 that `crypto.py`'s own
596
+ `THREAT_MODEL` says "can be recomputed by anyone". With **no sidecar on disk
597
+ and `arena_cache="off"`** — the full scan that re-reads every block — a forged
598
+ row is served at **cosine 0.999997**
599
+ (`tests/test_arena_cache.py::test_a_plaintext_vault_is_forgeable_with_no_cache_in_sight`).
600
+
601
+ So: the sidecar does not lower the vault's threat model, it inherits it, and in
602
+ plaintext that model is **corruption, not adversaries**. `verify` checks the
603
+ cache's digest and re-reads every vault block; it costs **0.1478 s against
604
+ 0.1656 s** for simply rescanning — *cheaper* than a scan, because it skips the
605
+ decode, the record parse and the per-row interning — so what it buys is the
606
+ mapped memory profile at a scan's price, with a scan's corruption checking. The
607
+ word "authentication" no longer appears anywhere in this feature except where a
608
+ passphrase exists, and a passphrase disqualifies a vault from having a cache at
609
+ all, which is exactly why no keyed fix is available here.
610
+
611
+ **One integrity case was closed rather than described.** A vault edited in place
612
+ at the same length with its trailer left stale was refused by `off` and `verify`
613
+ (`IntegrityError`) and served by `map` with no error, no warning and no
614
+ `integrity_errors` entry — a silent downgrade of the **default**. The cache
615
+ header now records the vault's `(size, mtime)` as of the scan it was built from,
616
+ captured the instant the scan returns rather than at write time, so a concurrent
617
+ in-place rewrite cannot bless a cache built from the old bytes. All three modes
618
+ now raise. Cost: **+4.08 us** at 71,433 rows and +6.42 us at 10,000, on a
619
+ ~0.000175 s open, measured paired against the same package with the check
620
+ deleted. It is one `os.stat` and it is a corruption check: **`os.utime` defeats
621
+ it**, which is itself a passing test, and so does an edit to the prefix of a
622
+ vault that is afterwards appended to.
623
+
624
+ `ARENA_CACHE_VERSION` 1 → 2 for the two new header fields. The header is a fixed
625
+ 4 KiB page, so the file is not one byte larger; v1 caches are dropped and
626
+ rebuilt silently.
627
+
628
+ **Added with it:** `VaultEngine.arena_cache_path()`,
629
+ `VaultEngine.arena_cache_info()` — including `vault_changed_since_cache` and
630
+ `vault_blocks_checked` (`"all"` / `"appended tail only"` / `"none"`, so a
631
+ default that stops re-reading blocks is *answerable* instead of silent) — and
632
+ `arena_cache_refresh_rows`.
633
+
634
+ ### Added — `screen="pca"` (default OFF), an exactness-preserving latency option
635
+
636
+ A **latency** flag with no recall knob and no accuracy trade-off: its recall
637
+ delta is identically 0 by construction. It bounds every document's cosine from
638
+ above in a 256-dimensional subspace, skips the rows whose bound provably puts
639
+ them below the `top_k`-th score already in hand, and scores the survivors with
640
+ the same fp32 kernel the full scan uses. `nanomem/screen.py` derives the bound;
641
+ it holds for *any* basis, which is why an append cannot invalidate it.
642
+
643
+ 71,433 HotpotQA paragraphs, 200 questions × 5 paired interleaved cycles, one
644
+ engine with the flag toggled per query (`scratch/refound/bench_screen.py` →
645
+ `pca_screen_results.json`, `phaseC_latency_full`, re-run against this build at
646
+ load 1.28):
647
+
648
+ | arm | p50 | p95 | mean |
649
+ |---|---|---|---|
650
+ | `screen="off"` | 1.9495 ms | 2.0898 ms | 1.9328 ms |
651
+ | `screen="pca"` | 1.1350 ms | 1.7318 ms | 1.1829 ms |
652
+ | **speedup** | **1.718x** | 1.207x | 1.634x |
653
+
654
+ Per-cycle p50 ratios 1.709–1.731; bootstrap CI [1.695, 1.736]. An earlier run on
655
+ a machine at load 4.5–5.4 read 1.705x [1.689, 1.725] — the same answer, which is
656
+ the point of interleaving. **Read that file's `phase*` blocks, not its
657
+ `verdict`**: `bench_screen.py` merges a new run over the old file with
658
+ `prior.update(res)`, so `verdict.C2_speed` still holds the previous run's
659
+ numbers.
660
+
661
+ **Exactness**, proven and then measured anyway: 0 bound violations in
662
+ **71,433,000** document checks, and over 3,000 searches at k = 1, 4 and 10, zero
663
+ with a different id list, zero with a non-bitwise-identical float32 score list,
664
+ zero real misses. A basis fitted from 6,000 rows with 24,000 rows appended
665
+ afterwards and never refitted: 0 and 0 again over 2,400,000 checks.
666
+
667
+ **Cost**: 1032 bytes per document resident (71.6 MiB at 71,433, +34.2% over the
668
+ 209.3 MiB arena) and a one-off 0.191 s build. The basis is **recomputed**, not
669
+ persisted — it is admissible however stale it is, so persisting it would buy
670
+ latency only.
671
+
672
+ **Stands down automatically**, changing no result and raising nothing, below
673
+ `screen_min_rows=20_000`, under `residency="int8"`, with a `metadata_filter`, on
674
+ an entity-tagged vault, for an explicit `temporal_direction="historical"`, when
675
+ survivors would exceed `screen_max_frac`, and if the basis cannot be built. The
676
+ floor comes from a measured sweep: forced on, the screen is 0.478x at 1,000
677
+ documents, 0.919x at 5,000, 0.977x at 10,000, 1.370x at 20,000, 1.468x at
678
+ 40,000. At the shipped default the p50 ratio at 1,000/5,000/10,000 is
679
+ 1.0005/1.0002/0.9949.
680
+
681
+ **What it does not claim**: any recall improvement. The pre-registered rule in
682
+ `exotic_routing_results.json` was a confidence interval on a recall delta, which
683
+ is unreachable for this mechanism **by construction** rather than merely unmet.
684
+ A new criterion — exactness, ≥1.25x p50 at 71,433, no regression below 10,000 —
685
+ was written before the mechanism went into the engine, and that substitution is
686
+ stated here rather than buried.
687
+
688
+ ### Changed — top-10 output can REORDER against 0.3.2, always between tied rows
689
+
690
+ `_select_top_k` now breaks ties on the **row id** instead of inheriting
691
+ `np.argpartition`'s unspecified order among equal elements. This was required to
692
+ make the screen's "identical top-k" clause true as written rather than restated
693
+ as "identical up to ties": `argpartition` resolves a tie differently depending
694
+ on how long the array it is handed is, and the screen hands it a shorter one.
695
+
696
+ Consequences, both real:
697
+
698
+ * **8 of 500** top-10 lists at 71,433 documents and **1 of 500** at 10,000 come
699
+ back in a different order than 0.3.2 gave. Every changed position was scored
700
+ and the maximum |score gap| is **exactly 0.0** — bit-identical duplicate
701
+ paragraphs, where both orders were always correct
702
+ (`reopen_results.json`, `tie_forensics`, which fails loudly if one is not).
703
+ The count is insertion-order dependent: an independent re-derivation with a
704
+ different insertion order measured 7.
705
+ * It costs **16 us per search on the default exact path** (0.8% of its p50),
706
+ paid whether or not `screen="pca"` is ever turned on. What it buys is a result
707
+ that is a function of the *set* of rows scored and nothing else, which is also
708
+ reproducibility across numpy versions.
709
+
710
+ `search_batch` is untouched and its tie order remains numpy-dependent.
711
+
712
+ ### Changed — `stats()["arena_bytes"]` reads 0 on a cached open
713
+
714
+ `arena_bytes` counts **anonymous** memory only, deliberately — counting clean
715
+ evictable pages as RAM is the over-reporting 0.3.2 removed. On a cached open the
716
+ vectors are a mapping, so it reads 0 and the 219,442,176 bytes appear under
717
+ `arena_cache_mapped_bytes` / `arena_mapped_bytes`, with `arena_used_bytes`
718
+ unchanged. `arena_from_cache` and `arena_vectors_mapped` say which shape you
719
+ have. Anything trending `arena_bytes` will see a step change.
720
+
721
+ ### Also in this entry
722
+
723
+ `tests/test_round5_temporal.py` no longer reads the chat fixtures. It globbed
724
+ `scratch/refound` for `*chat_benchmark*.json` and `*personas*.json` and read
725
+ every match, so **every `pytest` run opened the quarantined
726
+ `clean_chat_benchmark_persona4.json` and `clean_chat_benchmark_heldout.json`**.
727
+ `test_entities.py` had already been moved onto the committed digest table
728
+ `tests/data/fixture_vocab.txt`; this sibling was missed. It now uses the same
729
+ table and names no path under `scratch/`. A new suite-wide guard,
730
+ `tests/test_screen.py::test_no_test_module_names_a_quarantined_fixture`, parses
731
+ every test module so the next one cannot be missed the same way — the old audit
732
+ only inspected itself. Found by wrapping `builtins.open` for a whole suite run
733
+ and reading the paths back; an access timestamp would not have settled it, since
734
+ reads do not reliably bump `atime` on this filesystem. Verified again on this
735
+ build: a full instrumented suite run opens exactly 6 paths under
736
+ `scratch/refound`, none of them quarantined.
737
+
738
+ `tests/test_arena_residency.py::test_the_allocator_not_a_narrower_dtype_is_what_took_the_ram_off_the_default`
739
+ was amended, because integrating the arena cache made its assertion false
740
+ without making its claim false. It read `arena_bytes` on a reopened vault, which
741
+ is now 0 (above); it now reads whichever key carries the vectors and pins
742
+ **both** open paths. Its `legacy_303_fp32_doubling` proxy is also asserted only
743
+ when both arms took the same open path: `bench_memory.py` does not pin
744
+ `arena_cache=`, so whichever arm runs first for a given dtype scans and writes
745
+ the sidecar and the next arm with that dtype maps it, which makes its
746
+ `reopen_only` RSS column no longer a clean residency comparison. Repairing that
747
+ harness is not work this release did.
748
+
749
+ **Not shipped, and recorded as a negative:** four mechanisms — conversation
750
+ context, per-user attribute priors, multi-intent retrieval and session recency —
751
+ pre-registered against a +5.0 pt bar on the personal-memory chat task and scored
752
+ on a held-out persona split. None cleared it; the best reached +1.0 pt
753
+ [-0.3, +2.7] (not significant) and conversation context was significantly
754
+ **negative** at -3.3 pt [-6.0, -0.7]. See COMPETITIVE_POSITION.md, Axis 6, and
755
+ `scratch/refound/context_lever_results.json`.
756
+
757
+ ---
758
+
759
+ ## 0.3.2 — engine 3.0.5, container format 3 (unchanged)
760
+
761
+ One change, in one place: **the resident arena stopped paying for its own
762
+ growth.** No stored byte, no answer and no public default moves. `format_version`
763
+ is still 3 and files written by 0.3.0/0.3.1 open unchanged.
764
+
765
+ ### The measurement
766
+
767
+ `ru_maxrss` high-water mark over the whole loader shape — build 71,433
768
+ documents, close, reopen, answer 500 queries — measured by
769
+ `scratch/refound/bench_memory.py` (arm `fp32_reserved`, the shipped default),
770
+ today's harness on today's machine against a checkout of the committed 0.3.1
771
+ build whose `arena.py` is byte-identical to `git show 6aa6923:`:
772
+
773
+ | corpus | 0.3.1 (engine 3.0.4) | 0.3.2 (engine 3.0.5) | ratio |
774
+ |---|---|---|---|
775
+ | 71,433 documents | 819.1, 818.6 MB | 286.4, 286.9 MB | **0.35x** |
776
+ | 10,000 documents | 102.0, 103.0 MB | 41.2, 41.1 MB | **0.40x** |
777
+
778
+ Two runs per cell, both printed rather than averaged. The same file's single
779
+ driver pass reads 305.1 MB at 71,433, so the honest range for the new arm at
780
+ that size across today's three readings is **286.4–305.1 MB**; a separate
781
+ harness (`scratch/refound/ingest_ram_results.json`, four runs plus two
782
+ post-report checks) puts it at **287.9–310.0 MB** with a median of 298.4. Quote a
783
+ range, not a point.
784
+
785
+ **The server shape did not move, and was not supposed to.** Opening a vault
786
+ another process wrote and serving from it: 286.0 MB → 285.2 MB at 71,433
787
+ (`memory_results.json`, `reopen_only`). 0.3.1 already fixed that shape; this
788
+ release fixes the *loader* shape, which was the one still losing.
789
+
790
+ ### Nothing an answer depends on changed
791
+
792
+ * The SHA-256 of the concatenated fp32 score vectors of all 500 queries against
793
+ all rows is **identical** between the two builds at both corpus sizes
794
+ (`ingest_ram_results.json`, `exactness`). That check does not depend on the
795
+ machine.
796
+ * **0 of 500** top-10 lists changed, in all 20 (arm, corpus, phase) cells.
797
+ * Evidence recall@4 is **70.4%** at 10,000 and **60.6%** at 71,433 — the same
798
+ numbers, still equal to exhaustive numpy and FAISS `IndexFlatIP`.
799
+ * p50 **1.7495 ms** at 71,433 against a 0.9605 ms bare-numpy floor measured in
800
+ the same process (`memory_results.json`, `arms.n71433.fp32_reserved.timing`);
801
+ published 0.3.1 figure was 1.774 ms. A 7-cycle paired duel reads the change as
802
+ 0.9966x (`ingest_ram_results.json`, `latency_duel`) — no measurable cost.
803
+ * The temporal-supersession benchmark re-run against this build reproduces
804
+ **all 19 arms to the digit**: 90.5% top-1 at shipped defaults, 47.9% ranking-off
805
+ floor, 86.2% for the strongest competitor arm
806
+ (`scratch/refound/temporal_bench_results.json`).
807
+
808
+ ### How it works
809
+
810
+ `nanomem/arena.py` holds the fp32 vectors in a lazily committed anonymous
811
+ mapping (`mmap.mmap(-1, ...)`, `MAP_PRIVATE|MAP_ANON`) reserved larger than the
812
+ rows in it; growth is a **new view over the same pages** — no allocation, no
813
+ copy, nothing discarded. The array itself is still cut to the exact row count.
814
+ When a reservation is outgrown a larger one is taken, the live rows are copied
815
+ once and the old mapping is `munmap`ped — which matters because a freed numpy
816
+ buffer measurably is *not* returned to the OS on this platform, and that is why
817
+ doubling's peak tracked live-plus-everything-ever-discarded. Reaching 71,433
818
+ rows costs **2 copies**, of 2.3 MB and 37 MB.
819
+
820
+ Allocator in isolation, 71,433 x 768 fp32 grown 50 rows at a time, one
821
+ subprocess per policy (`memory_results.json`, `growth_policy`, and
822
+ `ingest_ram_results.json`):
823
+
824
+ | policy | final capacity | growths | bytes discarded | peak RSS delta |
825
+ |---|---|---|---|---|
826
+ | capacity doubling (0.3.1) | 131,072 rows | 12 | 383.8 MB | 592.2 MB |
827
+ | exact fit, reallocating | 71,433 rows | 1,429 | 149,458.9 MB | 11,729.9 MB |
828
+ | **reserved view (0.3.2)** | 71,433 rows | **2 copies** | **0 MB** | **208.9 MB** |
829
+ | explicit `reserve` up front | 71,433 rows | 1 | 0 MB | 209.3 MB |
830
+
831
+ Doubling is **kept** for the small per-row column arrays (2.7 MiB live at 71k).
832
+
833
+ **Fallbacks, all tested.** No anonymous mapping available → a plain zero-filled
834
+ array with the same view discipline. No headroom available (strict overcommit,
835
+ an rlimit) → retry at exactly the rows needed. An impossible hint (10^13 rows)
836
+ is dropped, not raised.
837
+
838
+ ### Added
839
+
840
+ * **`VaultEngine.reserve_additional_rows(n)`** — batch-relative pre-sizing, for
841
+ callers that know a batch length but not the eventual total. `reserve_rows(n)`
842
+ remains exact and total-relative.
843
+ * **`nanomem ingest --expect-docs N`**.
844
+ * **Automatic hinting on every bulk path.** `Vault.add_batch` hints its own
845
+ `len(records)`; `ingest_file` flows through it; `ingest_directory` hints once
846
+ up front from total bytes / `BYTES_PER_CHUNK_ESTIMATE` (2048, chosen from a
847
+ measured 1,640 / 1,755 / 2,339 / 9,703 bytes per chunk on four real trees and
848
+ leaning low on purpose); `merge`, `export` and `split` reach it through
849
+ `add_batch`; `_rebuild` → `replace_all` gets the container's exact count.
850
+ `Vault.add()`'s auto-split path (a handful of chunks per call) is **not**
851
+ hinted.
852
+ * **Three `stats()` keys**: `arena_reservation_bytes`,
853
+ `arena_reservation_is_mapped`, `arena_growth_copies`.
854
+
855
+ The hint is now worth **3%** (298.4 → 289.5 MB at 71,433), not the 2.85x it was
856
+ worth under doubling. It is wired anyway because it also guarantees zero growth
857
+ copies and a reservation no larger than the rows — on a reopen the arena comes
858
+ back with `arena_bytes == arena_used_bytes == arena_reservation_bytes` =
859
+ 219,442,176 and `arena_growth_copies` = 0.
860
+
861
+ ### Changed
862
+
863
+ * **`stats()["arena_bytes"]` no longer overstates.** Under doubling it reported
864
+ the *capacity*, up to **1.83x** the rows that existed. It now reports the rows.
865
+ Anything trending that key will see a step change that is a reporting fix.
866
+ * **`Arena.vec` is a read-only property.** Assigning to it was never supported
867
+ and now raises.
868
+ * `arena_reservation_bytes` is **address space, not RAM** — an untouched page of
869
+ an anonymous mapping is not resident. Unhinted, a 209 MB arena sits behind
870
+ 552 MB of reserved VM (721 MB via `add_batch`); hinted, exactly the rows.
871
+ Never add it to `arena_bytes`.
872
+
873
+ ### Considered and declined
874
+
875
+ **Fixed-size chunk list.** Same RAM, no reservation needed — and it turns one
876
+ BLAS call into `ceil(n/chunk)`. Paired in cycles at 71,433 rows: 0.9634 ms
877
+ contiguous against 1.0207 / 1.0847 / 1.1915 / 1.5541 ms at 32,768 / 16,384 /
878
+ 8,192 / 4,096-row chunks, i.e. **1.06x to 1.61x** on latency, the axis nanomem
879
+ is already second on. Scores bitwise identical at every chunk size, so it is a
880
+ pure latency-for-simplicity trade. Declined: the reserved view buys the same RAM
881
+ at 0.997x the p50. (`ingest_ram_results.json`, `chunked_alternative`.)
882
+
883
+ ### Tests
884
+
885
+ 313 → **322** (`python3 -m pytest -q`, 0 failed).
886
+ `test_growth_is_still_capacity_doubling` is deleted — it encoded the policy that
887
+ was replaced — and five tests encode the new one (no copies, exact fit, repeated
888
+ hints, refused reservation, impossible hint) and five cover the bulk wiring. The
889
+ directory-estimate test asserts a *bound* (0.2 ≤ hint/chunks ≤ 5.0), not a
890
+ fitted constant.
891
+
892
+ One further test was **rewritten, not added**:
893
+ `test_reserve_is_what_actually_took_the_ram_off_the_default` compared the
894
+ pre-sized arm against the harness's `legacy_303_fp32_doubling` arm and required
895
+ it to be under 55% of it. Re-running `bench_memory.py` against this build made
896
+ that assertion fail — correctly, because the legacy arm stopped being a baseline
897
+ (see "Known limits" below). It is now
898
+ `test_the_allocator_not_a_narrower_dtype_is_what_took_the_ram_off_the_default`
899
+ and asserts the claim directly: on a reopened 71,433-document vault
900
+ `arena_bytes == arena_used_bytes == arena_reservation_bytes` = rows x 768 x 4
901
+ with `arena_growth_copies` = 0, resident cost below 1.45x the vector bytes, the
902
+ arena still `float32`, and zero changed answers. Its last assertion pins the
903
+ dead proxy — `legacy ≈ presized` within 5% — so the arm cannot quietly be
904
+ re-read as a 3.0.4 baseline, and so that the test fails loudly if anyone repairs
905
+ it into a real one.
906
+
907
+ ### Known limits in 0.3.2
908
+
909
+ * **Peak RSS is better, not won.** 298.4 MB at 71,433 against FAISS's 230.2 MB
910
+ is still **1.296x** — see `COMPETITIVE_POSITION.md`, "Where nanomem loses".
911
+ The remaining ~290 MB is no longer the allocator: the loader now peaks at the
912
+ *server* floor, i.e. what a process that only reads the file pays.
913
+ * **The reserved arm is bimodal at 71,433 rows.** Readings cluster near 288 and
914
+ near 305–310 MB, a 7% spread. The committed-plus-hint arm shows the same jump,
915
+ so it is an ingest transient rather than the reservation. Not chased.
916
+ * **`scratch/refound/bench_memory.py`'s `legacy_303_fp32_doubling` arm no longer
917
+ reconstructs what it claims to.** It disables `Arena.reserve`, which was the
918
+ whole difference under 3.0.4, but the reserved view sizes exactly regardless;
919
+ its own `_next_capacity` assertion still passes because the *column* arrays
920
+ still double, so the harness does not self-detect this. Measured: that arm's
921
+ server-shape figure went 643.6 MB → 285.7 MB between the two builds, which is
922
+ the proof. **Do not read that arm as a 0.3.1 baseline.** This changelog's
923
+ before-column comes from a real checkout instead.
924
+
925
+ ---
926
+
927
+ ## 0.3.1 — engine 3.0.4, container format 3 (unchanged)
928
+
929
+ Recorded late; this entry was missing when 0.3.2 was written. Three measured
930
+ default-behaviour changes, each named in `nanomem/engine.py`'s `ENGINE_VERSION`
931
+ comment and each citing its own results file.
932
+
933
+ * **The temporal direction is read off the question's own wording.** A default
934
+ `search()` returns a different record for a historically-worded question than
935
+ 0.3.0 did, with no argument change. Temporal-supersession top-1 at shipped
936
+ defaults **44.2% → 90.5%**; historical questions 9.0% → 75.0%; previous-value
937
+ questions 1.8% → 98.2%; sibling attributes 57.5% → 90.0%
938
+ (`scratch/refound/temporal_bench_results.json`, before-column via
939
+ `final_scorecard.json`). Passing the two-way `temporal_direction` argument
940
+ explicitly now *costs* 13.2 points (90.5% → 77.3%) because it overrides that
941
+ reading — see `COMPETITIVE_POSITION.md`.
942
+ * **The arena is pre-sized on reopen** from the container's header row count.
943
+ Server shape at 71,433 documents: **643.6 MB → 286.0 MB**, 0 of 500 top-10
944
+ lists changed, no latency cost (`scratch/refound/memory_results.json`,
945
+ `reopen_only`). This did **not** fix the loader shape; 0.3.2 does.
946
+ * **The router gate was re-run and the default confirmed OFF**
947
+ (`scratch/refound/router_gate_results.json`).
948
+
949
+ ---
950
+
951
+ ## 0.3.0 — engine 3.0.3, container format 3
952
+
953
+ A rebuild of the storage, retrieval and ranking layers, and a rewrite of the
954
+ documentation to match what is measured. **This release is not source-compatible
955
+ with 0.1.x for anything that reads `score`.**
956
+
957
+ ### Breaking
958
+
959
+ * **`score` is a cosine.** In 0.1.x it was a squashed non-linear value. A hit now
960
+ carries `score` (cosine plus explicit, documented boosts, bounded by
961
+ `stats()['max_boost']` = 0.70) and `cosine` (the plain cosine). Every shipped
962
+ threshold was retuned: **0.25 → 0.42, 0.32 → 0.53, 0.35 → 0.58**
963
+ (`cli.py`, `chat.py`, `vault.py` forget default, `proxy.py`, `server.py`).
964
+ `nanomem.engine.legacy_score_to_cosine(old_threshold)` converts any other one.
965
+ `min_score` is applied to the pre-boost cosine, so a threshold means the same
966
+ thing whether or not the entity/temporal layer fires.
967
+ * **`Vault.add()` and `VaultEngine.add_fact()` return the document id (`str`).**
968
+ They returned `None` before. `POST /v1/memory/add` now returns the id the
969
+ record is really stored under; in 0.1.x the response invented an id nothing
970
+ could look up. The CLI's `add` prints the id and its own elapsed time.
971
+ * **Plaintext is the default.** A vault is a plain file unless you supply a
972
+ passphrase. `stats()['encrypted_at_rest']` reports which one you have. In
973
+ 0.1.x `stats()` hard-coded `encrypted_at_rest: true` regardless.
974
+ * **`stats()` values are measured.** `active_heap_ram_kb` was the constant
975
+ `160.0`; it is now a real sum of allocated buffers, with an
976
+ `active_heap_ram_method` string stating what it excludes and a
977
+ `process_rss_kb` beside it. `cipher` reports the real construction. New keys:
978
+ `engine_version`, `format_version`, `vector_dtype`, `router`, `n_exhaustive`,
979
+ `resident_arena_mb`, `max_boost`, `integrity_errors`, `truncated_tail_bytes`.
980
+ * **`search(multihop=True)` uses a text bridge.** `alpha`, `num_hops` and
981
+ `beam_width` are accepted for compatibility and no longer change the result;
982
+ alpha-steering was removed after measuring significantly worse than the query
983
+ alone.
984
+ * **`prune()` returns bytes freed**, not a record count. (`delete()` returns a
985
+ count.)
986
+ * **`password=""` is an error**, not a silently plaintext vault. A `bytes`
987
+ password is an error rather than being coerced with `str()`.
988
+
989
+ ### Migration from 0.1.x (`format_version` 2)
990
+
991
+ Open the file. Migration runs once and the original survives beside it as
992
+ `<path>.v2.bak`, byte-identical to the source, readable by the archived v2
993
+ reader. `stats()['format_version']` is `3` afterwards, and the second open is an
994
+ ordinary v3 open.
995
+
996
+ Verified on two golden fixtures in `scratch/refound/golden/`: a chat vault
997
+ answers **12 of 12** expected top-1 queries after migration, against 10 of 12 for
998
+ the v2 engine on the same fixture (`scratch/refound/ranking_dev_r4_shipped.json`,
999
+ `golden_chat_v2`); and a book vault migrates **845 of 845** documents with **0 of
1000
+ 20** top-4 differences from exhaustive fp32 cosine computed over
1001
+ `iter_records()` — that second arm was run by hand and is not in a results JSON.
1002
+
1003
+ Junk v2 entity tags are re-derived with the generic tagger; the original value
1004
+ is preserved in `metadata["entity_v2"]`. `nanomem user delete` ignores and
1005
+ removes `.v2.bak` and `.tmp-*` siblings. `Vault(..., migrate=False)` opens a v2
1006
+ file read-only through the legacy reader; supplying a passphrase to that path is
1007
+ now an error instead of being silently dropped.
1008
+
1009
+ ### Retrieval
1010
+
1011
+ Real HotpotQA paragraphs in random insertion order, 500 held-out questions,
1012
+ `top_k=4`, ingest → close → re-open → search. Before:
1013
+ `scratch/refound/scale_results_current_engine.json`. After:
1014
+ `scratch/refound/scale_results_v3r4.json`.
1015
+
1016
+ | Corpus | recall@4 | p50 | index |
1017
+ | :--- | :--- | :--- | :--- |
1018
+ | 10,000 | 24.0 % → **70.4 %** | 44.01 ms → **0.345 ms** | 71.8 MB → **22.0 MB** |
1019
+ | 71,433 | 5.8 % → **60.6 %** | 461.22 ms → **1.762 ms** | 511.5 MB → **155.9 MB** |
1020
+
1021
+ 70.4 % and 60.6 % are exactly what an exhaustive fp32 numpy scan and FAISS
1022
+ `IndexFlatIP` score on the same data. At 1,190 documents the engine matches
1023
+ exhaustive fp32 cosine in *ordering* as well: 0/120 top-4 order differences and
1024
+ 0/120 rank-1 differences with the real question strings
1025
+ (`scratch/refound/exactness_v3r2.json`, `headtohead_v3.json`). At 10,000 and
1026
+ 71,433 documents recall is still identical, but 2 of 500 and 5 of 500 questions
1027
+ differ in top-4 *order* — every case a tie or near-tie, largest cosine gap
1028
+ 2.2e-05, caused by the fp16 vectors on disk
1029
+ (`scratch/refound/verify_round3_v3r3.json`).
1030
+
1031
+ The 0.1.x collapse was the block-page router on randomly ordered blocks: in
1032
+ random insertion order it recalled 3.3–21.7 % against 68.3 % exhaustive
1033
+ (`scratch/refound/sweep_routing_1190.txt`). 3.0 scans exhaustively below
1034
+ `n_exhaustive` (50,000) instead.
1035
+
1036
+ ### Ranking
1037
+
1038
+ * Revision resolution: the current revision is ranked first in **14 of 16**
1039
+ generic probes against **3 of 16** for plain cosine, while **40 of 40**
1040
+ adjacent-but-different attributes are left exactly where plain cosine puts
1041
+ them. Historical lookups 16/16. `scratch/refound/ranking_dev_r4_shipped.json`.
1042
+ * Third-party statements are namespaced separately, so "his number is …" can no
1043
+ longer be stored as revision 2 of your own number and returned as the answer to
1044
+ your own question (`scratch/refound/third_party_v3r4.json`).
1045
+ * Third-person questions now resolve into that namespace; in 0.1.x and 3.0.2 they
1046
+ received a boost of exactly +0.000 and fell back to raw cosine.
1047
+ * Score contract: 0 violations over a 5,856-hit fuzz across 120 randomly shaped
1048
+ vaults; `cosine` matched the true stored-vector cosine on 5,856 of 5,856.
1049
+
1050
+ Chat benchmarks, gold-store top-1 (the expected memory ranked first, with the
1051
+ correct records already stored):
1052
+
1053
+ | Set | before | after | top-3 after |
1054
+ | :--- | ---: | ---: | ---: |
1055
+ | 3-persona selection (n=36) | 50.0 % | **91.7 %** | 100.0 % |
1056
+ | 2-persona held out (n=24) | 33.3 % | **75.0 %** | 95.8 % |
1057
+ | 3-persona dev set (n=24) | — | 95.8 % | 100.0 % |
1058
+
1059
+ `scratch/refound/clean_chat_results_current_engine.json`,
1060
+ `clean_chat_results_heldout_baseline.json`,
1061
+ `clean_chat_results_v3r4_engine*.json`.
1062
+
1063
+ **The release target was ≥ 80 % on both persona sets; the held-out set missed it
1064
+ at 75.0 % (18 of 24).** Five of the six failures are ordering errors inside a
1065
+ group that was retrieved (top-3 is 23 of 24); the sixth misses the top 3. The held-out set was also inspected during development this
1066
+ round, so 75.0 % is an upper bound rather than a clean out-of-sample estimate.
1067
+
1068
+ ### Write gate (classifier)
1069
+
1070
+ Replaced. The 0.1.x gate had four rule layers whose phrases were copied verbatim
1071
+ from a benchmark fixture, plus a prototype asset built from paraphrases of the
1072
+ same benchmark; both are gone, and the asset was deleted. The new gate is a numpy
1073
+ logistic head over the embedding plus generic surface features.
1074
+
1075
+ Out of sample, on two unseen personas, 231 turns
1076
+ (`scratch/refound/write_classifier_v2_results.json`):
1077
+
1078
+ | | accuracy | F1 |
1079
+ | :--- | ---: | ---: |
1080
+ | 0.1.x gate | 76.2 % | 69.6 |
1081
+ | **3.0 gate** | **90.5 %** | **87.2** |
1082
+ | 3.0 surface-only fallback (no embedder) | 87.0 % | 82.8 |
1083
+
1084
+ The decision itself costs 0.06 ms given an embedding; 13.2 ms p50 end to end
1085
+ through `Vault`, dominated by the embedding call.
1086
+
1087
+ ### Storage, durability, concurrency
1088
+
1089
+ The storage and concurrency figures in this subsection, and the tamper battery
1090
+ under *Password mode*, came from one-off scripts run during implementation and
1091
+ reproduced during verification. They are not written into a results JSON in
1092
+ `scratch/refound/`; the performance and recall tables elsewhere in this file all
1093
+ are.
1094
+
1095
+ * fp16 vectors on disk, fp32 arena in RAM. Worst per-row cosine error
1096
+ 0.99999988; no ordering changes.
1097
+ * 2,209 bytes per document at 1,190 docs — 0.609× the raw text plus fp32 vectors
1098
+ it replaces. Holds at scale: 2,307 B/doc at 10k, 2,289 at 71,433.
1099
+ * Rebuilds (`update`, `delete`, `prune`, `unmerge`, `export --purge`) go through
1100
+ one atomic `engine.replace_all()`: temp file plus `os.replace`.
1101
+ * `SIGKILL` mid-ingest at 809,000 records: reopens in 887 ms, all records
1102
+ contiguous and in insertion order, `integrity_errors` empty,
1103
+ `truncated_tail_bytes` 0, appendable afterwards.
1104
+ * A single-byte flip at four different structural offsets raises four different
1105
+ specific `NanomemError` subclasses; no silent wrong data.
1106
+ * 4 processes × 150 appends, 8 × 80, and 4 × 60 racing to *create* a vault that
1107
+ does not exist: every record present, unique ids, no torn tail.
1108
+ * `VaultEngine.__exit__` flushes; a `with` block no longer discards pending
1109
+ records.
1110
+ * Non-finite embeddings and timestamps are refused on the write path instead of
1111
+ being stored unreachable.
1112
+
1113
+ ### Password mode
1114
+
1115
+ Optional, off by default. scrypt (n=2¹⁶, r=8, p=1) → three HMAC-SHA256 sub-keys →
1116
+ SHAKE256 keystream XOR → HMAC-SHA256 encrypt-then-MAC bound to the vault uuid,
1117
+ verified with `compare_digest` before decryption. Header key-check rejects a wrong
1118
+ passphrase before any block is read.
1119
+
1120
+ Measured (`scratch/refound/crypto_overhead_v3r3.json`): 95.67 ms per scrypt
1121
+ derivation (≈ 10.5 offline guesses/s/core); open +99.67 ms at 1,190 docs,
1122
+ +104.88 at 5,000, +165.36 at 40,000; per-search −0.0003 / +0.002 / −0.0043 ms.
1123
+ Encrypted and plaintext files are byte-for-byte the same size.
1124
+
1125
+ 18 tamper mutations — ciphertext flips, nonce flips with repaired CRC, reserved
1126
+ header bytes, block swaps, replays, appended duplicates, deleted blocks, blocks
1127
+ spliced from another vault and from an earlier generation of the same vault, KDF
1128
+ downgrade, flag stripping — were all rejected, each with a specific
1129
+ `NanomemError`.
1130
+
1131
+ `nanomem.THREAT_MODEL` now states the three things it does not do: truncation and
1132
+ rollback are undetectable and `on_torn_tail="raise"` covers neither; the size
1133
+ leak is exact rather than approximate; and a block tag binds the vault uuid, not
1134
+ the file path.
1135
+
1136
+ ### HTTP services
1137
+
1138
+ * `POST /v1/vault/init` was unauthenticated, took an arbitrary filesystem path
1139
+ and opened it with a bare `Vault(name)` — a remote caller could downgrade a
1140
+ password-protected proxy to a plaintext vault anywhere on disk. Now: names are
1141
+ confined to the active vault's directory (absolute paths and `../` → `400`),
1142
+ the operator's passphrase is applied to every `Vault()` the handler opens, and
1143
+ changing the *active* vault requires `--allow-vault-switch` (otherwise `403`).
1144
+ `server.py`'s `/load` is confined the same way.
1145
+ * The proxy binds `127.0.0.1` by default and warns otherwise. Neither service
1146
+ authenticates; that is stated in the docs rather than implied away.
1147
+ * `/v1/memory/stats` answered `500` on a freshly created empty vault, because
1148
+ `Vault.__len__` made `if self.vault:` false. Fixed; a new vault can now learn.
1149
+ * `server.py`'s `/health` service string is "NanoMem Continuous Memory Engine".
1150
+ It was "NanoMem 4D Latent Continuous Memory Engine"; there is no 4D component
1151
+ in this package.
1152
+
1153
+ ### Multi-hop
1154
+
1155
+ * The shipped bridge is a text hop: re-embed the question with the hop-1 winner's
1156
+ text, search again, merge under the same `top_k`. 68.3 % → 79.2 % evidence
1157
+ recall@4 at `top_k=4` on 1,190 documents
1158
+ (`scratch/refound/multihop_texthop_v3r2_1190.json`). Single-pass `top_k=8` is
1159
+ 90.0 %, so widening `top_k` remains the larger lever.
1160
+ * Alpha-steering removed: 95 % CI [−0.050, −0.033] on MRR against the query
1161
+ alone, i.e. significantly worse.
1162
+ * No trained latent bridge shipped. Linear residual, residual MLP and an RK4
1163
+ neural ODE (3 seeds each) all had CIs at or below zero on the primary held-out
1164
+ set of 1,600 questions over 71,433 documents
1165
+ (`scratch/refound/experiment_4d_bridge_results.json`).
1166
+
1167
+ ### Routing
1168
+
1169
+ `router="auto"` (opt-in) uses global spherical k-means cells and triggers one
1170
+ `compact(recluster=True)`, recorded in the file header so later opens do not
1171
+ rewrite again (12 of 12 concurrent first opens succeed;
1172
+ `scratch/refound/router_persist_v3r4.json`). On the reclustered 71,433-document
1173
+ corpus it reaches 78.60 % recall@4 against exhaustive 78.65 % (−0.05 pt, CI
1174
+ [−0.20, +0.10]) scanning 29 % of the corpus — the recall gate passes. **It ships
1175
+ off**, because in the engine it is slower at equal recall: p50 3.604 ms versus
1176
+ 1.919 ms, plus 7.51 s on every open (`scratch/refound/router_gate_v3r3.json`).
1177
+
1178
+ ### Tests
1179
+
1180
+ A pytest suite ships at `nanomem_standalone/tests/`: 237 tests, `python3 -m
1181
+ pytest -q`, no network. 0.1.x had no tests at all.
1182
+
1183
+ ### Documentation claims withdrawn
1184
+
1185
+ The following appeared in 0.1.x documentation and are not supported by any
1186
+ results file in this repository. They have been removed, and where a real
1187
+ measurement exists it replaces them:
1188
+
1189
+ * "160 KB active heap", "< 500 KB RAM", "strictly 160 KB", "~160 KB per open
1190
+ vault" → measured 8–9 KB resident **per document** (`rss_v3r4.json`).
1191
+ * "256-bit encrypted", "AES-256", "Projected Stream Cipher (256-Bit)", "Zero
1192
+ plaintext leakage", "military/bank-grade" → plaintext by default; the optional
1193
+ mode is scrypt + SHAKE256 + HMAC-SHA256 and is explicitly not AES.
1194
+ * `"cipher": "256-bit Projected Stream Cipher"` as a documented `stats()` value →
1195
+ the code never produced it.
1196
+ * "Sub-millisecond retrieval", "0.44 ms", "34–100× faster", "retrieval scales
1197
+ sub-linearly", "searches 20+ projects in under 20 ms", "< 10 ms at 20,000
1198
+ chunks" → the measured p50 table, which is linear in corpus size.
1199
+ * "100 % recall", "100 % Hop-2 recall@1", "100 % rank-1 needle precision",
1200
+ "98 %+ accuracy", "95.8 % LLM factuality" → the measured recall tables.
1201
+ * The 321 ms / 648 ms / 978 ms multi-hop latency table and its 70.8 % / 70.8 % /
1202
+ 67.5 % recall row → `multihop_texthop_v3r2_1190.json` and
1203
+ `experiment_4d_bridge_results.json`.
1204
+ * The "single vault vs divided vs re-merged" table (586.5 / 1,221.5 / 580.9 ms,
1205
+ 160 KB, 95.8 % factuality, 100 % composite recall) → removed; its benchmark's
1206
+ fixtures had leaked into the engine's own rule layers.
1207
+ * "α = 0.35 balanced optimal default" for multi-hop steering → measured
1208
+ significantly worse than no steering; the mechanism is gone.
1209
+ * "prime kernel", "manifold resonance", "non-linear harmonic scores",
1210
+ "4D Latent Continuous Memory Engine" as a service name → removed; none names
1211
+ anything in this package.
1212
+ * "Run `python3 test_security.py`" → that file never existed. Run
1213
+ `python3 -m pytest -q`.
1214
+ * "MCP: reserved for Pro (v2)" → there is no `mcp.py` in this package; the
1215
+ capability is absent, not withheld.
1216
+ * "+16.7 % EM multi-hop improvement" → a measurement of a different, MLX-based
1217
+ research prototype on synthetic data, not of `nanomem.Vault`.
1218
+
1219
+ ### Packaging and distribution
1220
+
1221
+ The 0.1.0 wheel was built on 2026-09-15 from a *v2* source tree and then
1222
+ hand-copied to five places, while three older generations of it sat in the
1223
+ platform folders. Eleven `nanomem-0.1.0-py3-none-any.whl` files were on disk in
1224
+ three mutually different generations (md5 `a7a29809` ×5, `84bf1848` ×3,
1225
+ `513a1f1c` ×3) and none of them contained this engine. All eleven are deleted.
1226
+
1227
+ * **One wheel: `nanomem-0.3.0-py3-none-any.whl`**, 139,166 B, md5
1228
+ `900c16a08b86c1da588ef28fcf9ace4b`, 22 entries, built from `Launch 1/shared`.
1229
+ It is byte-identical in all twelve locations that previously held a 0.1.0
1230
+ wheel. Every `nanomem/*.py` inside it md5-matches the canonical
1231
+ `nanomem_standalone/nanomem/` source.
1232
+ * **There is no 0.2.0.** The version is not written in `pyproject.toml` or
1233
+ `setup.py` any more; both read `nanomem.__version__`, so the distribution
1234
+ version and the imported version cannot disagree again. That disagreement is
1235
+ what made a 0.1.0 install indistinguishable from this one.
1236
+ * **`assets/write_classifier.npz` is now declared package data.** A wheel built
1237
+ without `[tool.setuptools.package-data]` contains no `nanomem/assets/` entry
1238
+ at all — verified by building one — and the classifier then falls back to its
1239
+ surface-only constants without saying so. On the held-out 2-persona set that
1240
+ fallback costs 3.5 pts of accuracy and 4.5 pts of F1 (90.48 % / 87.21 % with
1241
+ the head, 87.01 % / 82.76 % without; `scratch/refound/write_classifier_v2_results.json`,
1242
+ keys `heldout.full` and `heldout.surface`).
1243
+ * **`assets/manifold_prototypes.npz` is gone** (271,607 B in every 0.1.x copy).
1244
+ It cached the 95 archetype embeddings of the retired manifold classifier,
1245
+ 30 of which were paraphrases of a single benchmark's turns. Nothing reads it.
1246
+ * **Both build paths agree.** `python3 -m pip wheel . --no-deps -w dist` and the
1247
+ `python3 setup.py bdist_wheel` that `Mac/build_mac.sh`, `Linux/build_linux.sh`
1248
+ and `Windows/build_windows.py` invoke produce the same 22-entry archive. The
1249
+ package is pure Python; all three platform folders hold the same
1250
+ `py3-none-any` wheel.
1251
+ * **Install check**: `pip install --force-reinstall` of the wheel into a clean
1252
+ CPython **3.9.6** venv (the floor of `requires-python = ">=3.8"` available
1253
+ here), then `import nanomem; from nanomem.vault import Vault` →
1254
+ `0.3.0 3.0.3`, `classifier.model_info['source'] == 'write_classifier.npz'`,
1255
+ and the `nanomem` console script resolves.
1256
+ * **MCP**: `nanomem/mcp.py` is still absent from the package and from the wheel.
1257
+ A 0.1.x-era copy of it survives in the two `nanomem_mac_bundle/` folders only,
1258
+ where it was preserved rather than shipped as a supported feature.
1259
+
1260
+ ### Shipped fixture vaults migrated
1261
+
1262
+ Opened once with this engine, in place, each leaving its original beside it as
1263
+ `<path>.v2.bak` (verified byte-identical to the pre-migration file):
1264
+
1265
+ | fixture | docs | v2 bytes | v3 bytes | migrate ms |
1266
+ |---|---:|---:|---:|---:|
1267
+ | `hands_on_llm_vault.dat` (×3 copies) | 845 | 6,352,820 | 2,159,838 | 205 / 84 / 83 |
1268
+ | `Launch 1/Mac/nanomem_mac_bundle/my_demo_vault.dat` | 4 | 35,956 | 7,276 | 1 |
1269
+ | `personal_memory.dat` | 0 | 64 | 256 | 1 |
1270
+
1271
+ The book vault is **66.0 % smaller** (fp16 vectors on disk plus block
1272
+ compression) with all 845 documents intact: reading every record back out of the
1273
+ three migrated copies gives one identical content digest. The three files are
1274
+ *not* byte-identical to each other any more — each migration stamps its own
1275
+ creation time and vault uuid into the header — which is a change from 0.1.x,
1276
+ where the three copies had one md5.
1277
+
1278
+ ### Known limits in 0.3.0
1279
+
1280
+ 1. Held-out chat ranking is 75.0 % gold-store top-1 against an 80 % target.
1281
+ 2. Search is linear; no sub-linear index ships. The opt-in router is slower
1282
+ in-engine at equal recall.
1283
+ 3. `delete` / `update` / `prune` are full rewrites (7.5 ms at 1k, 70.5 ms at 10k,
1284
+ ≈ 1 s at 71k) and block appenders. Tombstones are planned for 3.1.
1285
+ 4. Resident memory is 8–9 KB per document; there is no fixed ceiling.
1286
+ 5. Password mode does not detect truncation or rollback and leaks exact sizes.
1287
+ 6. The write gate is English-only.
1288
+ 7. `durable="full"` (F_FULLFSYNC) is implemented but unmeasured.
1289
+ 8. No MCP server.
1290
+ 9. A filter that matches nothing still costs a full scan.
1291
+ 10. `_ContainerView` (`.toc`, `.read_payload`, `.close`) is a deprecated compat
1292
+ shim for 3.0 and will be removed in 3.1.
1293
+
1294
+ ---
1295
+
1296
+ ## 0.1.0
1297
+
1298
+ Initial release. Superseded; see the withdrawn-claims list above before relying
1299
+ on anything written about it.