@hviana/sema 0.9.3 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. package/AGENTS.md +16 -7
  2. package/dist/src/mind/learning.js +11 -12
  3. package/dist/src/mind/mind.js +4 -4
  4. package/dist/src/mind/types.d.ts +10 -15
  5. package/dist/src/store.d.ts +10 -10
  6. package/dist/src/store.js +5 -5
  7. package/docs/INDEX.md +62 -59
  8. package/docs/INVARIANTS.md +36 -18
  9. package/docs/PHILOSOPHY.md +317 -0
  10. package/docs/architecture/bounded-reads.md +31 -71
  11. package/docs/architecture/caches.md +61 -81
  12. package/docs/architecture/closure.md +88 -107
  13. package/docs/architecture/commonality.md +47 -38
  14. package/docs/architecture/cost-model.md +57 -79
  15. package/docs/architecture/determinism.md +43 -55
  16. package/docs/architecture/evidence.md +158 -235
  17. package/docs/architecture/exact-vs-approximate.md +41 -35
  18. package/docs/architecture/factored-machinery.md +34 -20
  19. package/docs/architecture/fold-contract.md +110 -118
  20. package/docs/architecture/halo-sketch.md +105 -96
  21. package/docs/architecture/match-project.md +51 -42
  22. package/docs/architecture/mechanism-market.md +87 -91
  23. package/docs/architecture/memoization.md +60 -74
  24. package/docs/architecture/meter.md +37 -47
  25. package/docs/architecture/saturation.md +75 -101
  26. package/docs/architecture/store.md +118 -99
  27. package/docs/architecture/thresholds.md +66 -73
  28. package/docs/failures/tempting-but-wrong.md +139 -165
  29. package/docs/harness/gates.md +27 -32
  30. package/docs/mechanisms/alu.md +22 -69
  31. package/docs/mechanisms/cast.md +76 -71
  32. package/docs/mechanisms/confluence.md +22 -29
  33. package/docs/mechanisms/cover.md +58 -66
  34. package/docs/mechanisms/extraction.md +33 -37
  35. package/docs/mechanisms/prefix-completion.md +36 -39
  36. package/docs/mechanisms/recall.md +60 -53
  37. package/docs/mechanisms/reference.md +63 -49
  38. package/jsr.json +1 -1
  39. package/package.json +1 -1
  40. package/src/alu/README.md +90 -298
  41. package/src/derive/README.md +94 -256
  42. package/src/mind/learning.ts +11 -12
  43. package/src/mind/mind.ts +4 -4
  44. package/src/mind/types.ts +10 -15
  45. package/src/rabitq-ivf/README.md +11 -8
  46. package/src/store.ts +5 -5
@@ -0,0 +1,317 @@
1
+ # Philosophy — The Path of Information
2
+
3
+ `docs/INDEX.md` routes the laws. This file follows one piece of information from
4
+ the moment it enters Sema to the moment it is used, and says at each step what
5
+ happens, why it must happen that way, and what it makes possible. One principle
6
+ runs through the whole path:
7
+
8
+ > **Keep exactly what was given. Build ways of finding it. Decide only on what
9
+ > was kept. Charge for what was not found.**
10
+
11
+ Every structure below is either something kept (exact, it may decide) or a way
12
+ of finding (approximate, it may only propose). Holding that distinction is what
13
+ lets a reader predict how any part of Sema behaves.
14
+
15
+ ## 1. Information enters
16
+
17
+ **What enters.** A deposit is a pair, a context and what followed it, or a bare
18
+ experience. That is the only relation Sema assumes of the world: _this came
19
+ after that_. It imposes no schema, no entities and no types. Everything else
20
+ must be discovered from what follows what.
21
+
22
+ **As bytes.** There is no tokenizer, no vocabulary and no language. A modality
23
+ is only a reading order: text is read as written, and a grid along a Hilbert
24
+ curve, which keeps most of the plane's locality in the stream. Whatever can be
25
+ read as a stream can be learnt.
26
+
27
+ **Into a tree, cut by the content itself.** A rolling hash runs over the bytes.
28
+ Where it vanishes, a cut falls, and the more digits it vanishes on, the higher
29
+ the cut's level. Higher cuts nest inside lower ones, so the stream folds into a
30
+ tree of about `W` children per node (`W` = 4 by default), level by level
31
+ (`fold-contract.md`). There are three reasons for this shape:
32
+
33
+ - **Content-defined cuts make identity independent of position.** A phrase is
34
+ cut the same way wherever it occurs, so in two different deposits it is the
35
+ same subtree. After a shift, 99.7% of cuts survive, against 14.3% for a fixed
36
+ grid.
37
+ - **A tree, not a flat list.** Recurrence happens at every scale: a word, a
38
+ phrase, a sentence. The hierarchy makes every scale a node, so recurrence is
39
+ caught wherever it happens, and an edit touches only the path from the change
40
+ to the root.
41
+ - **Nothing is imposed that a later question could not reproduce.** No turn
42
+ boundaries, no metadata: a question will be cut by the same rule from its own
43
+ bytes, and any cut it could not reproduce would make the same thing two
44
+ things. When the two sides disagreed, alignment went quadratic (5.2M cells
45
+ against 0) and turns asked verbatim stopped resolving to themselves.
46
+
47
+ **Interned by content.** Every subtree becomes a node named by what it contains:
48
+ a branch by its children, a short run by its bytes. The same subtree in a
49
+ thousand deposits is one node with a thousand parents, so the store is a DAG,
50
+ not a forest. Every window of `W−1` and `W` bytes is also indexed as a node of
51
+ its own and linked to the chunks that contain it, so a span can be found
52
+ whatever the fold did around it. Teaching the same thing twice creates no
53
+ structure.
54
+
55
+ **Linked by succession.** The context's root gets an edge to the continuation's
56
+ root. Each suffix of the context that is already a known form inherits that edge
57
+ too, and inside a bare experience the parts are chained in order. An edge is a
58
+ unique pair, so how many contexts lead to a continuation counts distinct
59
+ contexts, not repetitions.
60
+
61
+ **What this structure can answer, exactly:**
62
+
63
+ - **Is this stored?** Fold it and look the name up. One byte apart is another
64
+ name.
65
+ - **What contains this, and what is it part of?** Climb the parents and
66
+ containers.
67
+ - **What followed this?** Read its edges.
68
+ - **How common is this?** Count the contexts that reach it: a minority
69
+ discriminates, and what reaches nearly everything is scaffolding
70
+ (`commonality.md`).
71
+ - **What of a question is already known?** Recognition decomposes the question
72
+ into every stored form inside it that leads somewhere, with a bounded number
73
+ of probes per byte.
74
+
75
+ **What it cannot answer:** what is _near_. A content address destroys locality
76
+ on purpose, because that is what makes it a name. `colour` and `colours` are as
77
+ far apart, by name, as `colour` and `Zanzibar`. Everything from here on exists
78
+ because of that gap.
79
+
80
+ ## 2. Two ways of being near
81
+
82
+ Every node also gets vectors in a high-dimensional space (a vector-symbolic
83
+ architecture: Plate 1995; Kanerva 2009). There are two of them, because there
84
+ are two independent ways for things to be near.
85
+
86
+ **The gist: nearness of form.** Each byte value has a vector from an alphabet
87
+ built by refinement (16 coarse directions, then 64, then 256), so neighbouring
88
+ values resemble each other. A node's gist binds each child to its position, by a
89
+ fixed permutation per seat, counted from both ends so that growth at one edge
90
+ keeps the other edge's coordinates, and then adds them up. The fold is linear,
91
+ so the cosine of two gists reads how many bytes they share in place. The gist
92
+ exists at birth: it is computed from the bytes alone. It puts `colour` near
93
+ `colours`.
94
+
95
+ **The halo: nearness of use.** Each time a fact is deposited, two pours happen:
96
+
97
+ - every newly seen part of the context receives the continuation's signature, in
98
+ the seat for _what it led to_;
99
+ - the continuation receives each part's signature, in the seat for _what led to
100
+ it_.
101
+
102
+ A signature is keyed on a node's identity, never on its gist. Read as whole
103
+ partners, company would almost never recur, since whole deposits rarely repeat.
104
+ So each partner contributes its own signature together with a sketch of its
105
+ constituents: `√D` of them, the capacity of a superposition before each term
106
+ falls below noise, with scaffolding excluded (`halo-sketch.md`). Two words that
107
+ never meet can therefore still share the _kinds_ of things around them. Each
108
+ episode adds one unit, by addition, so order is forgotten and proportion kept.
109
+ The halo is earned by experience. It puts `colour` near `hue`.
110
+
111
+ **Why two, and why kept apart.** Form and use are different axes of meaning, and
112
+ neither predicts the other. The halo is built from identities and never from
113
+ gists, so that resemblance of spelling cannot leak into resemblance of use. When
114
+ the two agree, it is corroboration, not echo.
115
+
116
+ **Why vectors at all, and why they never decide.** A vector restores the
117
+ locality the name destroyed. An index over the vectors (RaBitQ-IVF) finds the
118
+ near ones among millions in one query. But a vector is a lossy summary, and its
119
+ score is an estimate. So the space proposes and the structure decides
120
+ (`exact-vs-approximate.md`). The bars are set against the space's own chance:
121
+ random vectors are nearly orthogonal, with noise `1/√D`, so a resemblance counts
122
+ at `3/√D`, and two halos share a concept at `0.5 + 0.5/√D` (`thresholds.md`).
123
+ The single exception is the store's near-merge at deposit, bounded to one window
124
+ of difference.
125
+
126
+ **Two memories.** The deposits are a record: verbatim, enumerable, each with
127
+ provenance. The halos are a statistic accumulated over that record. This is the
128
+ episodic/semantic distinction (Tulving 1972) with fixed roles: **the statistic
129
+ proposes, the record decides.** A halo can say that two things are of a kind; it
130
+ never says they are the same thing, and it never establishes a fact.
131
+
132
+ **How the three readings are ordered.** Where Sema looks for a span, it tries
133
+ exact bytes first, then the halo's role, then the gist (`match.ts`). A form
134
+ counts as knowledge only if it _leads somewhere_: it has a continuation, or it
135
+ has a halo. Something that never led anywhere is not used.
136
+
137
+ **What the vectors add to reasoning.** They add three things:
138
+
139
+ - **Paraphrase:** a question one byte off its stored twin is found by its gist.
140
+ - **Substitution:** a form with no continuation of its own may borrow a halo
141
+ sibling's, at the price of a concept hop.
142
+ - **Imagination, bounded:** binding stored parts gives the gist of a whole never
143
+ seen, which can be searched for. That search is the most approximate tier of
144
+ all, and is gated hardest, because nothing about it is contained in bytes.
145
+
146
+ ## 3. A question arrives
147
+
148
+ The question is folded by the same rule as every deposit, and the identity fold
149
+ names each part by asking the store as it folds. So perceiving a question is
150
+ already recognising what of it memory holds; representation and search begin as
151
+ one act. Recognition returns the question's _sites_: every stored form inside it
152
+ that leads somewhere, each with its exact identity.
153
+
154
+ ## 4. Attention, built on the structure
155
+
156
+ The sites say what the question contains. They do not say what the question is
157
+ _about_, meaning which stored contexts its parts point to together. That is
158
+ attention's job, the consensus climb (`attention.ts`).
159
+
160
+ 1. **Regions.** Every node of the question's tree, and every recognised site, is
161
+ a region.
162
+ 2. **Lookup.** A region that is a stored form looks itself up exactly. Any other
163
+ region searches by gist, and pays a margin: it must beat the best rival
164
+ conclusion by the noise floor, scaled by how much of it is not stored.
165
+ 3. **Climb.** Each region climbs the DAG, through parents and containers, to the
166
+ stored contexts it reaches. The climb stops at a named saturation when a node
167
+ reaches more than `√N` contexts, because past that point reading further
168
+ cannot discriminate (`saturation.md`).
169
+ 4. **Weigh by rarity.** A region that reaches `c` of `N` contexts weighs
170
+ `ln(N/c)`. This is inverse document frequency, read over structure.
171
+ 5. **Agree.** Independent regions add, pooled in the (+, +) semiring of the same
172
+ deduction engine the search uses.
173
+ 6. **Join.** When two regions point to different contexts, the stored whole that
174
+ contains both is sought, by exact junction first, then through halo synonyms,
175
+ then by an imagined whole. Exact joint evidence explains the separate votes
176
+ away.
177
+
178
+ The output is a set of points of attention: stored contexts where independent
179
+ parts of the question agree, ranked.
180
+
181
+ Compared with a transformer's attention, which is soft content addressing over a
182
+ context window:
183
+
184
+ - **The keys are stored contexts,** reached by containment across the whole
185
+ memory, not tokens in a window. This is cross-attention, from the question
186
+ into memory.
187
+ - **The weights are rarity,** derived and not learned.
188
+ - **The votes are absolute.** A softmax must attend somewhere, because its
189
+ weights sum to one. Sema's anchors count only above floors derived from `D`
190
+ and `N`, so attention can come back empty.
191
+ - **Nothing is blended.** Mixing values would produce bytes nobody deposited.
192
+ Attention selects, and does not mix.
193
+ - **Exact evidence outranks resemblance.** Only exact evidence may explain a
194
+ vote away.
195
+
196
+ Attention is also spent, not assumed. The climb is the shared analysis the
197
+ market defers until no cheaper bound can rule it out (`mechanism-market.md`).
198
+
199
+ **Why this is the right notion of relevance.** Every judgement of relevance in
200
+ Sema divides a population into what it shares (frame) and what varies (filler).
201
+ Commonality and discrimination are the two sides of that one cut, and the answer
202
+ depends on the population. Sema reads three populations and never confuses them:
203
+ the corpus, the cohort of structures aligned to this question, and the
204
+ containers of a window (`commonality.md`). The rarity weighting is the graded
205
+ form of the cut over the corpus. Attention's deeper role is to _choose the
206
+ population_: its points are the cohort the next steps cut.
207
+
208
+ ## 5. How it is used
209
+
210
+ **Many ways of thinking.** Each grounding mechanism reads the same material
211
+ differently:
212
+
213
+ - `cover` composes the question from its sites and follows their edges, by graph
214
+ search.
215
+ - CAST aligns the attention points to the question and cuts that cohort into
216
+ frame and filler. It then carries structure between them: substitution,
217
+ redirection, comparison.
218
+ - `confluence` intersects what independent anchors share, as long as it is not
219
+ scaffolding.
220
+ - `extraction` reads out a located frame.
221
+ - `reference` learns a frame from worked examples and voices its slot with the
222
+ asker's own bytes.
223
+ - `recall` takes the nearest stored form by gist.
224
+ - `prefix-completion` completes a known beginning.
225
+ - The ALU computes, and what it computes is authoritative.
226
+
227
+ **One price.** Every candidate weighs `moves + PASS·unaccounted`, and `PASS` per
228
+ byte of unexplained question outweighs any move (`cost-model.md`). The price is
229
+ not confidence. It is how much of the question remains unaccounted for, so the
230
+ winner is the reading that explains the most. When nothing accounts for the
231
+ question, silence wins, and silence is a first-class answer. An answer that is
232
+ only near says so.
233
+
234
+ **Reasoning that answers to the question.** The winner may be extended, step by
235
+ step, along succession. A step is admitted only if it closes the derivation,
236
+ moves to structure not yet consumed, or carries what the question still owes
237
+ (`closure.md`). A step counts as _named_ when the question, together with the
238
+ node the step stands on, witnesses one of the contexts that establish it
239
+ (`evidence.md`). A step the question did not name is paid from its remaining
240
+ debt. The question owns the inference, so a chain cannot wander off to a fact
241
+ nobody asked for.
242
+
243
+ **Generalization, read and never stored.** When no context is named outright,
244
+ attention's points may hold another instance of the question.
245
+ `Where was Peter Jackson born?` shares the frame of
246
+ `Where was the director of film Beat Girl born?`. Peter Jackson's fact is also
247
+ established by `Peter Jackson place of birth`, which is how the corpus spells
248
+ the relation. The node at hand, `Edmond T. Gréville`, is put into that frame,
249
+ and the result is looked up by content.
250
+
251
+ This is anti-unification (Plotkin 1970): keep what two instances share, put a
252
+ variable where they differ. It is admitted only under three conditions:
253
+
254
+ - the variable is a thing the store knows;
255
+ - two instances agree;
256
+ - what the corpus files under one filler is not credited to the frame.
257
+
258
+ The third condition comes from a real failure. Two people born in Wellington
259
+ once gave an unknown `Zorblax` the same birthplace (`test/76`), Goodman's (1955)
260
+ accidental generalization. The frame is never stored, and neither is any answer.
261
+ A conclusion kept as a deposit would become evidence for itself.
262
+
263
+ **Every answer replays.** The derivation is a hyperpath in an AND/OR hypergraph
264
+ whose axioms are stored nodes (`src/derive/`). Its trace replays byte for byte,
265
+ and ties are broken by the order of teaching, never by chance
266
+ (`determinism.md`). A correction erases nothing. It is a further deposit, and it
267
+ prevails by evidence.
268
+
269
+ ## 6. The path, seen whole
270
+
271
+ | Step | What it adds | Kept or finding | It may |
272
+ | ----------------- | ---------------------------------------------- | --------------- | ---------------- |
273
+ | bytes, fold | one tree per stream, cut by content | kept | decide identity |
274
+ | DAG, edges | identity, parthood, succession, commonality | kept | decide |
275
+ | gist | nearness of form | finding | propose |
276
+ | halo | nearness of use | finding | propose |
277
+ | recognition | what of the question is known | kept | decide |
278
+ | attention | where the question's parts agree; a population | finding | propose |
279
+ | mechanisms | readings of the same material | both | offer candidates |
280
+ | price and closure | what remains unexplained; what may follow | kept | decide |
281
+
282
+ Representation, search and reasoning are separate modules, but not separate
283
+ ideas. One fold serves deposit and question. One identity serves every lookup.
284
+ One price serves every mechanism. One law admits every step. Everything that
285
+ finds is kept out of every decision, and everything that decides rests on what
286
+ was given.
287
+
288
+ ## 7. Hypotheses the path invites
289
+
290
+ Each of these follows from a gap visible on the path, and none is a plan.
291
+
292
+ - **One cut.** Frame against filler is computed in several places: the three
293
+ commonality measures, `frameSlots`, `coInstanceFiller` and CAST's `depth[]`.
294
+ Is it one operation read over different populations, or several, as
295
+ `commonality.md` holds?
296
+ - **Form, use and identity together.** Two forms with near halos that can stand
297
+ for each other in every stored frame — Leibniz's substitution _salva veritate_
298
+ — would be one referent. That is aliases (`Clara Novello` and
299
+ `Clara Anastasia Novello`) seen by the record, with the halo proposing.
300
+ - **Imagination as hypothesis.** Binding stored parts proposes wholes never
301
+ seen, and today it only joins regions. Proposals made this way, then checked
302
+ by content, are abduction with a guaranteed verifier.
303
+ - **Derivations as instances.** Anti-unify derivations, not texts, and `father`
304
+ twice reads as `grandfather` wherever an instance shows it. Derived material
305
+ must never count as a new context.
306
+ - **Richer company.** The halo has two seats, _what it led to_ and _what led to
307
+ it_. What other relations of use would a seat capture, and what would they let
308
+ attention see?
309
+ - **Other streams.** Whatever has a reading order that preserves locality can
310
+ enter by the same path.
311
+
312
+ ## References
313
+
314
+ Goodman (1955), _Fact, Fiction, and Forecast_ · Kanerva (2009), _Cognitive
315
+ Computation_ 1(2) · Plate (1995), _IEEE TNN_ 6(3) · Plotkin (1970), _Machine
316
+ Intelligence_ 5 · Tulving (1972), "Episodic and semantic memory", in
317
+ _Organization of Memory_.
@@ -1,85 +1,45 @@
1
1
  # Bounded Reads — No Per-Query Read Grows With the Corpus
2
2
 
3
- > **Law:** the cost of one query is proportional to the query, not to how much
4
- > was learned. No per-query read may grow with corpus size N.
3
+ > **Law:** a query's cost is proportional to the query, not to how much was
4
+ > learned. Every fan-out, walk and disambiguation reads at most the oldest
5
+ > `hubBound = ⌈√N⌉` entries, and the store enforces it.
5
6
 
6
- Every fan-out, walk, and disambiguation reads at most the OLDEST `hubBound` —
7
- `ceil(sqrt(N))`, floored at 2 for a near-empty store. A better-supported
8
- candidate beyond that prefix is invisible: a trade, and there is no second
9
- convention.
7
+ **Why.** Keeping everything means some nodes sit inside everything. A read that
8
+ follows them in full makes a bigger memory a slower thinker. The cap is a trade:
9
+ a better-supported candidate beyond the oldest `√N` is invisible. There is one
10
+ convention for this trade, and no second one. Deciding _when to stop_ inside the
11
+ cap is the walk's own saturation (`saturation.md`).
10
12
 
11
- ## Scale
13
+ ## Scale — defined once (`mind/traverse.ts`)
12
14
 
13
15
  ```
14
- corpusN(ctx) = max(2, store.edgeSourceCount()) // distinct learnt contexts
15
- hubBound(ctx) = ceil(sqrt(corpusN(ctx))) // >= 2, the store cap
16
- hubCap(ctx, ids) = ids.slice(0, hubBound(ctx)) // list-side reading
17
- boundFor(n) = ceil(sqrt(max(2, n))) // ctx-free reading
16
+ corpusN(ctx) = max(2, store.edgeSourceCount()) // distinct learnt contexts
17
+ hubBound(ctx) = ⌈√corpusN⌉ // the store cap
18
+ hubCap(ctx, ids) = ids.slice(0, hubBound(ctx)) // the list-side reading
19
+ boundFor(n) = ⌈√max(2, n)⌉ // the ctx-free reading
18
20
  ```
19
21
 
20
- Defined once in `mind/traverse.ts` (`corpusN`, `hubBound`, `hubCap`,
21
- `boundFor`). Every consumer imports them; never re-derive them inline.
22
+ Import these, and never re-derive them inline: no `edgeSourceCount()` or
23
+ `Math.sqrt` at a call site, and no private per-walk limit.
22
24
 
23
- ## Enforcement at the store level
25
+ ## Enforcement — the store, not the caller
24
26
 
25
- The cap is not advisory — adapters must make bounded reads bounded in SQL.
27
+ | Kind | Methods | Contract |
28
+ | ------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
29
+ | LIMITed reads | `nextFirst`, `prevFirst`, `parentsFirst`, `containersSlice` | A real `LIMIT ?`, with the same `ORDER BY` as the full read, never a materialise-then-slice. Reading `bound + 1` decides "hub or not" exactly. |
30
+ | existence probes | `hasNext`, `hasParents`, `hasContainers`, `hasHalo`, `prevCount` | One indexed `EXISTS`/`COUNT`, with no vector decode and no blob unpack. Use these, never `next(id).length > 0`. |
31
+ | prefix-capped reads | `bytesPrefix(id, cap)`, `contentLen(id, cap)` | They stop at the cap: `contentLen` is exact below it and reports `≥ cap` otherwise. An oversized candidate is rejected on the length probe alone. Uncapped reads in the weave, junction walks or bridge cost seconds per query on a large store. |
32
+ | transparent runs | `chainRun(id)` | One recursive CTE climbs a run of single-parent, edge-less nodes. It is cached for the store's lifetime and dropped on writes that break transparency. |
26
33
 
27
- ### 1. LIMITed reads — real `LIMIT ?`
28
-
29
- `nextFirst(id, limit)`, `prevFirst(id, limit)`, `parentsFirst(id, limit)`,
30
- `containersSlice(child, offset, limit)`.
31
-
32
- Same statement and `ORDER BY` as the full read, with `LIMIT ?`. Never
33
- "materialise then slice". Reading `hubBound + 1` parents decides "hub or not"
34
- exactly without reading the rest. Implemented as thin wrappers in
35
- `store-sqlite.ts` over `AbstractStore` in `store.ts`.
36
-
37
- ### 2. Existence probes — indexed point probes
38
-
39
- `hasNext(id)`, `hasParents(id)`, `hasContainers(child)`, `hasHalo(id)`,
40
- `prevCount(id)`.
41
-
42
- One indexed `EXISTS` / `COUNT` probe that never decodes vectors or unpacks
43
- blobs. Use them for every "does this lead anywhere?" question instead of
44
- `next(id).length > 0` or `prev(id).length`. `prevCount` is the reverse-edge
45
- support count for `chooseNext`/`chooseAmong`; `hasNext`/`hasHalo` gate the
46
- `leadsSomewhere` admission predicate in `mind/traverse.ts`.
47
-
48
- ### 3. Prefix-capped reads — reject without reconstruction
49
-
50
- `bytesPrefix(id, cap)` and `contentLen(id, cap)`.
51
-
52
- `contentLen` under a cap returns an exact length below the cap and `>= cap`
53
- otherwise — an indexed memo walk that stops early, never a full subtree walk.
54
- `bytesPrefix` stops after `cap` bytes. A candidate exceeding the cap is rejected
55
- on the length probe alone; the weave, junction walks, and bridge all read this
56
- way. Uncapped reads there cost seconds per query on a large store.
57
-
58
- ### 4. Transparent scaffolding — one bounded read
59
-
60
- `chainRun(id)` climbs a run of transparent nodes (exactly one parent, no edges)
61
- in a single recursive CTE, cached for the store lifetime and dropped on writes
62
- that break transparency. The climber hops the whole run where a node-at-a-time
63
- ascent would pay three probes per node.
64
-
65
- ## Maintenance only
66
-
67
- The full materialising reads — `next(id)`, `prev(id)`, `parents(id)`,
68
- `containers(child)` — exist for inspection, repair, and compaction only
69
- (`compactContentIndex`, `repairContentIndex`). Keep them off hot paths.
70
-
71
- ## Adding a walk
72
-
73
- Any new fan-out walk uses `hubBound`/`hubCap`. Do not call `edgeSourceCount()`
74
- or `Math.ceil(Math.sqrt(...))` inline, and do not invent a per-walk limit. The
75
- walk's saturation decision (when to stop) is separate from the cap (the safety
76
- net); a walk with only a cap drifts to the cap.
34
+ The full reads (`next`, `prev`, `parents`, `containers`) exist for inspection
35
+ and maintenance only (`compactContentIndex`, `repairContentIndex`).
77
36
 
78
37
  ## Pins
79
38
 
80
- - `test/14` — sublinear inference in corpus size and constant-rate in input
81
- length; training throughput floor; exact recall at scale.
82
- - `test/89` — completion recursion stays output-sensitive (nested searches/pops
83
- sublinear); guards the count of reads, not just per-read size.
84
- - `test/90` — connector probe (`offerConnectors`) reads by the query length
85
- (`QUERY.length + 1`), not by the learnt continuation; per-read size bound.
39
+ - `test/14` — inference is sublinear in corpus size and linear in input length;
40
+ recall stays exact at scale.
41
+ - `test/89` — completion recursion is output-sensitive: the _number_ of reads is
42
+ bounded, not just their size.
43
+ - `test/90` — the connector probe reads by the query's length, not by the learnt
44
+ continuation's.
45
+ - `test/119` — the derivation's work is flat in `N` for a byte-identical answer.
@@ -1,90 +1,70 @@
1
- # Caches — Every Acceleration Is a BoundedMap
1
+ # Caches — Every Acceleration Is a Budget
2
2
 
3
- > **Law:** every acceleration is a `BoundedMap` with a byte budget. A miss
4
- > re-derives from durable state. Degradation order is speed/reach lost, never
5
- > identity.
3
+ > **Law:** every acceleration is a byte-budgeted `BoundedMap`, and a miss
4
+ > re-derives from durable state. Eviction may cost speed or reach. It never
5
+ > changes what is stored, what is resolved or what tree is folded.
6
6
 
7
- No cache may change what is stored, what is resolved, or what tree is folded.
8
- Eviction costs a re-read, a re-walk, or a narrower reach — never a wrong answer
9
- or a wrong tree.
7
+ **Why.** Resident memory is capped by configuration, not by how much was learnt,
8
+ so a large store does not need a large machine. That holds only if every cache
9
+ can forget without being wrong.
10
10
 
11
- ## `BoundedMap` — the one cache primitive
11
+ ## `BoundedMap` — the one cache primitive (`src/store.ts`)
12
12
 
13
- `src/store.ts:BoundedMap<K,V>` — LRU with byte accounting (`maxBytes`, `sizeOf`,
14
- `evict`, `recency`).
13
+ An LRU with byte accounting (`maxBytes`, `sizeOf`), whose eviction is amortised
14
+ O(1) over a persistent cursor. It has two settings:
15
15
 
16
- - `evict: "lru"` — uniform-cost entries (dedup, vectors, records).
17
- - `evict: "smallest"` — variable-cost reconstruction (`_bytesCache`): protects
16
+ **`evict`, which entry goes:**
17
+
18
+ - `"lru"` — for entries of uniform cost.
19
+ - `"smallest"` — for variable-cost reconstructions (`_bytesCache`). It protects
18
20
  expensive large branches over cheap leaves.
19
- - `recency: "reorder"` (default) — exact LRU via `delete+set`; required when
20
- eviction choice is load-bearing (`_depositTrees` — 8 entries, victim changes
21
- fold).
22
- - `recency: "clock"` — bit instead of reorder; only for transparent caches where
23
- wrong victim costs a re-read (`_bytesCache`, `_recCache`). Measured: same
24
- entries cached, hot-path time 55% → bit.
25
-
26
- Persistent cursor over V8 insertion order makes eviction amortised O(1);
27
- candidate window for `"smallest"` never rescans from front.
28
-
29
- ## Store caches — budgets in `src/config.ts:StoreConfig`
30
-
31
- | Cache | Field | Budget | `sizeOf` | Eviction |
32
- | ------------------- | -------------------------- | ---------------------------- | ------------------- | -------------- |
33
- | dedup leaf/branch | `_leafKey` / `_branchKey` | `dedupCacheMax` 1M entries | 1 | lru |
34
- | flat-branch hits | `_flatKey` | `dedupCacheMax` 1M entries | 1 | lru+clock |
35
- | reconstructed bytes | `_bytesCache` | `bytesCacheMax` 20 MB | `byteLength` | smallest+clock |
36
- | content length | `_lenCache` | `bytesCacheMax` | 16 | lru |
37
- | node records | `_recCache` | `recCacheBytes` 10 MB | leaf+4·kids+12 | lru+clock |
38
- | pending gists | `_pendingGist` | `pendingGistBytes` 16 MB | `byteLength` (D·4) | lru |
39
- | halo exact / norm | `_haloExact` / `_haloNorm` | `haloCacheBytes` 16 MB each | `byteLength` | lru |
40
- | skipped interiors | `_coveredIds` | `coveredIdsMax` 100K entries | 1 | lru |
41
- | indexed ids | `_indexedIds` | `coveredIdsMax` | 1 | lru |
42
- | transparent chains | `_chainMemo` | `chainCacheBytes` 16 MB | 4·len+32 | lru |
43
- | ingest memo | `CachedIngest._memo` | `ingestCacheBytes` 50 MB | vector+ids+keyBytes | lru |
44
-
45
- ANN read caches (`_resonateCache`, `_resonateHaloCache`) are `Map<string,Hit[]>`
46
- keyed by `vecKey(v)+":"+k`, dropped on any index mutation.
47
- `vectorCacheMb`/`sqliteCacheMb` are pure page-cache latency knobs.
48
-
49
- `_bytesCache` only caches complete reconstructions — `bytesPrefix(id,cap)` with
50
- `got < cap`; a truncated prefix is never stored. `_chainMemo` is dropped on any
51
- write that could break transparency; `_pendingGist` eviction falls back to DAG
52
- climb; halo eviction re-decodes the durable 2-bit row.
53
-
54
- ## Mind caches — session and per-response
55
-
56
- | Cache | Location | Budget | Scope / invalidation |
57
- | ------------------- | ------------------------------------------------ | --------- | ------------------------------------------------------------------ |
58
- | `_gistCache` | `Mind._gistCache` | 32 MB | session-lifetime, never invalidated (perception pure) |
59
- | `_depositTrees` | `Mind._depositTrees` | 8 entries | session; `perceiveDeposit` only when `conversational` |
60
- | `_depositLens` | `Mind._depositLens` | — | byte lengths for prefix probes; cleared with map when >64 |
61
- | `_internIds` | `Mind._internIds: WeakMap<Sema,number>` | — | Mind lifetime; ids permanent |
62
- | `_resolvedSubtrees` | `Mind._resolvedSubtrees: WeakMap<Sema,{id,len}>` | — | per-response/conversation; fast path only when `visit===undefined` |
63
-
64
- `REACH_MEMO_MAX` / `STRUCT_MEMO_MAX` 100K (`src/mind/traverse.ts`) — whole-climb
65
- and per-node structural probes (`hasNext`/`prevCount`/`hasParents`); cleared on
66
- write or when cap reached. `reachMemo`/`structCaches` keyed by `_structMemoKey`,
67
- bypassed under trace.
68
-
69
- ## Deposit caches — offset-keyed, caller-discharged
70
-
71
- `_depositTrees`/`_depositLens`/`_internIds`/`_resolvedSubtrees` key **offsets**,
72
- not bytes, for O(1) reuse. Offsets alone cannot witness byte agreement — caller
73
- must discharge it.
74
-
75
- - Correct: conversation append — each turn extends the prior cumulative context
76
- by its own bytes; longest cached proper prefix hit (`L < bytes.length`) reuses
77
- `contentFoldIncremental` segments bit-identically.
78
- - Wrong: mismatched `prev` reused by offset produced wrong tree (336 vs 400
79
- bytes) — a coincidental prefix length aliased unrelated content.
80
- - Now: `perceiveDeposit` keys by `latin1(bytes.subarray(0,L))` (prefix bytes),
81
- probes longest cached proper prefix first; `_depositTrees` populated only for
82
- conversational deposits (budget discipline), otherwise cold path always
83
- correct.
21
+
22
+ **`recency`, how use is recorded:**
23
+
24
+ - `"reorder"`, the default — exact LRU. It is required wherever the choice of
25
+ victim is load-bearing.
26
+ - `"clock"` — a use bit. It is only for transparent caches, where a wrong victim
27
+ costs a re-read: `_bytesCache` and `_recCache`. It caches the same entries as
28
+ `"reorder"`, at a fraction of the hot-path time.
29
+
30
+ ## Store caches — budgets in `StoreConfig` (`src/config.ts`)
31
+
32
+ | Cache | Field | Default budget | Eviction | A miss costs |
33
+ | ------------------------------ | ----------------------------- | ---------------------------- | ---------------- | -------------------------------------------------- |
34
+ | dedup keys | `_leafKey` / `_branchKey` | `dedupCacheMax`, 1M entries | lru | a durable content probe |
35
+ | flat-branch hits | `_flatKey` | `dedupCacheMax` | lru + clock | a hashed probe |
36
+ | reconstructed bytes | `_bytesCache` | `bytesCacheMax`, 20 MB | smallest + clock | a subtree walk |
37
+ | content length | `_lenCache` | `bytesCacheMax` | lru | a capped walk |
38
+ | node records | `_recCache` | `recCacheBytes`, 10 MB | lru + clock | a row read |
39
+ | pending gists | `_pendingGist` | `pendingGistBytes`, 16 MB | lru | a DAG climb |
40
+ | exact halos / norms | `_haloExact` / `_haloNorm` | `haloCacheBytes`, 16 MB each | lru | decoding the 2-bit row |
41
+ | skipped interiors, indexed ids | `_coveredIds` / `_indexedIds` | `coveredIdsMax`, 100K | lru | a re-check |
42
+ | transparent chains | `_chainMemo` | `chainCacheBytes`, 16 MB | lru | one CTE; dropped on writes that break transparency |
43
+ | ingest memo | `CachedIngest._memo` | `ingestCacheBytes`, 50 MB | lru | a re-fold (the ids are hash-consed) |
44
+
45
+ - `_bytesCache` stores only complete reconstructions. A truncated prefix is
46
+ never cached.
47
+ - The ANN read caches (`_resonateCache`, `_resonateHaloCache`) are keyed by
48
+ `vecKey(v):k`, dropped on any index mutation, and cleared at
49
+ `RESONATE_CACHE_MAX`.
50
+ - `vectorCacheMb` and `sqliteCacheMb` only tune page-cache latency.
51
+
52
+ ## Mind caches — the session
53
+
54
+ - **`_gistCache`** (32 MB): node gists, kept for the session's lifetime and
55
+ never invalidated, since perception is pure.
56
+ - **`_depositTrees` / `_depositLens`**: up to 8 folds, keyed by their bytes,
57
+ written only by conversational deposits, so that a growing context re-folds
58
+ only its suffix (`fold-contract.md`). Both reset together when the length set
59
+ exceeds 64.
60
+ - **`_internIds`** (a `WeakMap` from tree node to id): skips re-interning a
61
+ shared subtree. The ids it holds are permanent.
62
+
63
+ Per-response memos are in `memoization.md`.
84
64
 
85
65
  ## Pins
86
66
 
87
- - `test/91` — `chainRun` via capped `_prefix`: bounded transparent-chain hop,
88
- not per-node probes.
89
- - `test/96` — `_bytesCache` is byte-accounted `BoundedMap` that evicts; miss
90
- re-derives.
67
+ - `test/96` — `_bytesCache` is a byte-accounted `BoundedMap` that evicts, and a
68
+ miss re-derives.
69
+ - `test/91` — `chainRun` hops a transparent chain in one bounded read, not with
70
+ probes per node.