@hviana/sema 0.9.4 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/AGENTS.md +16 -7
  2. package/dist/src/mind/learning.js +11 -12
  3. package/dist/src/mind/mind.js +4 -4
  4. package/dist/src/mind/types.d.ts +10 -15
  5. package/dist/src/store.d.ts +10 -10
  6. package/dist/src/store.js +5 -5
  7. package/docs/INDEX.md +62 -63
  8. package/docs/INVARIANTS.md +36 -18
  9. package/docs/architecture/bounded-reads.md +31 -71
  10. package/docs/architecture/caches.md +61 -81
  11. package/docs/architecture/closure.md +88 -107
  12. package/docs/architecture/commonality.md +47 -38
  13. package/docs/architecture/cost-model.md +57 -79
  14. package/docs/architecture/determinism.md +43 -55
  15. package/docs/architecture/evidence.md +158 -235
  16. package/docs/architecture/exact-vs-approximate.md +41 -35
  17. package/docs/architecture/factored-machinery.md +34 -20
  18. package/docs/architecture/fold-contract.md +110 -118
  19. package/docs/architecture/halo-sketch.md +105 -96
  20. package/docs/architecture/match-project.md +51 -42
  21. package/docs/architecture/mechanism-market.md +87 -91
  22. package/docs/architecture/memoization.md +60 -74
  23. package/docs/architecture/meter.md +37 -47
  24. package/docs/architecture/saturation.md +75 -101
  25. package/docs/architecture/store.md +118 -99
  26. package/docs/architecture/thresholds.md +66 -73
  27. package/docs/failures/tempting-but-wrong.md +139 -165
  28. package/docs/harness/gates.md +27 -32
  29. package/docs/mechanisms/alu.md +22 -69
  30. package/docs/mechanisms/cast.md +76 -71
  31. package/docs/mechanisms/confluence.md +22 -29
  32. package/docs/mechanisms/cover.md +58 -66
  33. package/docs/mechanisms/extraction.md +33 -37
  34. package/docs/mechanisms/prefix-completion.md +36 -39
  35. package/docs/mechanisms/recall.md +60 -53
  36. package/docs/mechanisms/reference.md +63 -49
  37. package/jsr.json +1 -1
  38. package/package.json +1 -1
  39. package/src/alu/README.md +90 -298
  40. package/src/derive/README.md +94 -256
  41. package/src/mind/learning.ts +11 -12
  42. package/src/mind/mind.ts +4 -4
  43. package/src/mind/types.ts +10 -15
  44. package/src/rabitq-ivf/README.md +11 -8
  45. package/src/store.ts +5 -5
@@ -1,295 +1,133 @@
1
1
  # derive
2
2
 
3
- A small, dependency-free library for computing the **lightest derivation** in a
4
- weighted deduction system — equivalently, an implicit **AND/OR hypergraph** — by
5
- A\* search. It is the generic graph-exploration core of symbolic processing: the
6
- engine knows nothing about what its items _are_, only how to canonicalise them,
7
- enumerate their rules, score them, and recognise the goal. Anything that can be
8
- phrased as "derive a global structure from weighted rules" — parsing, shortest
9
- paths, segmentation, rewriting, planning — is one call to the same search.
10
-
11
- It has no dependency on the rest of the codebase and is intended to be reused
12
- across mechanisms, in the spirit of a self-contained sublibrary.
3
+ A small, dependency-free library that computes the **lightest derivation** in a
4
+ weighted deduction system, equivalently an implicit **AND/OR hypergraph**, by
5
+ A\* search. The engine knows nothing about what its items _are_. It only knows
6
+ how to key them, enumerate their rules, bound them and recognise the goal.
7
+ Parsing, shortest paths, segmentation, rewriting and planning are each one call
8
+ to the same search. It imports nothing from the rest of the codebase.
13
9
 
14
10
  ## The algorithm
15
11
 
16
- The library implements **adapted A\* Lightest Derivation (adapted A\*LD)**:
17
-
18
- > P. F. Felzenszwalb and D. McAllester. _The Generalized A\* Architecture._
19
- > Journal of Artificial Intelligence Research 29 (2007) 153–190.
12
+ The library implements an adapted A\* Lightest Derivation (A\*LD; P. F.
13
+ Felzenszwalb & D. McAllester, _The Generalized A\* Architecture_, JAIR 29,
14
+ 2007). It unifies two classical results:
20
15
 
21
- adapted A\*LD generalises A\* from shortest paths to the problem of computing a
22
- lightest derivation of a goal from a set of weighted rules, searching an AND/OR
23
- graph bottom-up. It rests on two classical results and unifies them:
24
-
25
- - **Knuth (1977)**, _A generalization of Dijkstra's algorithm_ — the
26
- lightest-derivation problem and its Dijkstra-like solution: process items in
27
- priority order, and an item's cost is final the instant it is removed from the
16
+ - **Knuth (1977)**, _A generalization of Dijkstra's algorithm_: process items in
17
+ order of priority, and an item's cost is final the moment it leaves the
28
18
  agenda.
29
- - **A\* parsing** (Klein & Manning, 2003) — adding an admissible heuristic so
30
- that partial derivations which cannot lead cheaply to the goal are never
31
- expanded.
32
-
33
- For a problem that happens to be a shortest path, adapted A\*LD reduces exactly
34
- to A\*. With a small number of antecedents per rule it runs in **O(M log N)** (M
35
- rules, N items), and — crucially — it is **output-sensitive**: only items `B`
36
- with `ℓ(B) ≤ ℓ(goal)` are ever expanded, where `ℓ(B)` is the cost of `B`'s
37
- lightest derivation. Work is proportional to the goal, not to the size of the
38
- implicit graph, so there is no need for length caps or other artificial bounds
39
- to keep it tractable.
40
-
41
- > **Evidence pooling via semiring generalization.** The `derive` engine is no
42
- > the pure classic A\*LD algorithm. It has been extended with **additive
43
- > evidence accumulation** — a second combining mode (`combine: "sum"`) that
44
- > operates in the **arithmetic (+, +) semiring** instead of the standard
45
- > **tropical (min, +) semiring** of shortest-path search. Under the min-cost
46
- > regime, only the single cheapest route to a conclusion survives; under the
47
- > arithmetic semiring, every independent line of evidence corroborating the same
48
- > conclusion is **pooled** — its cost is summed, its contribution is recorded,
49
- > and nothing is discarded for not being the cheapest. This is how Sema forms
50
- > consensus votes from multiple independent query regions: each region
51
- > contributes its evidence, and the pooled aggregate is the conclusion's total
52
- > evidential support. A pooled conclusion never competes in the min-cost chart,
53
- > never re-enters the agenda, and is read back by the caller once the search
54
- > exhausts its axioms — making evidence pooling a zero-cost opt-in that coexists
55
- > with the classic algorithm in a single search.
56
-
57
- ## What "lightest derivation" means
58
-
59
- A **weighted deduction system** is a set of _items_ combined by inference rules
60
-
61
- ```
62
- premise₁ ∧ … ∧ premiseₖ --localCost--> conclusion
63
- ```
64
-
65
- A _derivation_ of an item is a tree: the root is a rule, its children are
66
- derivations of that rule's premises, and the leaves are axioms. A derivation's
67
- cost is the sum of the local costs of the rules it uses. `ℓ(B)` is the minimum
68
- such cost over all derivations of `B`. The search returns a derivation achieving
69
- `ℓ(goal)`, or `null` if the goal is underivable.
70
-
71
- A rule with one premise is an ordinary (OR) edge; a rule with several premises
72
- is an **AND** node that _composes_ its premises — this is what makes the
73
- structure a hypergraph rather than a plain graph, and it is how partial results
74
- are joined (for example, combining two recognised spans into one).
75
-
76
- ### Bridges: composing premises into a coherent whole
77
-
78
- A multi-premise rule is a **bridge**: it derives a conclusion from two (or more)
79
- already-derived items and charges a local cost for the join. Bridges are the
80
- mechanism by which independently-derived parts are assembled — and because a
81
- bridge's conclusion is itself an ordinary item, it can serve as a premise of a
82
- _further_ bridge, so bridges chain associatively: `A∧B → AB`, then `AB∧C → ABC`,
83
- all within one search. The cost of the join is paid inside the derivation, so
84
- the lightest derivation is the globally most coherent assembly of the parts, not
85
- a left-to-right stitch decided after the fact.
86
-
87
- This is what backs the connector ("bridge") mechanism in the surrounding mind:
88
- two rewritten spans are joined by a bridge whose local cost reflects how well a
89
- learned connector fits between them, and a third span chains onto the result by
90
- the same rule — so a multi-part answer is found as one whole, with no
91
- accumulated gaps, and never as a post-processing pass.
92
-
93
- Two properties make bridges safe to lean on:
94
-
95
- - **Lazy, order-free firing.** A bridge is emitted from `rules` when _either_
96
- premise is finalised; the engine fires it only once _all_ its premises are
97
- known (see `lightestDerivation`'s readiness check). So a bridge may be yielded
98
- twice — once from each side — and the engine deduplicates by waiting for the
99
- full conjunction. The caller need not know which premise finalises first.
100
- - **Robust reconstruction.** Recovering the derivation tree is iterative (an
101
- explicit post-order stack), so it handles both arbitrarily long single-premise
102
- chains — which, on long inputs, would overflow a recursive walk — and
103
- multi-premise bridges uniformly. A bridged conclusion's `premises` array holds
104
- one `Derivation` per premise, in rule order.
105
-
106
- ## The four mechanisms
107
-
108
- The search stays proportional to the goal because of four reductions, three of
109
- which the caller participates in through the `DeductionSystem` interface:
110
-
111
- 1. **Canonical chart memoization** — items are keyed by `key(item)`; equivalent
112
- partial derivations collapse to a single chart entry, the cheapest one.
113
- 2. **Backward demand filtering** — `rules` only emits rules whose conclusion can
114
- still reach the goal, so work unrelated to the goal is never generated.
115
- 3. **A\* lower-bound pruning** — `heuristic` keeps the agenda ordered by
116
- `g + h`, so only competitive items are expanded.
117
- 4. **Lazy hyperedge generation** — rules are produced by `rules` only when one
118
- of their premises is finalised, never enumerated up front.
19
+ - **A\* parsing** (Klein & Manning, 2003): an admissible heuristic keeps partial
20
+ derivations that cannot reach the goal cheaply from ever being expanded.
21
+
22
+ On a shortest-path problem it reduces exactly to A\*. With a small number of
23
+ premises per rule it runs in `O(M log N)`. It is **output-sensitive**: only
24
+ items whose lightest cost is at most the goal's are ever expanded, so its work
25
+ is proportional to the goal, not to the size of the implicit graph, and it needs
26
+ no length caps.
27
+
28
+ A **derivation** of an item is a tree: its root is a rule, its children derive
29
+ that rule's premises, and its leaves are axioms. Its cost is the sum of the
30
+ local costs of the rules it uses. A rule with one premise is an OR edge. A rule
31
+ with several premises is an AND node that **composes** them. That is what makes
32
+ the structure a hypergraph, and it is how separately derived parts are joined
33
+ (`A ∧ B → AB`, then `AB ∧ C → ABC`). Because the join is paid inside the search,
34
+ the lightest derivation is the most coherent assembly of the parts overall, not
35
+ a left-to-right stitch decided afterwards.
36
+
37
+ ## Two combinators
38
+
39
+ | `Rule.combine` | Semiring | At the conclusion |
40
+ | ----------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
41
+ | `"min"` (default) | (min, +), tropical | the cheapest route wins and every other is discarded. This is the search proper. |
42
+ | `"sum"` | (+, +) | every firing adds its cost to the same conclusion, in `system.pool`. Independent lines of evidence corroborate instead of competing. |
43
+
44
+ A pooled conclusion never enters the agenda and is never a premise. It is a
45
+ terminal aggregate the caller reads from `pool` once the search has exhausted
46
+ its axioms. A system without `pool` never takes this branch, so pooling costs
47
+ nothing to systems that do not use it.
48
+
49
+ ## Why the search stays proportional to the goal
50
+
51
+ 1. **Canonical chart memoization.** Items are keyed by `key(item)`, so
52
+ equivalent partial derivations collapse to one entry, the cheapest.
53
+ 2. **Backward demand filtering.** `rules` emits only rules whose conclusion can
54
+ still reach the goal.
55
+ 3. **A\* pruning.** `heuristic` orders the agenda by `g + h`.
56
+ 4. **Lazy hyperedges.** A rule is produced only when one of its premises is
57
+ finalised, and fires only once all of its premises are known. It may be
58
+ yielded from either side, and the engine waits for the full conjunction.
59
+
60
+ The derivation tree is reconstructed iteratively, so very long chains cannot
61
+ overflow the stack.
119
62
 
120
63
  ## API
121
64
 
122
65
  ```ts
123
- import {
124
- type DeductionSystem,
125
- type Derivation,
126
- lightestDerivation,
127
- type Rule,
128
- } from "derive";
129
- ```
130
-
131
- ### `lightestDerivation<I>(system: DeductionSystem<I>): Derivation<I> | null`
66
+ lightestDerivation<I>(system: DeductionSystem<I>, stats?: SearchStats): Derivation<I> | null
132
67
 
133
- Runs the search and returns the root of a lightest derivation tree (`null` if
134
- the goal cannot be derived). `root.cost` is the total derivation cost; the tree
135
- is walked through `root.premises`.
136
-
137
- ### `DeductionSystem<I>`
138
-
139
- The problem you hand the solver. `I` is your item type — anything at all.
140
-
141
- ```ts
142
68
  interface DeductionSystem<I> {
143
- // Canonical key for chart memoization (the item's "boundary signature":
144
- // everything that can affect how it later combines).
145
- key(item: I): string;
146
-
147
- // The atomic items and their base costs — the search's seeds.
148
- axioms(): Iterable<{ item: I; cost: number }>;
149
-
150
- // Lazily yield the rules that have `item` among their premises. Called once,
151
- // when `item` is finalised. `costOf(other)` returns another item's finalised
152
- // cost (Infinity if not yet known) — use it to drop rules whose other
153
- // premises are still open or whose conclusion can no longer beat the goal.
154
- rules(item: I, costOf: (other: I) => number): Iterable<Rule<I>>;
155
-
156
- // Whether `item` satisfies the goal. The first finalised goal wins.
157
- isGoal(item: I): boolean;
158
-
159
- // Optional admissible, consistent lower bound on the cost from `item` to a
160
- // goal. Omit it (or return 0) for plain Knuth/Dijkstra.
161
- heuristic?(item: I): number;
162
- }
163
- ```
164
-
165
- ### `Rule<I>` and `Derivation<I>`
166
-
167
- ```ts
168
- interface Rule<I> {
169
- premises: readonly I[]; // the conjunction (one premise = an OR edge; many = an AND node)
170
- conclusion: I;
171
- cost: number; // local cost, non-negative
172
- }
173
-
174
- interface Derivation<I> {
175
- item: I;
176
- cost: number; // this item's lightest-derivation cost (its g)
177
- rule: Rule<I> | null; // the producing rule, or null for an axiom
178
- premises: Array<Derivation<I>>; // derivations of the rule's premises
69
+ key(item: I): string; // chart key: everything that can affect later combination
70
+ axioms(): Iterable<{ item: I; cost: number }>; // the seeds
71
+ rules(item: I, costOf: (other: I) => number): Iterable<Rule<I>>; // called once, when `item` is finalised
72
+ isGoal(item: I): boolean; // the first finalised goal wins
73
+ heuristic?(item: I): number; // admissible, consistent; omit for Knuth/Dijkstra
74
+ pool?: Map<string, PooledConclusion<I>>; // present only for `combine: "sum"` systems
179
75
  }
180
- ```
181
-
182
- ## Example: shortest path is a special case
183
-
184
- A weighted directed graph is a deduction system in which every node is an item
185
- and every edge is a one-premise rule. The lightest derivation of the target is
186
- the shortest path — adapted A\*LD collapses to A\*.
187
-
188
- ```ts
189
- import { type DeductionSystem, lightestDerivation } from "derive";
190
-
191
- const edges: Record<string, Array<[string, number]>> = {
192
- a: [["b", 1], ["c", 4]],
193
- b: [["c", 1], ["d", 5]],
194
- c: [["d", 1]],
195
- d: [],
196
- };
197
76
 
198
- const shortestPath: DeductionSystem<string> = {
199
- key: (n) => n,
200
- *axioms() {
201
- yield { item: "a", cost: 0 };
202
- },
203
- *rules(node) {
204
- for (const [to, w] of edges[node] ?? []) {
205
- yield { premises: [node], conclusion: to, cost: w };
206
- }
207
- },
208
- isGoal: (n) => n === "d",
209
- heuristic: () => 0, // any admissible lower bound focuses the search
210
- };
211
-
212
- const best = lightestDerivation(shortestPath);
213
- // best.cost === 3, and a → b → c → d is recovered from best.premises
77
+ interface Rule<I> { premises: readonly I[]; conclusion: I; cost: number; combine?: "min" | "sum" }
78
+ interface Derivation<I> { item: I; cost: number; rule: Rule<I> | null; premises: Derivation<I>[] }
79
+ interface SearchStats { pops: number; pushes: number }
214
80
  ```
215
81
 
216
- To compose rather than merely traverse, give a rule **several** premises: the
217
- search will derive each one and only fire the rule once they are all finalised,
218
- charging `cost` on top of their combined cost.
219
-
220
- ## Example: a bridge composes two parts into one
82
+ `costOf(other)` returns another item's finalised cost, or `Infinity` if it is
83
+ not yet known. Use it to drop rules whose other premises are still open, or
84
+ whose conclusion can no longer beat the goal.
221
85
 
222
- Two axioms `A` (3) and `B` (4) and a bridge `A ∧ B → AB` (1). The bridge is
223
- yielded from _either_ premise; the engine waits until both are finalised, then
224
- charges the join. The only derivation of `AB` costs `3 + 4 + 1 = 8`, and its
225
- `premises` array holds the derivations of `A` and `B`.
86
+ ## Example — a bridge composes two parts
226
87
 
227
88
  ```ts
228
- import { type DeductionSystem, lightestDerivation } from "derive";
229
-
230
89
  const system: DeductionSystem<string> = {
231
90
  key: (s) => s,
232
91
  axioms: () => [{ item: "A", cost: 3 }, { item: "B", cost: 4 }],
233
92
  isGoal: (s) => s === "AB",
234
93
  *rules(item) {
235
- // Fired from either side; the engine composes once both are known.
236
- if (item === "A") yield { premises: ["A", "B"], conclusion: "AB", cost: 1 };
237
- if (item === "B") yield { premises: ["A", "B"], conclusion: "AB", cost: 1 };
94
+ if (item === "A" || item === "B") {
95
+ yield { premises: ["A", "B"], conclusion: "AB", cost: 1 };
96
+ }
238
97
  },
239
98
  };
240
-
241
- const best = lightestDerivation(system);
242
- // best.cost === 8, best.premises.length === 2 — it really used the bridge.
99
+ lightestDerivation(system); // cost 8 = 3 + 4 + 1; premises hold A's and B's derivations
243
100
  ```
244
101
 
245
- Add a rule `AB ∧ C → ABC` and the bridge chains: the lightest derivation of
246
- `ABC` composes all three parts, the join cost paid inside the search.
102
+ A weighted graph is the special case where every rule has one premise. There,
103
+ the lightest derivation of the target is the shortest path.
247
104
 
248
105
  ## Correctness conditions
249
106
 
250
- The engine is correct provided the caller upholds the conditions adapted A\*LD
251
- requires:
252
-
253
- - **Non-negative local costs** (more generally, monotone): adding a rule never
254
- lowers a derivation's cost.
255
- - **Admissible, consistent heuristic**: `heuristic` never overestimates the cost
256
- remaining to a goal, and is hyperedge-consistent,
257
- `h(conclusion) ≤ ruleCost + Σ h(premiseᵢ)`. The default (`0`) is trivially
258
- consistent and yields plain Knuth/Dijkstra.
259
- - **Faithful `key`**: two items with the same key must be interchangeable in
260
- every rule. The key must preserve every part of an item that can affect future
261
- composition; anything it drops is asserted irrelevant.
262
-
263
- ## Companion utilities
264
-
265
- Three small, independently useful pieces share the package, all aligned with the
266
- same "only the work the goal demands" philosophy:
267
-
268
- - **`Trie<P>`** — a forward prefix matcher used as a _lazy recognition index_.
269
- `matchesAt(seq, pos)` walks forward from the root and reports every stored
270
- pattern beginning at `pos` in time proportional to the longest match; a
271
- position with nothing learned dead-ends at once, and no length bound is
272
- imposed — the structure of what was inserted is the bound. A cursor API
273
- (`root`, `step`, `terminal`) supports incremental walks; `scan(seq)` reports
274
- all matches across all positions.
107
+ - **Local costs are non-negative,** or more generally monotone.
108
+ - **The heuristic is admissible and consistent:**
109
+ `h(conclusion) ≤ ruleCost + Σ h(premiseᵢ)`. The default `0` is trivially so.
110
+ - **`key` is faithful:** two items with the same key must be interchangeable in
111
+ every rule. Whatever the key drops is asserted to be irrelevant.
275
112
 
276
- - **`coverSequence<P>(length, candidates)`** — the segmentation primitive: the
277
- lightest set of non-overlapping spans covering a sequence, computed on the
278
- engine. It is the principled, corpus-independent replacement for "scan with an
279
- automaton, then greedily keep the longest non-overlapping matches" — the same
280
- linear cost, but the _optimal_ cover.
113
+ ## Companions
281
114
 
282
- - **`MinHeap<T>`** — the allocation-light binary heap the agenda is built on.
115
+ - **`Trie<P>`** is a lazy forward prefix matcher. `matchesAt(seq, pos)` reports
116
+ every stored pattern that starts at `pos`, in time proportional to the longest
117
+ match, and `scan(seq)` reports all of them. A cursor API (`root`, `step`,
118
+ `terminal`) walks incrementally.
119
+ - **`coverSequence(length, candidates)`** returns the lightest set of
120
+ non-overlapping spans covering a sequence. It is the optimal replacement for
121
+ "keep the longest matches greedily", at the same linear cost.
122
+ - **`MinHeap<T>`** is the agenda's heap.
283
123
 
284
124
  ## Layout
285
125
 
286
126
  ```
287
- derive/
288
- ├── README.md this file
289
- └── src/
290
- ├── deduction.ts lightestDerivation — the adapted A*LD engine
291
- ├── trie.ts Trie — lazy forward prefix matcher
292
- ├── rewrite.ts coverSequence — optimal sequence cover
293
- ├── priority-queue.ts MinHeap — the agenda's heap
294
- └── index.ts public surface
127
+ src/deduction.ts lightestDerivation, the engine
128
+ src/trie.ts Trie
129
+ src/rewrite.ts coverSequence
130
+ src/priority-queue.ts MinHeap
131
+ src/index.ts public surface
132
+ test/derive.test.ts self-contained tests
295
133
  ```
@@ -122,18 +122,17 @@ export async function deposit(
122
122
  { tree: Sema; rootId: number; ids: Map<Sema, number>; changed: Sema[] }
123
123
  > {
124
124
  const bytes = inputBytes(ctx, input);
125
- // Deposit-shaped perception: stable-prefix tree SEEDING (see
126
- // perceiveDeposit) — an accumulated context re-folds only its new suffix,
127
- // O(turn) instead of O(context) per conversation turn. Cache-only here
128
- // (no store-probe fallback): a knownPrefixLength scan on every novel fact
129
- // would cost O(n²) hashing, while conversation replays are always warm —
130
- // re-deposition replays from the first turn, rebuilding the cache as it
131
- // goes. `conversational` scopes the STABLE-PREFIX variant (turn-boundary
132
- // folding, matching query-time perception) to ingestPair's own growing
133
- // context argument — a bare ingestOne deposit whose bytes merely happen
134
- // to extend an earlier UNRELATED deposit (no conversational relationship)
135
- // must keep the plain fold, or two coincidentally-prefix-sharing facts
136
- // would stop sharing structure with each other.
125
+ // Deposit-shaped perception (perceiveDeposit): the plain content fold, the
126
+ // same tree inference computes for these bytes. An accumulated context
127
+ // reuses the already-folded segments of its cached prefix
128
+ // (contentFoldIncremental), so it re-folds only its new suffix — O(turn)
129
+ // instead of O(context) per conversation turn. The reuse is transparent:
130
+ // a hit saves time and never changes the tree. Cache-only here (no
131
+ // store-probe fallback): conversation replays are always warm, because
132
+ // re-deposition replays from the first turn and rebuilds the cache as it
133
+ // goes. `conversational` only decides which deposits WRITE the cache —
134
+ // ingestPair's growing context, not every unrelated fact — a budget
135
+ // choice, not a correctness one (fold-contract.md).
137
136
  const tree = perceiveDeposit(ctx, bytes, conversational);
138
137
 
139
138
  const ids = new Map<Sema, number>();
package/src/mind/mind.ts CHANGED
@@ -1063,10 +1063,10 @@ export class Mind implements MindContext {
1063
1063
  // No recognise-memo pre-seeding here: that used to be necessary because
1064
1064
  // the flat/positional fold lost visibility into an earlier turn's own
1065
1065
  // structure once later bytes shifted its position (foldTree no longer
1066
- // visited the turn's root node). The STABLE-PREFIX fold (see {@link
1067
- // ConversationData}) makes every turn's subtree independent of what
1068
- // follows it by construction, so recognise() finds it correctly on its
1069
- // own, first-touch, exactly once per turn.
1066
+ // visited the turn's root node). The content-defined fold (see {@link
1067
+ // ConversationData}) cuts by the bytes, never by position, so an earlier
1068
+ // turn's structure is the same whatever follows it, and recognise() finds
1069
+ // it on its own, first-touch, exactly once per turn.
1070
1070
  this.beginResponse(
1071
1071
  inspectRationale,
1072
1072
  this._canonFor(typeof turn === "string" ? textCanon : null),
package/src/mind/types.ts CHANGED
@@ -400,9 +400,8 @@ export interface MindContext extends GraphSearchHost {
400
400
  * with `bytesToTree` on every turn, so every key was fresh and this cache
401
401
  * could not hit even once — the O(suffix) claim above described an
402
402
  * intention rather than the code. It now grows the context through
403
- * {@link stablePrefixFoldIncremental}, which reuses each already-folded
404
- * segment: measured over four turns, turn 4 shared 69 of its 95 nodes with
405
- * turn 3 (26 new ≈ the new turn's own size). */
403
+ * contentFoldIncremental, which reuses each already-folded segment as the
404
+ * same object (~92% of nodes reused by identity across turns). */
406
405
  _resolvedSubtrees: WeakMap<Sema, { id: number; len: number }> | null;
407
406
  /** Completed assistant-turn byte spans in the current cumulative query.
408
407
  * Empty for ordinary respond(); response-scoped structural context for
@@ -435,18 +434,14 @@ export interface MindContext extends GraphSearchHost {
435
434
  * never a correctness risk. */
436
435
  _gistCache: BoundedMap<number, Vec>;
437
436
  /** DEPOSIT-path perception cache: content key (latin1) of a deposited
438
- * input → its accumulated turn BOUNDARIES plus reusable fold state. A
439
- * deposit whose content extends a cached entry IS a conversation context
440
- * grown by one turn — the cached length is the new boundary — so it
441
- * folds with the SAME stable-prefix fold query-time perception uses
442
- * (structural train/inference agreement, load-bearing for recall),
443
- * reusing every already-folded segment via `stable` (see StableFold) —
444
- * O(turn) per deposit instead of O(context). A first-seen input takes the
445
- * same fold with no boundaries at all, and caches the segments it produced
446
- * so a later turn of the same conversation reuses them. Purely a
447
- * performance cache for the FOLD STATE; the boundaries are semantic but
448
- * derived only from the deposit sequence itself (an evicted chain falls
449
- * back to plain-fold behavior, exactly the pre-boundary shape). */
437
+ * input → its reusable content-fold state ({@link DepositCacheEntry}). A
438
+ * deposit whose bytes extend a cached entry reuses that entry's
439
+ * already-folded segments (contentFoldIncremental) — O(turn) per deposit
440
+ * instead of O(context) — and gets exactly the tree a cold fold would
441
+ * give, the same one query-time perception computes. It holds no turn
442
+ * boundaries: the fold imposes none (fold-contract.md). Written only by
443
+ * conversational deposits, so the 8-entry budget keeps the live chains;
444
+ * an evicted chain costs a re-fold, never a different tree. */
450
445
  _depositTrees: BoundedMap<string, DepositCacheEntry>;
451
446
  /** The byte lengths present in {@link _depositTrees} — the candidate
452
447
  * prefix lengths probed (longest first). Drifts on eviction (a stale
@@ -25,20 +25,23 @@ detected by length.
25
25
  - **Clusters, not a graph.** The collection is partitioned into clusters, each
26
26
  with a binary pivot code and its member codes packed in fixed-size chunk
27
27
  blobs.
28
- - **Insert = route + append.** Find the nearest pivot (one linear Hamming scan
29
- of the RAM-resident pivot table) and append to that cluster's tail chunk. No
30
- beam search, no neighbour rewiring — per-insert cost is essentially flat in
31
- collection size.
28
+ - **Insert = route + append.** Find a nearest pivot and append to that cluster's
29
+ tail chunk. Routing is two-level: Hamming-scan the RAM-resident majority-bit
30
+ super-pivots (one per 64 clusters), then the members of the 4 nearest groups.
31
+ There is no beam search and no neighbour rewiring, so the cost of an insert is
32
+ essentially flat in collection size.
32
33
  - **Query = probe + scan.** Rank all pivots with the accurate RaBitQ estimator,
33
34
  scan the `ceil(efSearch/4)` nearest clusters with the same estimator, keep the
34
35
  top k. Per-query storage reads are bounded by nprobe × chunks-per-cluster —
35
36
  constant once the collection has split.
36
37
  - **Adaptive, deterministic splits.** A cluster reaching 4096 entries is
37
38
  median-split on the margin between two farthest-point seeds (two exact halves,
38
- cascade-proof), and both halves get fresh majority-bit pivots. There is no
39
- RNG: the index is a pure function of the insertion sequence.
40
- - **Same durability discipline as the rest of Sema**: WAL, batched caller-owned
41
- transactions (`upsertMany`), 1 KiB pages, 64 MiB WAL autocheckpoint.
39
+ cascade-proof), and both halves get fresh majority-bit pivots. Splitting and
40
+ routing use no randomness, and RaBitQ's rotation is seeded, so the index is a
41
+ pure function of the seed and the insertion sequence.
42
+ - **The same durability discipline as the rest of Sema:** WAL, batched
43
+ caller-owned transactions (`upsertMany`), 1 KiB pages, 64 MiB WAL
44
+ autocheckpoint.
42
45
 
43
46
  ## Usage
44
47
 
package/src/store.ts CHANGED
@@ -136,11 +136,11 @@ export class BoundedMap<K, V> {
136
136
  /** How a HIT records recency.
137
137
  *
138
138
  * `"reorder"` (default) promotes the entry to most-recent by
139
- * `m.delete(k); m.set(k, v)` — exact LRU, and the only policy that is
140
- * safe for a cache whose CONTENTS are load-bearing rather than merely
141
- * warm. `_depositTrees` (8 entries, feeds stablePrefixFoldIncremental)
142
- * is exactly that: which of its entries survives changes how the next
143
- * turn FOLDS, so test/13 D1 flips answer when the victim changes.
139
+ * `m.delete(k); m.set(k, v)` — exact LRU, the policy for any cache whose
140
+ * choice of victim must follow use exactly. `_depositTrees` (8 entries,
141
+ * feeds contentFoldIncremental) keeps it so the live conversation chains
142
+ * stay warm; its reuse is transparent, so a wrong victim costs a re-fold,
143
+ * never a different tree (fold-contract.md).
144
144
  *
145
145
  * `"clock"` records recency as a BIT instead of as position, spent by
146
146
  * the eviction sweep (see `nextOldest`). Correct only for a TRANSPARENT