@hviana/sema 0.9.4 → 0.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +16 -7
- package/dist/src/mind/learning.js +11 -12
- package/dist/src/mind/mind.js +4 -4
- package/dist/src/mind/types.d.ts +10 -15
- package/dist/src/store.d.ts +10 -10
- package/dist/src/store.js +5 -5
- package/docs/INDEX.md +62 -63
- package/docs/INVARIANTS.md +36 -18
- package/docs/architecture/bounded-reads.md +31 -71
- package/docs/architecture/caches.md +61 -81
- package/docs/architecture/closure.md +88 -107
- package/docs/architecture/commonality.md +47 -38
- package/docs/architecture/cost-model.md +57 -79
- package/docs/architecture/determinism.md +43 -55
- package/docs/architecture/evidence.md +158 -235
- package/docs/architecture/exact-vs-approximate.md +41 -35
- package/docs/architecture/factored-machinery.md +34 -20
- package/docs/architecture/fold-contract.md +110 -118
- package/docs/architecture/halo-sketch.md +105 -96
- package/docs/architecture/match-project.md +51 -42
- package/docs/architecture/mechanism-market.md +87 -91
- package/docs/architecture/memoization.md +60 -74
- package/docs/architecture/meter.md +37 -47
- package/docs/architecture/saturation.md +75 -101
- package/docs/architecture/store.md +118 -99
- package/docs/architecture/thresholds.md +66 -73
- package/docs/failures/tempting-but-wrong.md +139 -165
- package/docs/harness/gates.md +27 -32
- package/docs/mechanisms/alu.md +22 -69
- package/docs/mechanisms/cast.md +76 -71
- package/docs/mechanisms/confluence.md +22 -29
- package/docs/mechanisms/cover.md +58 -66
- package/docs/mechanisms/extraction.md +33 -37
- package/docs/mechanisms/prefix-completion.md +36 -39
- package/docs/mechanisms/recall.md +60 -53
- package/docs/mechanisms/reference.md +63 -49
- package/jsr.json +1 -1
- package/package.json +1 -1
- package/src/alu/README.md +90 -298
- package/src/derive/README.md +94 -256
- package/src/mind/learning.ts +11 -12
- package/src/mind/mind.ts +4 -4
- package/src/mind/types.ts +10 -15
- package/src/rabitq-ivf/README.md +11 -8
- package/src/store.ts +5 -5
package/src/derive/README.md
CHANGED
|
@@ -1,295 +1,133 @@
|
|
|
1
1
|
# derive
|
|
2
2
|
|
|
3
|
-
A small, dependency-free library
|
|
4
|
-
weighted deduction system
|
|
5
|
-
A\* search.
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
paths, segmentation, rewriting, planning — is one call to the same search.
|
|
10
|
-
|
|
11
|
-
It has no dependency on the rest of the codebase and is intended to be reused
|
|
12
|
-
across mechanisms, in the spirit of a self-contained sublibrary.
|
|
3
|
+
A small, dependency-free library that computes the **lightest derivation** in a
|
|
4
|
+
weighted deduction system, equivalently an implicit **AND/OR hypergraph**, by
|
|
5
|
+
A\* search. The engine knows nothing about what its items _are_. It only knows
|
|
6
|
+
how to key them, enumerate their rules, bound them and recognise the goal.
|
|
7
|
+
Parsing, shortest paths, segmentation, rewriting and planning are each one call
|
|
8
|
+
to the same search. It imports nothing from the rest of the codebase.
|
|
13
9
|
|
|
14
10
|
## The algorithm
|
|
15
11
|
|
|
16
|
-
The library implements
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
> Journal of Artificial Intelligence Research 29 (2007) 153–190.
|
|
12
|
+
The library implements an adapted A\* Lightest Derivation (A\*LD; P. F.
|
|
13
|
+
Felzenszwalb & D. McAllester, _The Generalized A\* Architecture_, JAIR 29,
|
|
14
|
+
2007). It unifies two classical results:
|
|
20
15
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
graph bottom-up. It rests on two classical results and unifies them:
|
|
24
|
-
|
|
25
|
-
- **Knuth (1977)**, _A generalization of Dijkstra's algorithm_ — the
|
|
26
|
-
lightest-derivation problem and its Dijkstra-like solution: process items in
|
|
27
|
-
priority order, and an item's cost is final the instant it is removed from the
|
|
16
|
+
- **Knuth (1977)**, _A generalization of Dijkstra's algorithm_: process items in
|
|
17
|
+
order of priority, and an item's cost is final the moment it leaves the
|
|
28
18
|
agenda.
|
|
29
|
-
- **A\* parsing** (Klein & Manning, 2003)
|
|
30
|
-
that
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
A
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
is an **AND** node that _composes_ its premises — this is what makes the
|
|
73
|
-
structure a hypergraph rather than a plain graph, and it is how partial results
|
|
74
|
-
are joined (for example, combining two recognised spans into one).
|
|
75
|
-
|
|
76
|
-
### Bridges: composing premises into a coherent whole
|
|
77
|
-
|
|
78
|
-
A multi-premise rule is a **bridge**: it derives a conclusion from two (or more)
|
|
79
|
-
already-derived items and charges a local cost for the join. Bridges are the
|
|
80
|
-
mechanism by which independently-derived parts are assembled — and because a
|
|
81
|
-
bridge's conclusion is itself an ordinary item, it can serve as a premise of a
|
|
82
|
-
_further_ bridge, so bridges chain associatively: `A∧B → AB`, then `AB∧C → ABC`,
|
|
83
|
-
all within one search. The cost of the join is paid inside the derivation, so
|
|
84
|
-
the lightest derivation is the globally most coherent assembly of the parts, not
|
|
85
|
-
a left-to-right stitch decided after the fact.
|
|
86
|
-
|
|
87
|
-
This is what backs the connector ("bridge") mechanism in the surrounding mind:
|
|
88
|
-
two rewritten spans are joined by a bridge whose local cost reflects how well a
|
|
89
|
-
learned connector fits between them, and a third span chains onto the result by
|
|
90
|
-
the same rule — so a multi-part answer is found as one whole, with no
|
|
91
|
-
accumulated gaps, and never as a post-processing pass.
|
|
92
|
-
|
|
93
|
-
Two properties make bridges safe to lean on:
|
|
94
|
-
|
|
95
|
-
- **Lazy, order-free firing.** A bridge is emitted from `rules` when _either_
|
|
96
|
-
premise is finalised; the engine fires it only once _all_ its premises are
|
|
97
|
-
known (see `lightestDerivation`'s readiness check). So a bridge may be yielded
|
|
98
|
-
twice — once from each side — and the engine deduplicates by waiting for the
|
|
99
|
-
full conjunction. The caller need not know which premise finalises first.
|
|
100
|
-
- **Robust reconstruction.** Recovering the derivation tree is iterative (an
|
|
101
|
-
explicit post-order stack), so it handles both arbitrarily long single-premise
|
|
102
|
-
chains — which, on long inputs, would overflow a recursive walk — and
|
|
103
|
-
multi-premise bridges uniformly. A bridged conclusion's `premises` array holds
|
|
104
|
-
one `Derivation` per premise, in rule order.
|
|
105
|
-
|
|
106
|
-
## The four mechanisms
|
|
107
|
-
|
|
108
|
-
The search stays proportional to the goal because of four reductions, three of
|
|
109
|
-
which the caller participates in through the `DeductionSystem` interface:
|
|
110
|
-
|
|
111
|
-
1. **Canonical chart memoization** — items are keyed by `key(item)`; equivalent
|
|
112
|
-
partial derivations collapse to a single chart entry, the cheapest one.
|
|
113
|
-
2. **Backward demand filtering** — `rules` only emits rules whose conclusion can
|
|
114
|
-
still reach the goal, so work unrelated to the goal is never generated.
|
|
115
|
-
3. **A\* lower-bound pruning** — `heuristic` keeps the agenda ordered by
|
|
116
|
-
`g + h`, so only competitive items are expanded.
|
|
117
|
-
4. **Lazy hyperedge generation** — rules are produced by `rules` only when one
|
|
118
|
-
of their premises is finalised, never enumerated up front.
|
|
19
|
+
- **A\* parsing** (Klein & Manning, 2003): an admissible heuristic keeps partial
|
|
20
|
+
derivations that cannot reach the goal cheaply from ever being expanded.
|
|
21
|
+
|
|
22
|
+
On a shortest-path problem it reduces exactly to A\*. With a small number of
|
|
23
|
+
premises per rule it runs in `O(M log N)`. It is **output-sensitive**: only
|
|
24
|
+
items whose lightest cost is at most the goal's are ever expanded, so its work
|
|
25
|
+
is proportional to the goal, not to the size of the implicit graph, and it needs
|
|
26
|
+
no length caps.
|
|
27
|
+
|
|
28
|
+
A **derivation** of an item is a tree: its root is a rule, its children derive
|
|
29
|
+
that rule's premises, and its leaves are axioms. Its cost is the sum of the
|
|
30
|
+
local costs of the rules it uses. A rule with one premise is an OR edge. A rule
|
|
31
|
+
with several premises is an AND node that **composes** them. That is what makes
|
|
32
|
+
the structure a hypergraph, and it is how separately derived parts are joined
|
|
33
|
+
(`A ∧ B → AB`, then `AB ∧ C → ABC`). Because the join is paid inside the search,
|
|
34
|
+
the lightest derivation is the most coherent assembly of the parts overall, not
|
|
35
|
+
a left-to-right stitch decided afterwards.
|
|
36
|
+
|
|
37
|
+
## Two combinators
|
|
38
|
+
|
|
39
|
+
| `Rule.combine` | Semiring | At the conclusion |
|
|
40
|
+
| ----------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
41
|
+
| `"min"` (default) | (min, +), tropical | the cheapest route wins and every other is discarded. This is the search proper. |
|
|
42
|
+
| `"sum"` | (+, +) | every firing adds its cost to the same conclusion, in `system.pool`. Independent lines of evidence corroborate instead of competing. |
|
|
43
|
+
|
|
44
|
+
A pooled conclusion never enters the agenda and is never a premise. It is a
|
|
45
|
+
terminal aggregate the caller reads from `pool` once the search has exhausted
|
|
46
|
+
its axioms. A system without `pool` never takes this branch, so pooling costs
|
|
47
|
+
nothing to systems that do not use it.
|
|
48
|
+
|
|
49
|
+
## Why the search stays proportional to the goal
|
|
50
|
+
|
|
51
|
+
1. **Canonical chart memoization.** Items are keyed by `key(item)`, so
|
|
52
|
+
equivalent partial derivations collapse to one entry, the cheapest.
|
|
53
|
+
2. **Backward demand filtering.** `rules` emits only rules whose conclusion can
|
|
54
|
+
still reach the goal.
|
|
55
|
+
3. **A\* pruning.** `heuristic` orders the agenda by `g + h`.
|
|
56
|
+
4. **Lazy hyperedges.** A rule is produced only when one of its premises is
|
|
57
|
+
finalised, and fires only once all of its premises are known. It may be
|
|
58
|
+
yielded from either side, and the engine waits for the full conjunction.
|
|
59
|
+
|
|
60
|
+
The derivation tree is reconstructed iteratively, so very long chains cannot
|
|
61
|
+
overflow the stack.
|
|
119
62
|
|
|
120
63
|
## API
|
|
121
64
|
|
|
122
65
|
```ts
|
|
123
|
-
|
|
124
|
-
type DeductionSystem,
|
|
125
|
-
type Derivation,
|
|
126
|
-
lightestDerivation,
|
|
127
|
-
type Rule,
|
|
128
|
-
} from "derive";
|
|
129
|
-
```
|
|
130
|
-
|
|
131
|
-
### `lightestDerivation<I>(system: DeductionSystem<I>): Derivation<I> | null`
|
|
66
|
+
lightestDerivation<I>(system: DeductionSystem<I>, stats?: SearchStats): Derivation<I> | null
|
|
132
67
|
|
|
133
|
-
Runs the search and returns the root of a lightest derivation tree (`null` if
|
|
134
|
-
the goal cannot be derived). `root.cost` is the total derivation cost; the tree
|
|
135
|
-
is walked through `root.premises`.
|
|
136
|
-
|
|
137
|
-
### `DeductionSystem<I>`
|
|
138
|
-
|
|
139
|
-
The problem you hand the solver. `I` is your item type — anything at all.
|
|
140
|
-
|
|
141
|
-
```ts
|
|
142
68
|
interface DeductionSystem<I> {
|
|
143
|
-
//
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
// Lazily yield the rules that have `item` among their premises. Called once,
|
|
151
|
-
// when `item` is finalised. `costOf(other)` returns another item's finalised
|
|
152
|
-
// cost (Infinity if not yet known) — use it to drop rules whose other
|
|
153
|
-
// premises are still open or whose conclusion can no longer beat the goal.
|
|
154
|
-
rules(item: I, costOf: (other: I) => number): Iterable<Rule<I>>;
|
|
155
|
-
|
|
156
|
-
// Whether `item` satisfies the goal. The first finalised goal wins.
|
|
157
|
-
isGoal(item: I): boolean;
|
|
158
|
-
|
|
159
|
-
// Optional admissible, consistent lower bound on the cost from `item` to a
|
|
160
|
-
// goal. Omit it (or return 0) for plain Knuth/Dijkstra.
|
|
161
|
-
heuristic?(item: I): number;
|
|
162
|
-
}
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
### `Rule<I>` and `Derivation<I>`
|
|
166
|
-
|
|
167
|
-
```ts
|
|
168
|
-
interface Rule<I> {
|
|
169
|
-
premises: readonly I[]; // the conjunction (one premise = an OR edge; many = an AND node)
|
|
170
|
-
conclusion: I;
|
|
171
|
-
cost: number; // local cost, non-negative
|
|
172
|
-
}
|
|
173
|
-
|
|
174
|
-
interface Derivation<I> {
|
|
175
|
-
item: I;
|
|
176
|
-
cost: number; // this item's lightest-derivation cost (its g)
|
|
177
|
-
rule: Rule<I> | null; // the producing rule, or null for an axiom
|
|
178
|
-
premises: Array<Derivation<I>>; // derivations of the rule's premises
|
|
69
|
+
key(item: I): string; // chart key: everything that can affect later combination
|
|
70
|
+
axioms(): Iterable<{ item: I; cost: number }>; // the seeds
|
|
71
|
+
rules(item: I, costOf: (other: I) => number): Iterable<Rule<I>>; // called once, when `item` is finalised
|
|
72
|
+
isGoal(item: I): boolean; // the first finalised goal wins
|
|
73
|
+
heuristic?(item: I): number; // admissible, consistent; omit for Knuth/Dijkstra
|
|
74
|
+
pool?: Map<string, PooledConclusion<I>>; // present only for `combine: "sum"` systems
|
|
179
75
|
}
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
## Example: shortest path is a special case
|
|
183
|
-
|
|
184
|
-
A weighted directed graph is a deduction system in which every node is an item
|
|
185
|
-
and every edge is a one-premise rule. The lightest derivation of the target is
|
|
186
|
-
the shortest path — adapted A\*LD collapses to A\*.
|
|
187
|
-
|
|
188
|
-
```ts
|
|
189
|
-
import { type DeductionSystem, lightestDerivation } from "derive";
|
|
190
|
-
|
|
191
|
-
const edges: Record<string, Array<[string, number]>> = {
|
|
192
|
-
a: [["b", 1], ["c", 4]],
|
|
193
|
-
b: [["c", 1], ["d", 5]],
|
|
194
|
-
c: [["d", 1]],
|
|
195
|
-
d: [],
|
|
196
|
-
};
|
|
197
76
|
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
yield { item: "a", cost: 0 };
|
|
202
|
-
},
|
|
203
|
-
*rules(node) {
|
|
204
|
-
for (const [to, w] of edges[node] ?? []) {
|
|
205
|
-
yield { premises: [node], conclusion: to, cost: w };
|
|
206
|
-
}
|
|
207
|
-
},
|
|
208
|
-
isGoal: (n) => n === "d",
|
|
209
|
-
heuristic: () => 0, // any admissible lower bound focuses the search
|
|
210
|
-
};
|
|
211
|
-
|
|
212
|
-
const best = lightestDerivation(shortestPath);
|
|
213
|
-
// best.cost === 3, and a → b → c → d is recovered from best.premises
|
|
77
|
+
interface Rule<I> { premises: readonly I[]; conclusion: I; cost: number; combine?: "min" | "sum" }
|
|
78
|
+
interface Derivation<I> { item: I; cost: number; rule: Rule<I> | null; premises: Derivation<I>[] }
|
|
79
|
+
interface SearchStats { pops: number; pushes: number }
|
|
214
80
|
```
|
|
215
81
|
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
## Example: a bridge composes two parts into one
|
|
82
|
+
`costOf(other)` returns another item's finalised cost, or `Infinity` if it is
|
|
83
|
+
not yet known. Use it to drop rules whose other premises are still open, or
|
|
84
|
+
whose conclusion can no longer beat the goal.
|
|
221
85
|
|
|
222
|
-
|
|
223
|
-
yielded from _either_ premise; the engine waits until both are finalised, then
|
|
224
|
-
charges the join. The only derivation of `AB` costs `3 + 4 + 1 = 8`, and its
|
|
225
|
-
`premises` array holds the derivations of `A` and `B`.
|
|
86
|
+
## Example — a bridge composes two parts
|
|
226
87
|
|
|
227
88
|
```ts
|
|
228
|
-
import { type DeductionSystem, lightestDerivation } from "derive";
|
|
229
|
-
|
|
230
89
|
const system: DeductionSystem<string> = {
|
|
231
90
|
key: (s) => s,
|
|
232
91
|
axioms: () => [{ item: "A", cost: 3 }, { item: "B", cost: 4 }],
|
|
233
92
|
isGoal: (s) => s === "AB",
|
|
234
93
|
*rules(item) {
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
94
|
+
if (item === "A" || item === "B") {
|
|
95
|
+
yield { premises: ["A", "B"], conclusion: "AB", cost: 1 };
|
|
96
|
+
}
|
|
238
97
|
},
|
|
239
98
|
};
|
|
240
|
-
|
|
241
|
-
const best = lightestDerivation(system);
|
|
242
|
-
// best.cost === 8, best.premises.length === 2 — it really used the bridge.
|
|
99
|
+
lightestDerivation(system); // cost 8 = 3 + 4 + 1; premises hold A's and B's derivations
|
|
243
100
|
```
|
|
244
101
|
|
|
245
|
-
|
|
246
|
-
|
|
102
|
+
A weighted graph is the special case where every rule has one premise. There,
|
|
103
|
+
the lightest derivation of the target is the shortest path.
|
|
247
104
|
|
|
248
105
|
## Correctness conditions
|
|
249
106
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
-
|
|
254
|
-
|
|
255
|
-
- **Admissible, consistent heuristic**: `heuristic` never overestimates the cost
|
|
256
|
-
remaining to a goal, and is hyperedge-consistent,
|
|
257
|
-
`h(conclusion) ≤ ruleCost + Σ h(premiseᵢ)`. The default (`0`) is trivially
|
|
258
|
-
consistent and yields plain Knuth/Dijkstra.
|
|
259
|
-
- **Faithful `key`**: two items with the same key must be interchangeable in
|
|
260
|
-
every rule. The key must preserve every part of an item that can affect future
|
|
261
|
-
composition; anything it drops is asserted irrelevant.
|
|
262
|
-
|
|
263
|
-
## Companion utilities
|
|
264
|
-
|
|
265
|
-
Three small, independently useful pieces share the package, all aligned with the
|
|
266
|
-
same "only the work the goal demands" philosophy:
|
|
267
|
-
|
|
268
|
-
- **`Trie<P>`** — a forward prefix matcher used as a _lazy recognition index_.
|
|
269
|
-
`matchesAt(seq, pos)` walks forward from the root and reports every stored
|
|
270
|
-
pattern beginning at `pos` in time proportional to the longest match; a
|
|
271
|
-
position with nothing learned dead-ends at once, and no length bound is
|
|
272
|
-
imposed — the structure of what was inserted is the bound. A cursor API
|
|
273
|
-
(`root`, `step`, `terminal`) supports incremental walks; `scan(seq)` reports
|
|
274
|
-
all matches across all positions.
|
|
107
|
+
- **Local costs are non-negative,** or more generally monotone.
|
|
108
|
+
- **The heuristic is admissible and consistent:**
|
|
109
|
+
`h(conclusion) ≤ ruleCost + Σ h(premiseᵢ)`. The default `0` is trivially so.
|
|
110
|
+
- **`key` is faithful:** two items with the same key must be interchangeable in
|
|
111
|
+
every rule. Whatever the key drops is asserted to be irrelevant.
|
|
275
112
|
|
|
276
|
-
|
|
277
|
-
lightest set of non-overlapping spans covering a sequence, computed on the
|
|
278
|
-
engine. It is the principled, corpus-independent replacement for "scan with an
|
|
279
|
-
automaton, then greedily keep the longest non-overlapping matches" — the same
|
|
280
|
-
linear cost, but the _optimal_ cover.
|
|
113
|
+
## Companions
|
|
281
114
|
|
|
282
|
-
- **`
|
|
115
|
+
- **`Trie<P>`** is a lazy forward prefix matcher. `matchesAt(seq, pos)` reports
|
|
116
|
+
every stored pattern that starts at `pos`, in time proportional to the longest
|
|
117
|
+
match, and `scan(seq)` reports all of them. A cursor API (`root`, `step`,
|
|
118
|
+
`terminal`) walks incrementally.
|
|
119
|
+
- **`coverSequence(length, candidates)`** returns the lightest set of
|
|
120
|
+
non-overlapping spans covering a sequence. It is the optimal replacement for
|
|
121
|
+
"keep the longest matches greedily", at the same linear cost.
|
|
122
|
+
- **`MinHeap<T>`** is the agenda's heap.
|
|
283
123
|
|
|
284
124
|
## Layout
|
|
285
125
|
|
|
286
126
|
```
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
├── priority-queue.ts MinHeap — the agenda's heap
|
|
294
|
-
└── index.ts public surface
|
|
127
|
+
src/deduction.ts lightestDerivation, the engine
|
|
128
|
+
src/trie.ts Trie
|
|
129
|
+
src/rewrite.ts coverSequence
|
|
130
|
+
src/priority-queue.ts MinHeap
|
|
131
|
+
src/index.ts public surface
|
|
132
|
+
test/derive.test.ts self-contained tests
|
|
295
133
|
```
|
package/src/mind/learning.ts
CHANGED
|
@@ -122,18 +122,17 @@ export async function deposit(
|
|
|
122
122
|
{ tree: Sema; rootId: number; ids: Map<Sema, number>; changed: Sema[] }
|
|
123
123
|
> {
|
|
124
124
|
const bytes = inputBytes(ctx, input);
|
|
125
|
-
// Deposit-shaped perception:
|
|
126
|
-
//
|
|
127
|
-
//
|
|
128
|
-
// (
|
|
129
|
-
//
|
|
130
|
-
//
|
|
131
|
-
//
|
|
132
|
-
//
|
|
133
|
-
//
|
|
134
|
-
//
|
|
135
|
-
//
|
|
136
|
-
// would stop sharing structure with each other.
|
|
125
|
+
// Deposit-shaped perception (perceiveDeposit): the plain content fold, the
|
|
126
|
+
// same tree inference computes for these bytes. An accumulated context
|
|
127
|
+
// reuses the already-folded segments of its cached prefix
|
|
128
|
+
// (contentFoldIncremental), so it re-folds only its new suffix — O(turn)
|
|
129
|
+
// instead of O(context) per conversation turn. The reuse is transparent:
|
|
130
|
+
// a hit saves time and never changes the tree. Cache-only here (no
|
|
131
|
+
// store-probe fallback): conversation replays are always warm, because
|
|
132
|
+
// re-deposition replays from the first turn and rebuilds the cache as it
|
|
133
|
+
// goes. `conversational` only decides which deposits WRITE the cache —
|
|
134
|
+
// ingestPair's growing context, not every unrelated fact — a budget
|
|
135
|
+
// choice, not a correctness one (fold-contract.md).
|
|
137
136
|
const tree = perceiveDeposit(ctx, bytes, conversational);
|
|
138
137
|
|
|
139
138
|
const ids = new Map<Sema, number>();
|
package/src/mind/mind.ts
CHANGED
|
@@ -1063,10 +1063,10 @@ export class Mind implements MindContext {
|
|
|
1063
1063
|
// No recognise-memo pre-seeding here: that used to be necessary because
|
|
1064
1064
|
// the flat/positional fold lost visibility into an earlier turn's own
|
|
1065
1065
|
// structure once later bytes shifted its position (foldTree no longer
|
|
1066
|
-
// visited the turn's root node). The
|
|
1067
|
-
// ConversationData})
|
|
1068
|
-
// follows it
|
|
1069
|
-
// own, first-touch, exactly once per turn.
|
|
1066
|
+
// visited the turn's root node). The content-defined fold (see {@link
|
|
1067
|
+
// ConversationData}) cuts by the bytes, never by position, so an earlier
|
|
1068
|
+
// turn's structure is the same whatever follows it, and recognise() finds
|
|
1069
|
+
// it on its own, first-touch, exactly once per turn.
|
|
1070
1070
|
this.beginResponse(
|
|
1071
1071
|
inspectRationale,
|
|
1072
1072
|
this._canonFor(typeof turn === "string" ? textCanon : null),
|
package/src/mind/types.ts
CHANGED
|
@@ -400,9 +400,8 @@ export interface MindContext extends GraphSearchHost {
|
|
|
400
400
|
* with `bytesToTree` on every turn, so every key was fresh and this cache
|
|
401
401
|
* could not hit even once — the O(suffix) claim above described an
|
|
402
402
|
* intention rather than the code. It now grows the context through
|
|
403
|
-
*
|
|
404
|
-
*
|
|
405
|
-
* turn 3 (26 new ≈ the new turn's own size). */
|
|
403
|
+
* contentFoldIncremental, which reuses each already-folded segment as the
|
|
404
|
+
* same object (~92% of nodes reused by identity across turns). */
|
|
406
405
|
_resolvedSubtrees: WeakMap<Sema, { id: number; len: number }> | null;
|
|
407
406
|
/** Completed assistant-turn byte spans in the current cumulative query.
|
|
408
407
|
* Empty for ordinary respond(); response-scoped structural context for
|
|
@@ -435,18 +434,14 @@ export interface MindContext extends GraphSearchHost {
|
|
|
435
434
|
* never a correctness risk. */
|
|
436
435
|
_gistCache: BoundedMap<number, Vec>;
|
|
437
436
|
/** DEPOSIT-path perception cache: content key (latin1) of a deposited
|
|
438
|
-
* input → its
|
|
439
|
-
* deposit whose
|
|
440
|
-
*
|
|
441
|
-
*
|
|
442
|
-
*
|
|
443
|
-
*
|
|
444
|
-
*
|
|
445
|
-
*
|
|
446
|
-
* so a later turn of the same conversation reuses them. Purely a
|
|
447
|
-
* performance cache for the FOLD STATE; the boundaries are semantic but
|
|
448
|
-
* derived only from the deposit sequence itself (an evicted chain falls
|
|
449
|
-
* back to plain-fold behavior, exactly the pre-boundary shape). */
|
|
437
|
+
* input → its reusable content-fold state ({@link DepositCacheEntry}). A
|
|
438
|
+
* deposit whose bytes extend a cached entry reuses that entry's
|
|
439
|
+
* already-folded segments (contentFoldIncremental) — O(turn) per deposit
|
|
440
|
+
* instead of O(context) — and gets exactly the tree a cold fold would
|
|
441
|
+
* give, the same one query-time perception computes. It holds no turn
|
|
442
|
+
* boundaries: the fold imposes none (fold-contract.md). Written only by
|
|
443
|
+
* conversational deposits, so the 8-entry budget keeps the live chains;
|
|
444
|
+
* an evicted chain costs a re-fold, never a different tree. */
|
|
450
445
|
_depositTrees: BoundedMap<string, DepositCacheEntry>;
|
|
451
446
|
/** The byte lengths present in {@link _depositTrees} — the candidate
|
|
452
447
|
* prefix lengths probed (longest first). Drifts on eviction (a stale
|
package/src/rabitq-ivf/README.md
CHANGED
|
@@ -25,20 +25,23 @@ detected by length.
|
|
|
25
25
|
- **Clusters, not a graph.** The collection is partitioned into clusters, each
|
|
26
26
|
with a binary pivot code and its member codes packed in fixed-size chunk
|
|
27
27
|
blobs.
|
|
28
|
-
- **Insert = route + append.** Find
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
28
|
+
- **Insert = route + append.** Find a nearest pivot and append to that cluster's
|
|
29
|
+
tail chunk. Routing is two-level: Hamming-scan the RAM-resident majority-bit
|
|
30
|
+
super-pivots (one per 64 clusters), then the members of the 4 nearest groups.
|
|
31
|
+
There is no beam search and no neighbour rewiring, so the cost of an insert is
|
|
32
|
+
essentially flat in collection size.
|
|
32
33
|
- **Query = probe + scan.** Rank all pivots with the accurate RaBitQ estimator,
|
|
33
34
|
scan the `ceil(efSearch/4)` nearest clusters with the same estimator, keep the
|
|
34
35
|
top k. Per-query storage reads are bounded by nprobe × chunks-per-cluster —
|
|
35
36
|
constant once the collection has split.
|
|
36
37
|
- **Adaptive, deterministic splits.** A cluster reaching 4096 entries is
|
|
37
38
|
median-split on the margin between two farthest-point seeds (two exact halves,
|
|
38
|
-
cascade-proof), and both halves get fresh majority-bit pivots.
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
39
|
+
cascade-proof), and both halves get fresh majority-bit pivots. Splitting and
|
|
40
|
+
routing use no randomness, and RaBitQ's rotation is seeded, so the index is a
|
|
41
|
+
pure function of the seed and the insertion sequence.
|
|
42
|
+
- **The same durability discipline as the rest of Sema:** WAL, batched
|
|
43
|
+
caller-owned transactions (`upsertMany`), 1 KiB pages, 64 MiB WAL
|
|
44
|
+
autocheckpoint.
|
|
42
45
|
|
|
43
46
|
## Usage
|
|
44
47
|
|
package/src/store.ts
CHANGED
|
@@ -136,11 +136,11 @@ export class BoundedMap<K, V> {
|
|
|
136
136
|
/** How a HIT records recency.
|
|
137
137
|
*
|
|
138
138
|
* `"reorder"` (default) promotes the entry to most-recent by
|
|
139
|
-
* `m.delete(k); m.set(k, v)` — exact LRU,
|
|
140
|
-
*
|
|
141
|
-
*
|
|
142
|
-
*
|
|
143
|
-
*
|
|
139
|
+
* `m.delete(k); m.set(k, v)` — exact LRU, the policy for any cache whose
|
|
140
|
+
* choice of victim must follow use exactly. `_depositTrees` (8 entries,
|
|
141
|
+
* feeds contentFoldIncremental) keeps it so the live conversation chains
|
|
142
|
+
* stay warm; its reuse is transparent, so a wrong victim costs a re-fold,
|
|
143
|
+
* never a different tree (fold-contract.md).
|
|
144
144
|
*
|
|
145
145
|
* `"clock"` records recency as a BIT instead of as position, spent by
|
|
146
146
|
* the eviction sweep (see `nextOldest`). Correct only for a TRANSPARENT
|