@hviana/sema 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +470 -0
- package/AUTHORS.md +12 -0
- package/COMMERCIAL-LICENSE.md +19 -0
- package/CONTRIBUTING.md +15 -0
- package/HOW_IT_WORKS.md +4210 -0
- package/LICENSE.md +117 -0
- package/README.md +290 -0
- package/TRADEMARKS.md +19 -0
- package/example/demo.ts +46 -0
- package/example/train_base.ts +2675 -0
- package/index.html +893 -0
- package/package.json +25 -0
- package/src/alphabet.ts +34 -0
- package/src/alu/README.md +332 -0
- package/src/alu/src/alu.ts +541 -0
- package/src/alu/src/expr.ts +339 -0
- package/src/alu/src/index.ts +115 -0
- package/src/alu/src/kernel-arith.ts +377 -0
- package/src/alu/src/kernel-bits.ts +199 -0
- package/src/alu/src/kernel-logic.ts +102 -0
- package/src/alu/src/kernel-nd.ts +235 -0
- package/src/alu/src/kernel-numeric.ts +466 -0
- package/src/alu/src/operation.ts +344 -0
- package/src/alu/src/parser.ts +574 -0
- package/src/alu/src/resonance.ts +161 -0
- package/src/alu/src/text.ts +83 -0
- package/src/alu/src/value.ts +327 -0
- package/src/alu/test/alu.test.ts +1004 -0
- package/src/bytes.ts +62 -0
- package/src/config.ts +227 -0
- package/src/derive/README.md +295 -0
- package/src/derive/src/deduction.ts +263 -0
- package/src/derive/src/index.ts +24 -0
- package/src/derive/src/priority-queue.ts +70 -0
- package/src/derive/src/rewrite.ts +127 -0
- package/src/derive/src/trie.ts +242 -0
- package/src/derive/test/derive.test.ts +137 -0
- package/src/extension.ts +42 -0
- package/src/geometry.ts +511 -0
- package/src/index.ts +38 -0
- package/src/ingest-cache.ts +224 -0
- package/src/mind/articulation.ts +137 -0
- package/src/mind/attention.ts +1061 -0
- package/src/mind/canonical.ts +107 -0
- package/src/mind/graph-search.ts +1100 -0
- package/src/mind/index.ts +18 -0
- package/src/mind/junction.ts +347 -0
- package/src/mind/learning.ts +246 -0
- package/src/mind/match.ts +507 -0
- package/src/mind/mechanisms/alu.ts +33 -0
- package/src/mind/mechanisms/cast.ts +568 -0
- package/src/mind/mechanisms/confluence.ts +268 -0
- package/src/mind/mechanisms/cover.ts +248 -0
- package/src/mind/mechanisms/extraction.ts +422 -0
- package/src/mind/mechanisms/recall.ts +241 -0
- package/src/mind/mind.ts +483 -0
- package/src/mind/pipeline-mechanism.ts +326 -0
- package/src/mind/pipeline.ts +300 -0
- package/src/mind/primitives.ts +201 -0
- package/src/mind/rationale.ts +275 -0
- package/src/mind/reasoning.ts +198 -0
- package/src/mind/recognition.ts +234 -0
- package/src/mind/resonance.ts +0 -0
- package/src/mind/trace.ts +108 -0
- package/src/mind/traverse.ts +556 -0
- package/src/mind/types.ts +281 -0
- package/src/rabitq-hnsw/README.md +303 -0
- package/src/rabitq-hnsw/src/database.ts +492 -0
- package/src/rabitq-hnsw/src/heap.ts +90 -0
- package/src/rabitq-hnsw/src/hnsw.ts +514 -0
- package/src/rabitq-hnsw/src/index.ts +15 -0
- package/src/rabitq-hnsw/src/prng.ts +39 -0
- package/src/rabitq-hnsw/src/rabitq.ts +308 -0
- package/src/rabitq-hnsw/src/store.ts +994 -0
- package/src/rabitq-hnsw/test/hnsw.test.ts +1213 -0
- package/src/store-sqlite.ts +928 -0
- package/src/store.ts +2148 -0
- package/src/vec.ts +124 -0
- package/test/00-extract.test.mjs +151 -0
- package/test/01-floor.test.mjs +20 -0
- package/test/02-roundtrip.test.mjs +83 -0
- package/test/03-recall.test.mjs +98 -0
- package/test/04-think.test.mjs +627 -0
- package/test/05-concepts.test.mjs +84 -0
- package/test/08-storage.test.mjs +948 -0
- package/test/09-edges.test.mjs +65 -0
- package/test/11-universality.test.mjs +180 -0
- package/test/13-conversation.test.mjs +228 -0
- package/test/14-scaling.test.mjs +503 -0
- package/test/15-decomposition-gap.test.mjs +0 -0
- package/test/16-bridge.test.mjs +250 -0
- package/test/17-intelligence.test.mjs +240 -0
- package/test/18-alu.test.mjs +209 -0
- package/test/19-nd.test.mjs +254 -0
- package/test/20-stability.test.mjs +489 -0
- package/test/21-partial.test.mjs +158 -0
- package/test/22-multihop.test.mjs +130 -0
- package/test/23-rationale.test.mjs +316 -0
- package/test/24-generalization.test.mjs +416 -0
- package/test/25-embedding.test.mjs +75 -0
- package/test/26-cooperative.test.mjs +89 -0
- package/test/27-saturation-drop.test.mjs +637 -0
- package/test/28-unknowable.test.mjs +88 -0
- package/test/29-counterfactual.test.mjs +476 -0
- package/test/30-conflict-resolution.test.mjs +166 -0
- package/test/31-audit.test.mjs +288 -0
- package/test/32-confluence.test.mjs +173 -0
- package/test/33-multi-candidate.test.mjs +222 -0
- package/test/34-cross-region.test.mjs +252 -0
- package/tsconfig.json +16 -0
package/HOW_IT_WORKS.md
ADDED
|
@@ -0,0 +1,4210 @@
|
|
|
1
|
+
# How Sema Works
|
|
2
|
+
|
|
3
|
+
**A complete account of the theory and the algorithm — from the mathematical
|
|
4
|
+
foundations to the full inference pipeline, in plain language.**
|
|
5
|
+
|
|
6
|
+
This document explains _concepts_ and _algorithms_. It deliberately contains no
|
|
7
|
+
repository-level detail (file layout, module names, build and test concerns):
|
|
8
|
+
for that, see [AGENTS.md](AGENTS.md), the development manual. Everything here is
|
|
9
|
+
stated so that a careful reader with no background in machine learning — human
|
|
10
|
+
or machine — can follow it from first principles.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Table of contents
|
|
15
|
+
|
|
16
|
+
- **Part I — Foundations**
|
|
17
|
+
- [1. What kind of AI is this?](#1-what-kind-of-ai-is-this)
|
|
18
|
+
- [2. Vector Symbolic Architectures](#2-vector-symbolic-architectures)
|
|
19
|
+
- [3. Content-addressable memory and the Merkle DAG](#3-content-addressable-memory-and-the-merkle-dag)
|
|
20
|
+
- [4. Distributional structure](#4-distributional-structure)
|
|
21
|
+
- [5. Automated deduction: the lightest derivation](#5-automated-deduction-the-lightest-derivation)
|
|
22
|
+
- [6. Approximate nearest-neighbour search](#6-approximate-nearest-neighbour-search)
|
|
23
|
+
- **Part II — The big picture**
|
|
24
|
+
- [7. How the five foundations connect](#7-how-the-five-foundations-connect)
|
|
25
|
+
- [8. Derived thresholds: the geometry of every decision](#8-derived-thresholds-the-geometry-of-every-decision)
|
|
26
|
+
- [9. The concept inventory](#9-the-concept-inventory)
|
|
27
|
+
- **Part III — The ingestion pipeline**
|
|
28
|
+
- [10. Perception: from bytes to a tree](#10-perception-from-bytes-to-a-tree)
|
|
29
|
+
- [11. Deposition: interning the tree into the graph](#11-deposition-interning-the-tree-into-the-graph)
|
|
30
|
+
- [12. Learning relations: edges and halos](#12-learning-relations-edges-and-halos)
|
|
31
|
+
- [13. Ingestion, end to end](#13-ingestion-end-to-end)
|
|
32
|
+
- **Part IV — The inference pipeline**
|
|
33
|
+
- [14. The shape of an answer](#14-the-shape-of-an-answer)
|
|
34
|
+
- [15. Recognition: decomposing the query](#15-recognition-decomposing-the-query)
|
|
35
|
+
- [16. Computation: extensions and the ALU](#16-computation-extensions-and-the-alu)
|
|
36
|
+
- [17. The consensus climb: points of attention](#17-the-consensus-climb-points-of-attention)
|
|
37
|
+
- [18. Grounding I — counterfactual transfer (CAST)](#18-grounding-i--counterfactual-transfer-cast)
|
|
38
|
+
- [19. Grounding II — cover: the graph search](#19-grounding-ii--cover-the-graph-search)
|
|
39
|
+
- [20. Grounding III — extraction by skill](#20-grounding-iii--extraction-by-skill)
|
|
40
|
+
- [21. Grounding IV — recall by resonance](#21-grounding-iv--recall-by-resonance)
|
|
41
|
+
- [22. Reasoning: the multi-hop chain](#22-reasoning-the-multi-hop-chain)
|
|
42
|
+
- [23. Fusion: multi-topic answers](#23-fusion-multi-topic-answers)
|
|
43
|
+
- [24. Articulation: answering in the asker's words](#24-articulation-answering-in-the-askers-words)
|
|
44
|
+
- [25. Disambiguation: choosing among alternatives](#25-disambiguation-choosing-among-alternatives)
|
|
45
|
+
- [26. Auditability: provenance and the rationale](#26-auditability-provenance-and-the-rationale)
|
|
46
|
+
- **Part V — The whole algorithm in pseudocode**
|
|
47
|
+
- [27. End-to-end pseudocode](#27-end-to-end-pseudocode)
|
|
48
|
+
- **Part VI — Reference**
|
|
49
|
+
- [28. Glossary](#28-glossary)
|
|
50
|
+
- [29. Complexity summary](#29-complexity-summary)
|
|
51
|
+
- [30. Bibliography](#30-bibliography)
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
# Part I — Foundations
|
|
58
|
+
|
|
59
|
+
Sema rests on five independent bodies of theory, each decades old and each well
|
|
60
|
+
established in the academic literature. None of them is a neural network, and
|
|
61
|
+
none of them requires training in the gradient-descent sense. Part I presents
|
|
62
|
+
each one on its own terms; Part II shows how they interlock.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## 1. What kind of AI is this?
|
|
67
|
+
|
|
68
|
+
### 1.1 The classification problem
|
|
69
|
+
|
|
70
|
+
Sema is not a large language model: it has no learned weight matrices, no
|
|
71
|
+
gradient descent, no probabilistic sampling. But calling it simply "symbolic AI"
|
|
72
|
+
is also inaccurate — classical symbolic AI (logic programming, production
|
|
73
|
+
systems, description logics) operates on discrete tokens with exact matching and
|
|
74
|
+
has no notion of _similarity_, _generalization by proximity_, or _graceful
|
|
75
|
+
degradation_, all of which Sema exhibits. And industry labels such as "RAG +
|
|
76
|
+
Reasoner" describe _pipelines of separate components_ (a retriever feeding a
|
|
77
|
+
generator), which misrepresents Sema's single-mechanism design and has no
|
|
78
|
+
standing in the academic literature as a category of system.
|
|
79
|
+
|
|
80
|
+
The academically precise classification is:
|
|
81
|
+
|
|
82
|
+
> **Sema is a non-parametric, instance-based reasoning system: a Vector Symbolic
|
|
83
|
+
> Architecture (VSA) coupled to a content-addressable memory, with inference
|
|
84
|
+
> performed by weighted automated deduction.**
|
|
85
|
+
|
|
86
|
+
Each term in that sentence is a recognised concept with its own literature:
|
|
87
|
+
|
|
88
|
+
| Term | Meaning | Field |
|
|
89
|
+
| :-------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- |
|
|
90
|
+
| **Non-parametric** | The system has no fixed-size parameter vector that training adjusts. Capacity grows with the data itself; the stored instances _are_ the model. | Statistics; machine learning |
|
|
91
|
+
| **Instance-based** (memory-based) | Learning is storing experiences; generalization happens at _query time_ by comparing the query to stored instances, not at training time by fitting a function. | Machine learning; cognitive science exemplar theory |
|
|
92
|
+
| **Vector Symbolic Architecture** | Structured knowledge (sequences, trees, role–filler bindings) is encoded into fixed-width high-dimensional vectors using algebraic operations (binding, superposition), so that _similarity of structure_ becomes _proximity of vectors_. | Connectionist/cognitive modelling (Plate 1995; Kanerva 2009; Gayler 2003) |
|
|
93
|
+
| **Content-addressable memory** | Items are retrieved by _what they are_ (their content or something similar to it), not by _where they are_ (an address or key assigned externally). | Computer architecture and associative-memory theory |
|
|
94
|
+
| **Weighted automated deduction** | Answers are _derived_ by applying inference rules with costs, and the system returns the derivation of minimal total cost — a strict generalization of shortest-path search. | Automated reasoning; parsing theory (Knuth 1977; Goodman 1999; Felzenszwalb & McAllester 2007) |
|
|
95
|
+
|
|
96
|
+
### 1.2 What this system is _not_
|
|
97
|
+
|
|
98
|
+
To sharpen the category, the contrasts matter as much as the definition:
|
|
99
|
+
|
|
100
|
+
- **Not a parametric statistical model.** An LLM compresses its corpus into a
|
|
101
|
+
fixed number of floating-point weights and answers by sampling from a learned
|
|
102
|
+
conditional distribution. Sema stores its corpus as an explicit, losslessly
|
|
103
|
+
reconstructible graph and answers by _deduction over that graph_. There is no
|
|
104
|
+
distribution to sample; identical inputs always produce identical outputs.
|
|
105
|
+
|
|
106
|
+
- **Not classical (crisp) symbolic AI.** In a Prolog-style system, `mona_lisa`
|
|
107
|
+
and `monalisa` are unrelated atoms; nothing matches unless it unifies exactly.
|
|
108
|
+
In Sema, every stored structure also carries a high-dimensional vector (its
|
|
109
|
+
_gist_), and geometric proximity of gists gives the system the soft matching,
|
|
110
|
+
analogy, and noise tolerance that crisp symbols lack — while the underlying
|
|
111
|
+
identities remain exact and content-addressed.
|
|
112
|
+
|
|
113
|
+
- **Not a retrieval pipeline with a bolted-on reasoner.** In a
|
|
114
|
+
retrieval-augmented generation system, retrieval and generation are two
|
|
115
|
+
different mechanisms with an interface between them. In Sema there is one
|
|
116
|
+
mechanism: a single lightest-derivation search whose _axioms_ come from
|
|
117
|
+
recognition and resonance (the "retrieval") and whose _rules_ are the learned
|
|
118
|
+
edges and compositions (the "reasoning"). Retrieval and inference are two
|
|
119
|
+
descriptions of one search.
|
|
120
|
+
|
|
121
|
+
- **Not neuro-symbolic in the usual sense.** The term "neuro-symbolic" usually
|
|
122
|
+
denotes hybrids of neural networks with symbolic components. Sema contains no
|
|
123
|
+
neural network. Its vectors are constructed by deterministic algebra (random
|
|
124
|
+
projections, permutations, sums), not learned by backpropagation. The correct
|
|
125
|
+
lineage for its vector side is _hyperdimensional computing / VSA_, which grew
|
|
126
|
+
out of connectionism but does not require learning.
|
|
127
|
+
|
|
128
|
+
### 1.3 Where it sits in the literature
|
|
129
|
+
|
|
130
|
+
Systems close in spirit, each sharing one facet:
|
|
131
|
+
|
|
132
|
+
- **Holographic Reduced Representations** (Plate 1995) and **hyperdimensional
|
|
133
|
+
computing** (Kanerva 2009): the representational substrate. Sema shares the
|
|
134
|
+
algebra (superposition + binding); it differs by anchoring every vector to an
|
|
135
|
+
exact, reconstructible symbolic structure in a Merkle DAG.
|
|
136
|
+
- **Semiring-weighted deduction** (Goodman 1999) and the **Generalized A\***
|
|
137
|
+
architecture (Felzenszwalb & McAllester 2007): the inference engine. Sema
|
|
138
|
+
shares the formalism exactly; its contribution is _what_ the items and rules
|
|
139
|
+
are — spans of a query, nodes of a learned graph, learned continuations.
|
|
140
|
+
- **Case-based reasoning** (Kolodner 1992): solve new problems by retrieving and
|
|
141
|
+
adapting stored cases. Sema's counterfactual-transfer mechanism (Section 18)
|
|
142
|
+
is a formalized, byte-level version of case adaptation.
|
|
143
|
+
|
|
144
|
+
The short label used in the rest of this document: **a vector-symbolic,
|
|
145
|
+
memory-based reasoning system**.
|
|
146
|
+
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
## 2. Vector Symbolic Architectures
|
|
150
|
+
|
|
151
|
+
### 2.1 The problem VSAs solve
|
|
152
|
+
|
|
153
|
+
Fixed-width vectors are attractive as a representation: comparing two of them
|
|
154
|
+
(one dot product) is trivially fast, and "similar" has a natural meaning (small
|
|
155
|
+
angle between them). But a bag of numbers has no obvious way to represent
|
|
156
|
+
_structure_ — the difference between "the dog bit the man" and "the man bit the
|
|
157
|
+
dog" is not a difference in _which_ words occur but in _how they are arranged_.
|
|
158
|
+
|
|
159
|
+
A **Vector Symbolic Architecture** (VSA; also _hyperdimensional computing_) is
|
|
160
|
+
an algebra over high-dimensional vectors that solves exactly this problem. It
|
|
161
|
+
provides:
|
|
162
|
+
|
|
163
|
+
1. **Atoms.** Elementary symbols are assigned (usually random) high-dimensional
|
|
164
|
+
vectors.
|
|
165
|
+
2. **Superposition** (bundling): a way to combine several vectors into one
|
|
166
|
+
vector that is _similar to each of its inputs_ — typically element-wise
|
|
167
|
+
addition followed by normalization. Superposition represents _sets_: the
|
|
168
|
+
result "contains" its inputs in the sense that each input can be detected in
|
|
169
|
+
it by a similarity test.
|
|
170
|
+
3. **Binding**: a way to combine vectors into one that is _dissimilar to its
|
|
171
|
+
inputs_ — it represents an _association_ (a role filled by a filler, a
|
|
172
|
+
position occupied by an item) rather than a collection. Crucially, binding is
|
|
173
|
+
invertible (or at least, structurally distinguishable), so bound structure
|
|
174
|
+
can be probed.
|
|
175
|
+
4. **A similarity measure**, usually cosine similarity, under which all of the
|
|
176
|
+
above has meaning.
|
|
177
|
+
|
|
178
|
+
Different VSA families use different binding operators: circular convolution in
|
|
179
|
+
Plate's Holographic Reduced Representations (1995), element-wise XOR in
|
|
180
|
+
Kanerva's binary spatter codes, element-wise multiplication in Gayler's
|
|
181
|
+
Multiply–Add–Permute (2003). Sema uses **permutation binding**: applying a fixed
|
|
182
|
+
random permutation of the vector's coordinates.
|
|
183
|
+
|
|
184
|
+
### 2.2 Why high dimensions work: quasi-orthogonality
|
|
185
|
+
|
|
186
|
+
The entire edifice rests on a fact of high-dimensional geometry sometimes called
|
|
187
|
+
_concentration of measure_: **two independently chosen random unit vectors in D
|
|
188
|
+
dimensions are almost exactly orthogonal**. Their expected cosine similarity is
|
|
189
|
+
0, with standard deviation approximately 1/√D. At D = 1024, that standard
|
|
190
|
+
deviation is about 0.031 — so unrelated random vectors reliably score within a
|
|
191
|
+
few hundredths of zero, while a vector scores 1.0 against itself. There is an
|
|
192
|
+
enormous, dependable gap between "same" and "unrelated", and an entire usable
|
|
193
|
+
band in between for "partially similar".
|
|
194
|
+
|
|
195
|
+
This has three consequences that Sema uses constantly:
|
|
196
|
+
|
|
197
|
+
- **Capacity.** A superposition of a handful of random vectors is still clearly
|
|
198
|
+
similar to each of them and clearly dissimilar to everything else, because the
|
|
199
|
+
cross-talk between the components is O(1/√D) noise.
|
|
200
|
+
- **Statistical decision thresholds.** "Is this similarity real or chance?" has
|
|
201
|
+
a principled answer: a cosine of k/√D is k standard deviations above what
|
|
202
|
+
chance produces. Sema's significance bar is exactly 3/√D — three sigma
|
|
203
|
+
(Section 8).
|
|
204
|
+
- **Robustness.** Corrupting a few coordinates of a high-dimensional vector
|
|
205
|
+
barely moves it; every comparison degrades gracefully rather than breaking.
|
|
206
|
+
|
|
207
|
+
### 2.3 Permutation binding and why order becomes visible
|
|
208
|
+
|
|
209
|
+
A permutation π rearranges a vector's coordinates: the value at position π(i)
|
|
210
|
+
moves to position i. Two properties make permutations excellent binding
|
|
211
|
+
operators:
|
|
212
|
+
|
|
213
|
+
- **They preserve lengths and angles** (they are orthogonal linear maps), so
|
|
214
|
+
permuting a vector produces an equally well-behaved vector.
|
|
215
|
+
- **A random permutation decorrelates.** For a random vector v, the permuted
|
|
216
|
+
vector πv is (with overwhelming probability) nearly orthogonal to v — the
|
|
217
|
+
permutation "hides" the vector's identity behind the role.
|
|
218
|
+
|
|
219
|
+
Sema keeps a fixed **keyring** of independent random permutations π₀, π₁, π₂, …
|
|
220
|
+
— one per _seat_ (ordinal position in a group). To encode an ordered group of
|
|
221
|
+
children (c₀, c₁, …, cₖ), each child's vector is bound to its seat and the
|
|
222
|
+
results are superposed:
|
|
223
|
+
|
|
224
|
+
```
|
|
225
|
+
encode(c₀, c₁, …, cₖ) = π₀·v(c₀) + π₁·v(c₁) + … + πₖ·v(cₖ)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Because the seats are _different_ permutations, "A in seat 0, B in seat 1" and
|
|
229
|
+
"B in seat 0, A in seat 1" produce nearly orthogonal encodings — **order is part
|
|
230
|
+
of the representation**. And because independent permutations do not commute,
|
|
231
|
+
nesting the operation encodes _paths_: "the x that sits in seat 2 of the thing
|
|
232
|
+
in seat 1" has a distinct signature from "the x in seat 1 of the thing in seat
|
|
233
|
+
2". A whole tree can thus be folded, level by level, into one fixed-width vector
|
|
234
|
+
whose geometry reflects the tree's entire shape and content. Sema calls the
|
|
235
|
+
result of this fold the tree's **gist**.
|
|
236
|
+
|
|
237
|
+
Note that this encoding step is **not** followed by a normalize: unlike the
|
|
238
|
+
classical VSA recipe (which renormalizes after every superposition), Sema's fold
|
|
239
|
+
leaves every interior sum at its natural length and only ever normalizes the
|
|
240
|
+
_root_ of a fold — see §2.6, where this is the basis of the system's
|
|
241
|
+
angle-and-magnitude semantics.
|
|
242
|
+
|
|
243
|
+
The trade this makes is deliberate and important to understand: the fold is
|
|
244
|
+
**lossy** as a vector (a 1024-dimensional gist cannot losslessly hold a kilobyte
|
|
245
|
+
of text), but Sema never needs to _decode_ a gist — the exact content is always
|
|
246
|
+
recoverable from the symbolic side (the DAG, Section 3). Gists exist purely to
|
|
247
|
+
make _similarity of structured content computable in one dot product_. This
|
|
248
|
+
division of labour — vectors for geometry, the DAG for identity — is the single
|
|
249
|
+
most important design fact in Sema.
|
|
250
|
+
|
|
251
|
+
### 2.4 The alphabet: atoms with graded similarity
|
|
252
|
+
|
|
253
|
+
The atoms of Sema's VSA are the 256 possible byte values. Each byte value is
|
|
254
|
+
assigned a fixed unit vector at construction time, deterministically from a
|
|
255
|
+
seed. But the assignment is not uniformly random: the 256 vectors are built by
|
|
256
|
+
**recursive refinement** — 16 coarse random directions are each refined into 4
|
|
257
|
+
intermediate directions, each of which is refined into 4 final directions (16 →
|
|
258
|
+
64 → 256). Refinement means: keep a weighted portion of the parent direction and
|
|
259
|
+
mix in fresh randomness.
|
|
260
|
+
|
|
261
|
+
The result is an alphabet with _graded similarity_: byte values that share a
|
|
262
|
+
refinement ancestor have moderately similar vectors, while distant byte values
|
|
263
|
+
are quasi-orthogonal. This gives perception a mild, structured tolerance at the
|
|
264
|
+
very lowest level, while preserving the global quasi-orthogonality the algebra
|
|
265
|
+
needs. (The mixing ratio is the alphabet's _roughness_; the construction is
|
|
266
|
+
deterministic given the seed, which is what makes all of Sema reproducible.)
|
|
267
|
+
|
|
268
|
+
### 2.5 What the VSA contributes to Sema, in one sentence
|
|
269
|
+
|
|
270
|
+
> The VSA turns "how similar are these two _structures_?" into "what is the
|
|
271
|
+
> cosine of these two vectors?" — a question answerable in microseconds and
|
|
272
|
+
> indexable at scale — without ever being trusted to _store_ the structures
|
|
273
|
+
> themselves.
|
|
274
|
+
|
|
275
|
+
Sema's word for cosine similarity between gists is **resonance**, and this
|
|
276
|
+
document uses both terms interchangeably.
|
|
277
|
+
|
|
278
|
+
### 2.6 Magnitude: the second axis of similarity
|
|
279
|
+
|
|
280
|
+
A fold whose interior sums are never renormalized (§2.3) carries more
|
|
281
|
+
information than its angle alone. Because the leaf vectors it superposes are
|
|
282
|
+
close to orthogonal, the length of an unnormalized interior gist grows with the
|
|
283
|
+
_amount_ of content folded into it: a span of `len` bytes has a gist whose norm
|
|
284
|
+
is approximately √len. Only the root of a completed fold is normalized to unit
|
|
285
|
+
length (so that content and halo indexes, which compare roots, still compare
|
|
286
|
+
directions on the unit sphere); every gist below the root retains this natural,
|
|
287
|
+
byte-proportional magnitude.
|
|
288
|
+
|
|
289
|
+
This turns the fold into what the codebase calls a **linear** fold: it is a
|
|
290
|
+
genuine linear operator (superposition of seat-bound leaf vectors, nothing
|
|
291
|
+
else), and a resonance score between two such unnormalized quantities reads as
|
|
292
|
+
**byte-proportional overlap**, not scale-free cosine. Concretely: the cosine
|
|
293
|
+
between two spans' gists is (shared content) / √(len₁ · len₂) — the geometric
|
|
294
|
+
mean of their lengths in the denominator, exactly as an inner product between
|
|
295
|
+
two sums of near-orthogonal unit vectors predicts. Reading a magnitude back out
|
|
296
|
+
of a stored gist is therefore reading a byte count: a node's gist norm (or,
|
|
297
|
+
equivalently, its stored content length) IS its size in the fold's own units.
|
|
298
|
+
|
|
299
|
+
This is Sema's **angle-and-magnitude** semantics: the ANGLE between two gists
|
|
300
|
+
carries the _fraction_ of content they share; the MAGNITUDE (recovered from the
|
|
301
|
+
stored span length) converts that fraction into an absolute count of shared
|
|
302
|
+
bytes. Every mechanism that must ask a scale-sensitive question — "does this
|
|
303
|
+
score mean the query is entirely explained, or merely brushed by a much larger
|
|
304
|
+
stored form?" — reads both axes explicitly rather than trusting the cosine
|
|
305
|
+
alone. Two examples used constantly in Part IV:
|
|
306
|
+
|
|
307
|
+
- The **identity bar** for a whole-span claim (§8.1) tightens as the span grows,
|
|
308
|
+
because a fixed cosine tolerates a growing number of foreign bytes once the
|
|
309
|
+
span is long — the magnitude correction keeps "near-identical" meaning the
|
|
310
|
+
same absolute thing at every scale.
|
|
311
|
+
- **Recall's last-resort tier** (§21) and the **consensus climb's** per-region
|
|
312
|
+
vote (§17.3) both convert a raw resonance score into a _query-relative
|
|
313
|
+
fraction_ — how much of the smaller side the larger side's content actually
|
|
314
|
+
accounts for — using exactly this norm-as-byte-count reading, rather than
|
|
315
|
+
trusting the raw cosine, which conflates "small thing fully inside a big
|
|
316
|
+
thing" with "big thing loosely touching a small one".
|
|
317
|
+
|
|
318
|
+
Nothing about content addressing or the DAG changes: identity is still decided
|
|
319
|
+
by exact, byte-verified lookup (§3.1), never by a magnitude reading. Magnitude
|
|
320
|
+
is strictly a refinement of the _geometric_ half of the system — it makes
|
|
321
|
+
resonance scores honest about scale, it does not replace the exact half.
|
|
322
|
+
|
|
323
|
+
---
|
|
324
|
+
|
|
325
|
+
## 3. Content-addressable memory and the Merkle DAG
|
|
326
|
+
|
|
327
|
+
### 3.1 Content addressing
|
|
328
|
+
|
|
329
|
+
In a conventional memory, an item lives at an _address_, and you must know the
|
|
330
|
+
address to retrieve the item. In a **content-addressable memory** (CAM), the
|
|
331
|
+
item's own content determines where it is: to ask "do I know this?" you present
|
|
332
|
+
the content itself, and the memory answers with the stored item (or its
|
|
333
|
+
identity) directly. Associative memories of this kind have been studied since
|
|
334
|
+
the earliest days of computer architecture.
|
|
335
|
+
|
|
336
|
+
Sema's long-term store is content-addressed in the strict sense: **a node's
|
|
337
|
+
identity is a pure function of its content**. Storing the same content twice
|
|
338
|
+
yields the same node, always. This gives three properties for free:
|
|
339
|
+
|
|
340
|
+
- **Idempotent learning.** Re-ingesting a document changes nothing; there is no
|
|
341
|
+
duplicate to create.
|
|
342
|
+
- **Intrinsic deduplication.** Storage grows with _distinct_ content, not with
|
|
343
|
+
volume. A phrase seen a million times occupies one node.
|
|
344
|
+
- **Exact identity tests.** "Is this span something I have seen?" is a lookup,
|
|
345
|
+
not a similarity estimate. Sema leans on this constantly: soft (vector)
|
|
346
|
+
evidence _suggests_, but identity decisions are always made by
|
|
347
|
+
content-addressed lookup, never by a similarity score crossing 1.0.
|
|
348
|
+
|
|
349
|
+
### 3.2 The Merkle DAG
|
|
350
|
+
|
|
351
|
+
A **Merkle structure** (Merkle 1987) is one in which a composite object's
|
|
352
|
+
identifier is derived from the identifiers of its parts. If two composites have
|
|
353
|
+
the same parts in the same arrangement, they _are_ the same object. Applied to
|
|
354
|
+
trees this is also known in programming-language circles as **hash-consing**:
|
|
355
|
+
construct each node "modulo equality", so structurally equal subtrees are
|
|
356
|
+
physically shared.
|
|
357
|
+
|
|
358
|
+
Sema's memory is a Merkle **DAG** (directed acyclic graph) of nodes:
|
|
359
|
+
|
|
360
|
+
- A **leaf** node is a span of raw bytes.
|
|
361
|
+
- A **branch** node is an ordered list of child node identities.
|
|
362
|
+
- A node's identity is determined by exactly that content (bytes, or the ordered
|
|
363
|
+
child list). Equal content ⇒ same node.
|
|
364
|
+
|
|
365
|
+
Because identical subtrees collapse into one shared node, the "tree" of any
|
|
366
|
+
single perceived input becomes, in storage, a subgraph woven into every other
|
|
367
|
+
input that shares material with it. A sentence deposited yesterday and a
|
|
368
|
+
paragraph deposited today that contains that sentence _share the sentence's
|
|
369
|
+
node_; the paragraph's branch simply points at it. Three consequences:
|
|
370
|
+
|
|
371
|
+
1. **Every span ever perceived is individually addressable.** Not just whole
|
|
372
|
+
documents: every intermediate grouping the perception process produced is a
|
|
373
|
+
node with an identity, and can be the target of an association or a
|
|
374
|
+
similarity probe.
|
|
375
|
+
2. **The graph can be climbed.** From any node one can ask "which larger
|
|
376
|
+
structures contain me?" (its _parents_) — the reverse of the child lists.
|
|
377
|
+
Recognising a fragment of an experience thus gives a path _upward_ to the
|
|
378
|
+
whole experiences that contain it. This upward climb is the backbone of
|
|
379
|
+
Sema's attention mechanism (Section 17).
|
|
380
|
+
3. **Reconstruction is exact.** Concatenating a node's leaves, left to right,
|
|
381
|
+
reproduces the original bytes losslessly. The memory is not an approximation
|
|
382
|
+
of the corpus; it _is_ the corpus, shared and structured.
|
|
383
|
+
|
|
384
|
+
### 3.3 The marriage of CAM and VSA
|
|
385
|
+
|
|
386
|
+
Each node carries both halves of Sema's dual representation:
|
|
387
|
+
|
|
388
|
+
| Facet | Representation | Answers |
|
|
389
|
+
| :----------- | :------------------------------------------------- | :------------------------------------------------------------------------ |
|
|
390
|
+
| **Identity** | content-addressed node in the Merkle DAG | "Is this exactly something I know? What contains it? What are its parts?" |
|
|
391
|
+
| **Geometry** | the gist vector (the VSA fold of the same content) | "What does this _resemble_? How strongly?" |
|
|
392
|
+
|
|
393
|
+
The store maintains a vector index over gists (Section 6) so that "find the
|
|
394
|
+
stored nodes most similar to this vector" is fast. The two facets discipline
|
|
395
|
+
each other: geometric search proposes candidates cheaply; content-addressed
|
|
396
|
+
structure verifies and grounds them exactly. Nothing in Sema ever acts on a
|
|
397
|
+
similarity score alone when an exact structural check is available — a rule
|
|
398
|
+
stated once here and honoured throughout the pipeline.
|
|
399
|
+
|
|
400
|
+
One refinement deserves mention because it is conceptually load-bearing:
|
|
401
|
+
**near-deduplication is byte-verified**. When a freshly perceived experience is
|
|
402
|
+
geometrically almost identical to one just stored (cosine above the _merge
|
|
403
|
+
threshold_, Section 8), Sema considers treating them as the same node — but
|
|
404
|
+
geometric closeness alone is scale-blind, so the decision is made by the bytes:
|
|
405
|
+
the two contents must be identical except for **one local span no wider than the
|
|
406
|
+
perception window**. Geometric evidence proposes; bytes dispose.
|
|
407
|
+
|
|
408
|
+
---
|
|
409
|
+
|
|
410
|
+
## 4. Distributional structure
|
|
411
|
+
|
|
412
|
+
### 4.1 The distributional hypothesis
|
|
413
|
+
|
|
414
|
+
The **distributional hypothesis** (Harris 1954) — often summarized as "you shall
|
|
415
|
+
know a word by the company it keeps" — holds that linguistic items with similar
|
|
416
|
+
meanings occur in similar contexts — that _meaning_, to a useful approximation,
|
|
417
|
+
is _distribution of use_. It is the theoretical foundation of every modern
|
|
418
|
+
word-embedding method, but the hypothesis itself is prior to and independent of
|
|
419
|
+
neural networks: it is a claim about language, testable by counting.
|
|
420
|
+
|
|
421
|
+
Sema implements the distributional hypothesis directly and transparently.
|
|
422
|
+
Alongside its gist (which encodes _what a node is made of_), a node that takes
|
|
423
|
+
part in learned associations accumulates a second vector: its **halo** — a
|
|
424
|
+
superposition of **company signatures** of the _partners it appeared with_ (what
|
|
425
|
+
preceded it, what followed it, bound to a role seat so that "appears as context"
|
|
426
|
+
and "appears as answer" are distinguishable). The halo encodes _the company the
|
|
427
|
+
node keeps_.
|
|
428
|
+
|
|
429
|
+
A company signature is a deterministic unit vector derived from the partner's
|
|
430
|
+
**node identity** (a seeded function of the node id), not from the partner's
|
|
431
|
+
gist. This decouples content similarity from company similarity: two halos
|
|
432
|
+
correlate exactly as much as their episode-participation histories overlap,
|
|
433
|
+
never because their partners merely contain similar bytes. Pouring raw partner
|
|
434
|
+
gists instead would let any byte-overlap between partners leak _content_
|
|
435
|
+
similarity into _distributional_ similarity, silently shifting the halo null
|
|
436
|
+
model that the concept threshold's derivation (unrelated halos ⇒ cosine 0 ±
|
|
437
|
+
1/√D) depends on.
|
|
438
|
+
|
|
439
|
+
Two nodes whose halos are similar have occurred in similar circumstances — they
|
|
440
|
+
are **distributional siblings**: synonyms, paraphrases, items of the same
|
|
441
|
+
category, two names for one thing. Note the complementarity:
|
|
442
|
+
|
|
443
|
+
| Vector | Encodes | Two nodes are close when… |
|
|
444
|
+
| :------- | :----------------------------------- | :--------------------------------------------------------------------------- |
|
|
445
|
+
| **Gist** | the node's own content and structure | they are _made of_ similar material ("colour" ≈ "colours") |
|
|
446
|
+
| **Halo** | the node's contexts of use | they are _used_ the same way ("colour" ≈ "hue"), even with zero shared bytes |
|
|
447
|
+
|
|
448
|
+
### 4.2 What halos do in the pipeline
|
|
449
|
+
|
|
450
|
+
Halos give Sema its capacity for synonymy and analogy without any trained
|
|
451
|
+
embedding model:
|
|
452
|
+
|
|
453
|
+
- **Concept hops.** A recognised form that has no learned continuation of its
|
|
454
|
+
own can borrow the continuation of a distributional sibling — the system
|
|
455
|
+
answers about "hue" using what it learned about "colour" (Section 19's concept
|
|
456
|
+
rule).
|
|
457
|
+
- **Articulation.** An answer is re-voiced in the asker's own vocabulary by
|
|
458
|
+
substituting answer forms with query forms that share a halo (Section 24).
|
|
459
|
+
- **Analogy strength.** Whether two entities are genuinely analogous — the gate
|
|
460
|
+
on counterfactual comparison (Section 18) — is measured by halo similarity,
|
|
461
|
+
directly or through shared siblings (a second-order distributional test).
|
|
462
|
+
- **Evidence weight.** How many episodes poured into a node's halo (its _mass_)
|
|
463
|
+
is a direct count of distributional corroboration, consulted when choosing
|
|
464
|
+
among competing continuations (Section 25).
|
|
465
|
+
|
|
466
|
+
Like gists, halos live in a vector index of their own so that "which nodes keep
|
|
467
|
+
this kind of company?" is a fast query.
|
|
468
|
+
|
|
469
|
+
### 4.3 A note on scientific hygiene
|
|
470
|
+
|
|
471
|
+
Because the halo is an explicit superposition of explicit episode signatures,
|
|
472
|
+
distributional claims in Sema are _auditable_: one can enumerate exactly which
|
|
473
|
+
learning events contributed to a halo and with what role. This distinguishes
|
|
474
|
+
Sema's distributional layer from learned embeddings, whose geometry is real but
|
|
475
|
+
whose provenance is diffused across an entire training run.
|
|
476
|
+
|
|
477
|
+
---
|
|
478
|
+
|
|
479
|
+
## 5. Automated deduction: the lightest derivation
|
|
480
|
+
|
|
481
|
+
### 5.1 Weighted deduction systems
|
|
482
|
+
|
|
483
|
+
**Automated deduction** is the field concerned with deriving conclusions from
|
|
484
|
+
premises by mechanical application of inference rules. A **weighted deduction
|
|
485
|
+
system** attaches a non-negative cost to each rule application:
|
|
486
|
+
|
|
487
|
+
```
|
|
488
|
+
premise₁ ∧ premise₂ ∧ … ∧ premiseₖ --(cost c)--> conclusion
|
|
489
|
+
```
|
|
490
|
+
|
|
491
|
+
A **derivation** of an item is a proof tree: leaves are axioms, and each
|
|
492
|
+
internal node is a rule application whose children derive its premises. The
|
|
493
|
+
derivation's cost is the sum of the costs of the rules it uses. The **lightest
|
|
494
|
+
derivation** of a goal item is the derivation of minimal total cost. This
|
|
495
|
+
formalism, developed principally in parsing theory (Goodman 1999 gave the
|
|
496
|
+
general semiring formulation), strictly generalizes shortest-path search: a
|
|
497
|
+
graph is the special case in which every rule has exactly one premise.
|
|
498
|
+
|
|
499
|
+
Rules with _multiple_ premises are what make the formalism powerful. A
|
|
500
|
+
two-premise rule is a **conjunction**: it composes two independently derived
|
|
501
|
+
results into one, paying a join cost. The search space is therefore an AND/OR
|
|
502
|
+
**hypergraph**, not a graph — and finding the lightest derivation is the
|
|
503
|
+
hypergraph analogue of finding a shortest path.
|
|
504
|
+
|
|
505
|
+
### 5.2 Knuth's algorithm and the A\* generalization
|
|
506
|
+
|
|
507
|
+
Knuth (1977) showed that Dijkstra's algorithm generalizes from graphs to
|
|
508
|
+
weighted deduction: process items in order of cost; when an item is removed from
|
|
509
|
+
the priority queue, its cost is final (given non-negative, monotone rules).
|
|
510
|
+
Felzenszwalb & McAllester (2007) then generalized A\* the same way — **A\*
|
|
511
|
+
Lightest Derivation (A\*LD)**: if an admissible heuristic (a lower bound on the
|
|
512
|
+
cost remaining from an item to the goal) is available, the queue is ordered by
|
|
513
|
+
_cost so far + lower bound_, and provably no item is expanded whose lightest
|
|
514
|
+
derivation costs more than the goal's. The search is **output-sensitive**: its
|
|
515
|
+
work is proportional to the answer, not to the size of the (implicit,
|
|
516
|
+
potentially enormous) space of derivations.
|
|
517
|
+
|
|
518
|
+
Sema's inference engine is an implementation of A\*LD, with one deliberate
|
|
519
|
+
extension described next. Four standard disciplines keep it tractable:
|
|
520
|
+
|
|
521
|
+
1. **Chart memoization** — equivalent partial derivations collapse to one
|
|
522
|
+
canonical entry (the cheapest).
|
|
523
|
+
2. **Lazy rule generation** — rules are enumerated only when one of their
|
|
524
|
+
premises has been finalised, never up front.
|
|
525
|
+
3. **Demand filtering** — rules whose conclusions cannot reach the goal are
|
|
526
|
+
never emitted.
|
|
527
|
+
4. **Admissible heuristic pruning** — the A\* bound keeps the frontier focused
|
|
528
|
+
on the goal.
|
|
529
|
+
|
|
530
|
+
### 5.3 The semiring extension: evidence pooling
|
|
531
|
+
|
|
532
|
+
Classic lightest-derivation search operates in the **tropical semiring** (min,
|
|
533
|
+
+): among competing derivations of the same conclusion, only the cheapest
|
|
534
|
+
survives. That is the right regime for _choosing_ — one best answer, one best
|
|
535
|
+
parse.
|
|
536
|
+
|
|
537
|
+
But some of Sema's decisions are not choices; they are _accumulations of
|
|
538
|
+
evidence_. When several independent regions of a query each independently point
|
|
539
|
+
at the same stored fact, the fact should be credited with the _sum_ of their
|
|
540
|
+
support, not merely the strongest single vote. For those decisions the engine
|
|
541
|
+
supports a second combining mode operating in the **arithmetic semiring** (+,
|
|
542
|
+
+): every derivation of a marked conclusion _adds_ its cost into a pooled total,
|
|
543
|
+
and every contribution is recorded rather than discarded. Semiring-general
|
|
544
|
+
deduction is standard theory (Goodman 1999); running both regimes in one search
|
|
545
|
+
— minimum-cost for structure, sum for evidence — is how Sema keeps consensus
|
|
546
|
+
formation (Section 17) inside the same formal system as everything else, rather
|
|
547
|
+
than as an ad-hoc tally alongside it.
|
|
548
|
+
|
|
549
|
+
### 5.4 Why deduction, and not generation
|
|
550
|
+
|
|
551
|
+
The choice of weighted deduction as the inference engine is what makes Sema's
|
|
552
|
+
central claims true rather than aspirational:
|
|
553
|
+
|
|
554
|
+
- **Auditability.** The answer _is_ a proof tree. Every byte of output is the
|
|
555
|
+
conclusion of an explicit chain of rule applications over explicit stored
|
|
556
|
+
facts, and that chain can be read back (Section 26).
|
|
557
|
+
- **Determinism.** Lightest derivation is an optimization with a well-defined
|
|
558
|
+
optimum (ties broken by fixed conventions), not a sample from a distribution.
|
|
559
|
+
- **Honest silence.** If no derivation of the goal exists, the search returns
|
|
560
|
+
nothing. The system cannot "make something up": fabrication is not expressible
|
|
561
|
+
in the formalism.
|
|
562
|
+
|
|
563
|
+
---
|
|
564
|
+
|
|
565
|
+
## 6. Approximate nearest-neighbour search
|
|
566
|
+
|
|
567
|
+
### 6.1 The role of ANN search
|
|
568
|
+
|
|
569
|
+
Both of Sema's vector relations — gists and halos — need the same primitive:
|
|
570
|
+
_given a query vector, find the k stored vectors with highest cosine
|
|
571
|
+
similarity_, over a store that may hold millions of vectors, in milliseconds, on
|
|
572
|
+
a CPU, without holding everything in RAM. Exact search is linear in the
|
|
573
|
+
collection size; **approximate nearest-neighbour (ANN)** search trades a small,
|
|
574
|
+
controlled amount of recall for sub-linear query time.
|
|
575
|
+
|
|
576
|
+
Sema uses two established techniques in combination:
|
|
577
|
+
|
|
578
|
+
- **HNSW** — Hierarchical Navigable Small World graphs (Malkov & Yashunin 2018).
|
|
579
|
+
Vectors are connected into a multi-layer proximity graph; a query descends
|
|
580
|
+
greedily from a sparse top layer to a dense bottom layer, examining only a
|
|
581
|
+
logarithmic-ish neighbourhood of the collection. Empirically the work grows
|
|
582
|
+
around N^0.3 in Sema's configuration — decisively sub-linear.
|
|
583
|
+
- **RaBitQ 1-bit quantization** (Gao & Long 2024). Each stored vector is
|
|
584
|
+
randomly rotated and reduced to one _sign bit_ per dimension — a 32×
|
|
585
|
+
compression — with an unbiased, theoretically-grounded estimator of the
|
|
586
|
+
original cosine computable from the code alone. Sema stores _only_ the codes;
|
|
587
|
+
the original float vectors are never kept by the index.
|
|
588
|
+
|
|
589
|
+
### 6.2 The epistemological consequence
|
|
590
|
+
|
|
591
|
+
This layer is the one place where Sema's answers to "what is similar?" are
|
|
592
|
+
_estimates_: the scores returned by the index are RaBitQ estimates over 1-bit
|
|
593
|
+
codes, not exact cosines, and the ranking is approximate. Sema's discipline
|
|
594
|
+
about this is strict and worth stating as a principle, because it shapes several
|
|
595
|
+
pipeline decisions:
|
|
596
|
+
|
|
597
|
+
> **Approximate scores may rank and propose; they may never decide identity or
|
|
598
|
+
> be compared against exactness.** Any decision of the form "this _is_ that" is
|
|
599
|
+
> made by content-addressed resolution in the DAG. Thresholds compared against
|
|
600
|
+
> estimated scores gate broad regions (three-sigma bands, half-window bars),
|
|
601
|
+
> never knife-edge equalities.
|
|
602
|
+
|
|
603
|
+
### 6.3 Space-filling curves: geometry as reading order
|
|
604
|
+
|
|
605
|
+
One more piece of classical machinery belongs to this layer of fundamentals.
|
|
606
|
+
Sema's perception consumes _streams of bytes_; images, video, and other
|
|
607
|
+
grid-shaped data must first become a stream. Sema linearizes n-dimensional grids
|
|
608
|
+
along a **Hilbert curve** — the space-filling curve with the strongest locality
|
|
609
|
+
guarantees (points close on the curve are close in the grid, and vice versa to
|
|
610
|
+
the extent topology allows). This means spatial neighbourhoods in an image
|
|
611
|
+
become contiguous runs in the stream, so the same stream-folding perception that
|
|
612
|
+
reads text reads pixels — _geometry is only a reading order_, and every modality
|
|
613
|
+
meets the same memory.
|
|
614
|
+
|
|
615
|
+
---
|
|
616
|
+
|
|
617
|
+
---
|
|
618
|
+
|
|
619
|
+
# Part II — The big picture
|
|
620
|
+
|
|
621
|
+
## 7. How the five foundations connect
|
|
622
|
+
|
|
623
|
+
### 7.1 One structure, two verbs, one memory
|
|
624
|
+
|
|
625
|
+
Everything Sema does reduces to two operations over one store:
|
|
626
|
+
|
|
627
|
+
- **Deposit** (learn): perceive an input into a tree, intern the tree into the
|
|
628
|
+
Merkle DAG, and record its relations (continuation edges, halo pours).
|
|
629
|
+
- **Ask** (think): perceive the query the same way, and run one
|
|
630
|
+
lightest-derivation search whose axioms come from recognising the query
|
|
631
|
+
against the store and whose rules are the store's learned relations.
|
|
632
|
+
|
|
633
|
+
There is no third operation. There is no training phase distinct from
|
|
634
|
+
depositing, no fine-tuning, no consolidation pass required for correctness. The
|
|
635
|
+
store _is_ the model; a deposit is immediately available to every subsequent
|
|
636
|
+
ask.
|
|
637
|
+
|
|
638
|
+
### 7.2 The division of labour
|
|
639
|
+
|
|
640
|
+
Each foundation from Part I owns one aspect of the system, and the seams between
|
|
641
|
+
them are explicit:
|
|
642
|
+
|
|
643
|
+
```
|
|
644
|
+
┌─────────────────────────────────────────┐
|
|
645
|
+
│ INPUT (any modality) │
|
|
646
|
+
│ text · bytes · images · video │
|
|
647
|
+
└──────────────────┬──────────────────────┘
|
|
648
|
+
│ Hilbert linearization (§6.3)
|
|
649
|
+
▼
|
|
650
|
+
PERCEPTION ┌─────────────────────────────────────────┐
|
|
651
|
+
(VSA, §2) │ the river fold: bytes → leaves → tree │
|
|
652
|
+
│ every node gets a GIST (permutation- │
|
|
653
|
+
│ bind + superpose + normalize) │
|
|
654
|
+
└──────────────────┬──────────────────────┘
|
|
655
|
+
│ identical bytes ⇒ identical tree
|
|
656
|
+
▼
|
|
657
|
+
MEMORY ┌─────────────────────────────────────────┐
|
|
658
|
+
(CAM/Merkle, §3) │ the DAG: hash-consed nodes │
|
|
659
|
+
│ identity = content; │
|
|
660
|
+
│ parents ↑ / kids ↓ climbable │
|
|
661
|
+
├─────────────────────────────────────────┤
|
|
662
|
+
RELATIONS │ continuation edges (what follows what) │
|
|
663
|
+
(distributional, §4) │ halos (what company each node keeps) │
|
|
664
|
+
├─────────────────────────────────────────┤
|
|
665
|
+
INDEXES │ gist index + halo index │
|
|
666
|
+
(ANN, §6) │ (HNSW over 1-bit RaBitQ codes) │
|
|
667
|
+
└──────────────────┬──────────────────────┘
|
|
668
|
+
│ axioms & rule candidates
|
|
669
|
+
▼
|
|
670
|
+
INFERENCE ┌─────────────────────────────────────────┐
|
|
671
|
+
(deduction, §5) │ ONE lightest-derivation search: │
|
|
672
|
+
│ cover the query · follow edges · │
|
|
673
|
+
│ hop concepts · fuse & recompose · │
|
|
674
|
+
│ splice connectors · pool evidence │
|
|
675
|
+
└──────────────────┬──────────────────────┘
|
|
676
|
+
▼
|
|
677
|
+
┌─────────────────────────────────────────┐
|
|
678
|
+
│ answer bytes + provenance + rationale │
|
|
679
|
+
│ (a readable proof tree) │
|
|
680
|
+
└─────────────────────────────────────────┘
|
|
681
|
+
```
|
|
682
|
+
|
|
683
|
+
Read the seams carefully, because they are where the design earns its
|
|
684
|
+
properties:
|
|
685
|
+
|
|
686
|
+
- **Perception → Memory.** Perception is a _pure, deterministic function of
|
|
687
|
+
bytes_. The same bytes always fold into the same tree with the same gists.
|
|
688
|
+
This is what makes content addressing possible at all: if perception were
|
|
689
|
+
stochastic or context-dependent, the same content would not reproduce the same
|
|
690
|
+
nodes.
|
|
691
|
+
- **Memory → Indexes.** The vector indexes are _derived_ data — pure
|
|
692
|
+
accelerators. Every fact they suggest is verified against the DAG before it is
|
|
693
|
+
acted on. Deleting the indexes loses speed, never knowledge.
|
|
694
|
+
- **Memory → Inference.** The deduction system's rules are read off the store: a
|
|
695
|
+
continuation edge is a one-premise rule; a learned composite is a two-premise
|
|
696
|
+
fusion rule; a distributional sibling licenses a (more expensive) concept-hop
|
|
697
|
+
rule. Inference has no rules of its own beyond the cost algebra — _everything
|
|
698
|
+
it can do, it can do only because something was learned_ (plus the manual
|
|
699
|
+
computation rules of Section 16).
|
|
700
|
+
|
|
701
|
+
### 7.3 The two vector relations, side by side
|
|
702
|
+
|
|
703
|
+
It is worth fixing firmly, once, the two distinct vector spaces in play —
|
|
704
|
+
confusing them is the commonest way to misunderstand the system:
|
|
705
|
+
|
|
706
|
+
| | **Gist (content) space** | **Halo (concept) space** |
|
|
707
|
+
| :---------------------- | :-------------------------------------------- | :----------------------------------------------------------------- |
|
|
708
|
+
| A node's vector encodes | its own bytes and structure | the episodes it took part in |
|
|
709
|
+
| Built by | the perception fold (deterministic) | superposing seat-bound partner company signatures at learning time |
|
|
710
|
+
| Two nodes close means | similar content | similar usage (synonymy, categoryhood) |
|
|
711
|
+
| Typical query | "what stored form resembles this query span?" | "which nodes are used like this one?" |
|
|
712
|
+
| Indexed in | the content index | the halo index |
|
|
713
|
+
|
|
714
|
+
### 7.4 What "learning" and "generalizing" mean here
|
|
715
|
+
|
|
716
|
+
- **Learning a fact** = depositing a (context, continuation) pair: both sides
|
|
717
|
+
are interned, one continuation edge is recorded from the context's root to the
|
|
718
|
+
continuation's root, and each side's signature is poured into the other's
|
|
719
|
+
halo.
|
|
720
|
+
- **Generalization** happens at query time, by three distinct, inspectable
|
|
721
|
+
mechanisms rather than one opaque one: _geometric proximity_ (a query near a
|
|
722
|
+
learned form resonates with it), _distributional substitution_ (a form can
|
|
723
|
+
stand in for its halo siblings), and _structural analogy_ (a query that
|
|
724
|
+
aligns, byte-wise, with the shapes of several learned experiences can have
|
|
725
|
+
structure transferred between them — Section 18).
|
|
726
|
+
- **Forgetting** does not happen implicitly. Nodes are never silently discarded;
|
|
727
|
+
the store only ever grows more connected. (Index maintenance can prune
|
|
728
|
+
_acceleration_ entries, but that affects speed, not knowledge.)
|
|
729
|
+
|
|
730
|
+
---
|
|
731
|
+
|
|
732
|
+
## 8. Derived thresholds: the geometry of every decision
|
|
733
|
+
|
|
734
|
+
A system that makes soft (geometric) decisions needs thresholds, and thresholds
|
|
735
|
+
are where hidden empiricism usually creeps in ("0.7 worked on the dev set").
|
|
736
|
+
Sema's design rule is strict: **every threshold is derived from the geometry of
|
|
737
|
+
the representation itself — from the dimension D, the perception window W, or
|
|
738
|
+
the corpus size N — never tuned.** Any constant that cannot be derived is not
|
|
739
|
+
used. The five bars below govern the entire pipeline; each is stated with its
|
|
740
|
+
derivation.
|
|
741
|
+
|
|
742
|
+
Throughout: D is the vector dimension, W ("maxGroup") is the maximum number of
|
|
743
|
+
children a perception fold groups at once, N is the number of learned contexts
|
|
744
|
+
(nodes bearing at least one outgoing continuation edge).
|
|
745
|
+
|
|
746
|
+
### 8.1 The merge threshold: 1 − 1/√D — "geometrically the same"
|
|
747
|
+
|
|
748
|
+
The standard deviation of chance resonance between unrelated vectors is 1/√D
|
|
749
|
+
(§2.2). A cosine within one such unit _of 1.0_ is closer to identity than chance
|
|
750
|
+
can measure apart: the store treats two gists this close as candidates for being
|
|
751
|
+
the _same_ node (subject to the byte verification of §3.3). Recall reuses the
|
|
752
|
+
same bar to accept "the query is essentially a stored form".
|
|
753
|
+
|
|
754
|
+
This fixed bar is the identity threshold for two gists of comparable size (in
|
|
755
|
+
particular, two roots, which are always unit vectors — §2.6). A claim of the
|
|
756
|
+
form "this whole SPAN of `len` bytes is essentially identical to that stored
|
|
757
|
+
form" needs a **scale-aware** version of the same idea, because under the linear
|
|
758
|
+
fold a cosine reads as a byte-proportional overlap fraction (§2.6): a fixed
|
|
759
|
+
cosine bar admits a foreign-byte budget that _grows_ with the span, so naively
|
|
760
|
+
reusing 1 − 1/√D on a long span would tolerate far more corruption than
|
|
761
|
+
"essentially the same" should mean. The scale-aware bar instead fixes the
|
|
762
|
+
tolerated foreign-byte budget at one perception window W — the same
|
|
763
|
+
single-window budget near-dedup's byte check grants (§3.3, §11.1) — and converts
|
|
764
|
+
it into a cosine floor for the span's own length: **1 − W/len**, floored at the
|
|
765
|
+
fixed bar 1 − 1/√D (below which the RaBitQ estimator cannot certify identity
|
|
766
|
+
regardless of scale). The scale-aware bar `1 − W/len` tolerates exactly one
|
|
767
|
+
perception window W of foreign content — the same single-window budget the byte
|
|
768
|
+
check grants, now expressed in cosine space; a span barely longer than W
|
|
769
|
+
tolerates almost none. Derived from W, D, and the span length; never tuned.
|
|
770
|
+
|
|
771
|
+
### 8.2 The reach threshold: 1 − 1/(2W) — "related at all"
|
|
772
|
+
|
|
773
|
+
Perception folds children in groups of at most W. Two structures that differ in
|
|
774
|
+
_one whole child_ — the smallest difference perception can express — sit at
|
|
775
|
+
cosine ≈ 1 − 1/W (one of W superposed, permuted components differs). Half that
|
|
776
|
+
quantum, 1 − 1/(2W), is therefore _closer than any real single-child difference
|
|
777
|
+
can be_: anything scoring above it is a positional echo of the same content, and
|
|
778
|
+
anything whose _best_ match in the store falls below it is structurally
|
|
779
|
+
unrelated to everything stored. The reach threshold is Sema's confidence floor:
|
|
780
|
+
rather than answer from an unrelated neighbour, the system returns nothing.
|
|
781
|
+
**Silence is a first-class output.**
|
|
782
|
+
|
|
783
|
+
### 8.3 The significance bar: 3/√D — "not chance"
|
|
784
|
+
|
|
785
|
+
Chance resonance has mean 0 and standard deviation 1/√D, so a cosine of 3/√D is
|
|
786
|
+
three standard deviations above chance — the conventional statistical bar for
|
|
787
|
+
"this relationship is real". Whole-query evidence below this bar is not followed
|
|
788
|
+
into the more trusting inference tiers.
|
|
789
|
+
|
|
790
|
+
### 8.4 The estimator noise floor: 1/√D — "above quantisation noise"
|
|
791
|
+
|
|
792
|
+
One standard deviation of the cosine between two independent random vectors in D
|
|
793
|
+
dimensions (§2.2). It is the smallest difference in cosine that is
|
|
794
|
+
distinguishable from the rotation-uniformised RaBitQ estimation error: a
|
|
795
|
+
contrastive margin below it is quantisation noise, not evidence. The consensus
|
|
796
|
+
climb gates a region's vote on its _discriminative margin_ — the score gap
|
|
797
|
+
between the best and second-best anchor — clearing this floor. One σ, not the
|
|
798
|
+
stricter 3σ relatedness bar: the minimal "above noise" threshold. Derived from
|
|
799
|
+
D, never tuned.
|
|
800
|
+
|
|
801
|
+
### 8.5 The concept threshold: ½ + 1/(2√D) — "same concept"
|
|
802
|
+
|
|
803
|
+
Halos are superpositions of episode signatures. The structural midpoint 0.5
|
|
804
|
+
separates "more similar than dissimilar"; the added half-sigma 1/(2√D) widens
|
|
805
|
+
the bar slightly at low dimension (where chance noise is broader) and vanishes
|
|
806
|
+
as D grows. Two halos above this bar mark their nodes as distributional siblings
|
|
807
|
+
— eligible for concept hops, articulation substitutions, and analogy.
|
|
808
|
+
|
|
809
|
+
### 8.6 The consensus floor: ln N + ½ — "corroborated, not echoed"
|
|
810
|
+
|
|
811
|
+
In the consensus climb (Section 17), a query region's vote for an anchor is
|
|
812
|
+
weighted by _inverse document frequency_: reaching an anchor through c of the N
|
|
813
|
+
learned contexts is worth ln(N/c), so the maximum any single region can
|
|
814
|
+
contribute is ln N (a maximally specific region, c = 1). Requiring a pooled vote
|
|
815
|
+
to exceed ln N + ½ therefore demands _strictly more than any one region could
|
|
816
|
+
say alone_ — genuine multi-region corroboration at the current corpus scale —
|
|
817
|
+
before an anchor is trusted as an independent point of attention. The floor
|
|
818
|
+
grows with the corpus exactly as the maximum single-region vote does.
|
|
819
|
+
|
|
820
|
+
### 8.7 The half-dominance test: ½ — "a part that swallows its whole"
|
|
821
|
+
|
|
822
|
+
A span covering strictly more than half of its whole can no longer discriminate
|
|
823
|
+
the whole's own content — the test behind three pipeline decisions: liftAnswer
|
|
824
|
+
keeps the framing when a single recognised span dominates the query (the rest of
|
|
825
|
+
the cover is scaffolding), collectRegions excludes a wrapper region that would
|
|
826
|
+
drown multi-topic queries, and CAST's frame-depth majority classifies shared
|
|
827
|
+
material as non-discriminative structure. Derived from the structural midpoint:
|
|
828
|
+
half is the threshold at which the part outweighs what remains. Never tuned.
|
|
829
|
+
|
|
830
|
+
### 8.8 The hub bound: √N — "stop at non-discriminative fan-out"
|
|
831
|
+
|
|
832
|
+
Not a similarity bar but the same spirit: any walk over the graph's fan-out (a
|
|
833
|
+
node's parents, a context's continuations, an answer's reverse fan-in) is capped
|
|
834
|
+
at √N candidates. A node connected to more than √N others is a _hub_ — its
|
|
835
|
+
connections are so numerous that each individual one carries almost no
|
|
836
|
+
discriminative information. The cap is applied at the _store level_: the store
|
|
837
|
+
provides LIMITed read operations so that no per-query read ever materialises a
|
|
838
|
+
corpus-sized fan-out list. Consumers of partial fan-outs use the LIMITed reads
|
|
839
|
+
to decide their question exactly — "hub or not?", "saturated or voted?" —
|
|
840
|
+
without ever expanding past √N distinct contexts. The full materialising reads
|
|
841
|
+
remain available for maintenance and inspection paths, but every hot-path
|
|
842
|
+
decision consults only the LIMITed prefix or an indexed existence probe. Every
|
|
843
|
+
fan-out-limited decision in the pipeline uses this one bound, so the trade is
|
|
844
|
+
made once, consistently, and the cost of inference stays bounded by √N rather
|
|
845
|
+
than growing with the corpus.
|
|
846
|
+
|
|
847
|
+
### 8.9 The cost ladder: the one currency of every decision
|
|
848
|
+
|
|
849
|
+
The deduction system's rule costs form the **single cost currency of the whole
|
|
850
|
+
mind**: every grounding mechanism's candidate answer is weighed in these same
|
|
851
|
+
units, so a mechanism-level choice (should the answer come from CAST, the cover
|
|
852
|
+
search, or recall?) and a byte-level choice (should this span be a recognised
|
|
853
|
+
completion or a carried literal?) are the _same kind of decision_ — a lightest
|
|
854
|
+
derivation. The ladder is:
|
|
855
|
+
|
|
856
|
+
```
|
|
857
|
+
ε (MICRO: bridge a recognised span into the — essentially free; the
|
|
858
|
+
cover) per-byte unit of the
|
|
859
|
+
admissible A* heuristic
|
|
860
|
+
1 (STEP: follow one learned edge; one computed — the unit of inference
|
|
861
|
+
result; one projection; one frame location)
|
|
862
|
+
10 (CONCEPT: borrow a sibling's edge; one halo- — one order dearer than a
|
|
863
|
+
mediated act; one consensus climb) literal continuation
|
|
864
|
+
1000·bytes (PASS: carry an unrecognised literal) — coverage dominates
|
|
865
|
+
everything: the search
|
|
866
|
+
prefers to recognise
|
|
867
|
+
```
|
|
868
|
+
|
|
869
|
+
The constants are chosen as a strict _ordering_ — any set preserving the order
|
|
870
|
+
yields identical lightest derivations. The one quantitative role of ε: it is the
|
|
871
|
+
cheapest per-position cost, so "ε × (bytes remaining)" is an admissible A\*
|
|
872
|
+
lower bound on the cost to finish covering the query — the heuristic that keeps
|
|
873
|
+
the search output-sensitive (§5.2).
|
|
874
|
+
|
|
875
|
+
The grounding decider (§14.1) uses the same ladder: a candidate answer's weight
|
|
876
|
+
is its mechanism's moves (STEP per projection, CONCEPT per halo-mediated act)
|
|
877
|
+
plus PASS for every query byte the mechanism did _not_ account for (did not
|
|
878
|
+
match against learnt structure). The lightest grounding candidate wins — the
|
|
879
|
+
same elementary decision, lifted to the mechanism level.
|
|
880
|
+
|
|
881
|
+
### 8.10 Two measures of commonality
|
|
882
|
+
|
|
883
|
+
Every mechanism that asks "is this content discriminative?" must choose a
|
|
884
|
+
**reference set** — the population over which _commonality_ is measured. The
|
|
885
|
+
system provides exactly two, and they are formally independent: neither quantity
|
|
886
|
+
bounds the other, and no derived threshold can convert one into the other. The
|
|
887
|
+
choice between them is the single most consequential design decision a mechanism
|
|
888
|
+
makes, because it determines what counts as _scaffolding_ and what counts as
|
|
889
|
+
_evidence_.
|
|
890
|
+
|
|
891
|
+
#### Corpus-global commonality
|
|
892
|
+
|
|
893
|
+
The reference set is **every learned context in the store** — the durable,
|
|
894
|
+
corpus-wide population of edge-bearing nodes (counted by `corpusN`). A node's
|
|
895
|
+
corpus-global commonality is the number of distinct contexts whose
|
|
896
|
+
containment/edge climb reaches it — `reachOf(id, N)`, read through
|
|
897
|
+
`edgeAncestors`, capped at √N.
|
|
898
|
+
|
|
899
|
+
Corpus-global commonality is a **property of a node**, stable across queries.
|
|
900
|
+
The same node always reaches the same number of contexts (modulo new deposits).
|
|
901
|
+
It answers: _does this content discriminate anything in what the system has
|
|
902
|
+
learned?_
|
|
903
|
+
|
|
904
|
+
Content reaching a corpus **minority** of contexts (¬dominates(reach, N))
|
|
905
|
+
discriminates — it is an entity, a filler, a name. Content reaching a corpus
|
|
906
|
+
**majority** (dominates(reach, N)) is frame scaffolding — it discriminates
|
|
907
|
+
nothing anywhere. This is the half-dominance convention of §8.7, applied to the
|
|
908
|
+
entire store.
|
|
909
|
+
|
|
910
|
+
The climb's IDF weighting (§17.3), confluence's filler/scaffolding gate (§18.5),
|
|
911
|
+
and every decision of the form "is this node a hub?" use corpus-global
|
|
912
|
+
commonality. The halo index (§4, §12.2) is also corpus-global: a node's
|
|
913
|
+
distributional signature is the superposition of ALL episodes it took part in,
|
|
914
|
+
not just those relevant to the current query.
|
|
915
|
+
|
|
916
|
+
#### Weave-local commonality
|
|
917
|
+
|
|
918
|
+
The reference set is **the structures aligned with this query** — the transient,
|
|
919
|
+
query-specific population of anchors the consensus climb ranked and whose
|
|
920
|
+
contexts produced literal or distributional runs against the query bytes. The
|
|
921
|
+
population size is `aligned`, counted fresh per query; commonality at byte
|
|
922
|
+
position `i` is `depth[i]`, the sum of alignment weights covering that byte.
|
|
923
|
+
|
|
924
|
+
Weave-local commonality is a **property of a query-byte position**, not of a
|
|
925
|
+
node. The same byte can be frame for one query and content for another, because
|
|
926
|
+
the aligned population changes. It answers: _does this content discriminate
|
|
927
|
+
among the structures THIS query activates?_
|
|
928
|
+
|
|
929
|
+
A byte covered by a weave-local **minority** of aligned structures
|
|
930
|
+
(¬dominates(depth[i], aligned)) discriminates among them — it differentiates one
|
|
931
|
+
aligned context from another. A byte covered by a weave-local **majority**
|
|
932
|
+
(dominates(depth[i], aligned)) is shared scaffolding of the weave — it carries
|
|
933
|
+
no information about which aligned structure is which, regardless of how rare or
|
|
934
|
+
common it is in the corpus.
|
|
935
|
+
|
|
936
|
+
CAST's frame gate (§18.3) uses weave-local commonality. The grounding decider's
|
|
937
|
+
`unaccounted` bytes (§14.1) are also weave-local in spirit: they measure what
|
|
938
|
+
THIS query's mechanisms did not explain, priced against the query's own length.
|
|
939
|
+
|
|
940
|
+
#### Independence
|
|
941
|
+
|
|
942
|
+
The two measures are computed over **different data structures** with
|
|
943
|
+
**different stopping criteria**:
|
|
944
|
+
|
|
945
|
+
- Corpus-global: `edgeAncestors` walks the DAG's parent edges (`parentsFirst`,
|
|
946
|
+
`prevFirst`), counting distinct edge-bearing contexts, capped at √N. It
|
|
947
|
+
answers a question about a node's position in the permanent store.
|
|
948
|
+
|
|
949
|
+
- Weave-local: `alignGraded` aligns raw bytes (literal W-gram seed-and-extend,
|
|
950
|
+
then halo-matched recognised sites), incrementing a per-byte `depth` array. It
|
|
951
|
+
answers a question about this query's transient alignment.
|
|
952
|
+
|
|
953
|
+
Neither computation is a special case of the other. A phrase common to 2 of 3
|
|
954
|
+
aligned exemplars but rare in the corpus (low reach, high weave-local share)
|
|
955
|
+
**is** frame for the weave — it is shared scaffolding of this particular
|
|
956
|
+
analogy, not differentiating content. A phrase with high corpus reach (common
|
|
957
|
+
everywhere) that happens to appear in only 1 of 3 aligned exemplars **is**
|
|
958
|
+
content for the weave — it differentiates that exemplar from the others. The two
|
|
959
|
+
coincide often (semantically rich exemplars tend to share corpus-wide
|
|
960
|
+
scaffolding), but neither derives the other. They cannot be treated as
|
|
961
|
+
interchangeable: replacing CAST's weave-local gate with the structural IDF lets
|
|
962
|
+
the substitution branch fire on reordered single-fact queries.
|
|
963
|
+
|
|
964
|
+
#### Which measure for which question
|
|
965
|
+
|
|
966
|
+
The system provides both measures. Each mechanism picks the one that answers its
|
|
967
|
+
question:
|
|
968
|
+
|
|
969
|
+
| Mechanism | Question | Measure |
|
|
970
|
+
| ------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------- |
|
|
971
|
+
| Consensus climb (§17) | Does this region's anchor discriminate among learned contexts? | Corpus-global (IDF weight) |
|
|
972
|
+
| Confluence (§18.5) | Does this shared content name an entity or is it scaffolding? | Corpus-global (dominates(reach, N)) |
|
|
973
|
+
| CAST frame gate (§18.3) | Do the aligned structures share this byte, or does it differentiate them? | Weave-local (dominates(depth[i], aligned)) |
|
|
974
|
+
| Grounding decider (§14.1) | Which mechanism explains more of THIS query? | Weave-local (unaccounted bytes / query length) |
|
|
975
|
+
| Recognition (§15) | Is this span a stored form? | Neither — exact, content-addressed |
|
|
976
|
+
| Cover search (§19) | Can the query be covered by recognised completions? | Neither — cost-ladder, output-sensitive |
|
|
977
|
+
|
|
978
|
+
A mechanism that uses the wrong measure answers the wrong question. The system
|
|
979
|
+
cannot prevent this — both measures are available, and the architecture does not
|
|
980
|
+
enforce which one a mechanism consults. The distinction is a design discipline,
|
|
981
|
+
not a type-level guard.
|
|
982
|
+
|
|
983
|
+
Concretely: in a query that weaves 3 painting exemplars, the phrase " describe
|
|
984
|
+
it" has high weave-local depth (all 3 aligned contexts contain it) but low
|
|
985
|
+
corpus-global reach (only those 3 of 20 learned contexts do).
|
|
986
|
+
`dominates(depth, 3)` says frame — shared scaffolding of this particular
|
|
987
|
+
analogy. `dominates(reach, 20)` says content — a minority of the corpus, hence
|
|
988
|
+
discriminative. The frame gate correctly ignores the corpus and asks the
|
|
989
|
+
weave-local question. A mechanism that asked the corpus-global question here
|
|
990
|
+
would classify " describe it" as content and let substitution fire on a
|
|
991
|
+
reordered single-fact query. The same node, the same bytes, two different
|
|
992
|
+
answers — because two different questions were asked over two different
|
|
993
|
+
populations.
|
|
994
|
+
|
|
995
|
+
---
|
|
996
|
+
|
|
997
|
+
## 9. The concept inventory
|
|
998
|
+
|
|
999
|
+
Every named concept in the system, one line each, with its home section. This is
|
|
1000
|
+
the vocabulary the rest of the document (and the codebase) speaks.
|
|
1001
|
+
|
|
1002
|
+
**Representation**
|
|
1003
|
+
|
|
1004
|
+
- **Gist** — the VSA fold of a span's content; content similarity in one dot
|
|
1005
|
+
product (§2.3).
|
|
1006
|
+
- **Seat / keyring** — the fixed random permutations that bind ordinal position
|
|
1007
|
+
into a fold (§2.3).
|
|
1008
|
+
- **Alphabet** — the 256 deterministic byte vectors with graded similarity
|
|
1009
|
+
(§2.4).
|
|
1010
|
+
- **River fold** — the level-by-level grouping of leaves into a tree, W at a
|
|
1011
|
+
time (§10). **Linear**: only the fold's root is normalized, so every interior
|
|
1012
|
+
gist keeps a byte-proportional magnitude — the basis of angle-and-magnitude
|
|
1013
|
+
semantics (§2.6).
|
|
1014
|
+
- **Magnitude / contentLen** — the byte-proportional length an unnormalized
|
|
1015
|
+
interior gist carries (norm ≈ √len); read back from the store as a span's
|
|
1016
|
+
content length and used to convert a raw cosine into a query-relative or
|
|
1017
|
+
scale-aware fraction (§2.6, §8.1, §17.3, §21).
|
|
1018
|
+
- **Stable prefix** — the already-known head of a stream, folded independently
|
|
1019
|
+
so its structure is reproducible (§10.3).
|
|
1020
|
+
|
|
1021
|
+
**Memory**
|
|
1022
|
+
|
|
1023
|
+
- **Node** — a leaf (bytes) or branch (ordered children); identity = content
|
|
1024
|
+
(§3.2).
|
|
1025
|
+
- **isChunk** — the predicate for "children are all leaves" — the perception
|
|
1026
|
+
tree's smallest grouped unit, behind region collection, canonical seams, and
|
|
1027
|
+
sub-span indexing (§10, §11.3, §17.2).
|
|
1028
|
+
- **Interning / hash-consing** — storing a tree bottom-up so equal subtrees
|
|
1029
|
+
share one node (§11).
|
|
1030
|
+
- **Near-dedup** — merging a fresh root onto a geometrically identical,
|
|
1031
|
+
byte-verified stored root (§11.2).
|
|
1032
|
+
- **Containment edge** — a durable "this window of bytes occurs inside that
|
|
1033
|
+
chunk" record for sub-spans that are not structural children (§11.3).
|
|
1034
|
+
- **Transparent chain (chainRun)** — a run of nodes each with exactly one
|
|
1035
|
+
structural parent and no continuation edges in or out; climbed in a single
|
|
1036
|
+
bounded read instead of one probe per node. Used by the structural DAG climb
|
|
1037
|
+
to skip scaffolding when ascending to edge-bearing contexts.
|
|
1038
|
+
- **Continuation edge** — the learned relation "this followed that"; the atom of
|
|
1039
|
+
factual knowledge (§12.1).
|
|
1040
|
+
- **Company signature** — a deterministic unit vector derived from a node's
|
|
1041
|
+
identity (seeded by its id), used as the halo-pour unit instead of the node's
|
|
1042
|
+
gist (§4, §12.2). Decouples content similarity from distributional similarity.
|
|
1043
|
+
- **Halo / halo mass** — a node's distributional signature (superposition of
|
|
1044
|
+
partner company signatures) and the count of episodes poured into it (§4,
|
|
1045
|
+
§12.2).
|
|
1046
|
+
- **Resonance target** — a node whose gist is admitted to the content index
|
|
1047
|
+
(lazily: roots, edge/halo bearers, and interior forms of experiences) (§12.3).
|
|
1048
|
+
- **Junction** — a learnt whole that literally contains two forms, found by
|
|
1049
|
+
content-addressed DAG ascent (parents + containment links) from the two sides'
|
|
1050
|
+
canonical identities — not by a resonance guess. The walk is **order-free** (a
|
|
1051
|
+
junction evidences that two forms were learnt together; which one the query
|
|
1052
|
+
mentions first is a fact about the query, not the learnt whole — the
|
|
1053
|
+
byte-containment test probes both orders, costing two indexOf calls per
|
|
1054
|
+
visited node, never a second walk). Overlapping or abutting occurrences are
|
|
1055
|
+
accepted (grid fragments of one whole legitimately overlap inside it), with a
|
|
1056
|
+
strict-super-form requirement (holding both must be more than restating either
|
|
1057
|
+
side). The bridge's Tier 1 connector search (§19.5) and cross-region
|
|
1058
|
+
attention's joint-context recovery (§17.6) ascend by the same shared, bounded,
|
|
1059
|
+
cached walk. A per-response walk cache memoises every identity read across all
|
|
1060
|
+
walks of one response, and junction seeds are computed once per candidate and
|
|
1061
|
+
reused across all its pairs. Synonym junctions extend the ascent to halo
|
|
1062
|
+
siblings (Tier 2.5), sharing one expansion budget across all sibling walks.
|
|
1063
|
+
|
|
1064
|
+
**Inference**
|
|
1065
|
+
|
|
1066
|
+
- **Match-and-project** — the ONE elementary operation every generalising
|
|
1067
|
+
mechanism configures: match a learned structure under a matcher, project along
|
|
1068
|
+
a learned relation in a direction, accept past a derived gate (§14.4).
|
|
1069
|
+
- **Matcher** — the matching relation of a match-and-project: exact
|
|
1070
|
+
(content-addressed), locate (the graded exact→halo→gist ladder), aligned
|
|
1071
|
+
(literal W-gram runs), or distributional (analogy strength) (§14.4).
|
|
1072
|
+
- **Projection / direction** — the learned relation a matched node is projected
|
|
1073
|
+
along: forward (`follow`, to the continuation fixpoint), reverse
|
|
1074
|
+
(`reverseContext`, to the establishing context), both (`project`), read-out,
|
|
1075
|
+
insert, or substitute (§14.4).
|
|
1076
|
+
- **Hub cap** — the one √N fan-out convention (§8.8), applied in two forms:
|
|
1077
|
+
`hubBound` (≥ 2, the numerical cap passed to the store's LIMITed reads) and
|
|
1078
|
+
`hubCap` (the list-side reading). Every fan-out walk and disambiguation uses
|
|
1079
|
+
one of them; the store enforces the cap at read time so no per-query cost
|
|
1080
|
+
grows with the corpus.
|
|
1081
|
+
- **Estimator noise floor** — 1/√D, one standard deviation of chance cosine
|
|
1082
|
+
between random vectors. The smallest difference distinguishable from RaBitQ
|
|
1083
|
+
quantisation error (§8.4). The consensus climb gates a region's vote on its
|
|
1084
|
+
discriminative margin clearing this floor.
|
|
1085
|
+
- **Half-dominance** — a part covering strictly more than half of its whole can
|
|
1086
|
+
no longer discriminate it (§8.7). The structural midpoint, derived, never
|
|
1087
|
+
tuned. Used by liftAnswer, region collection, and CAST frame classification.
|
|
1088
|
+
- **Commonality (corpus-global vs. weave-local)** — the TWO reference sets over
|
|
1089
|
+
which "is this content shared or discriminative?" is measured (§8.10).
|
|
1090
|
+
Corpus-global: the durable population of all learned contexts (N); answers
|
|
1091
|
+
"does this discriminate anything in the store?" Weave-local: the transient
|
|
1092
|
+
population of structures aligned with this query (aligned); answers "does this
|
|
1093
|
+
discriminate among the structures this query activates?" The two are formally
|
|
1094
|
+
independent — a phrase rare in the corpus can be scaffolding of the weave, and
|
|
1095
|
+
a phrase common everywhere can differentiate two aligned exemplars. The system
|
|
1096
|
+
provides both; each mechanism chooses the one that answers its question.
|
|
1097
|
+
- **Corpus N** (`corpusN`) — the count of distinct learned contexts floored at
|
|
1098
|
+
2, so its derived readings (ln N, √N) stay meaningful on a near-empty store.
|
|
1099
|
+
Defined once; every consumer of the corpus scale reads it. (§8.8, §17)
|
|
1100
|
+
- **Expand-until-decided** — the climb's work is bounded by stopping the moment
|
|
1101
|
+
the answer (saturated vs. voted) is known, through LIMITed store reads only
|
|
1102
|
+
(§17.4). The walk is exact below √N distinct contexts and stops at the first
|
|
1103
|
+
proof of saturation past it.
|
|
1104
|
+
- **Canonical contract** — the write/read convention for the store's
|
|
1105
|
+
segmentation: the write side interns W−1 and W sliding windows and a
|
|
1106
|
+
whole-stream flat branch; the read side chains leaf ids up to W² positions and
|
|
1107
|
+
probes every prefix as a flat branch. Defined in one module; a drift between
|
|
1108
|
+
the sides silently breaks recognition. (§10.3, §11.3, §15.2)
|
|
1109
|
+
- **Window IDs** — the canonical content-addressed identity of every W-sized
|
|
1110
|
+
slice of a byte stream, offset → node id. Under this mapping, any content two
|
|
1111
|
+
deposits share IS the same node (hash-consing paid the comparison at write
|
|
1112
|
+
time). Confluence's meet and CAST's frame detection read shared content
|
|
1113
|
+
through this — never through a byte scan. (§18.1, §18.5)
|
|
1114
|
+
- **Reach (structural IDF)** — the number of distinct learnt contexts a single
|
|
1115
|
+
node's containment/edge climb reaches, or Infinity when it reaches none or
|
|
1116
|
+
saturates. Paired with the half-dominance convention: content reaching a
|
|
1117
|
+
corpus minority of contexts discriminates (an entity, a filler); content
|
|
1118
|
+
reaching a majority is frame scaffolding. (§18.3, §18.5)
|
|
1119
|
+
- **Recognition** — decomposing a query into every stored form it contains, by
|
|
1120
|
+
structural and canonical readings (§15).
|
|
1121
|
+
- **Site** — one recognised form: a query span plus the node it names (§15).
|
|
1122
|
+
- **Cover** — the lightest-derivation goal: the query covered left to right by
|
|
1123
|
+
recognised completions and carried literals (§19).
|
|
1124
|
+
- **Fuse / recompose** — the search's discovery that adjacent fragments spell a
|
|
1125
|
+
deeper learned form (§19.4).
|
|
1126
|
+
- **Connector (bridge)** — learned material that belongs _between_ two spans,
|
|
1127
|
+
found by a graded junction ladder: Tier 1 containment ascent by
|
|
1128
|
+
content-addressed identity, then Tier 2 edge junctions (a continuation/context
|
|
1129
|
+
carrying the glue), then Tier 2.5 synonym junctions (the same ascent over halo
|
|
1130
|
+
siblings), then resonance as last resort; disambiguated by the response guide,
|
|
1131
|
+
with the shortest interior preferred. The junction ascent is shared with
|
|
1132
|
+
cross-region attention (§19.5, §17.6).
|
|
1133
|
+
- **Concept hop** — borrowing a distributional sibling's continuation via
|
|
1134
|
+
`haloSiblings` (the unified halo-sibling enumeration) and `guidedFirst` (the
|
|
1135
|
+
guided-or-first convention for edge picks) (§19.3).
|
|
1136
|
+
- **Join with bridge** — the shared composition step for out-of-search assembly:
|
|
1137
|
+
a learned connector between two spans, or a bare join with a visible
|
|
1138
|
+
`bridgeMiss` trace step. Used by multi-topic fusion and CAST. (§19.5, §23)
|
|
1139
|
+
- **Recompletion** — covering a produced answer's own bytes with the same
|
|
1140
|
+
machinery, recursively, to let composites resolve deeper (§19.6).
|
|
1141
|
+
- **Consensus climb / point of attention** — regions of the query vote, through
|
|
1142
|
+
the DAG's parents, for the learned contexts they belong to. Regions come from
|
|
1143
|
+
TWO sources: perceived subtrees (the river fold's positional chunks) and
|
|
1144
|
+
recognised sites (content-addressed nodes — exact anchors that skip the ANN
|
|
1145
|
+
resonance step). Sites capture whole words that cross W-boundaries, which
|
|
1146
|
+
perceived chunks alone miss. Pooled votes select the query's independent
|
|
1147
|
+
topics (§17).
|
|
1148
|
+
- **Saturation** — a region whose upward climb hits hub fan-out abstains rather
|
|
1149
|
+
than voting noise (§17.4).
|
|
1150
|
+
- **Cross-region attention** — direct region-to-region interaction: two regions
|
|
1151
|
+
that independently voted (at least one strongly) pair to recover their joint
|
|
1152
|
+
context — the learnt whole containing both — by the same order-free junction
|
|
1153
|
+
ascent the bridge uses. Corpus-independent: any voted region composes, and a
|
|
1154
|
+
known but non-voting region may serve as the weak side of a pair whose other
|
|
1155
|
+
side voted (a word never trained standalone still binds through its stored
|
|
1156
|
+
byte fragments); two non-voting regions never pair (the shared-prefix trap).
|
|
1157
|
+
N-ary: pair containers are filtered by the remaining candidate forms — the
|
|
1158
|
+
container covering the most of the query's composable forms wins, so three
|
|
1159
|
+
cross-cutting attributes resolve to their unique triple at the cost of one
|
|
1160
|
+
cached read + indexOf per (container, extra), never an extra walk. Explaining
|
|
1161
|
+
away: when a junction binds, any individual vote whose bytes the joint
|
|
1162
|
+
container literally contains and whose roots are fully disjoint from the
|
|
1163
|
+
junction's is superseded — the exact joint evidence explains those bytes away;
|
|
1164
|
+
partial agreement corroborates and is kept. Self-evidence guard: a container
|
|
1165
|
+
whose joined occurrence is itself a substring of the query is rejected —
|
|
1166
|
+
binding is only evidence when the query mentions the forms apart. Consumed
|
|
1167
|
+
candidates never re-pair. A joint container is exact evidence, voting at full
|
|
1168
|
+
strength. Additive pooling alone cannot surface a context zero regions
|
|
1169
|
+
individually voted for; cross-region evidence fills that gap (§17.6).
|
|
1170
|
+
- **CAST (counterfactual transfer)** — substitution / redirection / comparison
|
|
1171
|
+
between independently learned structures the query weaves together. Alignment
|
|
1172
|
+
is **graded** (literal W-grams → halo-matched sites); frame gates are
|
|
1173
|
+
**derived** (`MIN_WEAVE` from the weave minimum, `dominates` from
|
|
1174
|
+
half-dominance) and **weave-local** (majority of _aligned_ structures, not
|
|
1175
|
+
corpus-global IDF). (§18)
|
|
1176
|
+
- **Confluence join** — the meet of independent constraint streams by
|
|
1177
|
+
content-addressed identity: window IDs present in both anchors and absent from
|
|
1178
|
+
the query name the entity satisfying all constraints at once. Answers
|
|
1179
|
+
conjunctive queries ("Which X is A and B?") that no single-fact mechanism can
|
|
1180
|
+
resolve. (§18.5)
|
|
1181
|
+
- **Skill / exemplar** — a learned fact shaped "answer-is-a-span-of-context",
|
|
1182
|
+
reusable as an extraction template on unseen text (§20).
|
|
1183
|
+
- **Recall tiers** — the graded fallback for whole-query resonance, from exact
|
|
1184
|
+
self-match to honest echo. Each tier reports _what it matched_ (`accounted`),
|
|
1185
|
+
its _moves_, and `unexplained` — a human-readable label for the query bytes it
|
|
1186
|
+
left on the table — so the grounding decider can compare it against every
|
|
1187
|
+
other mechanism in the same currency with full diagnostic visibility. (§21)
|
|
1188
|
+
- **Accounted spans** — the query byte ranges a mechanism's own structural
|
|
1189
|
+
evidence explains (aligned runs, located frames, voted regions, constraint
|
|
1190
|
+
content). Query bytes outside them are priced at PASS each — the same rate the
|
|
1191
|
+
cover search pays for a literal connective — so "which mechanism explains more
|
|
1192
|
+
of the query with learnt structure" is the primary axis of the grounding
|
|
1193
|
+
decider. (§14.1, §20, §21)
|
|
1194
|
+
- **Open seat (read-out content is not evidence)** — the span extraction reads
|
|
1195
|
+
between located frames is structurally explained (we know _where_ to read it)
|
|
1196
|
+
but content-novel (we do not know _what_ it says). It is the variable being
|
|
1197
|
+
read, not the structure doing the reading — the same role the cover's
|
|
1198
|
+
unrecognised literals play, and priced the same way (PASS each, by exclusion).
|
|
1199
|
+
Counting it as explained would let a mechanism claim credit for bytes it
|
|
1200
|
+
merely copied from the query. (§20)
|
|
1201
|
+
- **Forward asymmetry (reverse is not derivation)** — the deduction system has
|
|
1202
|
+
no backward rule; a reading against the edge direction (`reverseContext`)
|
|
1203
|
+
produces bytes but no forward derivation. The grounding decider expresses this
|
|
1204
|
+
exactly: reverse readings get `accounted = []`, their weight the full
|
|
1205
|
+
PASS·|query|. The decider derives this from the evidence the formalism itself
|
|
1206
|
+
declares. (§21)
|
|
1207
|
+
- **Weave-local vs. corpus-global commonality** — CAST's frame gates are
|
|
1208
|
+
weave-local (majority of _aligned_ points, per query), not corpus-global
|
|
1209
|
+
(majority of contexts, across the store). The two quantities are formally
|
|
1210
|
+
independent — a phrase common to 2 of 3 aligned exemplars but rare in the
|
|
1211
|
+
corpus IS frame for CAST's purposes; substituting global IDF misfires on
|
|
1212
|
+
reordered single-fact queries. (§18.2)
|
|
1213
|
+
- **Free-will architecture** — the grounding decider as a market: mechanisms are
|
|
1214
|
+
decoupled (zero cross-imports), self-gating (binary structural preconditions),
|
|
1215
|
+
budget-capped (√N, k, LIMITed reads), and evidence-carrying (`accounted`,
|
|
1216
|
+
`moves`, `unexplained`). The decider compares weights in one currency; it does
|
|
1217
|
+
not know which mechanism produced which candidate. The same four constraints —
|
|
1218
|
+
decoupling, declared competence, visible budget, traveling evidence — are the
|
|
1219
|
+
structural principle that makes any budget-limited reasoner honest, from
|
|
1220
|
+
Sema's `√N` to a model's `max_tokens`. (§14.5)
|
|
1221
|
+
- **Grounding decider** — the unified choice among grounding mechanisms: every
|
|
1222
|
+
self-gating mechanism yields a candidate answer weighed in the one cost
|
|
1223
|
+
ladder, and the lightest grounding derivation wins. Moves (STEP per
|
|
1224
|
+
projection, CONCEPT per halo-mediated act) discriminate residually; PASS per
|
|
1225
|
+
unexplained byte dominates. Ties keep the mechanism list's order (cover, CAST,
|
|
1226
|
+
confluence, extract, recall). The decider uses admissible-floor pruning (a
|
|
1227
|
+
mechanism whose best-case floor cannot beat the incumbent is never run) — and
|
|
1228
|
+
a mechanism whose floor itself needs expensive precomputation to refine checks
|
|
1229
|
+
the SAME incumbent before paying for it (§14.1, §14.2).
|
|
1230
|
+
- **Pivot** — the longest unconsumed learned context contained in the current
|
|
1231
|
+
answer; the stepping stone of multi-hop reasoning (§22).
|
|
1232
|
+
- **Fusion** — grounding each independent point of attention and joining the
|
|
1233
|
+
results with learned connectors (§23).
|
|
1234
|
+
- **Articulation** — re-voicing the answer in the asker's own words via halo
|
|
1235
|
+
siblings (§24).
|
|
1236
|
+
- **Echo** — the last-resort output that returns a stored form verbatim,
|
|
1237
|
+
explicitly flagged as not grounded (§21.4, §26).
|
|
1238
|
+
- **Provenance** — which grounding mechanism produced the answer; part of every
|
|
1239
|
+
response (§26).
|
|
1240
|
+
- **Rationale** — the replayable trace of every rule application behind an
|
|
1241
|
+
answer (§26).
|
|
1242
|
+
|
|
1243
|
+
**Computation**
|
|
1244
|
+
|
|
1245
|
+
- **PipelineMechanism** — the ONE uniform interface every grounding mechanism
|
|
1246
|
+
(CAST, confluence, cover, extraction, recall, the ALU, any user extension)
|
|
1247
|
+
implements: an optional `parse` (authoritative computed spans, pre-loop), a
|
|
1248
|
+
`floor` (admissible lower bound), and a `run` (candidate answers). The
|
|
1249
|
+
pipeline never special-cases any mechanism by name or kind (§14.1, §16).
|
|
1250
|
+
- **Precomputed** — the shared, response-scoped container every mechanism's
|
|
1251
|
+
`floor`/`run` (and the post-grounding stages) receive: eager fields
|
|
1252
|
+
(recognition, computed spans, guide) plus lazily-cached methods for the
|
|
1253
|
+
expensive structural analyses (the consensus climb `attention()`, the weave,
|
|
1254
|
+
span-shape classification, the identity-window reads), each computed at most
|
|
1255
|
+
once, reused across every consumer, and never computed at all when no
|
|
1256
|
+
mechanism asks (§14.1, §14.2).
|
|
1257
|
+
- **Extension** — a user- or built-in-supplied `PipelineMechanism` whose `parse`
|
|
1258
|
+
recognises computations (arithmetic, logic, …) the mind should not have to
|
|
1259
|
+
learn fact-by-fact; joins via `mechanismFactories` (§16).
|
|
1260
|
+
- **ALU** — the built-in extension: arithmetic, logic, and numerical computation
|
|
1261
|
+
derived from an irreducible kernel (§16.2).
|
|
1262
|
+
- **Masking** — computed spans override colliding learned facts for exactly
|
|
1263
|
+
their bytes ("computation always wins") (§16.3).
|
|
1264
|
+
|
|
1265
|
+
---
|
|
1266
|
+
|
|
1267
|
+
---
|
|
1268
|
+
|
|
1269
|
+
# Part III — The ingestion pipeline
|
|
1270
|
+
|
|
1271
|
+
Ingestion is the learning half of the system: input bytes in, a more
|
|
1272
|
+
knowledgeable graph out. It has three stages — perception, deposition, relation
|
|
1273
|
+
learning — and the whole of it is deterministic: the same inputs in the same
|
|
1274
|
+
order always produce a structurally identical store.
|
|
1275
|
+
|
|
1276
|
+
```
|
|
1277
|
+
THE INGESTION PIPELINE
|
|
1278
|
+
|
|
1279
|
+
input (text / bytes / grid / frames)
|
|
1280
|
+
│
|
|
1281
|
+
│ 1. modality flattening (grids → Hilbert-ordered byte stream)
|
|
1282
|
+
▼
|
|
1283
|
+
byte stream b₀ b₁ b₂ … bₙ
|
|
1284
|
+
│
|
|
1285
|
+
│ 2. leaf lift: each byte → a leaf carrying its alphabet vector
|
|
1286
|
+
▼
|
|
1287
|
+
leaves [l₀][l₁][l₂] … [lₙ]
|
|
1288
|
+
│
|
|
1289
|
+
│ 3. RIVER FOLD: group ≤ W siblings per level, seat-bind + superpose
|
|
1290
|
+
│ (splitting at the stable-prefix boundary when the head of the
|
|
1291
|
+
│ stream is already known)
|
|
1292
|
+
▼
|
|
1293
|
+
perceived tree (every node: bytes-or-kids + gist vector)
|
|
1294
|
+
│
|
|
1295
|
+
│ 4. INTERN bottom-up into the Merkle DAG
|
|
1296
|
+
│ exact dedup → near-dedup (byte-verified) → mint new node
|
|
1297
|
+
▼
|
|
1298
|
+
root node id + id of every subtree
|
|
1299
|
+
│
|
|
1300
|
+
├─ 5a. sub-span windows + containment edges (recognition seams)
|
|
1301
|
+
├─ 5b. whole-stream flat branch (canonical byte identity)
|
|
1302
|
+
│
|
|
1303
|
+
│ 6. RELATIONS
|
|
1304
|
+
│ single input: chain part → part continuation edges
|
|
1305
|
+
│ pair (context, continuation):
|
|
1306
|
+
│ edge context-root ──▶ continuation-root
|
|
1307
|
+
│ halos: pour each side's seat-bound company signature into the other
|
|
1308
|
+
▼
|
|
1309
|
+
updated store (DAG + edges + halos + lazily-updated vector indexes)
|
|
1310
|
+
```
|
|
1311
|
+
|
|
1312
|
+
---
|
|
1313
|
+
|
|
1314
|
+
## 10. Perception: from bytes to a tree
|
|
1315
|
+
|
|
1316
|
+
### 10.1 Every modality is a stream
|
|
1317
|
+
|
|
1318
|
+
Perception's contract is minimal: it accepts a _byte stream_ and returns a tree.
|
|
1319
|
+
Text becomes bytes by UTF-8 encoding; raw binary is already bytes; an image, a
|
|
1320
|
+
volume, or a stack of video frames is linearized along a Hilbert curve (§6.3) so
|
|
1321
|
+
that spatial locality becomes stream locality. Nothing downstream knows or cares
|
|
1322
|
+
which modality produced the stream — _geometry is only a reading order_.
|
|
1323
|
+
|
|
1324
|
+
### 10.2 The river fold
|
|
1325
|
+
|
|
1326
|
+
The fold builds the tree level by level, like a river merging tributaries:
|
|
1327
|
+
|
|
1328
|
+
```
|
|
1329
|
+
riverFold(leaves):
|
|
1330
|
+
level ← leaves
|
|
1331
|
+
while |level| > 1:
|
|
1332
|
+
next ← []
|
|
1333
|
+
for each complete group g of W consecutive items in level:
|
|
1334
|
+
next.append( foldGroup(g) ) # one parent node
|
|
1335
|
+
append the trailing incomplete items (fewer than W) unchanged
|
|
1336
|
+
level ← next # recurse upward
|
|
1337
|
+
normalize( level[0].gist ) # ONLY the finished root
|
|
1338
|
+
return level[0]
|
|
1339
|
+
|
|
1340
|
+
foldGroup(children c₀ … cₖ): # k < W
|
|
1341
|
+
gist ← Σᵢ πᵢ · gist(cᵢ) # seat-bind + superpose (§2.3)
|
|
1342
|
+
# — NOT normalized here
|
|
1343
|
+
return branch node with kids (c₀ … cₖ) and that gist
|
|
1344
|
+
```
|
|
1345
|
+
|
|
1346
|
+
Properties worth noting:
|
|
1347
|
+
|
|
1348
|
+
- **Determinism.** Grouping is purely positional; the same stream always yields
|
|
1349
|
+
the same tree shape and, therefore (given the fixed alphabet and keyring), the
|
|
1350
|
+
same gists everywhere.
|
|
1351
|
+
- **Logarithmic depth.** Each level shrinks by roughly a factor of W, so a
|
|
1352
|
+
stream of n bytes folds in ⌈log_W n⌉ levels.
|
|
1353
|
+
- **Every level is meaningful.** Intermediate nodes are not scaffolding to be
|
|
1354
|
+
discarded; each one is a content-addressable span with a gist — perception
|
|
1355
|
+
manufactures the _addressable sub-structure_ that recognition and attention
|
|
1356
|
+
later depend on.
|
|
1357
|
+
- **Linear, not renormalized per level.** Only the completed root is normalized
|
|
1358
|
+
to unit length; every interior gist is left at its raw superposed length. This
|
|
1359
|
+
is a deliberate choice of similarity semantics, not a shortcut: an interior
|
|
1360
|
+
node's magnitude grows with the amount of content folded into it (§2.6), so
|
|
1361
|
+
the fold gives every span both an angle (what it resembles) and a magnitude
|
|
1362
|
+
(how much of it there is) for free, in the same vector.
|
|
1363
|
+
- **W is the resolution quantum.** W bounds how much material one fold step
|
|
1364
|
+
mixes; it reappears throughout the system as the "one perceptual step" unit
|
|
1365
|
+
(the reach threshold, the near-dedup window, the fusible span ceiling,
|
|
1366
|
+
alignment seed size).
|
|
1367
|
+
|
|
1368
|
+
### 10.3 The stable prefix
|
|
1369
|
+
|
|
1370
|
+
One refinement protects structure across growing inputs. Consider training on a
|
|
1371
|
+
dialogue where each turn's context is the previous context plus one more
|
|
1372
|
+
exchange. Folded naively, adding bytes at the end can shift every group
|
|
1373
|
+
boundary, so the shared prefix folds _differently_ in each deposit — the store
|
|
1374
|
+
would never notice that the prefix is the same knowledge.
|
|
1375
|
+
|
|
1376
|
+
Perception therefore checks, before folding, whether some head of the stream is
|
|
1377
|
+
_already a known form_ (a store-recognised sequence of leaves — an exact,
|
|
1378
|
+
content-addressed check, not a similarity guess). If a known **proper** prefix
|
|
1379
|
+
of length p exists, the fold is split at p: the prefix folds exactly as it did
|
|
1380
|
+
when it was learned, the suffix folds independently, and the two join only at
|
|
1381
|
+
the top. Identical prefixes thus produce identical subtrees — and hash-consing
|
|
1382
|
+
then collapses them to the very same nodes — regardless of what follows them.
|
|
1383
|
+
|
|
1384
|
+
(The prefix must be _proper_ — shorter than the whole input — because a
|
|
1385
|
+
full-length match would mean the entire input is already stored, and splitting
|
|
1386
|
+
there would hide the true internal structure.)
|
|
1387
|
+
|
|
1388
|
+
### 10.4 Perception pseudocode, complete
|
|
1389
|
+
|
|
1390
|
+
```
|
|
1391
|
+
perceive(input):
|
|
1392
|
+
bytes ← flatten(input) # UTF-8 / identity / Hilbert
|
|
1393
|
+
if bytes is empty: return the empty tree
|
|
1394
|
+
leaves ← [ leaf(bᵢ, alphabet[bᵢ]) for each byte bᵢ ]
|
|
1395
|
+
p ← longest proper prefix of `bytes` whose leaf-sequence is a known form
|
|
1396
|
+
return riverFold(leaves, splitAt = p if p > 0 else none)
|
|
1397
|
+
|
|
1398
|
+
# riverFold with a split: at every level, items are partitioned at the
|
|
1399
|
+
# boundary containing byte-offset p; each side folds as if standalone.
|
|
1400
|
+
```
|
|
1401
|
+
|
|
1402
|
+
---
|
|
1403
|
+
|
|
1404
|
+
## 11. Deposition: interning the tree into the graph
|
|
1405
|
+
|
|
1406
|
+
### 11.1 Bottom-up interning
|
|
1407
|
+
|
|
1408
|
+
The perceived tree is interned into the DAG bottom-up:
|
|
1409
|
+
|
|
1410
|
+
```
|
|
1411
|
+
intern(tree node n) → node id:
|
|
1412
|
+
if n is a leaf:
|
|
1413
|
+
return internLeaf(n.bytes, n.gist)
|
|
1414
|
+
kidIds ← [ intern(k) for k in n.kids ] # children first
|
|
1415
|
+
return internBranch(kidIds, n.gist)
|
|
1416
|
+
```
|
|
1417
|
+
|
|
1418
|
+
Both interning operations follow the same ladder:
|
|
1419
|
+
|
|
1420
|
+
```
|
|
1421
|
+
internLeaf / internBranch(content, gist):
|
|
1422
|
+
1. EXACT DEDUP if a node with this exact content exists → return it
|
|
1423
|
+
1b. CROSS-REP if this is a branch whose kid ids flatten to a known
|
|
1424
|
+
flat branch's bytes, reuse that flat branch's id
|
|
1425
|
+
(§3.2 — content addressing across representations)
|
|
1426
|
+
2. NEAR DEDUP (branches only, against whole-experience roots only)
|
|
1427
|
+
if some fresh root's gist is within the merge
|
|
1428
|
+
threshold (§8.1) AND the two byte strings differ by
|
|
1429
|
+
at most ONE local span of ≤ W bytes → return that id
|
|
1430
|
+
3. MINT otherwise create a new node; record, for each child,
|
|
1431
|
+
the reverse (child → parent) structural edge
|
|
1432
|
+
```
|
|
1433
|
+
|
|
1434
|
+
Points of principle:
|
|
1435
|
+
|
|
1436
|
+
- **Exact dedup is the primary compression** and is intrinsic: it is what makes
|
|
1437
|
+
identity a function of content (§3.1). It works for leaves and branches alike.
|
|
1438
|
+
- **Near-dedup is deliberately narrow.** It applies only to branches, only
|
|
1439
|
+
against _genuine whole experiences_ (roots that bear edges or halos — never
|
|
1440
|
+
interior scaffolding), and only with byte verification. The geometric bar
|
|
1441
|
+
alone is scale-blind: in a deep fold, a large localized difference dilutes
|
|
1442
|
+
toward cosine 1, so _any_ fixed bar below 1 would eventually merge things that
|
|
1443
|
+
differ in exactly the span that matters. The byte check — identical except one
|
|
1444
|
+
window of at most W bytes — is the perception system's own definition of "the
|
|
1445
|
+
smallest real difference", so the merge can never corrupt reconstruction.
|
|
1446
|
+
- **The parent edges minted in step 3 are the climb.** Every branch records
|
|
1447
|
+
itself as a parent of each distinct child. These reverse edges are what later
|
|
1448
|
+
lets a recognised fragment climb to the experiences containing it (§17).
|
|
1449
|
+
Single-byte leaves are exempt: a byte occurs in nearly everything, so its
|
|
1450
|
+
parent set would be a useless corpus-sized hub.
|
|
1451
|
+
|
|
1452
|
+
### 11.2 Why interior nodes matter
|
|
1453
|
+
|
|
1454
|
+
After interning, _every_ subtree of the deposit — not just the root — is an
|
|
1455
|
+
addressable node. This is not an implementation convenience; it is the mechanism
|
|
1456
|
+
behind three capabilities:
|
|
1457
|
+
|
|
1458
|
+
- **Partial recall**: a query naming only a slice of an experience can resonate
|
|
1459
|
+
with that slice's node directly.
|
|
1460
|
+
- **Multi-topic attention**: different regions of one query can anchor to
|
|
1461
|
+
interior nodes of _different_ experiences (§17).
|
|
1462
|
+
- **Compositional generalization**: fresh input that shares any sub-span with
|
|
1463
|
+
old input shares nodes with it, so the new is literally built out of the old.
|
|
1464
|
+
|
|
1465
|
+
### 11.3 Sub-span windows and containment
|
|
1466
|
+
|
|
1467
|
+
Perception's grouping is positional, so a meaningful unit (say, a name) may
|
|
1468
|
+
straddle a group boundary and never be a node of any tree. Deposition therefore
|
|
1469
|
+
additionally interns **sliding windows** of W and W−1 leaves across the stream,
|
|
1470
|
+
as flat branches. A window that does not coincide with a structural child of any
|
|
1471
|
+
chunk is linked to the chunk(s) it overlaps by a durable **containment edge** —
|
|
1472
|
+
a second, weaker parent relation meaning "these bytes occur inside that chunk".
|
|
1473
|
+
When a later climb starts from such a window (which has no structural parents of
|
|
1474
|
+
its own), it climbs through its containment parents instead. This closes the
|
|
1475
|
+
recognition seams that pure positional chunking would leave.
|
|
1476
|
+
|
|
1477
|
+
Deposition also interns the **whole stream as one flat branch** (the sequence of
|
|
1478
|
+
its byte-leaves). This gives every deposit a canonical byte-level identity
|
|
1479
|
+
independent of tree shape — the form the stable-prefix check of §10.3 looks up,
|
|
1480
|
+
and a second content-addressed route to the same experience.
|
|
1481
|
+
|
|
1482
|
+
---
|
|
1483
|
+
|
|
1484
|
+
## 12. Learning relations: edges and halos
|
|
1485
|
+
|
|
1486
|
+
### 12.1 Continuation edges: the atom of factual knowledge
|
|
1487
|
+
|
|
1488
|
+
A **fact**, in Sema, is an ordered association: _this_ was followed by _that_.
|
|
1489
|
+
Depositing a pair (context, continuation) records one continuation edge from the
|
|
1490
|
+
context's root node to the continuation's root node. Edges are:
|
|
1491
|
+
|
|
1492
|
+
- **Idempotent** — the same pair deposited twice is one edge (though its halo
|
|
1493
|
+
evidence accumulates; see below).
|
|
1494
|
+
- **Directional** — "what follows X" (forward) and "what does X follow"
|
|
1495
|
+
(reverse) are both readable, and both are used: forward for answering, reverse
|
|
1496
|
+
for recognising that a query _is_ some context's answer (reverse recall,
|
|
1497
|
+
§21.1) and for counting evidence (§25).
|
|
1498
|
+
- **Plural** — a context may accumulate many continuations (the same question
|
|
1499
|
+
answered differently across a corpus). Choosing among them is a first-class
|
|
1500
|
+
disambiguation problem (§25), not an error.
|
|
1501
|
+
|
|
1502
|
+
A _single_ input (no pair) still learns sequence: the parts of its root are
|
|
1503
|
+
chained by edges at stride W, so a long document is traversable as a sequence of
|
|
1504
|
+
its chunks.
|
|
1505
|
+
|
|
1506
|
+
### 12.2 Halo pours: distributional bookkeeping
|
|
1507
|
+
|
|
1508
|
+
When a pair (context, continuation) is deposited, each side's **company
|
|
1509
|
+
signature** — a deterministic unit vector derived from the partner's node
|
|
1510
|
+
identity — is superposed ("poured") into the other side's halo, bound to a role
|
|
1511
|
+
seat so that "I appeared as context" and "I appeared as answer" are
|
|
1512
|
+
geometrically distinct:
|
|
1513
|
+
|
|
1514
|
+
```
|
|
1515
|
+
pour( halo(contextPart) , π₁ · companySignature(continuation) )
|
|
1516
|
+
pour( halo(continuation) , π₀ · companySignature(contextPart) )
|
|
1517
|
+
```
|
|
1518
|
+
|
|
1519
|
+
The company signature is seeded by the partner's node id. Node ids are
|
|
1520
|
+
content-addressed (mint order), stable for a given corpus (including
|
|
1521
|
+
checkpoint/resume, which re-derives identical ids), so the same partner always
|
|
1522
|
+
contributes the same signature. But two partners with the _same bytes_ and
|
|
1523
|
+
_different ids_ contribute nearly orthogonal signatures — so two halos correlate
|
|
1524
|
+
only through shared episode history, never through accidental byte-level content
|
|
1525
|
+
overlap.
|
|
1526
|
+
|
|
1527
|
+
Over many episodes, a node's halo becomes the superposition of everything it has
|
|
1528
|
+
kept company with — the distributional signature of §4. Each pour also
|
|
1529
|
+
increments the node's **halo mass**, the direct count of corroborating episodes.
|
|
1530
|
+
|
|
1531
|
+
One subtlety guards the signal's quality: in a _tracked_ sequence of deposits
|
|
1532
|
+
(e.g. a growing dialogue), the context's pour targets only the **changed nodes**
|
|
1533
|
+
— the subtree that is new relative to the previous deposit — so a boilerplate
|
|
1534
|
+
prefix repeated in every turn does not soak up halo mass and drown the
|
|
1535
|
+
discriminating content.
|
|
1536
|
+
|
|
1537
|
+
### 12.3 Lazy indexing: what enters the vector indexes, and when
|
|
1538
|
+
|
|
1539
|
+
The DAG holds every node, but the _content index_ (the ANN structure over gists,
|
|
1540
|
+
§6) holds only nodes worth resonating to — and admission is lazy:
|
|
1541
|
+
|
|
1542
|
+
- A node's gist is _captured_ at intern time but _indexed_ only when the node
|
|
1543
|
+
becomes a **resonance target**: it gains an edge, gains a halo, is a deposit
|
|
1544
|
+
root, or is an interior form of an experience (when either end of an edge is
|
|
1545
|
+
learned, that experience's whole subtree is admitted, because partial queries
|
|
1546
|
+
must be able to resonate with its interior).
|
|
1547
|
+
- A node that structurally **bridges** two experiences (its parent count crosses
|
|
1548
|
+
1 → 2) is promoted at exactly that moment — the moment it becomes useful for
|
|
1549
|
+
cross-experience recall.
|
|
1550
|
+
- Pure intermediate scaffolding that never becomes any of those is never indexed
|
|
1551
|
+
at all. It still exists in the DAG (identity, reconstruction, climbing all
|
|
1552
|
+
work); it simply is not a resonance destination.
|
|
1553
|
+
|
|
1554
|
+
The halo index is maintained on a geometric schedule (a node's halo is
|
|
1555
|
+
re-indexed when its mass is small or crosses a power of two), since a halo's
|
|
1556
|
+
_direction_ stabilizes as mass grows.
|
|
1557
|
+
|
|
1558
|
+
The principle: **the DAG is the truth; the indexes are lazy, rebuildable views
|
|
1559
|
+
of the parts of the truth that queries actually land on.**
|
|
1560
|
+
|
|
1561
|
+
---
|
|
1562
|
+
|
|
1563
|
+
## 13. Ingestion, end to end
|
|
1564
|
+
|
|
1565
|
+
The complete deposit algorithm, in pseudocode:
|
|
1566
|
+
|
|
1567
|
+
```
|
|
1568
|
+
ingest(input, second = none):
|
|
1569
|
+
|
|
1570
|
+
# ── forms ────────────────────────────────────────────────────────────
|
|
1571
|
+
if input is a list of items / (context, continuation) pairs:
|
|
1572
|
+
for each element: ingest it by the rules below
|
|
1573
|
+
return
|
|
1574
|
+
|
|
1575
|
+
if second is given: ingestPair(input, second)
|
|
1576
|
+
else: ingestOne(input)
|
|
1577
|
+
|
|
1578
|
+
|
|
1579
|
+
deposit(input, tracked):
|
|
1580
|
+
bytes ← flatten(input)
|
|
1581
|
+
tree ← perceive(bytes) # §10
|
|
1582
|
+
ids ← intern every node of tree, bottom-up # §11.1
|
|
1583
|
+
intern sliding W / W−1 windows; record containment edges # §11.3
|
|
1584
|
+
intern the whole stream as a flat branch # §11.3
|
|
1585
|
+
changed ← if tracked and a previous deposit exists:
|
|
1586
|
+
the maximal new subtree vs. the previous deposit # §12.2
|
|
1587
|
+
else: [ tree ]
|
|
1588
|
+
return (tree, rootId, ids, changed)
|
|
1589
|
+
|
|
1590
|
+
|
|
1591
|
+
ingestOne(input): # a bare experience
|
|
1592
|
+
(tree, root, ids, _) ← deposit(input, tracked = true)
|
|
1593
|
+
mark root as a resonance target # §12.3
|
|
1594
|
+
parts ← the root's immediate children
|
|
1595
|
+
if |parts| > W:
|
|
1596
|
+
link parts[i] ──▶ parts[i+W] for each stride-W step # §12.1
|
|
1597
|
+
else:
|
|
1598
|
+
mark each part as a resonance target
|
|
1599
|
+
|
|
1600
|
+
|
|
1601
|
+
ingestPair(context, continuation): # a fact
|
|
1602
|
+
(ctxTree, ctxRoot, ctxIds, changed) ← deposit(context, tracked = true)
|
|
1603
|
+
(conTree, conRoot, _, _ ) ← deposit(continuation, tracked = false)
|
|
1604
|
+
|
|
1605
|
+
link ctxRoot ──▶ conRoot # the fact itself
|
|
1606
|
+
for each part in changed: # distributional evidence
|
|
1607
|
+
pour halo(part) += π₁ · companySignature(conRoot)
|
|
1608
|
+
pour halo(conRoot) += π₀ · companySignature(part)
|
|
1609
|
+
# linking / pouring admits both subtrees' interiors to the content
|
|
1610
|
+
# index (lazily), per §12.3
|
|
1611
|
+
```
|
|
1612
|
+
|
|
1613
|
+
Costs, in broad strokes: perception is linear in the input length; interning is
|
|
1614
|
+
one content-addressed lookup per tree node (the tree has O(n/W · logᵂ n) nodes
|
|
1615
|
+
but is dominated by its O(n) leaves); relation learning is O(1) edges plus
|
|
1616
|
+
O(changed parts) halo pours. Nothing in the deposit path scans the corpus.
|
|
1617
|
+
**Training a fact takes one pass over the fact.**
|
|
1618
|
+
|
|
1619
|
+
### 13.1 Why storage stays viable: the economics of the store
|
|
1620
|
+
|
|
1621
|
+
The claims above ("one pass", "nothing scans the corpus", "bounded RAM") are not
|
|
1622
|
+
free consequences of the data model — they are earned by a specific set of
|
|
1623
|
+
cost-control mechanisms in the store. They deserve their own account, because
|
|
1624
|
+
each one is a _deliberate trade_ whose failure mode is well understood, and
|
|
1625
|
+
together they are what makes a corpus-scale store run on ordinary hardware. The
|
|
1626
|
+
organizing principle:
|
|
1627
|
+
|
|
1628
|
+
> **Exactness is mandatory only for identity and reconstruction. Everything else
|
|
1629
|
+
> — caches, indexes, buffers — is a bounded, rebuildable accelerator whose miss
|
|
1630
|
+
> costs work, never correctness.**
|
|
1631
|
+
|
|
1632
|
+
The mechanisms, each with its trade:
|
|
1633
|
+
|
|
1634
|
+
**1. Implicit leaves and flat branches (representation compression).** Single
|
|
1635
|
+
bytes are not stored at all — a byte's node id is derived from its value (the
|
|
1636
|
+
negative range), so the 256 most common nodes in existence cost zero rows. A
|
|
1637
|
+
branch whose children are all single-byte leaves (the vast majority of small
|
|
1638
|
+
spans) stores its _bytes_ (1 byte per child) instead of a packed child-id list
|
|
1639
|
+
(4 bytes per child) — a 4× saving on the store's most numerous row shape,
|
|
1640
|
+
content-addressed through the same lookup path.
|
|
1641
|
+
|
|
1642
|
+
**2. Bounded caches everywhere (RAM viability).** Every in-memory acceleration
|
|
1643
|
+
structure — the exact-dedup key maps, the reconstructed-bytes cache, the
|
|
1644
|
+
node-record cache, the pending-gist capture, the exact halo accumulators — is an
|
|
1645
|
+
LRU map with a _byte budget_, not an entry count. Reconstruction caches evict
|
|
1646
|
+
smallest-first (protecting entries that are expensive to rebuild); all others
|
|
1647
|
+
evict least-recently used. A miss re-derives from durable state. Resident memory
|
|
1648
|
+
is therefore capped by configuration regardless of corpus size.
|
|
1649
|
+
|
|
1650
|
+
**3. Lazy, selective vector indexing (index viability).** The ANN index is the
|
|
1651
|
+
most expensive thing the store maintains — every entry costs an encode, graph
|
|
1652
|
+
edges, and future query work. So admission is lazy and selective (§12.3): a
|
|
1653
|
+
node's gist is _captured_ cheaply at intern time into a bounded buffer, and
|
|
1654
|
+
_indexed_ only at the moment the node demonstrably becomes a resonance
|
|
1655
|
+
destination — when it gains an edge (which also admits its whole subtree's
|
|
1656
|
+
interior forms), gains a halo, is a deposit root, or crosses the 1→2 parent
|
|
1657
|
+
transition that makes it a bridge between experiences. Pure scaffolding is never
|
|
1658
|
+
indexed. The trade: an evicted-before-promotion gist means that node is
|
|
1659
|
+
reachable only by the structural climb until a batch repair pass regenerates it
|
|
1660
|
+
— reduced reach, never wrong answers.
|
|
1661
|
+
|
|
1662
|
+
**4. No per-branch ANN probes on the write path (write viability).**
|
|
1663
|
+
Near-deduplication (§11.1 step 2) consults only the _write buffer's_ few
|
|
1664
|
+
whole-experience roots — an O(buffer) exact scan — never the flushed ANN index.
|
|
1665
|
+
Probing the ANN index for every new branch is both the dominant potential
|
|
1666
|
+
training cost and _unsound_: 1-bit estimates can rank a byte-distinct branch
|
|
1667
|
+
nearest, and merging on that corrupts reconstruction. The principle:
|
|
1668
|
+
**approximate structures are kept off the write path entirely.**
|
|
1669
|
+
|
|
1670
|
+
**5. Write batching with deferred durability (I/O viability).** Node rows,
|
|
1671
|
+
edges, halos, containment sets, and vector-index entries accumulate in buffers
|
|
1672
|
+
and commit in coalesced batches (one transaction, one index upsert) once a size
|
|
1673
|
+
threshold is reached — turning what would be per-node fsync-bounded writes into
|
|
1674
|
+
large sequential ones. Within a batch, repeated halo pours to the same node
|
|
1675
|
+
coalesce to one index write.
|
|
1676
|
+
|
|
1677
|
+
**6. Two quantizations, two purposes (halo viability).** Halo accumulators would
|
|
1678
|
+
otherwise be the largest table in an episodically-trained store (one float
|
|
1679
|
+
vector per fact-bearing node). Gists avoid this problem entirely — a gist is a
|
|
1680
|
+
deterministic function of a node's content (`perceive → fold`), so it can live
|
|
1681
|
+
in a volatile buffer and be regenerated on demand. A halo is a function of
|
|
1682
|
+
_training history_ (the sum of every episode signature poured into it) — it
|
|
1683
|
+
cannot be regenerated from the node's bytes and must be durable. The system uses
|
|
1684
|
+
two different quantizations because durability and search have different
|
|
1685
|
+
requirements:
|
|
1686
|
+
|
|
1687
|
+
- **Durable row: 2-bit Lloyd–Max, reversible.** The halo vector is stored on
|
|
1688
|
+
disk at 16× compression (260 bytes for D=1024 vs. 4096 for float32). It must
|
|
1689
|
+
be decodable back to an approximate float32 vector because it serves as an
|
|
1690
|
+
_accumulator_: a session loads it, adds new pours to it in float32, and writes
|
|
1691
|
+
it back. The round-trip through the quantizer preserves ≥ 0.88 correlation
|
|
1692
|
+
with the exact accumulator — the coarsest grain that survives repeated
|
|
1693
|
+
load→accumulate→flush cycles without the direction drifting. One bit would
|
|
1694
|
+
_not_ suffice for this purpose: decoded back, a sign-only vector is binary
|
|
1695
|
+
(±1), and accumulating on top of it degrades with every cycle.
|
|
1696
|
+
|
|
1697
|
+
- **ANN index: 1-bit RaBitQ, irreversible.** The same halo vector, projected
|
|
1698
|
+
through a random rotation and reduced to one sign bit per dimension (32×
|
|
1699
|
+
compression), serves as a search code in the HNSW index. This code is _never_
|
|
1700
|
+
decoded back — it only answers "which halos are near this query?" The
|
|
1701
|
+
estimator is unbiased (expected cosine is recoverable from the bit count), so
|
|
1702
|
+
ranking quality is preserved despite the loss of reversibility.
|
|
1703
|
+
|
|
1704
|
+
The session's actively poured accumulators are kept exact in a bounded float32
|
|
1705
|
+
cache, so within-session accumulate-then-compare never round-trips through
|
|
1706
|
+
either quantizer. The ANN index re-enters only when a halo's mass is small or
|
|
1707
|
+
crosses a power of two (mechanism 7 below), at which point the durable 2-bit row
|
|
1708
|
+
is decoded to float32, normalized, rotated, and re-encoded to 1-bit RaBitQ.
|
|
1709
|
+
|
|
1710
|
+
**7. Geometric re-index schedule (index-write viability).** A halo's _direction_
|
|
1711
|
+
stabilizes as mass grows, so re-indexing it on every pour is waste. Halos
|
|
1712
|
+
re-enter the index only when their mass is small or crosses a power of two —
|
|
1713
|
+
O(log mass) index writes per node over its lifetime instead of O(mass) — while
|
|
1714
|
+
the durable (quantized) row is always current.
|
|
1715
|
+
|
|
1716
|
+
**8. Counting instead of materialising (read viability).** Evidence questions on
|
|
1717
|
+
the hot path ("how many distinct contexts predict this?", "how many learned
|
|
1718
|
+
contexts exist?", "does this lead somewhere?") are answered by indexed counts
|
|
1719
|
+
(`prevCount`, an incrementally-maintained distinct-source count for `corpusN`,
|
|
1720
|
+
`hasNext`, `hasHalo`) — never by materialising corpus-sized edge lists. Full
|
|
1721
|
+
materialising reads (`prev`, `parents`) exist for maintenance and inspection but
|
|
1722
|
+
are kept off every hot path. The hub bound √N (§8.8) is enforced at the _store
|
|
1723
|
+
level_: every fan-out read on the hot path uses a LIMITed variant
|
|
1724
|
+
(`nextFirst(id, √N)`, `prevFirst(id, √N)`, `parentsFirst(id, √N+1)`,
|
|
1725
|
+
`containersSlice(id, offset, √N)`) whose work is bounded by the cap regardless
|
|
1726
|
+
of the actual fan-out size. A consumer that needs the full fan-out (the
|
|
1727
|
+
consensus climb, §17) instead uses these to decide its question exactly — "does
|
|
1728
|
+
this reach cross √N distinct contexts?" — without ever materialising a
|
|
1729
|
+
corpus-sized list. The principle: **a per-query read must never grow with the
|
|
1730
|
+
corpus.**
|
|
1731
|
+
|
|
1732
|
+
**9. Compaction on write-volume cadence (long-run viability).** Updates and
|
|
1733
|
+
deletions in the vector indexes leave tombstones. After every configured volume
|
|
1734
|
+
of index writes, an index whose physical size exceeds 2× its live size is
|
|
1735
|
+
rebuilt from its surviving codes (lossless — the build is code-based).
|
|
1736
|
+
Post-training, a batch pass can additionally remove index entries for
|
|
1737
|
+
structurally isolated nodes (single-parent, no edges, no halo — they bridge
|
|
1738
|
+
nothing), and a converse repair pass re-indexes bridges that were missed.
|
|
1739
|
+
Failures of these best-effort passes are counted visibly, never silent.
|
|
1740
|
+
|
|
1741
|
+
### 13.2 The store's cost machinery, in pseudocode
|
|
1742
|
+
|
|
1743
|
+
```
|
|
1744
|
+
# ── the intern ladder with its caches (§11.1, refined) ────────────────
|
|
1745
|
+
intern(content, gist): # content = bytes | kidIds
|
|
1746
|
+
# 1. exact dedup: bounded LRU first, durable probe second — so dedup
|
|
1747
|
+
# survives a cold cache (a resumed run still recognises old content)
|
|
1748
|
+
id ≔ dedupCache[key(content)] ?? durableFind(content)
|
|
1749
|
+
if id ≠ ∅:
|
|
1750
|
+
dedupCache[key(content)] ≔ id
|
|
1751
|
+
captureIfUnindexed(id, gist) # keep the gist available for
|
|
1752
|
+
return id # lazy indexing (mech. 3)
|
|
1753
|
+
# 1b. content addressing across representations: a branch spanning the
|
|
1754
|
+
# same bytes as a stored flat branch reuses that id
|
|
1755
|
+
if content is kidIds and their leaf-id flattening is a known branch:
|
|
1756
|
+
return that id (capture as above)
|
|
1757
|
+
# 2. near-dedup: BUFFERED whole-experience roots only — never an ANN
|
|
1758
|
+
# probe (mech. 4); geometric proposal + byte verification (§11.1)
|
|
1759
|
+
if content is kidIds:
|
|
1760
|
+
best ≔ argmax over nearDedupBuffer of dot(gist, root.gist)
|
|
1761
|
+
if best ≥ MERGE and differsByOneWindow(content, best, W):
|
|
1762
|
+
return best.id
|
|
1763
|
+
# 3. mint
|
|
1764
|
+
id ≔ nextId++ # dense ids; leaves are implicit
|
|
1765
|
+
write node row (flat-branch encoding when applicable — mech. 1)
|
|
1766
|
+
for each real child c: # (byte leaves get no parent
|
|
1767
|
+
insert kid edge c → id # rows — hub avoidance)
|
|
1768
|
+
if |parents(c)| just became 2: # the 1→2 bridge transition
|
|
1769
|
+
indexGist(c) # promote NOW (mech. 3)
|
|
1770
|
+
pendingGist[id] ≔ gist # captured, not yet indexed
|
|
1771
|
+
maybeFlush()
|
|
1772
|
+
return id
|
|
1773
|
+
|
|
1774
|
+
# ── lazy index admission (mech. 3) ────────────────────────────────────
|
|
1775
|
+
indexGist(id, dedupTarget = false):
|
|
1776
|
+
if id already indexed (session set, else one durable point query):
|
|
1777
|
+
return # a resumed run replays deposits
|
|
1778
|
+
# at read speed — no re-upserts
|
|
1779
|
+
v ≔ pendingGist[id]; if v = ∅: return # evicted ⇒ retry on a future
|
|
1780
|
+
# encounter or repair pass
|
|
1781
|
+
contentBuffer += (id, v); mark indexed
|
|
1782
|
+
if dedupTarget: nearDedupBuffer += (id, v)
|
|
1783
|
+
|
|
1784
|
+
indexSubtree(root): # fired by link() on both ends
|
|
1785
|
+
indexGist(root, dedupTarget = true) # only the ROOT may be merged
|
|
1786
|
+
walk the subtree, pruning at already-classified nodes:
|
|
1787
|
+
indexGist(interior) # reach-only: partial queries
|
|
1788
|
+
# must resonate with interiors
|
|
1789
|
+
|
|
1790
|
+
# ── halo pour (mech. 6–7) ─────────────────────────────────────────────
|
|
1791
|
+
pourHalo(id, addVec):
|
|
1792
|
+
indexGist(id, dedupTarget = true) # a halo-bearer is a target
|
|
1793
|
+
acc ≔ exactCache[id] ?? dequantize(durableRow[id]) ?? 0⃗
|
|
1794
|
+
acc += addVec; exactCache[id] ≔ acc # exact in-session
|
|
1795
|
+
mass += 1
|
|
1796
|
+
durableRow[id] ≔ quantize2bit(acc), mass # always current, 16× small
|
|
1797
|
+
if mass ≤ 4 or mass is a power of two: # geometric schedule
|
|
1798
|
+
haloBuffer[id] ≔ normalize(acc) # O(log mass) index writes
|
|
1799
|
+
|
|
1800
|
+
quantize2bit(v): # Lloyd–Max for unit Gaussian
|
|
1801
|
+
store ‖v‖ exactly; per coordinate: sign bit + |x| ≷ 0.9816·σ bit
|
|
1802
|
+
(σ ≔ ‖v‖/√D; decode to ±0.4528σ / ±1.5104σ, rescaled to ‖v‖)
|
|
1803
|
+
|
|
1804
|
+
# ── batching and compaction (mech. 5, 9) ──────────────────────────────
|
|
1805
|
+
maybeFlush():
|
|
1806
|
+
if |contentBuffer| + |haloBuffer| ≥ batchSize: flush()
|
|
1807
|
+
|
|
1808
|
+
flush():
|
|
1809
|
+
merge containment buffer into packed rows
|
|
1810
|
+
upsert contentBuffer into the content index (one batch)
|
|
1811
|
+
upsert haloBuffer into the halo index (one batch)
|
|
1812
|
+
clear nearDedupBuffer (mirrors contentBuffer)
|
|
1813
|
+
commit the deferred write transaction (one commit per batch)
|
|
1814
|
+
writtenSinceCompact += batch size
|
|
1815
|
+
if writtenSinceCompact ≥ compactEvery:
|
|
1816
|
+
for each vector index with physicalSize > 2 × liveSize:
|
|
1817
|
+
rebuild it from its live codes (lossless; code-based)
|
|
1818
|
+
|
|
1819
|
+
# ── post-hoc maintenance (mech. 9) ────────────────────────────────────
|
|
1820
|
+
compactContentIndex(minParents = 2): # archived-store trade:
|
|
1821
|
+
remove entries with < minParents parents and no edges and no halo
|
|
1822
|
+
# they bridge nothing
|
|
1823
|
+
repairContentIndex(regenerate, minParents = 2):
|
|
1824
|
+
for each branch node not in the index with ≥ minParents parents and
|
|
1825
|
+
(edges or a halo):
|
|
1826
|
+
re-perceive its bytes; add its gist to the index
|
|
1827
|
+
```
|
|
1828
|
+
|
|
1829
|
+
Two summary facts fall out of this machinery. First, the _asymptotics_: storage
|
|
1830
|
+
is O(distinct subtrees); the ANN index is bounded by the number of distinct byte
|
|
1831
|
+
patterns that ever became resonance destinations (not by deposits, and not by
|
|
1832
|
+
corpus volume — hash-consing sees to that); every per-deposit cost is a bounded
|
|
1833
|
+
number of O(1) probes and amortized-O(1) buffered writes. Second, the
|
|
1834
|
+
_degradation order_: under memory pressure or eviction, what is lost is always
|
|
1835
|
+
acceleration (a duplicate probe, a re-perception, reduced resonance reach until
|
|
1836
|
+
repair) and never identity, reconstruction, or an already-learned relation.
|
|
1837
|
+
|
|
1838
|
+
---
|
|
1839
|
+
|
|
1840
|
+
---
|
|
1841
|
+
|
|
1842
|
+
# Part IV — The inference pipeline
|
|
1843
|
+
|
|
1844
|
+
## 14. The shape of an answer
|
|
1845
|
+
|
|
1846
|
+
### 14.1 The pipeline at a glance
|
|
1847
|
+
|
|
1848
|
+
Every ask travels one road:
|
|
1849
|
+
|
|
1850
|
+
```
|
|
1851
|
+
THE INFERENCE PIPELINE
|
|
1852
|
+
|
|
1853
|
+
query bytes
|
|
1854
|
+
│
|
|
1855
|
+
│ perceive (same fold as ingestion — the query gets a tree & gists)
|
|
1856
|
+
▼
|
|
1857
|
+
┌───────────────────────────────────────────────────────────────────┐
|
|
1858
|
+
│ RECOGNISE (§15) every stored form the query contains │
|
|
1859
|
+
│ COMPUTE (§16) every mechanism's parse() evaluates spans it │
|
|
1860
|
+
│ is authoritative for (arithmetic, logic, …) │
|
|
1861
|
+
│ PRECOMPUTE build Precomputed: recognition, computed │
|
|
1862
|
+
│ spans, the response's gist — shared, │
|
|
1863
|
+
│ response-scoped data every mechanism (and the │
|
|
1864
|
+
│ post-grounding stages) read; every expensive │
|
|
1865
|
+
│ analysis (consensus climb, weave, span-shape) │
|
|
1866
|
+
│ is a lazily-cached method, computed at most │
|
|
1867
|
+
│ once and only if some consumer asks │
|
|
1868
|
+
└──────────────────────────────┬────────────────────────────────────┘
|
|
1869
|
+
▼
|
|
1870
|
+
┌───────────────────────────────────────────────────────────────────┐
|
|
1871
|
+
│ GROUND — ONE lightest-derivation choice among UNIFORM mechanisms: │
|
|
1872
|
+
│ │
|
|
1873
|
+
│ The pipeline sees every mechanism through the SAME │
|
|
1874
|
+
│ PipelineMechanism interface (parse?/floor/run) — no branch │
|
|
1875
|
+
│ anywhere names a specific mechanism or asks "is this an │
|
|
1876
|
+
│ extension?". Each mechanism's `floor` yields an admissible │
|
|
1877
|
+
│ lower bound (or null when it structurally cannot fire); each │
|
|
1878
|
+
│ `run` yields CANDIDATEs weighed in the one cost ladder (§8.9), │
|
|
1879
|
+
│ and the lightest grounding derivation wins. A candidate's │
|
|
1880
|
+
│ weight is: │
|
|
1881
|
+
│ │
|
|
1882
|
+
│ moves + PASS · (query bytes the mechanism did not account │
|
|
1883
|
+
│ for — did not match against learnt │
|
|
1884
|
+
│ structure) │
|
|
1885
|
+
│ │
|
|
1886
|
+
│ — PASS per unexplained byte is the cover's own price for a │
|
|
1887
|
+
│ literal connective, so the primary axis is "which mechanism │
|
|
1888
|
+
│ explains more of the query", and move costs (STEP per │
|
|
1889
|
+
│ projection, CONCEPT per halo-mediated act) discriminate │
|
|
1890
|
+
│ residually. Ties keep the mechanism list's own order. │
|
|
1891
|
+
│ │
|
|
1892
|
+
│ Admissible-floor pruning, uniformly: `floor` is called for │
|
|
1893
|
+
│ EVERY mechanism, every time; `run` only for one whose floor │
|
|
1894
|
+
│ can still beat the incumbent. A mechanism whose floor itself │
|
|
1895
|
+
│ needs expensive precomputation to refine (CAST's weave │
|
|
1896
|
+
│ alignment) checks the SAME incumbent, via a `worthRunning` │
|
|
1897
|
+
│ predicate passed into `floor`, before paying for it — the │
|
|
1898
|
+
│ pipeline's own pruning tool, exposed one level earlier, so no │
|
|
1899
|
+
│ mechanism special-cases what beat it. │
|
|
1900
|
+
│ │
|
|
1901
|
+
│ The mechanisms, in list order (cover runs first: a computed │
|
|
1902
|
+
│ span — from the ALU or any user extension — masks in at near- │
|
|
1903
|
+
│ zero cost, so a cheap incumbent is established before CAST/ │
|
|
1904
|
+
│ confluence would otherwise invest in their own precomputation): │
|
|
1905
|
+
│ │
|
|
1906
|
+
│ 1. COVER (§19) the query's own decomposition composes an │
|
|
1907
|
+
│ answer — ONE lightest-derivation search; │
|
|
1908
|
+
│ computed spans (§16) mask colliding sites │
|
|
1909
|
+
│ and enter the search at zero cost │
|
|
1910
|
+
│ 2. CAST (§18) the query weaves ≥2 independent learned │
|
|
1911
|
+
│ structures → transfer structure between them │
|
|
1912
|
+
│ 3. CONFLUENCE (§18.5) the query carries ≥2 independent │
|
|
1913
|
+
│ constraints → intersect their evidence │
|
|
1914
|
+
│ 4. EXTRACT (§20) a learned span-in-context skill reads the │
|
|
1915
|
+
│ analogous span out of the query │
|
|
1916
|
+
│ 5. RECALL (§21) whole-query resonance, four graded tiers │
|
|
1917
|
+
│ (…or NOTHING — silence below the reach bar) │
|
|
1918
|
+
└──────────────────────────────┬────────────────────────────────────┘
|
|
1919
|
+
▼
|
|
1920
|
+
┌───────────────────────────────────────────────────────────────────┐
|
|
1921
|
+
│ REASON (§22) extend the grounded answer across facts, hop by hop │
|
|
1922
|
+
│ FUSE (§23) ground the query's OTHER points of attention and │
|
|
1923
|
+
│ join them with learned connectors │
|
|
1924
|
+
│ ARTICULATE(§24) re-voice the result in the asker's own vocabulary │
|
|
1925
|
+
└──────────────────────────────┬────────────────────────────────────┘
|
|
1926
|
+
▼
|
|
1927
|
+
answer bytes + provenance + (optionally) the full rationale
|
|
1928
|
+
```
|
|
1929
|
+
|
|
1930
|
+
The decider also emits three diagnostic signals — purely observational, never
|
|
1931
|
+
affecting the decision itself:
|
|
1932
|
+
|
|
1933
|
+
- **Unexplained label** — every candidate carries `unexplained`, a
|
|
1934
|
+
human-readable label for the query bytes its evidence left on the table.
|
|
1935
|
+
Appears in the rationale trace; does not affect the weight (the PASS-per-byte
|
|
1936
|
+
pricing already accounts for it arithmetically).
|
|
1937
|
+
- **Narrow decision** — when the winner beats the runner-up by ≤ 1 grade unit
|
|
1938
|
+
(⌊w/STEP⌋), the rationale records a `narrowDecision` step. A margin of 0 means
|
|
1939
|
+
the tie-break (the mechanism list's own order: cover, cast, confluence,
|
|
1940
|
+
extract, recall) decided — the answer could change with one more training
|
|
1941
|
+
fact.
|
|
1942
|
+
- **Thin grounding** — when the winning candidate's density (fraction of query
|
|
1943
|
+
bytes actually accounted for by learnt structure) falls below `1/W` (the
|
|
1944
|
+
smallest fraction the store's perceptual window can discriminate), the
|
|
1945
|
+
rationale records a `thinGrounding` step. The answer stands; the label is a
|
|
1946
|
+
signal for downstream consumers that the grounding is sparse.
|
|
1947
|
+
|
|
1948
|
+
### 14.2 Design invariants of the pipeline
|
|
1949
|
+
|
|
1950
|
+
Four rules hold everywhere and are worth reading the rest of Part IV against:
|
|
1951
|
+
|
|
1952
|
+
1. **Uniform mechanisms, self-gating, weighed together.** Every grounding
|
|
1953
|
+
mechanism (CAST, confluence, cover, extraction, recall, the ALU, any user
|
|
1954
|
+
extension) implements the SAME `PipelineMechanism` interface — an optional
|
|
1955
|
+
`parse` (pre-loop computed spans), a `floor` (admissible lower bound), and a
|
|
1956
|
+
`run` (candidate answers). The pipeline (`think`) never imports a
|
|
1957
|
+
mechanism-specific type and never branches on which mechanism it is holding;
|
|
1958
|
+
adding or removing a mechanism means adding or removing one object from the
|
|
1959
|
+
list. Each mechanism checks its own _structural preconditions_ (does the
|
|
1960
|
+
query weave two structures? did a cover compose? is any anchor a skill
|
|
1961
|
+
exemplar?) inside `floor`/`run` and abstains when they fail — no mechanism
|
|
1962
|
+
runs on a query it cannot structurally explain. Every mechanism whose gate
|
|
1963
|
+
passes yields a candidate answer **weighed in the one cost ladder** (§8.9):
|
|
1964
|
+
its moves plus PASS per query byte it did not account for. The lightest
|
|
1965
|
+
grounding derivation wins — the same elementary decision the cover search
|
|
1966
|
+
makes among spans, lifted to the mechanism level. Ties keep the mechanism
|
|
1967
|
+
list's own order (cover, cast, confluence, extract, recall).
|
|
1968
|
+
|
|
1969
|
+
`floor` is called for EVERY mechanism, every response, in list order, BEFORE
|
|
1970
|
+
any `run`; `run` is called only for a mechanism whose floor can still beat
|
|
1971
|
+
the incumbent (`worthRunning`). Every expensive analysis a floor might need
|
|
1972
|
+
(the consensus climb `pre.attention()`, the weave `pre.weave()`, span-shape
|
|
1973
|
+
classification) is a lazily-cached method on the response's `Precomputed`:
|
|
1974
|
+
computed at most once, shared by every consumer — mechanisms AND the
|
|
1975
|
+
post-grounding stages — and never computed at all if nobody asks (a query an
|
|
1976
|
+
extension decided outright never pays for a climb). The INVESTMENT DISCIPLINE
|
|
1977
|
+
closes the loop: `worthRunning` only gates `run` on the pipeline's side, so a
|
|
1978
|
+
`floor` that would first-touch an expensive analysis (CAST's climb + weave,
|
|
1979
|
+
which decide only whether its floor EXISTS — the number is always exactly
|
|
1980
|
+
2·STEP, never below) checks `worthRunning(cheapestBound)` first, and when
|
|
1981
|
+
that already fails it RETURNS THE BOUND uninvested — still admissible, and
|
|
1982
|
+
the pipeline's own check then prunes `run` and records the truthful "cannot
|
|
1983
|
+
beat incumbent" trace note. This is the same admissible-floor pruning the A\*
|
|
1984
|
+
search lives by, applied uniformly — no mechanism asks "did an extension
|
|
1985
|
+
already decide?"; it only ever asks "can I still beat what already won?", and
|
|
1986
|
+
cover running first in the list means a computed span's near-zero cost prunes
|
|
1987
|
+
everything after it the same way any other cheap incumbent would.
|
|
1988
|
+
2. **Read-only store.** Asking never writes. All the memoization inference uses
|
|
1989
|
+
is per-response and is possible _because_ the store cannot change mid-answer.
|
|
1990
|
+
3. **One guide.** The whole response shares the query's gist as its
|
|
1991
|
+
disambiguation guide, and shares its per-context choices, so every mechanism
|
|
1992
|
+
of one answer follows the _same_ reading of every ambiguous fact.
|
|
1993
|
+
4. **Honesty outlets.** At every stage there is a sanctioned way to say less:
|
|
1994
|
+
mechanisms abstain when their structural preconditions fail, recall returns
|
|
1995
|
+
silence below the reach bar, an un-grounded echo is labelled as such, a
|
|
1996
|
+
missing connector joins pieces bare and says so in the trace.
|
|
1997
|
+
5. **Bounded inference.** No per-query read grows with the corpus. Every fan-out
|
|
1998
|
+
walk is capped at √N, and the cap is enforced at the store level through
|
|
1999
|
+
LIMITed reads and indexed existence probes. The climb uses
|
|
2000
|
+
expand-until-decided: it stops the moment saturation or a concrete vote is
|
|
2001
|
+
determined, with work bounded by √N distinct contexts regardless of corpus
|
|
2002
|
+
size. The cost of answering a query is dominated by the query's own
|
|
2003
|
+
structure, not by how much the system has learned.
|
|
2004
|
+
|
|
2005
|
+
### 14.3 How the mind's mechanisms integrate
|
|
2006
|
+
|
|
2007
|
+
The pipeline of §14.1 shows the _order of execution_; this diagram shows the
|
|
2008
|
+
_order of dependency_ — which mechanism is built on which. It is a strict
|
|
2009
|
+
layering: every arrow points downward, a mechanism only ever calls mechanisms in
|
|
2010
|
+
layers below it, and there are no cycles. (The layers correspond one-to-one to
|
|
2011
|
+
the modules of the implementation; see AGENTS.md, "Where things live".)
|
|
2012
|
+
|
|
2013
|
+
```
|
|
2014
|
+
THE MIND'S MECHANISMS — DEPENDENCY LAYERS
|
|
2015
|
+
(arrows = "is built on"; every arrow points to a lower layer)
|
|
2016
|
+
|
|
2017
|
+
L6 ORCHESTRATION ┌───────────────────────────────────────────────┐
|
|
2018
|
+
│ respond ─▶ think (the grounding decider, §14.1)│
|
|
2019
|
+
│ ─▶ articulate │
|
|
2020
|
+
│ rationale/trace: cross-cuts every layer │
|
|
2021
|
+
└──────┬───────┬──────────┬──────────┬──────────┘
|
|
2022
|
+
│ │ │ │
|
|
2023
|
+
L5 GROUNDING & ┌──────▼──┐ ┌──▼────┐ ┌───▼─────┐ ┌──▼──────┐
|
|
2024
|
+
POST-GROUNDING │ cover │ │ CAST │ │ confl. │ │ extract │
|
|
2025
|
+
(§18–§23) │ (§19) │ │ (§18) │ │ (§18.5) │ │ (§20) │
|
|
2026
|
+
└──┬───┬──┘ └──┬───┬─┘ └──┬───┬──┘ └──┬───┬──┘
|
|
2027
|
+
┌──▼───▼───────▼──────────▼──────────────▼──────┐
|
|
2028
|
+
│ recall (§21) · reason (§22) · fuse (§23) │
|
|
2029
|
+
└──┬────────────────┬───────────────────┬──────┘
|
|
2030
|
+
│ │ │
|
|
2031
|
+
L4 QUERY-LEVEL ┌──▼────────────────▼───┐ ┌───────────▼──────┐
|
|
2032
|
+
EVIDENCE │ consensus climb (§17) │ │ graph search: │
|
|
2033
|
+
│ regions → votes → │ │ the deduction │
|
|
2034
|
+
│ cross-region → pool → │ │ system (§19.1–6) │
|
|
2035
|
+
│ commit │ │ │
|
|
2036
|
+
└──┬──────────┬─────────┘ └──┬────────┬──────┘
|
|
2037
|
+
│ │ │ │
|
|
2038
|
+
L3 MATCH & PROJECT ┌──▼──────────▼────────────────▼───┐ ┌──▼──────┐
|
|
2039
|
+
(the elementary │ match (§14.4): │ │ derive: │
|
|
2040
|
+
operation, §14.4) │ matchers: locate · alignGraded ·│ │ A*LD │
|
|
2041
|
+
│ analogyStrength │ │ engine │
|
|
2042
|
+
│ projections: follow · │ │ (§5) │
|
|
2043
|
+
│ conceptHop · reverseContext ·│ │ │
|
|
2044
|
+
│ project │ │ │
|
|
2045
|
+
├───────────────────────────────────┤ └─────────┘
|
|
2046
|
+
│ resonance (§19.5, §21, §22): │
|
|
2047
|
+
│ bridge (→ junction) · │
|
|
2048
|
+
│ recallByResonance · │
|
|
2049
|
+
│ pivotInto · meaningOf │
|
|
2050
|
+
└──┬──────────────┬────────────────┘
|
|
2051
|
+
│ │
|
|
2052
|
+
L2 DECOMPOSITION ┌──▼──────────┐ ┌─▼────────────────────────────┐
|
|
2053
|
+
& TRAVERSAL │ recognition │ │ traverse: edgeAncestors · │
|
|
2054
|
+
│ (§15): │ │ nextOf/prevOf · contains · │
|
|
2055
|
+
│ sites/ │ │ chooseNext / chooseAmong │
|
|
2056
|
+
│ leaves/ │ │ (§25) · hubCap (§8.8) │
|
|
2057
|
+
│ splits │ │ │
|
|
2058
|
+
└──┬──────────┘ └─┬────────────────────────────┘
|
|
2059
|
+
│ │
|
|
2060
|
+
L1 PRIMITIVES ┌──▼──────────────▼────────────────────────────┐
|
|
2061
|
+
│ perceive · gistOf · resolve · read (§10, §27)│
|
|
2062
|
+
└──┬───────────────────────────────────────────┘
|
|
2063
|
+
│
|
|
2064
|
+
L0 SUBSTRATE ┌──▼───────────────────────────────────────────┐
|
|
2065
|
+
│ the store: DAG (nodes, parents, containment) │
|
|
2066
|
+
│ + edges + halos + the two vector indexes │
|
|
2067
|
+
└──────────────────────────────────────────────┘
|
|
2068
|
+
|
|
2069
|
+
Sideways (same-layer) collaborations, all mediated by lower layers:
|
|
2070
|
+
· the ALU and any user extension are ORDINARY L5 mechanisms — the same
|
|
2071
|
+
`PipelineMechanism` shape as CAST, confluence, cover, extraction, and
|
|
2072
|
+
recall (§16). Only `parse()` (pre-loop, collected from every mechanism
|
|
2073
|
+
that implements it) makes them distinctive: its computed spans feed into
|
|
2074
|
+
cover's masking, which hands them INTO the L4 graph search as axioms —
|
|
2075
|
+
an extension never talks to another mechanism directly, and think never
|
|
2076
|
+
branches on "is this an extension?".
|
|
2077
|
+
· CAST, confluence, extract, and recall all consume the SAME memoised
|
|
2078
|
+
consensus climb (via the response's shared `Precomputed`); cover instead
|
|
2079
|
+
consumes recognition directly (its axioms are the query's own
|
|
2080
|
+
decomposition).
|
|
2081
|
+
· reason and fuseAttention run AFTER whichever grounding mechanism
|
|
2082
|
+
fired, and reuse its consumed-node set — the one piece of state that
|
|
2083
|
+
crosses between L5 mechanisms.
|
|
2084
|
+
· articulate re-enters the L4 graph search in substitution mode: the
|
|
2085
|
+
revoiced answer is itself a lightest derivation.
|
|
2086
|
+
```
|
|
2087
|
+
|
|
2088
|
+
How to read the layers:
|
|
2089
|
+
|
|
2090
|
+
- **L0–L1** are the shared substrate every mechanism stands on: the store's
|
|
2091
|
+
exact relations, and perception as a pure function. Nothing above them touches
|
|
2092
|
+
bytes or vectors except through them.
|
|
2093
|
+
- **L2** produces the two elementary readings of anything: _what stored forms
|
|
2094
|
+
does this byte string contain_ (recognition) and _what does this node connect
|
|
2095
|
+
to_ (traversal — raw edge reads, the two disambiguation regimes of §25, and
|
|
2096
|
+
the one √N fan-out cap of §8.8). The shared **junction** ascent belongs here:
|
|
2097
|
+
it climbs the DAG from content-addressed seeds to find containers holding two
|
|
2098
|
+
forms, serving both the bridge (L3) and cross-region attention (L4) from the
|
|
2099
|
+
same bounded, cached walk.
|
|
2100
|
+
- **L3** is the elementary **match-and-project** operation (§14.4): the matcher
|
|
2101
|
+
family (graded locate, literal alignment, distributional analogy) and the
|
|
2102
|
+
projection family (forward fixpoint, concept hop, reverse context, and their
|
|
2103
|
+
composition), plus the resonance patterns built directly on them.
|
|
2104
|
+
- **L4** holds the two big composite engines: the consensus climb (which turns
|
|
2105
|
+
L2/L3 readings into _pooled evidence_ through the derive engine's arithmetic
|
|
2106
|
+
semiring — voting each region independently, then recovering joint contexts
|
|
2107
|
+
through cross-region junction ascent), and the graph search (which turns them
|
|
2108
|
+
into a _cover_ through the tropical semiring). Both are clients of the same
|
|
2109
|
+
A\*LD engine — that is the sense in which all of Sema's thinking is one kind
|
|
2110
|
+
of computation. The junction ascent sits as a shared utility consumed by both
|
|
2111
|
+
L3 (the bridge's Tier 1 connector search) and L4 (cross-region attention's
|
|
2112
|
+
joint-context recovery).
|
|
2113
|
+
- **L5** are the five grounding strategies plus the two post-grounding
|
|
2114
|
+
extenders. Their candidates are **weighed together** by the grounding decider
|
|
2115
|
+
of §14.1 — no fixed priority ladder; the lightest grounding derivation wins.
|
|
2116
|
+
Their _dependency_ structure is what the diagram shows — e.g. extraction
|
|
2117
|
+
depends on the climb (to find an exemplar) and on resonance (to locate
|
|
2118
|
+
frames), but never on cover or CAST. Confluence (like CAST) depends on the
|
|
2119
|
+
climb and on canonical window identity.
|
|
2120
|
+
- **L6** is orchestration only: `think` sequences L5, `articulate` closes the
|
|
2121
|
+
loop, and the rationale tracer observes every layer without being depended on
|
|
2122
|
+
by any.
|
|
2123
|
+
|
|
2124
|
+
The layering is also the _isolation_ structure: a defect in one grounding
|
|
2125
|
+
strategy cannot corrupt another, because they share nothing above L4 except the
|
|
2126
|
+
memoised climb (read-only) and the consumed set (explicitly handed forward).
|
|
2127
|
+
|
|
2128
|
+
### 14.4 The elementary operation: match → project, under a gate
|
|
2129
|
+
|
|
2130
|
+
Every generalising mechanism in Part IV is a configuration of **one** elementary
|
|
2131
|
+
operation:
|
|
2132
|
+
|
|
2133
|
+
> **Match** a learned structure against bytes under some _matching relation_,
|
|
2134
|
+
> bind whatever did not match as the variable, then **project** along a learned
|
|
2135
|
+
> relation in some _direction_ — accepting the result only past a derived
|
|
2136
|
+
> **gate**.
|
|
2137
|
+
|
|
2138
|
+
The three parameters, and their complete value sets:
|
|
2139
|
+
|
|
2140
|
+
**The matcher** — how strictly "this query material IS that learned structure"
|
|
2141
|
+
is decided, from strictest to loosest:
|
|
2142
|
+
|
|
2143
|
+
| Matcher | Decides by | Cost | Used as |
|
|
2144
|
+
| :------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------- | :--------------------------------------------------------------- |
|
|
2145
|
+
| **exact** | content-addressed identity (the DAG) | O(1) probe | recognition's sites; recall tier 0; the pivot's byte containment |
|
|
2146
|
+
| **locate** | the graded ladder exact → halo → gist: literal bytes first, then distributional role (gate: concept threshold), then perceived gist (gate: merge threshold) | one resonance per relaxation | extraction locating an exemplar's frames in the query |
|
|
2147
|
+
| **aligned** | maximal literal runs by seed-and-extend over W-grams — every span two whole structures share, not one position | O(\|a\|+\|b\|+runs) | CAST detecting the structures a query weaves |
|
|
2148
|
+
| **distributional** (`analogyStrength`) | graded three-tier test: direct halo cosine, then mutual halo sibling, then shared learnt W-window frames (gate: significance bar) | ≤ 2 halo-index queries + O(\|a\|+\|b\|) scan | CAST validating a genuine analog |
|
|
2149
|
+
|
|
2150
|
+
**The direction** — which learned relation the matched node is projected along,
|
|
2151
|
+
and which way:
|
|
2152
|
+
|
|
2153
|
+
| Direction | Operation | Meaning |
|
|
2154
|
+
| :----------------------------- | :---------------------------------------------------------------------------------------------------- | :-------------------------------------------------------- |
|
|
2155
|
+
| **forward** (`follow`) | walk continuation edges to the fixpoint; the first hop may cross a concept (halo) link | "what does this lead to?" |
|
|
2156
|
+
| **reverse** (`reverseContext`) | the context this continuation follows — guide-resonance pick with a guide, halo-mass pick without one | "what establishes this?" |
|
|
2157
|
+
| **both** (`project`) | forward, else reverse | the universal grounding step |
|
|
2158
|
+
| **read-out** | read the query span the matched frame delimits | extraction: the variable comes _out_ of the query |
|
|
2159
|
+
| **insert** | place the matched filler into the learned structure's seat | CAST substitution: the variable goes _into_ the structure |
|
|
2160
|
+
| **substitute** | replace the matched form with its concept sibling's bytes | articulation's revoicing |
|
|
2161
|
+
|
|
2162
|
+
**The gate** — the derived threshold (§8) the match/projection must clear:
|
|
2163
|
+
concept threshold for anything distributional, merge threshold for gist
|
|
2164
|
+
identity, significance bar for analogy, reach threshold for accepting any final
|
|
2165
|
+
answer, the consensus floor for a further point of attention. Never a tuned
|
|
2166
|
+
constant.
|
|
2167
|
+
|
|
2168
|
+
Under this decomposition, the mechanism catalogue of Part IV reads as a
|
|
2169
|
+
configuration table:
|
|
2170
|
+
|
|
2171
|
+
| Mechanism | Matcher | Direction | Gate |
|
|
2172
|
+
| :---------------------- | :-------------------------------- | :----------------------- | :---------------------------- |
|
|
2173
|
+
| cover follow-edge (§19) | exact | forward | — (cost ladder) |
|
|
2174
|
+
| concept hop (§19.3) | halo sibling | forward | concept threshold |
|
|
2175
|
+
| recall tiers 0–1 (§21) | identity / whole-query gist | both | merge threshold |
|
|
2176
|
+
| skill extraction (§20) | locate (on the exemplar's frames) | read-out | per-step ladder gates |
|
|
2177
|
+
| CAST substitution (§18) | graded (literal → halo) | insert | frame (MIN_WEAVE + dominates) |
|
|
2178
|
+
| CAST redirection (§18) | graded (literal → halo) | both (on the substitute) | frame (MIN_WEAVE + dominates) |
|
|
2179
|
+
| CAST comparison (§18) | graded + distributional | reverse, juxtaposed | significance bar |
|
|
2180
|
+
| multi-hop pivot (§22) | exact (byte containment) | forward | — |
|
|
2181
|
+
| articulation (§24) | halo sibling | substitute | concept threshold |
|
|
2182
|
+
|
|
2183
|
+
**Why this is stated here, prominently.** The machinery is factored exactly
|
|
2184
|
+
once: the matcher and projection families are one shared family (with the one √N
|
|
2185
|
+
fan-out convention beside the disambiguators), and each mechanism states _only
|
|
2186
|
+
its configuration_. This is a standing architectural rule:
|
|
2187
|
+
|
|
2188
|
+
> **Before adding a new generalising mechanism, express it as a (matcher,
|
|
2189
|
+
> direction, gate) triple.** If its matcher, projection, and gate already exist,
|
|
2190
|
+
> the mechanism is a configuration — write only the configuration. If it
|
|
2191
|
+
> genuinely needs a new matcher or a new direction, add that _to the shared
|
|
2192
|
+
> family_ (with its gate derived in the geometry, §8), never as a private helper
|
|
2193
|
+
> inside the mechanism. A mechanism file that re-implements locating, aligning,
|
|
2194
|
+
> edge-following to a fixpoint, predecessor-picking, or fan-out capping is
|
|
2195
|
+
> re-introducing the exact duplication this decomposition removed.
|
|
2196
|
+
|
|
2197
|
+
Two boundaries of the unification are deliberate, not omissions:
|
|
2198
|
+
|
|
2199
|
+
- **The cover search keeps its own internal edge-chain.** Inside the deduction
|
|
2200
|
+
system (§19), following an edge is a _rule application_ — one step of the
|
|
2201
|
+
proof tree, individually costed and traced. Collapsing it into the shared
|
|
2202
|
+
forward projection would erase the derivation's granularity and cross the
|
|
2203
|
+
synchronous/asynchronous seam (the search is synchronous; the projections may
|
|
2204
|
+
touch the ANN indexes). The two implementations of "follow an edge" are
|
|
2205
|
+
therefore one _concept_ with two _obligations_: proof-grained inside the
|
|
2206
|
+
search, fixpoint-grained outside it.
|
|
2207
|
+
- **Arbitration between mechanisms is one lightest-derivation decision.** The
|
|
2208
|
+
grounding decider (§14.1) weighs every mechanism's candidate in the one cost
|
|
2209
|
+
ladder, so a mechanism-level choice and a byte-level choice are the same kind
|
|
2210
|
+
of decision. Ties (at STEP resolution — sub-STEP costs like MICRO are
|
|
2211
|
+
non-ordering bookkeeping and must not decide a cross-mechanism choice) keep
|
|
2212
|
+
the mechanism list's own order (cover, cast, confluence, extract, recall) —
|
|
2213
|
+
cover runs first not because it is prioritised over the others by fiat, but
|
|
2214
|
+
because a computed span (§16) masks in at near-zero cost, which then prunes
|
|
2215
|
+
the rest through the SAME admissible-floor mechanism every mechanism is
|
|
2216
|
+
subject to (§14.2), not a special rule.
|
|
2217
|
+
|
|
2218
|
+
### 14.5 The free-will architecture
|
|
2219
|
+
|
|
2220
|
+
The grounding decider is not a priority ladder with a special-case branch for
|
|
2221
|
+
every mechanism. It is a **market**: mechanisms produce candidates, each priced
|
|
2222
|
+
in one currency, and the lightest wins. This works because every mechanism obeys
|
|
2223
|
+
four constraints that are the same constraints a budgeted reasoner — human,
|
|
2224
|
+
language model, or automated deduction engine — must obey to be composable:
|
|
2225
|
+
|
|
2226
|
+
**1. Decoupling.** A mechanism imports nothing from other mechanisms — and
|
|
2227
|
+
implements exactly the same `PipelineMechanism` shape as every other one, so the
|
|
2228
|
+
pipeline imports nothing mechanism-specific either. CAST does not know
|
|
2229
|
+
extraction exists. Extraction does not know confluence exists (or that an
|
|
2230
|
+
extension might have fired). Each receives `ctx`, `query`, and the response's
|
|
2231
|
+
shared `Precomputed` (recognition, computed spans, the gist — plus lazily-cached
|
|
2232
|
+
methods for every expensive analysis a mechanism opts into: the consensus climb,
|
|
2233
|
+
the weave, span-shape classification, the identity-window reads), and returns a
|
|
2234
|
+
candidate or nothing. Adding a mechanism never touches an existing mechanism
|
|
2235
|
+
file.
|
|
2236
|
+
|
|
2237
|
+
**2. Declared competence.** A mechanism gates itself with a structural
|
|
2238
|
+
precondition — a binary, auditable condition, not a learned score — inside its
|
|
2239
|
+
own `floor`. CAST checks `query.length < 2W`. Extraction checks whether any
|
|
2240
|
+
ranked anchor is span-shaped. When a mechanism does not fire, the rationale says
|
|
2241
|
+
exactly why. A mechanism whose floor needs its own expensive precomputation to
|
|
2242
|
+
refine also receives `worthRunning`, so declaring "structurally impossible" and
|
|
2243
|
+
declaring "cannot beat the incumbent, don't bother computing further" cost the
|
|
2244
|
+
same: nothing.
|
|
2245
|
+
|
|
2246
|
+
**3. Visible budget.** Every loop over corpus-scale data is capped at a named
|
|
2247
|
+
constant: `k = 2·recallQueryK` for the alignment loop, `√N` for every fan-out
|
|
2248
|
+
walk. The cap is enforced at the store level through LIMITed reads and indexed
|
|
2249
|
+
existence probes. No per-query cost grows with the corpus.
|
|
2250
|
+
|
|
2251
|
+
**4. Evidence travels with the answer.** Every candidate carries `accounted`
|
|
2252
|
+
(what it explained), `moves` (what its acts cost), and `unexplained` (what it
|
|
2253
|
+
left on the table). The decider does not know which mechanism produced which
|
|
2254
|
+
candidate — it only sees weights in the one cost ladder.
|
|
2255
|
+
|
|
2256
|
+
These four constraints are the same structural principle that makes a
|
|
2257
|
+
budget-limited reasoner honest. A language model has a context window (visible
|
|
2258
|
+
budget), stops when it converges (declared competence via early-stopping),
|
|
2259
|
+
reports its chain of thought (evidence traveling with the answer), and composes
|
|
2260
|
+
with tools via structured output schemas (decoupling). The architecture is the
|
|
2261
|
+
same. What Sema calls `√N`, a model calls `max_tokens`. What Sema calls
|
|
2262
|
+
`admissible floor`, a model calls `skip-if-unpromising`. What Sema calls
|
|
2263
|
+
`unexplained`, a model calls `I don't know`.
|
|
2264
|
+
|
|
2265
|
+
The difference is that Sema's caps are **derived** from the system's own
|
|
2266
|
+
geometry (D, W, N) rather than chosen by an external budget. A mechanism's
|
|
2267
|
+
admissible floor is computed from the memoised climb before the mechanism runs.
|
|
2268
|
+
A fan-out cap is `Math.ceil(Math.sqrt(corpusN))` — the point at which further
|
|
2269
|
+
evidence no longer discriminates. The principle is the same; the derivation is
|
|
2270
|
+
internal.
|
|
2271
|
+
|
|
2272
|
+
---
|
|
2273
|
+
|
|
2274
|
+
## 15. Recognition: decomposing the query
|
|
2275
|
+
|
|
2276
|
+
Recognition answers: _which stored forms does this byte string contain, and
|
|
2277
|
+
where?_ Its output — the **sites** (span → node), the query's perceived
|
|
2278
|
+
**leaves**, and the **split** positions where a form boundary falls inside a
|
|
2279
|
+
leaf — is the raw material of every downstream mechanism.
|
|
2280
|
+
|
|
2281
|
+
Two complementary readings run over the query:
|
|
2282
|
+
|
|
2283
|
+
### 15.1 The structural reading
|
|
2284
|
+
|
|
2285
|
+
Perceive the query (the same fold as ingestion) and walk its own tree, asking
|
|
2286
|
+
the store, bottom-up, to name each subtree: leaves by their bytes, branches by
|
|
2287
|
+
their children's ids. Because perception is deterministic, any part of the query
|
|
2288
|
+
that was ever deposited _as it appears here_ folds into the identical subtree
|
|
2289
|
+
and is named exactly. Additionally, within each leaf-parent chunk, every
|
|
2290
|
+
contiguous sub-run of leaves is probed, so forms smaller than a chunk are found
|
|
2291
|
+
too.
|
|
2292
|
+
|
|
2293
|
+
### 15.2 The canonical reading
|
|
2294
|
+
|
|
2295
|
+
The query's own fold may cut the stream differently from how training cut it
|
|
2296
|
+
(different surroundings ⇒ different group boundaries). The canonical reading
|
|
2297
|
+
re-derives the _store's_ segmentation directly on the query's bytes: at each
|
|
2298
|
+
position, chain the known single-byte leaves forward and probe each growing
|
|
2299
|
+
sequence as a flat branch — recovering forms _as training stored them_,
|
|
2300
|
+
regardless of how the query's tree happens to chunk. Where such a form's
|
|
2301
|
+
boundary falls strictly inside one of the query's leaves, that position is
|
|
2302
|
+
recorded as a **split**: the search may later cut the leaf there (§19.4).
|
|
2303
|
+
|
|
2304
|
+
### 15.3 What counts as a site
|
|
2305
|
+
|
|
2306
|
+
A recognised span is admitted as a site only if its node **leads somewhere** —
|
|
2307
|
+
it bears a continuation edge or a halo. A form that leads nowhere contributes
|
|
2308
|
+
nothing to any derivation, so recognition filters it out at the source.
|
|
2309
|
+
|
|
2310
|
+
```
|
|
2311
|
+
recognise(query):
|
|
2312
|
+
sites, leaves, splits ← ∅
|
|
2313
|
+
# structural
|
|
2314
|
+
tree ← perceive(query)
|
|
2315
|
+
for each subtree s of tree (bottom-up, with byte offsets):
|
|
2316
|
+
id ← store lookup of s (leaf bytes / branch kid-ids)
|
|
2317
|
+
if id exists and (next(id) ≠ ∅ or halo(id) ≠ ∅):
|
|
2318
|
+
sites += (span(s), id)
|
|
2319
|
+
collect leaves; probe sub-runs within leaf-parents likewise
|
|
2320
|
+
# canonical
|
|
2321
|
+
for each position p in query:
|
|
2322
|
+
chain known leaves from p; for each chained prefix that is a known
|
|
2323
|
+
flat branch, resolve its span and admit it as above
|
|
2324
|
+
splits ← form boundaries that fall inside a perceived leaf
|
|
2325
|
+
return (sites, leaves, splits)
|
|
2326
|
+
```
|
|
2327
|
+
|
|
2328
|
+
Everything here is a bounded number of O(1) content-addressed probes per byte —
|
|
2329
|
+
never a scan of the corpus.
|
|
2330
|
+
|
|
2331
|
+
---
|
|
2332
|
+
|
|
2333
|
+
## 16. Computation: extensions and the ALU
|
|
2334
|
+
|
|
2335
|
+
### 16.1 Manual rules beside learned ones
|
|
2336
|
+
|
|
2337
|
+
Some knowledge is _rules_, not facts: nobody should teach a memory system
|
|
2338
|
+
arithmetic one sum at a time. Sema accommodates this with **extensions**:
|
|
2339
|
+
`PipelineMechanism`s (§14.1, §14.2) whose distinguishing feature is `parse` —
|
|
2340
|
+
consulted once per query, before the grounding loop, over EVERY mechanism that
|
|
2341
|
+
implements it. An extension's `parse` receives the raw query and returns
|
|
2342
|
+
**computed spans** — byte ranges it recognises as computations, together with
|
|
2343
|
+
the authoritative result bytes for each. The mind lends every extension the same
|
|
2344
|
+
four neutral capabilities it already has (resonant meaning-matching, grounded
|
|
2345
|
+
continuation lookup, geometric segmentation, and the perception window W)
|
|
2346
|
+
through the `ExtensionHost` port; it learns nothing about what the extension
|
|
2347
|
+
computes.
|
|
2348
|
+
|
|
2349
|
+
An extension joins through `Mind`'s `mechanismFactories` option: a factory
|
|
2350
|
+
receiving the `ExtensionHost` and returning a `PipelineMechanism`. Once
|
|
2351
|
+
constructed, it is indistinguishable to the pipeline from CAST, confluence,
|
|
2352
|
+
cover, extraction, or recall — same `floor`/`run` shape, same admissible-floor
|
|
2353
|
+
pruning, same weight ladder. The ONLY place its computed spans get special
|
|
2354
|
+
treatment is cover's masking (§16.3), which is a property of cover's own `run`,
|
|
2355
|
+
not of the pipeline singling out "extensions" as a concept.
|
|
2356
|
+
|
|
2357
|
+
### 16.2 The ALU
|
|
2358
|
+
|
|
2359
|
+
The built-in extension — wrapped into a `PipelineMechanism` by `aluToMechanism`
|
|
2360
|
+
and appended to the mechanism list at `Mind` construction (`cfg.alu.enabled`) —
|
|
2361
|
+
is a small **arithmetic–logic unit** built the way a mathematician would want
|
|
2362
|
+
it: an irreducible kernel — one logic gate (NAND), the field-and-order
|
|
2363
|
+
primitives (0, 1, add, negate, multiply, reciprocal, sign), one limit operator
|
|
2364
|
+
(converge-to-tolerance), and three structural list operations (construct,
|
|
2365
|
+
length, project) — from which everything else (comparison, powers, roots,
|
|
2366
|
+
calculus, equation solving, map/reduce over nested lists, element-wise
|
|
2367
|
+
broadcasting) is _derived by rewriting_, not separately implemented. It parses
|
|
2368
|
+
infix notation directly, and can also recognise an operation _by meaning_ — a
|
|
2369
|
+
span whose gist resonates with a registered operation's learned anchor — using
|
|
2370
|
+
the host's resonance capability. Results are computed exactly (or to declared
|
|
2371
|
+
tolerance) and deterministically rendered to bytes.
|
|
2372
|
+
|
|
2373
|
+
### 16.3 Masking: computation always wins
|
|
2374
|
+
|
|
2375
|
+
A computed span enters the cover search (§19) as a recognised completion at the
|
|
2376
|
+
cost of one inference step — an authoritative _derived fact_, on par with
|
|
2377
|
+
following a learned edge. Precedence over memory is enforced not by cost but by
|
|
2378
|
+
**masking**: any recognised site that overlaps a computed span is removed before
|
|
2379
|
+
the search, so within those bytes the computation is the only available
|
|
2380
|
+
completion. A corpus deliberately taught "2+2 → 5" therefore cannot outvote the
|
|
2381
|
+
ALU on "2+2", while remaining free to associate whatever it likes _around_ the
|
|
2382
|
+
computation — a computed result and an unrelated learned rewrite still compose
|
|
2383
|
+
within one answer.
|
|
2384
|
+
|
|
2385
|
+
---
|
|
2386
|
+
|
|
2387
|
+
## 17. The consensus climb: points of attention
|
|
2388
|
+
|
|
2389
|
+
### 17.1 The problem: what is this query about?
|
|
2390
|
+
|
|
2391
|
+
Several mechanisms need to know which learned context(s) a query is _about_ —
|
|
2392
|
+
especially when the query's wording matches nothing directly (scaffolding words
|
|
2393
|
+
dominate), or when it is about _two things at once_. The consensus climb answers
|
|
2394
|
+
this with the machinery already on hand: geometry proposes, structure climbs,
|
|
2395
|
+
and pooled weighted deduction decides.
|
|
2396
|
+
|
|
2397
|
+
### 17.2 The algorithm
|
|
2398
|
+
|
|
2399
|
+
```
|
|
2400
|
+
climbAttention(query, k):
|
|
2401
|
+
|
|
2402
|
+
# 1. REGIONS — TWO sources, both structural:
|
|
2403
|
+
# a) PERCEIVED SUBTREES of the query (every branch level of the river
|
|
2404
|
+
# fold). A region that dominates the query (covers more than half)
|
|
2405
|
+
# is admitted only if it is the sole structure.
|
|
2406
|
+
# b) RECOGNISED SITES — content-addressed nodes the query literally
|
|
2407
|
+
# contains (§15). A site IS an exact structural anchor; perceived
|
|
2408
|
+
# sub-regions are approximate (W-byte chunks whose gist must resonate
|
|
2409
|
+
# to find an anchor). Sites fill the gap perception creates: a word
|
|
2410
|
+
# crossing a W-boundary is split into chunks whose partial gists may
|
|
2411
|
+
# not resonate distinctively, but the site names the whole word by
|
|
2412
|
+
# exact content identity. Sites that overlap sub-regions add
|
|
2413
|
+
# corroborating evidence; sites in gaps fill them.
|
|
2414
|
+
# Recognition is memoised per response — adding sites costs zero.
|
|
2415
|
+
regions ← subtrees of perceive(query) ∪ recognise(query).sites
|
|
2416
|
+
|
|
2417
|
+
# 2. VOTE — each region finds its best anchor in the DAG and climbs to
|
|
2418
|
+
# the edge-bearing contexts ("roots") that contain it.
|
|
2419
|
+
# Perceived sub-regions resonate their gist into the content index;
|
|
2420
|
+
# site-regions already carry an exact node id and skip the ANN query
|
|
2421
|
+
# entirely — they climb directly from the resolved node.
|
|
2422
|
+
# The climb uses EXPAND-UNTIL-DECIDED (§17.4): it stops the moment the
|
|
2423
|
+
# answer (saturated vs. voted, with exact contextsReached) is known —
|
|
2424
|
+
# all reads through it are LIMITed at √N, so the cost is bounded by the
|
|
2425
|
+
# hub convention, never by the corpus.
|
|
2426
|
+
for each region r:
|
|
2427
|
+
anchor ← r.nodeId ?? (canonical chunk id ?? contentIndex.nearest(gist(r), k)[0])
|
|
2428
|
+
# contrastive-margin gate: for APPROXIMATE regions only, the gap
|
|
2429
|
+
# between the best and second-best score must exceed the estimator
|
|
2430
|
+
# noise floor (§8.4); otherwise the region abstains
|
|
2431
|
+
if not r.known and (bestScore − secondBestScore) ≤ NOISE: ABSTAIN
|
|
2432
|
+
reach ← expandUntilDecided(anchor):
|
|
2433
|
+
for each level: check prevCount (indexed O(1)) to decide
|
|
2434
|
+
"edge-bearing?" without materialising prev lists; read
|
|
2435
|
+
parentsFirst(√N+1) — getting √N+1 proves "hub" exactly;
|
|
2436
|
+
containment parents page via containersSlice(offset, √N)
|
|
2437
|
+
so a common window's climb stops at the first saturated
|
|
2438
|
+
page; distinct contexts past √N decide "saturated" by an
|
|
2439
|
+
indexed count, never a materialised list.
|
|
2440
|
+
if anchor climbs nowhere and is not saturated:
|
|
2441
|
+
try the remaining hits, nearest first # ranking is
|
|
2442
|
+
# approximate; an
|
|
2443
|
+
# orphan top hit is
|
|
2444
|
+
# an accident
|
|
2445
|
+
if saturated: the region ABSTAINS # §17.4
|
|
2446
|
+
mutual ← mutualExplanation(score, region, anchor) # angle + magnitude
|
|
2447
|
+
w ← mutual · ln(N / contextsReached) / |roots| # IDF weighting
|
|
2448
|
+
the region votes w for each root it reached
|
|
2449
|
+
|
|
2450
|
+
# 2b. CROSS-REGION — voteRegions climbs each region INDEPENDENTLY;
|
|
2451
|
+
# additive pooling can only surface contexts at least one region
|
|
2452
|
+
# already votes for. Two regions whose individual climbs land on
|
|
2453
|
+
# DIFFERENT contexts ("red" → `red square`, "circle" → `circle`)
|
|
2454
|
+
# leave their JOINT context (`red circle`) with no vote — no amount
|
|
2455
|
+
# of pooling can recover it. Cross-region attention recovers it by
|
|
2456
|
+
# the SAME content-addressed junction ascent the bridge uses
|
|
2457
|
+
# (the shared junction ascent):
|
|
2458
|
+
#
|
|
2459
|
+
# Candidate regions: ANY voted region composes — not just recognised
|
|
2460
|
+
# sites (corpus independence). At least one side must be a STRONG
|
|
2461
|
+
# voter (individually discriminative). A known but non-voting region
|
|
2462
|
+
# may serve as the WEAK side (a word never learnt standalone still
|
|
2463
|
+
# binds through its stored byte fragments). Two non-voting regions
|
|
2464
|
+
# never pair (the shared-prefix trap). Only MAXIMAL spans compose
|
|
2465
|
+
# (a span contained in another candidate is not independent of it).
|
|
2466
|
+
# A single known region covering both is skipped — the whole form
|
|
2467
|
+
# already votes directly. Byte-sorted left-to-right.
|
|
2468
|
+
#
|
|
2469
|
+
# junctionSeeds are precomputed ONCE per candidate across all its
|
|
2470
|
+
# pairs (cost hoisting). The junction ascent is ORDER-FREE (a
|
|
2471
|
+
# junction is evidence the forms were learnt together; which one the
|
|
2472
|
+
# query mentioned first is a fact about the query, not the learnt
|
|
2473
|
+
# whole — the byte-containment test probes both orders). All reads
|
|
2474
|
+
# go through the shared per-response walkCache.
|
|
2475
|
+
#
|
|
2476
|
+
# N-ARY SELECTION: the container covering the MOST remaining
|
|
2477
|
+
# candidate forms wins (then tightest interior, then lowest id).
|
|
2478
|
+
# A consumed candidate never re-pairs — its evidence is already
|
|
2479
|
+
# composed at full joint strength.
|
|
2480
|
+
#
|
|
2481
|
+
# SELF-EVIDENCE GUARD: a container whose joined occurrence (left
|
|
2482
|
+
# through right including interior) is a literal substring of the
|
|
2483
|
+
# query is rejected — shards of a contiguous phrase pairing "around"
|
|
2484
|
+
# a gap chunk would merely rediscover the phrase they are shards of.
|
|
2485
|
+
#
|
|
2486
|
+
# EXPLAINING AWAY: when a junction binds, any individual vote whose
|
|
2487
|
+
# bytes the joint container LITERALLY CONTAINS yet whose roots are
|
|
2488
|
+
# FULLY DISJOINT from the junction's is SUPERSEDED — the exact joint
|
|
2489
|
+
# evidence explains those bytes away (grid aliasing). Partial
|
|
2490
|
+
# agreement (shared roots) corroborates and is kept.
|
|
2491
|
+
crossVotes ← []
|
|
2492
|
+
superseded ← ∅
|
|
2493
|
+
seedsOf(ri) ≔ junctionSeeds(ctx, query[regions[ri].start..regions[ri].end])
|
|
2494
|
+
# computed once, reused across all pairs of this candidate
|
|
2495
|
+
consumed ← ∅ # a candidate in one junction never re-pairs
|
|
2496
|
+
for each pair (a, b) of eligible candidates (non-overlapping,
|
|
2497
|
+
at least one strong voter, not both covered by one known region,
|
|
2498
|
+
≤ k total probes, skipping consumed candidates):
|
|
2499
|
+
left ← query[a.start..a.end]
|
|
2500
|
+
right ← query[b.start..b.end]
|
|
2501
|
+
containers ← junctionContainersFrom(left, right, cap,
|
|
2502
|
+
seedsOf(a), seedsOf(b), undefined, unordered = true)
|
|
2503
|
+
if containers is empty:
|
|
2504
|
+
# Tier 2.5 fallback — same ascent over halo siblings
|
|
2505
|
+
containers ← junctionSynonyms(left, right, maxInterior,
|
|
2506
|
+
unordered = true)
|
|
2507
|
+
if containers not empty:
|
|
2508
|
+
best ← the container covering the MOST remaining candidates
|
|
2509
|
+
(cachedRead + indexOf per extra, never an extra walk);
|
|
2510
|
+
ties → shortest interior → lowest id
|
|
2511
|
+
if best's joined occurrence is a query substring: continue
|
|
2512
|
+
reach ← edgeAncestors(best.id)
|
|
2513
|
+
if reach is discriminative (not saturated, idf > 0):
|
|
2514
|
+
w ← mutual · ln(N / contextsReached) / |roots|
|
|
2515
|
+
crossVotes.push(vote for best.id's roots at weight w,
|
|
2516
|
+
span covering all composed candidates)
|
|
2517
|
+
consumed.add(a); consumed.add(b); consumed.add(all extras)
|
|
2518
|
+
# Explaining away: individual votes whose bytes the container
|
|
2519
|
+
# literally contains and whose roots are fully disjoint from
|
|
2520
|
+
# the junction's are superseded
|
|
2521
|
+
for each individual vote rv:
|
|
2522
|
+
if rv.roots shares any root with reach.roots: keep
|
|
2523
|
+
if containerBytes literally contains rv's query bytes:
|
|
2524
|
+
superseded.add(rv)
|
|
2525
|
+
break # a is consumed — move to next unconsumed candidate
|
|
2526
|
+
|
|
2527
|
+
# 3. POOL — votes (INDEPENDENT, minus any superseded by cross-region
|
|
2528
|
+
# evidence, + CROSS-REGION) accumulate through the
|
|
2529
|
+
# arithmetic semiring (§5.3): each region is an axiom; each
|
|
2530
|
+
# (region → anchor) contribution is a summing rule; a vote for a
|
|
2531
|
+
# TERMINAL answer node redistributes to the ≤ √N contexts that lead
|
|
2532
|
+
# to it (via prevFirst, capped at the store level). Independent
|
|
2533
|
+
# corroboration ADDS.
|
|
2534
|
+
votes ← pooledConclusions
|
|
2535
|
+
|
|
2536
|
+
# 4. COMMIT — rank anchors by vote. The dominant anchor always stands.
|
|
2537
|
+
# A FURTHER (non-overlapping) anchor becomes an independent point of
|
|
2538
|
+
# attention only if its vote clears BOTH:
|
|
2539
|
+
# · the natural break (the steepest ratio drop in the sorted votes —
|
|
2540
|
+
# a scale-free "where does signal end" test), and
|
|
2541
|
+
# · the consensus floor ln N + ½ (§8.6 — more than any single
|
|
2542
|
+
# region could contribute alone).
|
|
2543
|
+
return the surviving points, each with its query span and vote
|
|
2544
|
+
```
|
|
2545
|
+
|
|
2546
|
+
### 17.3 Why inverse document frequency
|
|
2547
|
+
|
|
2548
|
+
A region that climbs to few contexts is _specific_ — strong evidence about what
|
|
2549
|
+
the query concerns. A region that climbs to half the corpus (a common phrase)
|
|
2550
|
+
says almost nothing. Weighting a region's vote by ln(N/c) — the classical
|
|
2551
|
+
inverse-document-frequency form (Spärck Jones 1972) — expresses exactly this,
|
|
2552
|
+
with N the store's count of learned contexts, and dividing by the number of
|
|
2553
|
+
roots reached splits a region's voice among the candidates it cannot
|
|
2554
|
+
distinguish.
|
|
2555
|
+
|
|
2556
|
+
The geometric factor is not the raw resonance score but a **mutual-explanation
|
|
2557
|
+
weight** that reads both angle and magnitude (§2.6). Under the linear fold,
|
|
2558
|
+
cosine = shared / (‖region‖ · ‖hit‖), and each unnormalized norm is recoverable
|
|
2559
|
+
as a byte count (the region's own span length; the hit's, read from the store).
|
|
2560
|
+
Splitting the cosine by each side's own magnitude gives two fractions — how much
|
|
2561
|
+
of the _region_ the hit explains, and how much of the _hit_ the region pins down
|
|
2562
|
+
— each capped at 1 (an estimated cosine can imply more shared content than the
|
|
2563
|
+
smaller side even holds; left uncapped, that impossible surplus would let a
|
|
2564
|
+
small region echoing inside a large context, or the reverse, vote above its
|
|
2565
|
+
physical evidence). The product of the two capped fractions is the mutual weight
|
|
2566
|
+
that replaces a bare score: it is exactly the same quantity a plain squared
|
|
2567
|
+
cosine approximated implicitly (score² ≈ (shared/len_region)·(shared/len_hit)
|
|
2568
|
+
when the two sides are close in size), made explicit and safe at every scale.
|
|
2569
|
+
The margin gate that precedes this weighting (§17.2, the noise-floor check
|
|
2570
|
+
before the vote is computed) stays in raw cosine units deliberately: it tests
|
|
2571
|
+
the RaBitQ estimator's own noise floor, which lives in cosine space, not in
|
|
2572
|
+
byte-magnitude space.
|
|
2573
|
+
|
|
2574
|
+
### 17.4 Saturation: expand-until-decided
|
|
2575
|
+
|
|
2576
|
+
The climb's work is bounded by **expand-until-decided**: the walk stops as soon
|
|
2577
|
+
as it knows whether the reach is saturated (the material is too common to
|
|
2578
|
+
discriminate) or a concrete vote (exact roots and contextsReached). The decision
|
|
2579
|
+
uses only LIMITed store reads:
|
|
2580
|
+
|
|
2581
|
+
- **Is this node edge-bearing?** `prevCount` — an indexed O(1) count — answers
|
|
2582
|
+
"does this node follow at least one learned context?" without materialising
|
|
2583
|
+
the (potentially corpus-sized) predecessor list.
|
|
2584
|
+
- **Is this node a hub?** `parentsFirst(id, √N+1)` — reading √N+1 parents proves
|
|
2585
|
+
"more than √N" exactly; the walk aborts at that step. Below √N, the read IS
|
|
2586
|
+
the full parent list, so the walk is exact.
|
|
2587
|
+
- **Distinct contexts past √N?** If the set of learned contexts visited crosses
|
|
2588
|
+
√N, the region is saturated — decided by a set-size check, never by counting
|
|
2589
|
+
every reachable context.
|
|
2590
|
+
- **Containment paging.** A window's containers are paged in chunks of √N via
|
|
2591
|
+
`containersSlice`. A distinctive window's containers (few, converging on one
|
|
2592
|
+
context) are walked in full — exact. A common window's corpus-sized container
|
|
2593
|
+
list is abandoned at the first page whose climbs push contexts past √N, with
|
|
2594
|
+
O(√N) page-work.
|
|
2595
|
+
|
|
2596
|
+
A region whose climb triggers any of the "decided: saturated" conditions
|
|
2597
|
+
abstains rather than vote noise. Saturation is also _recorded_: a leading
|
|
2598
|
+
saturated stretch of the query (a boilerplate preamble) is treated as
|
|
2599
|
+
scaffolding, and further points of attention are only admitted beyond it. The
|
|
2600
|
+
dual use — abstain from voting, and mark scaffolding — is what keeps long
|
|
2601
|
+
templated queries from diluting their own payload.
|
|
2602
|
+
|
|
2603
|
+
### 17.5 What consumes the climb
|
|
2604
|
+
|
|
2605
|
+
The climb is computed once per response (memoised) and consumed by five
|
|
2606
|
+
mechanisms: recall's scaffolding tier (§21.3), extraction's search for a skill
|
|
2607
|
+
exemplar (§20), CAST's identification of woven structures (§18), confluence's
|
|
2608
|
+
constraint-stream detection (§18.5), and fusion's grounding of further topics
|
|
2609
|
+
(§23). The cross-region attention pass (§17.6) also runs inside the climb,
|
|
2610
|
+
consuming its region votes and the junction module.
|
|
2611
|
+
|
|
2612
|
+
### 17.6 Cross-region attention: the binding problem
|
|
2613
|
+
|
|
2614
|
+
Additive pooling has a blind spot. Two regions whose independent climbs land on
|
|
2615
|
+
_different_ contexts leave their **joint context** — the learnt whole that
|
|
2616
|
+
contains both — with zero votes. "red" votes for `red square`, "circle" for
|
|
2617
|
+
`circle`; nothing votes for `red circle`, the only fact holding both. No amount
|
|
2618
|
+
of pooling can recover it: addition can only amplify what at least one region
|
|
2619
|
+
already said. This is the attention counterpart of the binding problem —
|
|
2620
|
+
independent evidence disaggregates what belongs together.
|
|
2621
|
+
|
|
2622
|
+
Cross-region attention recovers the missing joint contexts by the **same
|
|
2623
|
+
content-addressed junction ascent** the bridge uses (§19.5), extracted into the
|
|
2624
|
+
shared junction ascent. "Which learnt whole contains these forms?" is a bounded
|
|
2625
|
+
DAG ascent from the forms' canonical identities — not a resonance guess on a
|
|
2626
|
+
synthesised gist.
|
|
2627
|
+
|
|
2628
|
+
**Why not fold the two vectors?** Folding two region gists cannot even
|
|
2629
|
+
reconstruct the stored joint form. Sema builds a multi-word gist from byte-chunk
|
|
2630
|
+
folds, so isolated word vectors superpose into a different direction and
|
|
2631
|
+
resonate to `red circle` and `red square` indistinguishably. The junction ascent
|
|
2632
|
+
sidesteps this by matching **bytes** (content-addressed identities), not
|
|
2633
|
+
vectors.
|
|
2634
|
+
|
|
2635
|
+
**Candidate selection — corpus independence.** Candidates are ANY region that
|
|
2636
|
+
participated in the independent vote, not only recognised sites. At least one
|
|
2637
|
+
side of each pair must be a **strong** voter (individually discriminative — idf
|
|
2638
|
+
|
|
2639
|
+
> 0, non-saturated). A **known** (content-addressed) region that did _not_ vote
|
|
2640
|
+
> (saturated, or idf ≤ 0) may still serve as the **weak side** of a pair whose
|
|
2641
|
+
> other side voted: saturation is an abstention about where the region _climbs_;
|
|
2642
|
+
> the junction asks a different question — "which whole holds both?" — and the
|
|
2643
|
+
> container's own idf gate still guards the conclusion. This is **corpus
|
|
2644
|
+
> independence**: a word never trained standalone has no site, but its stored
|
|
2645
|
+
> chunks still vote and their bytes still compose — the ascent matches byte
|
|
2646
|
+
> containment, so a fragment pair evidences the same joint container the whole
|
|
2647
|
+
> word would.
|
|
2648
|
+
|
|
2649
|
+
Two disciplines prevent this openness from becoming noise:
|
|
2650
|
+
|
|
2651
|
+
- **Two non-voting regions never pair.** That is exactly the shared-prefix trap:
|
|
2652
|
+
ascending from two non-discriminative fragments can land on an
|
|
2653
|
+
incidentally-unique descendant container, manufacturing confidence the query
|
|
2654
|
+
gave no reason to have. At least one side must be individually discriminative.
|
|
2655
|
+
- **Only maximal spans compose.** A span wholly contained in another candidate
|
|
2656
|
+
is a fragment of that candidate's evidence, never independent of it.
|
|
2657
|
+
Contiguous shards of one word that are both covered by a single known region
|
|
2658
|
+
are skipped — the whole form already votes directly, and re-deriving it from
|
|
2659
|
+
its own pieces would only double-count.
|
|
2660
|
+
|
|
2661
|
+
Pairs are byte-sorted left-to-right; searches are capped at k probes total.
|
|
2662
|
+
|
|
2663
|
+
**Cost hoisting.** Junction seeds are computed once per candidate for the whole
|
|
2664
|
+
pairing loop (a candidate recurs in up to |cand|−1 pairs, and its seeds are a
|
|
2665
|
+
pure function of its bytes). All reads go through a shared per-response walk
|
|
2666
|
+
cache — one response issues many walks whose ancestries overlap heavily, so
|
|
2667
|
+
every identity read is a cache hit instead of a durable read or byte
|
|
2668
|
+
reconstruction.
|
|
2669
|
+
|
|
2670
|
+
**Order-free binding.** The junction ascent is order-free: a junction is
|
|
2671
|
+
evidence the two forms were learnt _together_; which one the query happened to
|
|
2672
|
+
mention first is a fact about the query, not about the learnt whole. The walk is
|
|
2673
|
+
identical (the seed ascent does not depend on order) — only the byte-containment
|
|
2674
|
+
test gains a second probe, so order-freedom costs two `indexOf` calls per
|
|
2675
|
+
visited node, never a second walk. The test also accepts overlapping or abutting
|
|
2676
|
+
occurrences: two grid-aligned fragments of one whole ("red " at 0 and " cir" at
|
|
2677
|
+
3 in `red circle`) legitimately overlap inside it. A strict-super-form
|
|
2678
|
+
requirement applies (the container must be longer than either side alone):
|
|
2679
|
+
holding both must be more than restating either side.
|
|
2680
|
+
|
|
2681
|
+
**N-ary binding.** Binding is not intrinsically pairwise. When a pair's
|
|
2682
|
+
containers are found, each container is tested against every _remaining_
|
|
2683
|
+
unconsumed candidate form: the container covering the **most** of the query's
|
|
2684
|
+
composable forms (by total byte length of contained extras) wins, with ties
|
|
2685
|
+
going to shortest interior then lowest id. This means three cross-cutting
|
|
2686
|
+
attributes — where every pair is ambiguous across two contexts — resolve to
|
|
2687
|
+
their unique triple, at the cost of one cached byte read + `indexOf` per
|
|
2688
|
+
(container, extra), never an extra walk. A consumed candidate (one whose
|
|
2689
|
+
evidence is already composed in a junction) never re-pairs — its evidence is
|
|
2690
|
+
already at full joint strength, and re-pairing would vote the same container
|
|
2691
|
+
twice.
|
|
2692
|
+
|
|
2693
|
+
**Self-evidence guard.** A container whose joined occurrence (left through right
|
|
2694
|
+
including its interior) is literally a substring of the query is rejected.
|
|
2695
|
+
Binding is only evidence when the query mentions the forms _apart_. Without this
|
|
2696
|
+
guard, shards of a contiguous phrase ("s pa" + "d by" around "inte" in "was
|
|
2697
|
+
painted by") pair to rediscover the phrase they are shards of, then explain away
|
|
2698
|
+
its rivals — breaking multi-candidate tests where the query legitimately weaves
|
|
2699
|
+
several exemplars.
|
|
2700
|
+
|
|
2701
|
+
**Explaining away — the aliasing complement of corpus independence.** A chunk of
|
|
2702
|
+
the query can straddle the byte grid so that it exists verbatim in the _wrong_
|
|
2703
|
+
deposit (" cir" of "red then circle" is a stored chunk of `blue circle`, never
|
|
2704
|
+
of `red circle` — a pure alignment accident) and its independent climb then
|
|
2705
|
+
votes for a context the query gives no reason to believe. When a junction binds,
|
|
2706
|
+
any individual vote whose bytes the joint container **literally contains** AND
|
|
2707
|
+
whose climb roots are **fully disjoint** from the junction's is **superseded**:
|
|
2708
|
+
the exact joint evidence explains those bytes away, so their disagreeing vote is
|
|
2709
|
+
grid aliasing, not signal. A vote sharing even one root with the junction
|
|
2710
|
+
_corroborates_ it (partial agreement — a different slice of the same context)
|
|
2711
|
+
and is kept. Votes whose bytes the container does not hold at all (a genuine
|
|
2712
|
+
second topic) are untouched.
|
|
2713
|
+
|
|
2714
|
+
**Graded ladder.** The junction ascent follows the same evidence discipline as
|
|
2715
|
+
the bridge: Tier 1 (exact containers of the two forms themselves, order-free)
|
|
2716
|
+
first; only when it finds nothing does Tier 2.5 (synonym junctions — the same
|
|
2717
|
+
ascent over halo siblings of one side, sharing one expansion budget across all
|
|
2718
|
+
sibling walks) run as a fallback. Exact evidence outranks distributional
|
|
2719
|
+
approximation; a synonym junction can never override an exact one.
|
|
2720
|
+
|
|
2721
|
+
**Voting.** A joint container is **exact** evidence — it literally holds the
|
|
2722
|
+
composed forms — so it votes at full strength (score = 1, no estimator).
|
|
2723
|
+
Weighting uses the same mutual-explanation and IDF discipline as single-region
|
|
2724
|
+
votes, with the combined byte length of all composed candidates as the region
|
|
2725
|
+
size. The combined pool (independent, minus any superseded by cross-region
|
|
2726
|
+
evidence, + cross-region) means a joint context with no single-region support
|
|
2727
|
+
can still become a point of attention when its combined evidence clears the
|
|
2728
|
+
consensus floor (§8.6).
|
|
2729
|
+
|
|
2730
|
+
---
|
|
2731
|
+
|
|
2732
|
+
## 18. Grounding I — counterfactual transfer (CAST)
|
|
2733
|
+
|
|
2734
|
+
### 18.1 When it applies
|
|
2735
|
+
|
|
2736
|
+
Some queries do not ask about one learned thing; they _weave together several_ —
|
|
2737
|
+
"what if X had Y's property?", "compare X and Y", a sentence that grafts one
|
|
2738
|
+
learned frame onto another's subject. CAST (Counterfactual trAnSfer) detects the
|
|
2739
|
+
weave by **graded alignment** — the same evidence ladder as `locate()`: literal
|
|
2740
|
+
W-gram runs first, then distributional role (halo-matched recognised sites
|
|
2741
|
+
filling gaps where the query has no literal coverage). It transfers structure
|
|
2742
|
+
between the woven parts. CAST is the byte-level, formalized descendant of
|
|
2743
|
+
case-based reasoning's _adaptation_ step (Kolodner 1992).
|
|
2744
|
+
|
|
2745
|
+
Preconditions (all structural, per invariant §14.2): the query is at least two
|
|
2746
|
+
perception windows long; the climb (§17) ranks at least two anchors; aligning
|
|
2747
|
+
the anchors' contexts against the query (below) leaves at least two anchors with
|
|
2748
|
+
_free_ aligned runs; the dominant anchor is a committed point of attention; and
|
|
2749
|
+
at least one aligned run falls **outside every recognised site** — the
|
|
2750
|
+
definition of "woven": material from a learned structure appearing where
|
|
2751
|
+
recognition's normal reading has no account of it. If any of this fails, CAST
|
|
2752
|
+
returns null — the grounding decider considers the remaining candidates.
|
|
2753
|
+
|
|
2754
|
+
### 18.2 Graded alignment
|
|
2755
|
+
|
|
2756
|
+
Alignment follows the same graded-evidence ladder as `locate()` (§14.4):
|
|
2757
|
+
**literal first, then distributional role.** For each ranked anchor:
|
|
2758
|
+
|
|
2759
|
+
1. **Literal** — `alignRuns`: W-gram seed-and-extend. Every W-gram of the query
|
|
2760
|
+
is indexed; each W-gram of the context that matches seeds a run, extended
|
|
2761
|
+
greedily in both directions; overlaps resolved longest-first. Weight = 1.0
|
|
2762
|
+
(exact match is full evidence).
|
|
2763
|
+
|
|
2764
|
+
2. **Halo** — where the query has **no literal coverage** from this anchor,
|
|
2765
|
+
recognised sites with halos are matched to the exemplar context's own
|
|
2766
|
+
recognised sites via `bestHaloMate` (gate: `conceptThreshold`). A query site
|
|
2767
|
+
whose halo resonates with a context site above the bar produces a run with
|
|
2768
|
+
`cs` = the context site's structural byte position and `weight` = the cosine
|
|
2769
|
+
itself (measured evidence, not an invented constant).
|
|
2770
|
+
|
|
2771
|
+
The per-byte depth `depth[i]` is the **sum of weights** of aligned structures
|
|
2772
|
+
covering byte `i` — a byte covered by two literal runs has depth 2.0; a byte
|
|
2773
|
+
covered by one literal (1.0) and one 0.7-halo run has depth 1.7. The `cs`
|
|
2774
|
+
(position in the context) is a real byte offset regardless of run kind, so the
|
|
2775
|
+
substitution and redirection schemas work unchanged on conceptual alignment.
|
|
2776
|
+
|
|
2777
|
+
### 18.3 Frame gate
|
|
2778
|
+
|
|
2779
|
+
Both components of the frame gate are **derived from the weave itself**, not
|
|
2780
|
+
tuned:
|
|
2781
|
+
|
|
2782
|
+
1. **MIN_WEAVE** — the minimum number of aligned structures to form a weave: the
|
|
2783
|
+
same `2` that gates CAST entry (`points.length < 2`). Frame requires evidence
|
|
2784
|
+
**beyond** the minimum pair — a third structure agreeing — so the depth gate
|
|
2785
|
+
is `depth[i] > MIN_WEAVE`. One definition, two uses.
|
|
2786
|
+
|
|
2787
|
+
2. **Half-dominance** — `dominates(part, whole)` (§8.7), the same test
|
|
2788
|
+
`collectRegions`, `liftAnswer`, and confluence's filler gate all use. A byte
|
|
2789
|
+
is frame when `dominates(depth[i], aligned)` — the structures covering it are
|
|
2790
|
+
a majority of all aligned structures. A run is usable when the framed bytes
|
|
2791
|
+
are NOT a majority: `¬dominates(framedCount, runLen)`.
|
|
2792
|
+
|
|
2793
|
+
In full:
|
|
2794
|
+
|
|
2795
|
+
```
|
|
2796
|
+
frame(i) ⇔ depth[i] > MIN_WEAVE ∧ dominates(depth[i], aligned)
|
|
2797
|
+
usable(qs,qe) ⇔ ¬dominates(framedCount(qs, qe), qe − qs)
|
|
2798
|
+
```
|
|
2799
|
+
|
|
2800
|
+
The frame gate is the canonical example of **weave-local commonality** (§8.10):
|
|
2801
|
+
`aligned` counts the structures aligned with _this query_, not the corpus. The
|
|
2802
|
+
choice of reference set is load-bearing — replacing the weave-local majority
|
|
2803
|
+
with corpus-global IDF misfires on reordered single-fact queries. See §8.10 for
|
|
2804
|
+
the general theory and the full table of which mechanism uses which measure.
|
|
2805
|
+
|
|
2806
|
+
### 18.4 Three transfer schemas
|
|
2807
|
+
|
|
2808
|
+
Each schema is tried independently — every one that fires contributes its own
|
|
2809
|
+
candidate to the grounding decider, with its own `accounted` (the runs of
|
|
2810
|
+
exactly the points that schema used, not the whole weave's alignment) and
|
|
2811
|
+
`moves`. The decider's weight comparison replaces CAST's former internal
|
|
2812
|
+
priority order.
|
|
2813
|
+
|
|
2814
|
+
**Substitution** — _the query puts a new subject into a learned structure's
|
|
2815
|
+
seat._ Detected when one structure's run starts mid-context (its head — the seat
|
|
2816
|
+
— is displaced) and another structure, wholly present earlier in the query, fits
|
|
2817
|
+
that seat. The answer is built by projecting the displaced structure onto the
|
|
2818
|
+
new filler: filler + (learned connector if one exists, §19.5) + the displaced
|
|
2819
|
+
structure's tail, then following the projected structure's continuation if it
|
|
2820
|
+
adds something new.
|
|
2821
|
+
|
|
2822
|
+
**Redirection** — _the query names a substitute for the thing the dominant
|
|
2823
|
+
structure is about._ Detected when the latest-positioned structure's run starts
|
|
2824
|
+
at its context's very beginning (it is wholly, freshly named), and none of the
|
|
2825
|
+
dominant anchor's own continuations appears in the query (its usual answer is
|
|
2826
|
+
displaced). The answer is the substitute's own grounded fact: the named thing's
|
|
2827
|
+
knowledge replaces the displaced structure's.
|
|
2828
|
+
|
|
2829
|
+
**Comparison** — _the query juxtaposes two entities of the same kind._ Candidate
|
|
2830
|
+
analogs are the non-dominant points (and their continuation targets, which often
|
|
2831
|
+
name the entity a long exemplar sentence is about). The gate is **analogy
|
|
2832
|
+
strength** (§4.2), a graded three-tier test: (1) the direct halo cosine between
|
|
2833
|
+
dominant and candidate, thresholded at the significance bar; (2) failing that,
|
|
2834
|
+
the strongest _mutual_ halo sibling — two things are analogous if they keep
|
|
2835
|
+
company with the same third things; (3) failing that, **shared-frame strength**
|
|
2836
|
+
— a structural tier that measures what fraction of the shorter side's bytes are
|
|
2837
|
+
covered by a learnt W-window that also occurs in the longer side (two sentences
|
|
2838
|
+
sharing " is " measure as analogs even when their halos never overlapped). If no
|
|
2839
|
+
candidate passes any tier, a deterministic structural fallback picks the
|
|
2840
|
+
best-evidenced genuine hub among the candidates. The answer voices each analog
|
|
2841
|
+
by the context that establishes its role, joined by a learned connector when one
|
|
2842
|
+
exists.
|
|
2843
|
+
|
|
2844
|
+
Whatever CAST produces, the anchors it consumed are marked as such, so the
|
|
2845
|
+
reasoning stage (§22) does not re-walk the same facts. Each schema reports its
|
|
2846
|
+
own `accounted` — the runs of exactly the points that schema transferred
|
|
2847
|
+
between, not the whole weave's alignment. `castFloor` pre-computes the consensus
|
|
2848
|
+
climb (memoised per response, zero extra cost) and returns `null` when the climb
|
|
2849
|
+
cannot possibly support a weave; when non-null, its ranked anchors are reused by
|
|
2850
|
+
the alignment loop, eliminating a redundant second climb call. Every candidate
|
|
2851
|
+
also carries `unexplained` — a human-readable label for the query bytes CAST
|
|
2852
|
+
left unexplained (§14.1).
|
|
2853
|
+
|
|
2854
|
+
---
|
|
2855
|
+
|
|
2856
|
+
### 18.5 Confluence Join — the meet of independent constraints
|
|
2857
|
+
|
|
2858
|
+
#### When it applies
|
|
2859
|
+
|
|
2860
|
+
Some queries carry two or more _independent constraints_ — "Which material is
|
|
2861
|
+
translucent and featherlight?" — where each constraint reaches its own set of
|
|
2862
|
+
exemplars, and the entity satisfying both lives exactly where those sets
|
|
2863
|
+
**intersect**. No single-fact mechanism can answer this: any one evidence path
|
|
2864
|
+
grounds only one constraint, and fusion concatenates one fact per constraint
|
|
2865
|
+
from _different_ entities (wrong answer). Confluence detects independent
|
|
2866
|
+
constraint streams and intersects them by content-addressed identity.
|
|
2867
|
+
|
|
2868
|
+
Preconditions (all structural): the query is at least two perception windows
|
|
2869
|
+
long; the consensus climb (§17) ranks at least two anchors; at least two of them
|
|
2870
|
+
hold disjoint discriminative query windows (they are _independent_ constraints,
|
|
2871
|
+
each answering a different part of what was asked); and the intersection of
|
|
2872
|
+
those anchors' content — the **meet** — contains at least one discriminative
|
|
2873
|
+
span (content reaching a corpus minority of contexts, per the half-dominance
|
|
2874
|
+
convention).
|
|
2875
|
+
|
|
2876
|
+
#### The meet is native, not a byte scan
|
|
2877
|
+
|
|
2878
|
+
The store is content-addressed: any content two deposits share IS the same node
|
|
2879
|
+
id, interned once at write time (hash-consing). So "what do these two facts have
|
|
2880
|
+
in common?" is a **set intersection of identities** the write side already
|
|
2881
|
+
computed, asked through the canonical window read — `windowIds` (§9), the same
|
|
2882
|
+
write/read contract recognition runs on. Three identity/structure tests make the
|
|
2883
|
+
whole mechanism:
|
|
2884
|
+
|
|
2885
|
+
1. **Constraint streams.** The consensus climb's ranked anchors, each bound to
|
|
2886
|
+
the query spans whose _discriminative_ windows it holds by identity (a
|
|
2887
|
+
resonance-voted anchor holding none of the query's discriminative content is
|
|
2888
|
+
no constraint at all). Two streams are independent when the content they bind
|
|
2889
|
+
is disjoint.
|
|
2890
|
+
|
|
2891
|
+
2. **The meet.** Window ids present in _both_ anchors and _absent from the
|
|
2892
|
+
query_: shared-with-query windows are the constraint being re-named (or its
|
|
2893
|
+
scaffolding), so subtracting the query's own window ids leaves exactly the
|
|
2894
|
+
content the question asks _for_ — the open seat.
|
|
2895
|
+
|
|
2896
|
+
3. **Filler/scaffolding separation.** The same structural IDF the climb derives
|
|
2897
|
+
(`reachOf`, §9): shared content reaching a corpus minority of contexts is an
|
|
2898
|
+
entity (a filler); content reaching a majority is frame scaffolding. No
|
|
2899
|
+
statistics, no learning — the same global-quantity-from-capped-local-probes
|
|
2900
|
+
reading that makes the climb's IDF work, pointed at a new question.
|
|
2901
|
+
|
|
2902
|
+
The meet thus names content that _byte-literally exists_ in two independently
|
|
2903
|
+
learnt exemplars. An empty intersection yields null and the ordinary pipeline
|
|
2904
|
+
decides — confluence cannot fabricate.
|
|
2905
|
+
|
|
2906
|
+
#### Evidence for the grounding decider
|
|
2907
|
+
|
|
2908
|
+
Confluence reports the query spans whose constraint content the two streams hold
|
|
2909
|
+
by identity (`accounted`), its acts (`moves`: two constraint matches + one meet
|
|
2910
|
+
= 3×STEP), and `unexplained` — a human-readable label for query bytes outside
|
|
2911
|
+
those spans (§14.1).
|
|
2912
|
+
|
|
2913
|
+
---
|
|
2914
|
+
|
|
2915
|
+
## 19. Grounding II — cover: the graph search
|
|
2916
|
+
|
|
2917
|
+
The cover is the heart of the system: **one lightest-derivation search (§5) that
|
|
2918
|
+
composes an answer out of the query's own decomposition**. All of "retrieval",
|
|
2919
|
+
"rewriting", "multi-part answers", and "synonym use" are individual _rules_ of
|
|
2920
|
+
this one search.
|
|
2921
|
+
|
|
2922
|
+
### 19.1 Items
|
|
2923
|
+
|
|
2924
|
+
Three kinds of item populate the deduction system:
|
|
2925
|
+
|
|
2926
|
+
- **Cover(p)** — "the query is covered from position 0 up to p". The axiom is
|
|
2927
|
+
Cover(0); the **goal is Cover(len(query))**.
|
|
2928
|
+
- **Form(i, j, node)** — "the node `node` names the query span [i, j)". A form
|
|
2929
|
+
is a _foothold in the graph_: rules walk out of it.
|
|
2930
|
+
- **Out(i, j, bytes)** — "the span [i, j) will be answered by these bytes". An
|
|
2931
|
+
out may be _recognised_ (a grounded completion — essentially free to cover) or
|
|
2932
|
+
a _literal_ (unrecognised query bytes carried through at PASS cost).
|
|
2933
|
+
|
|
2934
|
+
### 19.2 Axioms
|
|
2935
|
+
|
|
2936
|
+
- Cover(0), at cost 0.
|
|
2937
|
+
- One Out per perceived leaf of the query (its own bytes, literal) — the
|
|
2938
|
+
covering fabric.
|
|
2939
|
+
- One Form per recognised site (§15) — the graph entry points.
|
|
2940
|
+
- One recognised Out per computed span (§16), at STEP cost.
|
|
2941
|
+
|
|
2942
|
+
### 19.3 Rules
|
|
2943
|
+
|
|
2944
|
+
Stated abstractly (costs from the ladder, §8.9):
|
|
2945
|
+
|
|
2946
|
+
```
|
|
2947
|
+
BRIDGE Cover(i) ∧ Out(i, j) → Cover(j)
|
|
2948
|
+
cost: ε if the out is recognised; PASS·(j−i) if literal.
|
|
2949
|
+
The frontier advances; literal connectives (spaces, commas —
|
|
2950
|
+
anything unrecognised) are CARRIED, so the asker's own linking
|
|
2951
|
+
material survives between rewritten parts.
|
|
2952
|
+
|
|
2953
|
+
FOLLOW-EDGE Form(i, j, n) → Form(i, j, n′) cost STEP
|
|
2954
|
+
where n ──▶ n′ is a learned continuation edge. With several
|
|
2955
|
+
continuations, the disambiguator (§25) picks n′.
|
|
2956
|
+
|
|
2957
|
+
CONCEPT-HOP Form(i, j, n), n edge-less → Form(i, j, s′) cost CONCEPT
|
|
2958
|
+
where s is a halo sibling of n above the concept threshold and
|
|
2959
|
+
s ──▶ s′ — answering through a synonym, one order dearer than
|
|
2960
|
+
a literal edge. (Siblings are pre-resolved before the search,
|
|
2961
|
+
since index queries are asynchronous.)
|
|
2962
|
+
|
|
2963
|
+
GROUND Form(i, j, n), n terminal, reached via edges
|
|
2964
|
+
→ Out(i, j, bytes(n)) cost 0
|
|
2965
|
+
The chain's endpoint becomes answer bytes for the span. Before
|
|
2966
|
+
emitting, RECOMPLETION (§19.6) may resolve deeper.
|
|
2967
|
+
|
|
2968
|
+
SPLIT Out literal, containing a split position k (§15.2)
|
|
2969
|
+
→ the two halves cost 0
|
|
2970
|
+
The query's own chunking is not sacred; a form boundary the
|
|
2971
|
+
store knows can cut a leaf.
|
|
2972
|
+
|
|
2973
|
+
FUSE Out(i, j) ∧ Out(j, k) adjacent → Out(i, k) cost 0
|
|
2974
|
+
The concatenation may name a known node (by content-addressed
|
|
2975
|
+
lookup: as a short leaf; as the branch of the two sides'
|
|
2976
|
+
nodes; or by canonical re-perception when a side is a
|
|
2977
|
+
completed rewrite). If it does, also:
|
|
2978
|
+
|
|
2979
|
+
RECOMPOSE the fused pair → Form(i, k, node) cost 0
|
|
2980
|
+
Two already-rewritten parts fusing into a node that itself
|
|
2981
|
+
continues is a RECOMPOSITION: its onward FOLLOW-EDGE is free,
|
|
2982
|
+
so the consolidated whole strictly beats leaving the parts
|
|
2983
|
+
split. A guard requires the fused node to be halo-bearing —
|
|
2984
|
+
i.e. learned as a meaningful unit, not an accidental interior
|
|
2985
|
+
chunk of some one-shot phrase.
|
|
2986
|
+
|
|
2987
|
+
SPLICE Out(recognised L) ∧ Out(recognised R), a learned connector
|
|
2988
|
+
exists between L's and R's answers → Out(L+connector+R) cost 0
|
|
2989
|
+
(§19.5. Fires only when the gap between them is empty or
|
|
2990
|
+
wholly recognised — never across the asker's own literal
|
|
2991
|
+
separator, which BRIDGE carries instead.)
|
|
2992
|
+
```
|
|
2993
|
+
|
|
2994
|
+
The A\* heuristic is ε per uncovered byte beyond an item's right edge —
|
|
2995
|
+
admissible because ε is the minimum per-position cost, and what keeps the search
|
|
2996
|
+
output-sensitive (§5.2, §8.9).
|
|
2997
|
+
|
|
2998
|
+
### 19.4 What the cost ladder buys, concretely
|
|
2999
|
+
|
|
3000
|
+
- Coverage dominates: the search _must_ account for every byte, and prefers
|
|
3001
|
+
recognising to carrying.
|
|
3002
|
+
- A literal continuation beats a synonym's (STEP < CONCEPT); either beats
|
|
3003
|
+
leaving a span unrewritten (≪ PASS).
|
|
3004
|
+
- Free fusion/recomposition means the search always finds the _deepest
|
|
3005
|
+
consolidated reading_: if "D E" recomposes into a learned "DE" that continues
|
|
3006
|
+
to F, the answer is F, not "D′ E′".
|
|
3007
|
+
- Ties resolve by the fixed conventions of §25 — deterministically.
|
|
3008
|
+
|
|
3009
|
+
### 19.5 Connectors: learned joins (the bridge)
|
|
3010
|
+
|
|
3011
|
+
When an answer has several parts, what belongs _between_ them? Sema asks the
|
|
3012
|
+
store through a **graded junction ladder** — exact evidence before approximate,
|
|
3013
|
+
the same discipline as `locate` (§14.4). The junction search is extracted into
|
|
3014
|
+
one shared procedure so that both the bridge (a connector between answer pieces)
|
|
3015
|
+
and cross-region attention (§17.6, the joint context of query regions) ascend by
|
|
3016
|
+
the same bounded, cached walk:
|
|
3017
|
+
|
|
3018
|
+
1. **Junction containers, by content-addressed identity.** Hash-consing means
|
|
3019
|
+
"which learned wholes ran L and R together?" is a structural question, not a
|
|
3020
|
+
similarity guess: any deposit containing L's bytes shares L's node (or L's
|
|
3021
|
+
canonical-window identities), so ascending the DAG's parent and containment
|
|
3022
|
+
links from the two sides reaches every containing whole _exactly_, under the
|
|
3023
|
+
one √N fan-out discipline. A container whose bytes literally hold L and then
|
|
3024
|
+
R yields the bytes between them as the **connector** ("and", ", ", " is the
|
|
3025
|
+
opposite of " — whatever the corpus actually joins such things with).
|
|
3026
|
+
2. **Edge junctions.** A continuation edge _is_ junction information: a learned
|
|
3027
|
+
continuation of L that contains R carries the glue as its prefix; a learned
|
|
3028
|
+
context of R that contains L carries it as its suffix. An _empty_ interior
|
|
3029
|
+
found this way is a confirmed adjacency — returned as such, never confused
|
|
3030
|
+
with a miss.
|
|
3031
|
+
3. **Synonym junctions.** The content-addressed junction search applied to halo
|
|
3032
|
+
siblings of the two sides: when L or R has no direct junction, one of its
|
|
3033
|
+
distributional siblings may. Container evidence stays exact (same DAG ascent
|
|
3034
|
+
as tier 1, with window-id-enhanced seeds); the relaxation is only in which
|
|
3035
|
+
form occupies one side.
|
|
3036
|
+
4. **Resonance** (the last resort): the gist of the bare concatenation is
|
|
3037
|
+
resonated into the content index and the nearest containment-passing form
|
|
3038
|
+
supplies the connector — approximate, but it still reaches containers whose
|
|
3039
|
+
identity links are absent or saturated.
|
|
3040
|
+
|
|
3041
|
+
When several junctions qualify, the **response guide** (the query's gist — the
|
|
3042
|
+
same disambiguator every projection uses) picks by resonance; ties prefer the
|
|
3043
|
+
shortest interior (a junction should not insert unnecessary glue), then the
|
|
3044
|
+
lowest node id (deterministic — a property of the corpus, not the seed). No
|
|
3045
|
+
learned evidence at any tier ⇒ no connector invented.
|
|
3046
|
+
|
|
3047
|
+
Connectors are pre-resolved for the query's adjacent site pairs (and for
|
|
3048
|
+
first-to-later pairs of longer groups), then handed to the search, where SPLICE
|
|
3049
|
+
applies them inside the derivation — so multi-part answers are assembled as one
|
|
3050
|
+
globally-coherent whole, not stitched afterwards.
|
|
3051
|
+
|
|
3052
|
+
### 19.6 Recompletion: answers that resolve deeper
|
|
3053
|
+
|
|
3054
|
+
A followed edge may land on a _composite_ node that leads nowhere as a whole
|
|
3055
|
+
(say, "p1 p2") yet whose parts each continue. Before emitting such a node as
|
|
3056
|
+
terminal, the search **re-covers the node's own bytes** — the very same solve,
|
|
3057
|
+
recursively: recognition, edges, fusion, recomposition. If that inner cover
|
|
3058
|
+
produces something new that itself names a learned node, the deeper completion
|
|
3059
|
+
becomes the answer for the span. Recursion needs no depth cap: a node already
|
|
3060
|
+
being recompleted is not re-entered (cycle guard), node identities are finite,
|
|
3061
|
+
and finished recompletions are memoised — so chains run exactly as deep as the
|
|
3062
|
+
graph licenses, and stop.
|
|
3063
|
+
|
|
3064
|
+
### 19.7 Reading out the answer
|
|
3065
|
+
|
|
3066
|
+
The finished derivation's chosen spans, left to right, are the cover.
|
|
3067
|
+
**Lifting** extracts the answer from the asker's framing: if one span is
|
|
3068
|
+
recognised, its bytes are the answer (unless it dominates — covers more than
|
|
3069
|
+
half of — the query, in which case the whole cover is kept); with several,
|
|
3070
|
+
everything from the first to the last recognised span (inclusive of carried
|
|
3071
|
+
connectives between them) is kept, and the unrecognised framing outside is
|
|
3072
|
+
dropped.
|
|
3073
|
+
|
|
3074
|
+
---
|
|
3075
|
+
|
|
3076
|
+
## 20. Grounding III — extraction by skill
|
|
3077
|
+
|
|
3078
|
+
### 20.1 Skills are facts with a shape
|
|
3079
|
+
|
|
3080
|
+
Nothing in ingestion marks anything as a "template". But some learned facts
|
|
3081
|
+
_have a shape_: the answer is literally a span of the context (or a few pieces
|
|
3082
|
+
of it), and the context is the frame around it — e.g. ("The Mona Lisa was
|
|
3083
|
+
painted by Leonardo da Vinci.", "Leonardo da Vinci"). Such a fact is a
|
|
3084
|
+
**span-in-context exemplar**, and it can be _applied_ to fresh text: find where
|
|
3085
|
+
the exemplar's frame appears in the query, and read out the analogous span. This
|
|
3086
|
+
is instance-based learning in its purest form (§1.1): the stored episode itself,
|
|
3087
|
+
unmodified, functions as the rule.
|
|
3088
|
+
|
|
3089
|
+
Span-shapedness is read at two deliberately different strengths, and they are
|
|
3090
|
+
not interchangeable:
|
|
3091
|
+
|
|
3092
|
+
- **OPEN reading** (exemplar acceptance): the answer is a sparse subsequence of
|
|
3093
|
+
the context — bytes in order, arbitrary gaps. Permissive, so a multi-piece
|
|
3094
|
+
answer stitched from several context runs validates. Used to _accept_ an
|
|
3095
|
+
exemplar candidate.
|
|
3096
|
+
- **STRONG reading** (answer decomposition): a greedy longest-run decomposition
|
|
3097
|
+
into contiguous pieces. Greedy-longest is strictly stronger than subsequence
|
|
3098
|
+
(a long late match can consume context an earlier shorter choice needed), so
|
|
3099
|
+
an accepted exemplar can still fail to decompose — the mechanism then falls
|
|
3100
|
+
through to recall. Used to _read pieces out_ of the query.
|
|
3101
|
+
|
|
3102
|
+
### 20.2 The algorithm
|
|
3103
|
+
|
|
3104
|
+
```
|
|
3105
|
+
extractBySkill(query):
|
|
3106
|
+
ranked ← climbAttention(query).ranked # §17 — every voted anchor
|
|
3107
|
+
exemplar ← the first ranked anchor that is span-shaped:
|
|
3108
|
+
context ← the anchor's bytes (or, for a terminal answer node, the
|
|
3109
|
+
longest span-shaped context among ≤ √N of its
|
|
3110
|
+
predecessors; query-gist resonance breaks length ties)
|
|
3111
|
+
answer ← its continuation
|
|
3112
|
+
span-shaped ⇔ answer is a contiguous span of context, a recognised
|
|
3113
|
+
subtree of it, or an ordered sparse subsequence
|
|
3114
|
+
(a multi-piece answer)
|
|
3115
|
+
if none: return ∅ # no skill applies; decider moves on
|
|
3116
|
+
|
|
3117
|
+
runs ← decompose the exemplar answer into its pieces within the context
|
|
3118
|
+
for each piece:
|
|
3119
|
+
framePre ← up to W bytes of context before the piece
|
|
3120
|
+
framePost ← up to W bytes after (or the next piece's pre-frame)
|
|
3121
|
+
locate framePre / framePost in the QUERY — by exact bytes first,
|
|
3122
|
+
else by halo resonance (the frame's distributional role
|
|
3123
|
+
matches a query form), else by gist resonance against the
|
|
3124
|
+
query's segments # graded matching, §4
|
|
3125
|
+
the query bytes between the located frames are this piece's analog
|
|
3126
|
+
answer ← the concatenated analogs
|
|
3127
|
+
```
|
|
3128
|
+
|
|
3129
|
+
The demo in the README is this mechanism: three "X was painted by Y" examples
|
|
3130
|
+
make ("…was painted by …", painter) a span-shaped exemplar; the unseen
|
|
3131
|
+
sentence's frames locate; the analogous span — a painter never taught as an
|
|
3132
|
+
answer — is read out of the query itself.
|
|
3133
|
+
|
|
3134
|
+
Extraction reports its **elementary evidence** for the grounding decider: the
|
|
3135
|
+
located frame occurrences in the query (the matched evidence), plus the read
|
|
3136
|
+
span itself when it is **bounded on both sides** — both the pre-border and
|
|
3137
|
+
post-border frames were located, so the read is structurally delimited and its
|
|
3138
|
+
content is a consequence of that match. An open-ended read (the answer reaches
|
|
3139
|
+
the context's end, with no right border located) is NOT accounted — it is the
|
|
3140
|
+
variable being read without a closing delimiter, priced by exclusion through the
|
|
3141
|
+
`unaccounted` bytes the decider charges at PASS each. The act is costed at one
|
|
3142
|
+
CONCEPT (the skill is an analogy) plus one STEP per accounted span.
|
|
3143
|
+
|
|
3144
|
+
This is the mechanism's defining asymmetry. Extraction reads an unknown by
|
|
3145
|
+
structural analogy to a known exemplar: the frames prove "this query has the
|
|
3146
|
+
same shape as this skill" (matched evidence). When both borders are located, the
|
|
3147
|
+
span between them is structurally explained (we know both _where_ and _that_ it
|
|
3148
|
+
should be read); an open-ended read is structurally explained only on one side
|
|
3149
|
+
(we know _where_ to start but not _where_ to stop) — it is the content-novel
|
|
3150
|
+
variable being read, priced the same way the cover's unrecognised literals are:
|
|
3151
|
+
at PASS each. Counting a bounded read as unexplained would let a single-frame
|
|
3152
|
+
extraction tie-weight with mechanisms that genuinely explain more of the query.
|
|
3153
|
+
|
|
3154
|
+
The same discipline applies to recall's consensus tier (§21): the climb's vote
|
|
3155
|
+
explains exactly the query region whose evidence carried the winning point of
|
|
3156
|
+
attention (`Attention.start`–`end`), not the whole query. A consensus vote for
|
|
3157
|
+
"ice" among scaffolding does not explain the word "steel".
|
|
3158
|
+
|
|
3159
|
+
The decider thus prices each mechanism's _actual_ explanatory work — never the
|
|
3160
|
+
bytes it reads out: counting an open-ended read as accounted would let
|
|
3161
|
+
extraction outweigh mechanisms (e.g. CAST on a reworded single-fact question)
|
|
3162
|
+
whose aligned runs are real structural evidence while the extraction's open seat
|
|
3163
|
+
is merely copied. `extractionFloor` pre-computes the consensus climb (zero extra
|
|
3164
|
+
cost) and returns `null` when no anchor exists; when non-null, its ranked
|
|
3165
|
+
anchors are reused by `extractBySkill`, eliminating a redundant climb call.
|
|
3166
|
+
Every extraction candidate carries `unexplained` — a human-readable label for
|
|
3167
|
+
query bytes its frames did not cover (§14.1).
|
|
3168
|
+
|
|
3169
|
+
---
|
|
3170
|
+
|
|
3171
|
+
## 21. Grounding IV — recall by resonance
|
|
3172
|
+
|
|
3173
|
+
Recall handles queries whose own decomposition composed nothing: resonate the
|
|
3174
|
+
_whole query's gist_ and ground the nearest learned form. It is the most
|
|
3175
|
+
fallback-like mechanism — its weight carries the full PASS·|query| for most
|
|
3176
|
+
tiers, so it can only win as the sole grounding (the honest price of an
|
|
3177
|
+
ungrounded answer) — but it participates in the same decider as every other
|
|
3178
|
+
mechanism. Four tiers, each gated on structural evidence, orderly degrading from
|
|
3179
|
+
exactness to an honest echo.
|
|
3180
|
+
|
|
3181
|
+
#### The asymmetry of forward and reverse
|
|
3182
|
+
|
|
3183
|
+
The deduction system (§5, §19) is a **forward** engine: its rules (FOLLOW-EDGE,
|
|
3184
|
+
CONCEPT-HOP, FUSE, SPLICE) all move from premises toward conclusions in the
|
|
3185
|
+
direction of the learned edges. There is no backward rule — no inference step
|
|
3186
|
+
that consumes a conclusion to produce a premise. This is not an omission; it is
|
|
3187
|
+
the formalism: a derivation is a directed hyperpath from axioms to a goal, and
|
|
3188
|
+
the cost ladder prices each forward step. A reading against the edge direction —
|
|
3189
|
+
`reverseContext`, which asks "what establishes this?" rather than "what does
|
|
3190
|
+
this lead to?" — produces bytes but no derivation. It explains nothing about the
|
|
3191
|
+
query in the forward direction the search operates in.
|
|
3192
|
+
|
|
3193
|
+
The grounding decider expresses this exactly: reverse readings get
|
|
3194
|
+
`accounted = []` (nothing matched against learnt structure in the forward
|
|
3195
|
+
direction), so their weight is the full PASS·|query| plus a STEP — the most
|
|
3196
|
+
expensive grounding, available when nothing composes forward, impossible to
|
|
3197
|
+
prefer when anything does. The decider _derives_ this from the evidence the
|
|
3198
|
+
formalism itself declares: a backward step carries no explanatory weight.
|
|
3199
|
+
|
|
3200
|
+
Every tier grounds through the shared projections of §14.4: `reverseContext` for
|
|
3201
|
+
the reverse cases, `project` (forward-else-reverse) for the rest — recall owns
|
|
3202
|
+
no grounding machinery of its own.
|
|
3203
|
+
|
|
3204
|
+
**Tier 0 — exact self-match (content-addressed).** If the query _resolves_ — it
|
|
3205
|
+
is literally a stored node — answer with the context that predicts it (the
|
|
3206
|
+
reverse projection; among several predecessors, the query gist picks by
|
|
3207
|
+
resonance). This tier never consults the ANN index: identity is exactly
|
|
3208
|
+
decidable, and an estimated score must never stand in for it (§6.2). The
|
|
3209
|
+
grounding is a pure reverse reading: `accounted = []`, `moves = STEP`. The
|
|
3210
|
+
decider prices this honestly: a reverse reading is the designated last resort,
|
|
3211
|
+
never a peer of forward evidence.
|
|
3212
|
+
|
|
3213
|
+
**Tier 1 — clean resonance.** If the top hit's estimated score clears the merge
|
|
3214
|
+
threshold (§8.1), the query essentially _is_ a learned form: `project` tries
|
|
3215
|
+
forward first (to the continuation fixpoint), then reverse (to the establishing
|
|
3216
|
+
context). A forward grounding accounts for the _whole_ query (identity-grade
|
|
3217
|
+
match, `accounted = [[0, query.length]]`); a reverse reading accounts for
|
|
3218
|
+
nothing. Cost: one STEP either way.
|
|
3219
|
+
|
|
3220
|
+
**Tier 2 — scaffolding-dominated.** If the top score clears only the
|
|
3221
|
+
significance bar (§8.3) — real but diluted, typically because shared boilerplate
|
|
3222
|
+
dominates the gist — run the consensus climb (§17) and ground its dominant
|
|
3223
|
+
anchor, provided the anchor's pooled vote clears the consensus floor (§8.6).
|
|
3224
|
+
Accounts for exactly the query spans whose evidence carried the winning point of
|
|
3225
|
+
attention — not the whole query. Cost: one CONCEPT (the climb is a halo-mediated
|
|
3226
|
+
act).
|
|
3227
|
+
|
|
3228
|
+
**Tier 3 — last resort.** This tier is gated on the **fraction of the query the
|
|
3229
|
+
grounding explains**, not the raw cosine (§2.6). Root gists are unit vectors,
|
|
3230
|
+
but under the linear fold cosine = shared / √(len_query · len_grounding), so the
|
|
3231
|
+
raw cosine of a query fully contained in a much longer grounded answer is
|
|
3232
|
+
√(len_query / len_grounding) — a number that shrinks the longer the honestly-
|
|
3233
|
+
containing answer is, and would refuse a perfectly good containment while
|
|
3234
|
+
letting a same-length answer that shares only scaffolding pass. Converting the
|
|
3235
|
+
cosine into `cos · √(len_grounding / len_query)` — a query-relative fraction —
|
|
3236
|
+
measures exactly what the reach bar is supposed to mean: how much of THE QUERY
|
|
3237
|
+
the store accounts for, regardless of how much longer the matched form is.
|
|
3238
|
+
|
|
3239
|
+
Walk the hits nearest-first and ground the first one whose query-relative
|
|
3240
|
+
fraction clears the reach threshold (§8.2). Failing that: if even the nearest
|
|
3241
|
+
hit's fraction is _below_ the reach threshold, **return nothing** — the store
|
|
3242
|
+
holds nothing related. Otherwise return the nearest form's own bytes verbatim,
|
|
3243
|
+
explicitly flagged as an **echo**: within reach, but not a grounded fact — it
|
|
3244
|
+
accounts for nothing and carries no move cost, so it can only win as the sole
|
|
3245
|
+
grounding (the honest price of an ungrounded answer). The flag travels in the
|
|
3246
|
+
response's provenance (§26) so a confident-looking parrot is always
|
|
3247
|
+
distinguishable from an answer. Each tier also carries `unexplained` — a
|
|
3248
|
+
human-readable label for the query bytes its evidence left on the table (§14.1)
|
|
3249
|
+
— appearing in the rationale trace alongside `accounted` and `moves`.
|
|
3250
|
+
|
|
3251
|
+
---
|
|
3252
|
+
|
|
3253
|
+
## 22. Reasoning: the multi-hop chain
|
|
3254
|
+
|
|
3255
|
+
Grounding produces a _first_ answer; reasoning asks what that answer _implies_,
|
|
3256
|
+
iteratively:
|
|
3257
|
+
|
|
3258
|
+
```
|
|
3259
|
+
reason(query, answer, consumed₀):
|
|
3260
|
+
if the query itself is some context's learned continuation:
|
|
3261
|
+
return answer # echo guard: hopping forward from an
|
|
3262
|
+
# answer-shaped query would chain through
|
|
3263
|
+
# the very fact that produced it
|
|
3264
|
+
consumed ← consumed₀ # everything grounding already spoke for,
|
|
3265
|
+
# expanded through halo siblings (synonyms
|
|
3266
|
+
# of consumed nodes are consumed too)
|
|
3267
|
+
repeat up to K times:
|
|
3268
|
+
1. FORWARD ABSORB: if the current answer resolves to a node with an
|
|
3269
|
+
unconsumed continuation, follow it (guided, §25) to its fixpoint
|
|
3270
|
+
and absorb — the answer is itself a learned fact; state what it
|
|
3271
|
+
leads to.
|
|
3272
|
+
2. else PIVOT: find the longest unconsumed learned CONTEXT whose
|
|
3273
|
+
bytes the answer literally CONTAINS (candidates proposed by
|
|
3274
|
+
resonating the answer's subtree gists and by exact recognition;
|
|
3275
|
+
confirmed only by byte containment — resonance alone never
|
|
3276
|
+
hops). Follow the pivot's continuation. If none: stop.
|
|
3277
|
+
mark the followed fact and its neighbours consumed
|
|
3278
|
+
return the fixpoint
|
|
3279
|
+
```
|
|
3280
|
+
|
|
3281
|
+
The consumed-set discipline is what makes the chain _progress_: each hop must
|
|
3282
|
+
bring in a fact not already spoken for, so the walk cannot circle, and the same
|
|
3283
|
+
content is never restated. This is how "The Weeping Woman was painted by Pablo
|
|
3284
|
+
Picasso" continues onward to what the store knows _about Picasso_: the extracted
|
|
3285
|
+
answer contains the learned context "Pablo Picasso", whose continuation is the
|
|
3286
|
+
Cubism fact.
|
|
3287
|
+
|
|
3288
|
+
---
|
|
3289
|
+
|
|
3290
|
+
## 23. Fusion: multi-topic answers
|
|
3291
|
+
|
|
3292
|
+
If the query carries several independent points of attention (§17) — and the
|
|
3293
|
+
grounded answer was _not_ drawn from the query's own text (extraction already
|
|
3294
|
+
spans all its pieces; fusing would add noise) — each further committed point
|
|
3295
|
+
grounds its own answer, and the pieces are joined in query order, with a learned
|
|
3296
|
+
connector (§19.5) between each adjacent pair where one exists. A missing
|
|
3297
|
+
connector joins the pieces bare and records the degradation in the trace. Thus
|
|
3298
|
+
"ice fire" (two topics) becomes "cold hot" — or "cold and hot", if the corpus
|
|
3299
|
+
ever joined such answers with "and".
|
|
3300
|
+
|
|
3301
|
+
---
|
|
3302
|
+
|
|
3303
|
+
## 24. Articulation: answering in the asker's words
|
|
3304
|
+
|
|
3305
|
+
The final pass adjusts _voice_, not content. The asker's query is decomposed
|
|
3306
|
+
into its recognised, halo-bearing forms — the asker's _vocabulary_. Each
|
|
3307
|
+
recognised form of the answer whose halo resonates (above the concept threshold,
|
|
3308
|
+
§8.5) with one of the asker's forms is a concept the two express in different
|
|
3309
|
+
words; the answer's wording is substituted with the asker's. The substitutions
|
|
3310
|
+
are spliced by the same cover search (§19), run over the _answer_ with
|
|
3311
|
+
substitute emissions as the only rules — so voicing is a derivation too, subject
|
|
3312
|
+
to the same composition discipline (and traced like everything else).
|
|
3313
|
+
Single-byte forms are excluded on principle: a one-byte form's halo keeps
|
|
3314
|
+
company with everything, so it licenses spurious substitutions. If the revoiced
|
|
3315
|
+
cover does not compose, the answer stands unchanged.
|
|
3316
|
+
|
|
3317
|
+
---
|
|
3318
|
+
|
|
3319
|
+
## 25. Disambiguation: choosing among alternatives
|
|
3320
|
+
|
|
3321
|
+
Learned knowledge is plural: a context may have many continuations; a
|
|
3322
|
+
continuation may follow many contexts. Sema's choices among them follow two
|
|
3323
|
+
fixed regimes — and which regime applies is a matter of _direction_:
|
|
3324
|
+
|
|
3325
|
+
- **Forward (which continuation?): structural evidence.** Candidates are often
|
|
3326
|
+
short spans whose gists are dominated by accidental byte correlations, so
|
|
3327
|
+
geometry is _not_ consulted. The winner is the candidate predicted by the most
|
|
3328
|
+
**distinct contexts** (diversity of independent evidence), tie-broken by
|
|
3329
|
+
**halo mass** (sheer episodic repetition), then by insertion order
|
|
3330
|
+
(first-learned). Candidates are capped at the hub bound √N.
|
|
3331
|
+
- **Reverse (which context?): geometric evidence.** Candidate contexts are whole
|
|
3332
|
+
learned experiences — long enough that their gists are semantically meaningful
|
|
3333
|
+
— so the winner is the context whose gist best resonates with the query's gist
|
|
3334
|
+
(again capped at √N). Without a query in flight, the most-corroborated
|
|
3335
|
+
(highest halo mass) context wins.
|
|
3336
|
+
|
|
3337
|
+
Two response-wide conventions (invariant §14.2): every mechanism of one response
|
|
3338
|
+
consults the same query-gist guide and shares one memo of picks, so an ambiguous
|
|
3339
|
+
fact reads the same everywhere in the answer; and all tie-breaks bottom out in
|
|
3340
|
+
fixed, corpus-determined orderings — never in anything nondeterministic.
|
|
3341
|
+
|
|
3342
|
+
Both regimes, and the √N cap they share, are defined exactly once: `corpusN` →
|
|
3343
|
+
`hubBound` (≥ 2, the count of contexts the cap reads from) is the numerical
|
|
3344
|
+
bound passed to the store's LIMITed reads (`nextFirst(id, hubBound)`,
|
|
3345
|
+
`prevFirst(id, hubBound)`); `hubCap` is the list-side reading of the same
|
|
3346
|
+
convention. The forward regime lives inside `chooseNext` (called by the shared
|
|
3347
|
+
`guidedFirst`, which merges guided-pick with the first-inserted fallback into
|
|
3348
|
+
one LIMITed read), and the reverse regime inside `chooseAmong` — both consumed
|
|
3349
|
+
by every mechanism only through the shared projections (`follow`,
|
|
3350
|
+
`reverseContext`, `project`). The store's existence probes (`hasNext`,
|
|
3351
|
+
`hasHalo`) answer "does this lead anywhere?" without materialising edge lists.
|
|
3352
|
+
No mechanism can drift onto a private disambiguation rule, and no per-query read
|
|
3353
|
+
grows with the corpus.
|
|
3354
|
+
|
|
3355
|
+
---
|
|
3356
|
+
|
|
3357
|
+
## 26. Auditability: provenance and the rationale
|
|
3358
|
+
|
|
3359
|
+
Sema's answers carry their epistemology with them, at two grains:
|
|
3360
|
+
|
|
3361
|
+
**Provenance** — every response is tagged with the mechanism that grounded it:
|
|
3362
|
+
`cast`, `join`, `cover`, `extract`, `recall`, or `recall-echo`. The `join` tag
|
|
3363
|
+
means the answer was produced by intersecting independent constraint streams
|
|
3364
|
+
(confluence, §18.5) — a conjunctive query where no single fact holds the answer.
|
|
3365
|
+
The `recall-echo` tag is the honesty flag of §21's tier 3: the bytes are a
|
|
3366
|
+
stored form returned verbatim for being _near_, not a derived fact. A consumer
|
|
3367
|
+
can gate on this tag mechanically.
|
|
3368
|
+
|
|
3369
|
+
**Rationale** — on request, the response includes the complete replayable trace:
|
|
3370
|
+
every mechanism's entries and exits, and — at the finest grain — every rule
|
|
3371
|
+
application of the lightest derivation itself (each FOLLOW-EDGE, CONCEPT-HOP,
|
|
3372
|
+
FUSE, RECOMPOSE, SPLICE, BRIDGE, and pooled vote, with its premises, conclusion,
|
|
3373
|
+
local cost, and data-flow edges to the steps that produced its premises). This
|
|
3374
|
+
is a direct serialization of the proof tree of §5.4: the answer _is_ this
|
|
3375
|
+
derivation, so the trace is not instrumentation bolted onto an opaque process —
|
|
3376
|
+
it is the process.
|
|
3377
|
+
|
|
3378
|
+
Together with determinism (same store + same query ⇒ same answer, always), this
|
|
3379
|
+
yields the property regulated and safety-critical settings actually require: any
|
|
3380
|
+
output can be reproduced exactly and attributed to enumerable stored facts and
|
|
3381
|
+
rules.
|
|
3382
|
+
|
|
3383
|
+
---
|
|
3384
|
+
|
|
3385
|
+
---
|
|
3386
|
+
|
|
3387
|
+
# Part V — The whole algorithm in pseudocode
|
|
3388
|
+
|
|
3389
|
+
## 27. End-to-end pseudocode
|
|
3390
|
+
|
|
3391
|
+
This section restates the entire system as one connected program, at a level of
|
|
3392
|
+
detail sufficient to reimplement it. Notation: `≔` binds; `∅` is empty; D, W, N
|
|
3393
|
+
as in §8; thresholds by their §8 names. Store operations (`resolve`, `next`,
|
|
3394
|
+
`prev`, `parents`, `halo`, `resonate`, …) are as defined in Parts I–III.
|
|
3395
|
+
|
|
3396
|
+
### 27.1 Shared primitives
|
|
3397
|
+
|
|
3398
|
+
```
|
|
3399
|
+
# ── geometry (VSA, §2) ────────────────────────────────────────────────
|
|
3400
|
+
alphabet[b] ≔ deterministic unit vector for byte b (recursive
|
|
3401
|
+
refinement 16→64→256, seeded)
|
|
3402
|
+
π₀ … π_{S−1} ≔ fixed independent random permutations (the keyring)
|
|
3403
|
+
fold(v₀ … vₖ) ≔ Σᵢ πᵢ·vᵢ # NOT normalized — only
|
|
3404
|
+
# a fold's finished ROOT is
|
|
3405
|
+
# (§2.6): interior gists
|
|
3406
|
+
# keep a byte-proportional
|
|
3407
|
+
# magnitude, ‖·‖ ≈ √len
|
|
3408
|
+
companySignature(id) ≔ seeded random unit vector from `id`
|
|
3409
|
+
# halo pours use identity-based signatures, not gists
|
|
3410
|
+
resonance(a, b) ≔ cosine(a, b)
|
|
3411
|
+
contentLen(id) ≔ the byte length recoverable from a stored gist's own
|
|
3412
|
+
(unnormalized) magnitude, or the store's exact record
|
|
3413
|
+
of it — the linear fold's ‖·‖ ≈ √len read backward
|
|
3414
|
+
fracOfQuery(cos, otherLen, qLen) ≔ min(1, cos · √(otherLen / max(1, qLen)))
|
|
3415
|
+
# converts a raw cosine into a query-relative fraction
|
|
3416
|
+
# of shared content (§2.6, §21)
|
|
3417
|
+
|
|
3418
|
+
# ── perception (§10) ──────────────────────────────────────────────────
|
|
3419
|
+
perceive(bytes):
|
|
3420
|
+
leaves ≔ [ node(bytes = bᵢ, gist = alphabet[bᵢ]) ]
|
|
3421
|
+
p ≔ longest proper prefix already known as a stored leaf-sequence
|
|
3422
|
+
level ≔ leaves
|
|
3423
|
+
while |level| > 1:
|
|
3424
|
+
partition level at the item containing offset p (if p > 0)
|
|
3425
|
+
within each partition, fold complete groups of W;
|
|
3426
|
+
carry incomplete trailing items up unchanged
|
|
3427
|
+
if nothing folded (stall): force-fold in groups of W
|
|
3428
|
+
level ≔ the folded row
|
|
3429
|
+
normalize(level[0].gist) # ONLY the finished root — every
|
|
3430
|
+
# interior gist keeps its raw,
|
|
3431
|
+
# byte-proportional magnitude (§2.6)
|
|
3432
|
+
return level[0] # tree: every node has gist + kids/bytes
|
|
3433
|
+
|
|
3434
|
+
gistOf(bytes) ≔ perceive(bytes).gist
|
|
3435
|
+
resolve(bytes) ≔ intern-lookup of perceive(bytes), bottom-up:
|
|
3436
|
+
leaves by findLeaf, branches by findBranch(kidIds);
|
|
3437
|
+
null the moment any part is unknown
|
|
3438
|
+
read(node) ≔ concatenation of the node's leaf bytes, left to right
|
|
3439
|
+
|
|
3440
|
+
# ── thresholds (§8) ───────────────────────────────────────────────────
|
|
3441
|
+
MERGE ≔ 1 − 1/√D REACH ≔ 1 − 1/(2W) SIG ≔ 3/√D
|
|
3442
|
+
NOISE ≔ 1/√D CONCEPT_BAR ≔ ½ + 1/(2√D)
|
|
3443
|
+
FLOOR(N) ≔ ln N + ½ HUB(N) ≔ ⌈√N⌉
|
|
3444
|
+
DOMINATES(pLen, wLen) ≔ pLen·2 > wLen # half-dominance (§8.7)
|
|
3445
|
+
corpusN ≔ max(2, edgeSourceCount) # N floored at 2 (§8.8)
|
|
3446
|
+
```
|
|
3447
|
+
|
|
3448
|
+
### 27.2 Ingestion
|
|
3449
|
+
|
|
3450
|
+
```
|
|
3451
|
+
ingestPair(context, continuation):
|
|
3452
|
+
(ctxTree, ctxRoot, ctxIds, changed) ≔ deposit(context, tracked)
|
|
3453
|
+
(conTree, conRoot, _, _) ≔ deposit(continuation, untracked)
|
|
3454
|
+
link(ctxRoot → conRoot)
|
|
3455
|
+
for part in changed:
|
|
3456
|
+
pourHalo(ctxIds[part], π₁·companySignature(conRoot)); massOf(part) += 1
|
|
3457
|
+
pourHalo(conRoot, π₀·companySignature(part)); massOf(conRoot) += 1
|
|
3458
|
+
# link/pour lazily admit both subtrees' interiors to the content index
|
|
3459
|
+
|
|
3460
|
+
deposit(input, tracked):
|
|
3461
|
+
tree ≔ perceive(flatten(input))
|
|
3462
|
+
for node in postorder(tree):
|
|
3463
|
+
id(node) ≔ intern(node) # §11.1 ladder:
|
|
3464
|
+
# exact-dedup →
|
|
3465
|
+
# byte-verified near-
|
|
3466
|
+
# dedup → mint (+ the
|
|
3467
|
+
# child→parent edges)
|
|
3468
|
+
intern sliding windows of W and W−1 leaves as flat branches;
|
|
3469
|
+
containment-link each to the chunks it overlaps # §11.3
|
|
3470
|
+
intern the whole stream as one flat branch # §11.3
|
|
3471
|
+
changed ≔ tracked ? maximal-new-subtree(tree, previousDeposit) : [tree]
|
|
3472
|
+
return (tree, rootId, ids, changed)
|
|
3473
|
+
```
|
|
3474
|
+
|
|
3475
|
+
### 27.3 respond
|
|
3476
|
+
|
|
3477
|
+
```
|
|
3478
|
+
respond(input):
|
|
3479
|
+
query ≔ flatten(input)
|
|
3480
|
+
guide ≔ gistOf(query) # the response-wide disambiguation guide
|
|
3481
|
+
thought ≔ think(query)
|
|
3482
|
+
if thought = ∅: return SILENCE
|
|
3483
|
+
answer ≔ articulate(thought.bytes, query) # §24
|
|
3484
|
+
return (answer, thought.provenance [, rationale])
|
|
3485
|
+
|
|
3486
|
+
think(query, mechanisms ≔ defaultMechanisms):
|
|
3487
|
+
rec ≔ recognise(query) # §15
|
|
3488
|
+
|
|
3489
|
+
# Phase 1 — parse: EVERY mechanism that implements it (only computational
|
|
3490
|
+
# ones do — the ALU, any user extension) contributes computed spans
|
|
3491
|
+
# BEFORE any floor()/run() is called. No mechanism-specific branch: the
|
|
3492
|
+
# pipeline just asks "does this one have parse?".
|
|
3493
|
+
computed ≔ ⋃ mech.parse(query) for each mech in mechanisms with parse
|
|
3494
|
+
|
|
3495
|
+
# Phase 2 — the shared precomputation container, response-scoped, read
|
|
3496
|
+
# by every mechanism's floor/run AND by the post-grounding stages:
|
|
3497
|
+
guide ≔ gistOf(query)
|
|
3498
|
+
pre ≔ Precomputed(rec, computed, guide, k) # eager fields only.
|
|
3499
|
+
# Every EXPENSIVE analysis is a lazily-cached method:
|
|
3500
|
+
# pre.attention() — the consensus climb (§17)
|
|
3501
|
+
# pre.weave() — graded alignment over ranked anchors
|
|
3502
|
+
# pre.spanShapedOf(a) — per-anchor skill classification
|
|
3503
|
+
# pre.windowsOf(a) / pre.queryWindows / pre.reachMemo — the
|
|
3504
|
+
# content-addressed identity reads
|
|
3505
|
+
# Computed at most once, shared by every consumer; NEVER computed
|
|
3506
|
+
# if no surviving mechanism asks — a query an extension decided
|
|
3507
|
+
# outright never pays for a climb.
|
|
3508
|
+
|
|
3509
|
+
# ── Phase 3 — Grounding: ONE lightest-derivation choice among UNIFORM
|
|
3510
|
+
# mechanisms. Every mechanism implements the SAME PipelineMechanism
|
|
3511
|
+
# shape (floor, run); think never imports a mechanism-specific type and
|
|
3512
|
+
# never branches on which one it is holding. Each yields a CANDIDATE
|
|
3513
|
+
# weighed in the one cost ladder (§8.9). A candidate's weight is:
|
|
3514
|
+
#
|
|
3515
|
+
# moves + PASS · unaccounted(query, accounted)
|
|
3516
|
+
#
|
|
3517
|
+
# — moves is the ladder cost of the mechanism's acts (STEP per projection/
|
|
3518
|
+
# locate, CONCEPT per halo-mediated act); unaccounted counts query bytes
|
|
3519
|
+
# NOT covered by the union of accounted spans. PASS per unexplained byte
|
|
3520
|
+
# is exactly the cover's own price for a literal connective.
|
|
3521
|
+
#
|
|
3522
|
+
# Weights are compared at STEP resolution (grade ≔ ⌊w/STEP⌋): sub-STEP
|
|
3523
|
+
# costs (MICRO) are non-ordering bookkeeping. Grade TIES keep the
|
|
3524
|
+
# earlier candidate — the mechanism list's own order (cover, cast,
|
|
3525
|
+
# confluence, extract, recall — see defaultMechanisms).
|
|
3526
|
+
|
|
3527
|
+
best ≔ ∅
|
|
3528
|
+
grade(w) ≔ ⌊w / STEP⌋
|
|
3529
|
+
unaccounted(spans) ≔ query.length − total bytes covered by the union of spans
|
|
3530
|
+
weigh(accounted, moves) ≔ moves + PASS · unaccounted(accounted)
|
|
3531
|
+
|
|
3532
|
+
consider(c):
|
|
3533
|
+
if c.bytes.length = 0: return
|
|
3534
|
+
if best = ∅ or grade(c.weight) < grade(best.weight): best ≔ c
|
|
3535
|
+
|
|
3536
|
+
worthRunning(floor) ≔ best = ∅ or grade(floor) < grade(best.weight)
|
|
3537
|
+
|
|
3538
|
+
# `floor` runs for EVERY mechanism, every time, in list order — BEFORE
|
|
3539
|
+
# `run` is even considered. `worthRunning` gates `run`. A mechanism
|
|
3540
|
+
# whose floor itself needs expensive precomputation to refine (CAST's
|
|
3541
|
+
# weave alignment: existence only, the number is always 2·STEP) receives
|
|
3542
|
+
# `worthRunning` too, and checks ITS OWN cheapest possible floor against
|
|
3543
|
+
# the incumbent before paying for that precomputation — the same
|
|
3544
|
+
# admissible-floor economy, applied one level earlier, uniformly. cover
|
|
3545
|
+
# runs FIRST in the list: a computed span masks in at near-zero cost, so
|
|
3546
|
+
# by the time CAST/confluence ask worthRunning, a cheap incumbent may
|
|
3547
|
+
# already have pruned them — not because they know "an extension fired",
|
|
3548
|
+
# only because they know "the incumbent is cheap".
|
|
3549
|
+
for mech in mechanisms:
|
|
3550
|
+
floor ≔ mech.floor(ctx, query, pre, worthRunning)
|
|
3551
|
+
if floor = ∅: continue # structurally can't fire
|
|
3552
|
+
if not worthRunning(floor): continue # can't beat the incumbent
|
|
3553
|
+
for r in mech.run(ctx, query, pre):
|
|
3554
|
+
consider({ bytes: r.bytes, provenance: r.provenance ?? mech.provenance,
|
|
3555
|
+
weight: r.weight ?? weigh(r.accounted, r.moves),
|
|
3556
|
+
used: r.used, accounted: r.accounted,
|
|
3557
|
+
unexplained: r.unexplained })
|
|
3558
|
+
|
|
3559
|
+
if best = ∅: return ∅
|
|
3560
|
+
# ── Diagnostics (observational, never affect the decision) ──────────
|
|
3561
|
+
if |candidates| > 1 and runnerUp exists:
|
|
3562
|
+
margin ≔ grade(runnerUp.weight) − grade(best.weight)
|
|
3563
|
+
if margin ≤ 1: emit narrowDecision trace with both candidates
|
|
3564
|
+
density ≔ |union(best.accounted)| / query.length
|
|
3565
|
+
if density < 1/W: emit thinGrounding trace
|
|
3566
|
+
|
|
3567
|
+
(answer, provenance, consumed) ≔ (best.bytes, best.provenance,
|
|
3568
|
+
best.used ?? ∅)
|
|
3569
|
+
|
|
3570
|
+
# ── Post-grounding ──────────────────────────────────────────────────
|
|
3571
|
+
consumed ≔ per provenance: cast.used | join.used | sites of
|
|
3572
|
+
recognise(answer) | ∅ (recall/recall-echo consume nothing)
|
|
3573
|
+
# cast and join pre-consume their own consumed set for reasoning
|
|
3574
|
+
answer ≔ reason(query, answer, consumed) # §22
|
|
3575
|
+
if provenance ∈ {recall, recall-echo}:
|
|
3576
|
+
answer ≔ fuseAttention(query, answer) # §23
|
|
3577
|
+
return (answer, provenance)
|
|
3578
|
+
```
|
|
3579
|
+
|
|
3580
|
+
### 27.4 The cover search (grounding II, §19)
|
|
3581
|
+
|
|
3582
|
+
```
|
|
3583
|
+
coverMechanism.run(query, pre): # rec, computed read from pre
|
|
3584
|
+
sites ≔ rec.sites minus any site overlapping a computed span # masking §16.3
|
|
3585
|
+
connectors ≔ resolveConnectors(sites) # §19.5, async pre-resolution
|
|
3586
|
+
concepts ≔ resolveConcepts(sites) # halo siblings with edges,
|
|
3587
|
+
# for edge-less sites (§19.3)
|
|
3588
|
+
solved ≔ lightestDerivation( system(query.len, sites, concepts,
|
|
3589
|
+
rec.leaves, rec.splits,
|
|
3590
|
+
connectors, computed) )
|
|
3591
|
+
if solved = ∅: return ∅
|
|
3592
|
+
segs ≔ solved.segs
|
|
3593
|
+
answer ≔ liftAnswer(segs) # §19.7
|
|
3594
|
+
# accounted = RECOGNISED cover spans only (PASS-carried bytes are priced
|
|
3595
|
+
# in `cost`; the diagnostic `unexplained` label reflects the same distinction)
|
|
3596
|
+
accounted ≔ [s.span for each span s in segs where s.rec]
|
|
3597
|
+
return { bytes: answer, cost: solved.cost, accounted,
|
|
3598
|
+
unexplained: unexplainedLabel(query, accounted) }
|
|
3599
|
+
|
|
3600
|
+
system(L, sites, concepts, leaves, splits, connectors, computed):
|
|
3601
|
+
axioms: Cover(0)@0; Out(leaf)@0 ∀ leaves; Form(site)@0 ∀ sites;
|
|
3602
|
+
Out(computed, recognised)@STEP ∀ computed
|
|
3603
|
+
goal: Cover(L)
|
|
3604
|
+
h(item): ε · (L − rightEdge(item)) # admissible (§8.9)
|
|
3605
|
+
rules(item):
|
|
3606
|
+
Cover(p): BRIDGE across every coverable Out starting at p
|
|
3607
|
+
— ε if recognised, PASS·width if literal
|
|
3608
|
+
Form(i,j,n):
|
|
3609
|
+
if substitutionMode: emit the substitute Out@0 (articulation only)
|
|
3610
|
+
elif next(n) ≠ ∅: Form(i,j, chooseNext(n))@(rcmp? 0 : STEP)
|
|
3611
|
+
elif reached-via-edge:
|
|
3612
|
+
deeper ≔ recomplete(n) # §19.6: re-cover n's own
|
|
3613
|
+
# bytes; cycle-guarded, memoised
|
|
3614
|
+
Out(i,j, deeper ?? read(n), recognised)@0
|
|
3615
|
+
elif concepts[n] exists: Form(i,j, concepts[n])@CONCEPT
|
|
3616
|
+
else: (no rule — the form leads nowhere)
|
|
3617
|
+
Out(i,j,b):
|
|
3618
|
+
SPLICE with any finalised partner Out whose (leftNode,rightNode)
|
|
3619
|
+
has a connector, gap empty-or-wholly-recognised @0
|
|
3620
|
+
SPLIT at any split position k ∈ (i,j) if literal @0
|
|
3621
|
+
BRIDGE from Cover(i) if already finalised (symmetric case)
|
|
3622
|
+
FUSE with any adjacent finalised Out:
|
|
3623
|
+
node ≔ findLeaf(bytes) if short | findBranch(nodes)
|
|
3624
|
+
| resolve(bytes) if a side is a completed rewrite
|
|
3625
|
+
require halo(node) ≠ ∅ when a completed rewrite fuses in
|
|
3626
|
+
yield Out(i,k,bytes)@0 [kept only while it could still grow]
|
|
3627
|
+
if node: yield Form(i,k,node, rcmp = both-sides-rewritten
|
|
3628
|
+
∧ next(node) ≠ ∅)@0
|
|
3629
|
+
```
|
|
3630
|
+
|
|
3631
|
+
### 27.5 The consensus climb (§17)
|
|
3632
|
+
|
|
3633
|
+
```
|
|
3634
|
+
climbAttention(query, k):
|
|
3635
|
+
regions ≔ subtrees of perceive(query), excluding any region that
|
|
3636
|
+
dominates (covers more than half of) the query unless it is
|
|
3637
|
+
the sole structure
|
|
3638
|
+
# sites name content-addressed nodes — exact anchors, no ANN needed
|
|
3639
|
+
regions ∪= recognise(query).sites (as {start, end, gist, nodeId})
|
|
3640
|
+
for each region:
|
|
3641
|
+
anchor ≔ region.nodeId # site: exact, skip resonance
|
|
3642
|
+
?? canonicalChunkId(region.bytes, HUB(N))
|
|
3643
|
+
?? contentIndex.nearest(region.gist, k)[0]
|
|
3644
|
+
reach ≔ expandUntilDecided(anchor, HUB(N)):
|
|
3645
|
+
# uses ONLY LIMITed store reads, bounded by √N:
|
|
3646
|
+
# · prevCount(id) — indexed O(1) "edge-bearing?" check
|
|
3647
|
+
# · parentsFirst(id, HUB(N)+1) — hub if |result| > HUB(N)
|
|
3648
|
+
# · containersSlice(anchor, offset, HUB(N)) — paged
|
|
3649
|
+
# · distinct contexts past HUB(N) → saturated
|
|
3650
|
+
# · below √N, every read IS the full set → exact
|
|
3651
|
+
# fall back to lower hits if orphaned and not saturated
|
|
3652
|
+
if reach.saturated: abstain
|
|
3653
|
+
mutual ≔ min(1, score · ratio) · min(1, score / ratio) # §17.3
|
|
3654
|
+
where ratio ≔ √( max(1, contentLen(anchor, region.len·D))
|
|
3655
|
+
/ max(1, region.len) )
|
|
3656
|
+
# contentLen capped at region.len·D — beyond that the
|
|
3657
|
+
# mutual weight approaches zero and the full walk is waste
|
|
3658
|
+
w ≔ mutual · ln(N / reach.contexts) / |reach.roots|
|
|
3659
|
+
vote w for each root (a terminal answer root redistributes its
|
|
3660
|
+
vote over prevFirst(root, HUB(N)) of the contexts that lead
|
|
3661
|
+
to it — capped at the store level, never materialised)
|
|
3662
|
+
# cross-region: any two regions (at least one strong voter) pair to
|
|
3663
|
+
# recover joint contexts their independent climbs missed (junction ascent).
|
|
3664
|
+
# Corpus-independent: known but non-voting regions may serve as weak side.
|
|
3665
|
+
# Order-free, n-ary, with self-evidence guard and explaining away.
|
|
3666
|
+
cross ≔ []
|
|
3667
|
+
superseded ≔ ∅
|
|
3668
|
+
seedsOf(ri) ≔ junctionSeeds(ctx, query[regions[ri].start..regions[ri].end])
|
|
3669
|
+
# precomputed once per candidate, reused across all its pairs
|
|
3670
|
+
consumed ≔ ∅
|
|
3671
|
+
for each pair (a, b) of eligible candidates (non-overlapping,
|
|
3672
|
+
at least one strong voter, not both covered by one known region,
|
|
3673
|
+
≤ k total probes, skipping consumed):
|
|
3674
|
+
containers ≔ junctionContainersFrom(left, right, cap,
|
|
3675
|
+
seedsOf(a), seedsOf(b), undefined, unordered = true)
|
|
3676
|
+
if containers = ∅:
|
|
3677
|
+
containers ≔ junctionSynonyms(left, right, maxInterior,
|
|
3678
|
+
unordered = true)
|
|
3679
|
+
if containers ≠ ∅:
|
|
3680
|
+
best ≔ the container covering the MOST remaining candidates
|
|
3681
|
+
(cached reads, never an extra walk); ties → shortest
|
|
3682
|
+
interior → lowest id
|
|
3683
|
+
if best's joined occurrence is a query substring: continue
|
|
3684
|
+
reach ≔ edgeAncestors(best.id, HUB(N))
|
|
3685
|
+
if not saturated and idf > 0:
|
|
3686
|
+
w ≔ mutual · ln(N / reach.contexts) / |reach.roots|
|
|
3687
|
+
cross.push(vote for best.id's roots at weight w,
|
|
3688
|
+
span covering all composed candidates)
|
|
3689
|
+
consumed.add(a); consumed.add(b); consumed.add(all extras)
|
|
3690
|
+
# Explaining away: supersede individual votes whose bytes the
|
|
3691
|
+
# container literally contains and whose roots are disjoint
|
|
3692
|
+
for each individual vote rv:
|
|
3693
|
+
if rv.roots shares any root with reach.roots: keep
|
|
3694
|
+
if containerBytes literally contains rv's query bytes:
|
|
3695
|
+
superseded.add(rv)
|
|
3696
|
+
break # a is consumed
|
|
3697
|
+
pooled ≔ lightestDerivation in the (+,+) semiring over the union
|
|
3698
|
+
of the independent votes (minus superseded) and cross # §5.3
|
|
3699
|
+
ranked ≔ anchors by pooled vote, descending
|
|
3700
|
+
cut ≔ steepest ratio drop in the sorted focus votes (natural break)
|
|
3701
|
+
roots ≔ [ranked[0]] ∪ { further non-overlapping anchors past any
|
|
3702
|
+
leading saturated stretch whose vote ≥ max(cut, FLOOR(N)) }
|
|
3703
|
+
return (roots, ranked)
|
|
3704
|
+
```
|
|
3705
|
+
|
|
3706
|
+
### 27.6 Recall, reasoning, fusion (§21–23)
|
|
3707
|
+
|
|
3708
|
+
```
|
|
3709
|
+
recallByResonance(query):
|
|
3710
|
+
whole_ ≔ [[0, query.length]] # identity-grade match: the whole query
|
|
3711
|
+
nothing ≔ [] # reverse readings/echoes: nothing matched
|
|
3712
|
+
q ≔ resolve(query)
|
|
3713
|
+
if q ≠ ∅: # tier 0
|
|
3714
|
+
g ≔ reverseContext(q, guide)
|
|
3715
|
+
if g ≠ ∅: return { bytes: g, accounted: nothing, moves: STEP }
|
|
3716
|
+
hits ≔ contentIndex.nearest(gistOf(query), k)
|
|
3717
|
+
if hits = ∅: return ∅
|
|
3718
|
+
if hits[0].score ≥ MERGE: # tier 1
|
|
3719
|
+
for h in hits:
|
|
3720
|
+
g ≔ project(h, guide) # forward first
|
|
3721
|
+
if g ≠ ∅: return { bytes: g, accounted: whole_, moves: STEP }
|
|
3722
|
+
# all reverse — accounted: nothing (no forward rule)
|
|
3723
|
+
g ≔ reverseContext(hits[0], guide)
|
|
3724
|
+
if g ≠ ∅: return { bytes: g, accounted: nothing, moves: STEP }
|
|
3725
|
+
if hits[0].score ≥ SIG: # tier 2
|
|
3726
|
+
forest ≔ climbAttention(query).roots
|
|
3727
|
+
if forest[0].vote ≥ FLOOR(N):
|
|
3728
|
+
g ≔ project(forest[0].anchor, guide)
|
|
3729
|
+
if g ≠ ∅: return { bytes: g,
|
|
3730
|
+
accounted: [[forest[0].start, forest[0].end]],
|
|
3731
|
+
moves: CONCEPT } # the climb
|
|
3732
|
+
for h in hits: # tier 3
|
|
3733
|
+
g ≔ project(h, guide)
|
|
3734
|
+
if g ≠ ∅ and fracOfQuery(resonance(gistOf(query), gistOf(g)),
|
|
3735
|
+
contentLen(g), query.length) ≥ REACH:
|
|
3736
|
+
return { bytes: g, accounted: nothing, moves: STEP }
|
|
3737
|
+
if fracOfQuery(hits[0].score, contentLen(hits[0]), query.length) < REACH:
|
|
3738
|
+
return ∅ # silence
|
|
3739
|
+
return { bytes: read(hits[0]), accounted: nothing,
|
|
3740
|
+
moves: 0, echoed: true } # honest echo
|
|
3741
|
+
|
|
3742
|
+
reason(query, answer, consumed₀): # §22
|
|
3743
|
+
q ≔ resolve(query)
|
|
3744
|
+
if q ≠ ∅ and prevCount(q) > 0: return answer # echo guard
|
|
3745
|
+
consumed ≔ consumed₀
|
|
3746
|
+
# synonym expansion — CAPPED at the hub bound: a common continuation's
|
|
3747
|
+
# reverse fan-in is corpus-sized, and no per-hop operation may grow
|
|
3748
|
+
# with the corpus (the same visibility trade chooseNext documents)
|
|
3749
|
+
for id in consumed₀:
|
|
3750
|
+
for sib in haloSiblings(id): # unified enumeration,
|
|
3751
|
+
consumeNode(sib.id) # above CONCEPT_BAR
|
|
3752
|
+
cur ≔ answer
|
|
3753
|
+
repeat up to K times:
|
|
3754
|
+
c ≔ resolve(cur); consumeNode(c)
|
|
3755
|
+
if c ≠ ∅ and nextFirst(c, hubBound).some(n ∉ consumed):
|
|
3756
|
+
fwd ≔ follow(c, guide) # forward absorb
|
|
3757
|
+
if fwd ≠ ∅ and fwd ≠ cur and resolve(fwd) ∉ consumed:
|
|
3758
|
+
consumeAll(c); cur ≔ fwd; continue
|
|
3759
|
+
consumeAll(c)
|
|
3760
|
+
pivot ≔ pivotInto(cur, consumed) # below
|
|
3761
|
+
if pivot = ∅: break
|
|
3762
|
+
fc ≔ follow(pivot, guide); consumeAll(pivot)
|
|
3763
|
+
if fc = ∅ or fc = cur: break
|
|
3764
|
+
cur ≔ fc
|
|
3765
|
+
return cur
|
|
3766
|
+
|
|
3767
|
+
# consume-set expansion: prevs and nexts are capped at hubBound —
|
|
3768
|
+
# a node suppressed only by a beyond-cap neighbour may still fire,
|
|
3769
|
+
# the same visibility trade the disambiguators make.
|
|
3770
|
+
|
|
3771
|
+
pivotInto(answer, consumed): # §22 — the stepping stone
|
|
3772
|
+
tree ≔ perceive(answer) # ONE perception, shared by the
|
|
3773
|
+
# probe budget and the walk
|
|
3774
|
+
candidates ≔ ∅
|
|
3775
|
+
for each branch node b of tree, breadth-first,
|
|
3776
|
+
at most min(number of branch nodes, k) probes:
|
|
3777
|
+
for hit in contentIndex.nearest(b.gist, k):
|
|
3778
|
+
if hit ∉ consumed and hasNext(hit): candidates += hit
|
|
3779
|
+
for site in recognise(answer).sites: # exact beats
|
|
3780
|
+
if site ∉ consumed and hasNext(site): # approximate
|
|
3781
|
+
candidates += site (full confidence)
|
|
3782
|
+
return the candidate whose bytes `answer` literally CONTAINS,
|
|
3783
|
+
longest such span wins; ∅ if none # resonance proposes,
|
|
3784
|
+
# bytes confirm
|
|
3785
|
+
|
|
3786
|
+
fuseAttention(query, primary): # §23
|
|
3787
|
+
if primary is strictly contained in query: return primary
|
|
3788
|
+
roots ≔ climbAttention(query).roots
|
|
3789
|
+
if |roots| ≤ 1: return primary
|
|
3790
|
+
qv ≔ guide (the response guide, already computed — once, not per root)
|
|
3791
|
+
pieces ≔ [primary] ∪ [ project(r.anchor, qv) for r in roots[1:] ,
|
|
3792
|
+
dropping ∅ and duplicates ]
|
|
3793
|
+
sort pieces by their supporting query span
|
|
3794
|
+
out ≔ pieces[0]
|
|
3795
|
+
for p in pieces[1:]:
|
|
3796
|
+
out ≔ joinWithBridge(out, p) # learned connector when one exists;
|
|
3797
|
+
# bare join + bridgeMiss trace step
|
|
3798
|
+
# otherwise — degradation is never
|
|
3799
|
+
# silent (§19.5, §23)
|
|
3800
|
+
return out
|
|
3801
|
+
|
|
3802
|
+
bridge(left, right): # §19.5 — the graded junction ladder
|
|
3803
|
+
# Tier 1 — junction containers, by content-addressed identity:
|
|
3804
|
+
# ascend parents + containment links from resolve(left)/resolve(right)
|
|
3805
|
+
# (or their canonical-window ids), √N-disciplined; collect ancestors
|
|
3806
|
+
# whose bytes contain left then right.
|
|
3807
|
+
cands ≔ junctionContainers(left, right)
|
|
3808
|
+
# Tier 2 — edge junctions: a continuation of left containing right
|
|
3809
|
+
# (glue = its prefix), or a context of right containing left (glue =
|
|
3810
|
+
# its suffix). An empty interior is a CONFIRMED adjacency, not a miss.
|
|
3811
|
+
if cands = ∅: cands ≔ junctionEdges(left, right)
|
|
3812
|
+
# Tier 2.5 — synonym junctions: tiers 1 + 2 applied to halo siblings
|
|
3813
|
+
if cands = ∅: cands ≔ junctionSynonyms(left, right)
|
|
3814
|
+
if cands ≠ ∅:
|
|
3815
|
+
# guide resonance picks; ties → shortest interior → lowest id
|
|
3816
|
+
return pick(cands, guide).interior
|
|
3817
|
+
# Tier 3 — the resonance fallback (last resort):
|
|
3818
|
+
for hit in contentIndex.nearest(gistOf(left ⧺ right), 2k), nearest first:
|
|
3819
|
+
f ≔ read(hit)
|
|
3820
|
+
if f contains left at position i, and right at position j > i+|left|:
|
|
3821
|
+
return f[i+|left| … j] # the bytes the corpus puts between
|
|
3822
|
+
return ∅
|
|
3823
|
+
|
|
3824
|
+
joinWithBridge(left, right): # the ONE out-of-search assembly step
|
|
3825
|
+
link ≔ bridge(left, right)
|
|
3826
|
+
if link = ∅: emit bridgeMiss trace; return left ⧺ right
|
|
3827
|
+
return left ⧺ link ⧺ right
|
|
3828
|
+
|
|
3829
|
+
# ── the projection family (§14.4) — shared by every mechanism above ──
|
|
3830
|
+
|
|
3831
|
+
follow(node, guide): # FORWARD: the continuation fixpoint
|
|
3832
|
+
nxt ≔ chooseNext(node, guide)
|
|
3833
|
+
if nxt = ∅:
|
|
3834
|
+
nxt ≔ conceptHop(node) # first hop may cross a synonym:
|
|
3835
|
+
# the first halo sibling above
|
|
3836
|
+
# CONCEPT_BAR that has an edge
|
|
3837
|
+
if nxt = ∅: return ∅
|
|
3838
|
+
walk chooseNext from nxt until revisit or dead end (cycle-guarded)
|
|
3839
|
+
return read(final node)
|
|
3840
|
+
|
|
3841
|
+
reverseContext(node, guide, rev?): # REVERSE: the establishing context
|
|
3842
|
+
candidates ≔ rev ?? prevFirst(node, hubBound) # CAPPED at √N: a common
|
|
3843
|
+
# continuation's reverse fan-in
|
|
3844
|
+
# is corpus-sized; prevFirst
|
|
3845
|
+
# reads only the first √N
|
|
3846
|
+
if candidates = ∅: return ∅
|
|
3847
|
+
pick ≔ |candidates| = 1 ? candidates[0] # skip needless gisting
|
|
3848
|
+
: guide ≠ ∅ ? chooseAmong(candidates, guide)
|
|
3849
|
+
: argmax haloMass over candidates (already capped)
|
|
3850
|
+
g ≔ read(pick)
|
|
3851
|
+
return |g| > 0 ? g : ∅ # empty bytes are no grounding
|
|
3852
|
+
|
|
3853
|
+
project(node, guide): # BOTH: the universal grounding step
|
|
3854
|
+
return follow(node, guide) ?? reverseContext(node, guide)
|
|
3855
|
+
|
|
3856
|
+
# ── the disambiguators + the one fan-out convention (§25, §8.8) ──
|
|
3857
|
+
|
|
3858
|
+
corpusN ≔ max(2, edgeSourceCount) # floored at 2 so ln N and √N stay
|
|
3859
|
+
hubBound ≔ ⌈√corpusN⌉ # meaningful on a near-empty store
|
|
3860
|
+
|
|
3861
|
+
hubCap(ids): # THE fan-out cap, defined once
|
|
3862
|
+
return the first hubBound of ids (insertion order); no copy when under
|
|
3863
|
+
|
|
3864
|
+
guidedFirst(node): # guided-or-first, for answer-shaped reads
|
|
3865
|
+
return chooseNext(node, guide) ?? nextFirst(node, 1)[0]
|
|
3866
|
+
# LIMIT 1 read when no guide is in flight
|
|
3867
|
+
|
|
3868
|
+
chooseNext(node, guide): # §25, forward regime
|
|
3869
|
+
nx ≔ nextFirst(node, hubBound) # only the first √N continuations
|
|
3870
|
+
# are ever candidates — a hub context's
|
|
3871
|
+
# full fan-out is never materialised
|
|
3872
|
+
if |nx| ≤ 1 or guide = ∅: return nx[0]
|
|
3873
|
+
among nx (already capped):
|
|
3874
|
+
maximize ( prevCount(candidate) , haloMass(candidate) ) # indexed,
|
|
3875
|
+
ties → first inserted # never materialised
|
|
3876
|
+
|
|
3877
|
+
chooseAmong(candidates, guide): # §25, reverse regime
|
|
3878
|
+
among candidates (already capped by the caller):
|
|
3879
|
+
maximize resonance(guide, gistOf(read(candidate)))
|
|
3880
|
+
return the winner
|
|
3881
|
+
```
|
|
3882
|
+
|
|
3883
|
+
### 27.7 Counterfactual transfer (grounding I, §18)
|
|
3884
|
+
|
|
3885
|
+
```
|
|
3886
|
+
counterfactualTransfer(query, sites, roots, ranked):
|
|
3887
|
+
# castFloor already checked |query|<2W, N=0, |ranked|<2 — these
|
|
3888
|
+
# gates are checked once in the floor, not duplicated here.
|
|
3889
|
+
# If roots/ranked not given (standalone call), compute the climb.
|
|
3890
|
+
|
|
3891
|
+
# ── graded alignment (literal → halo) ────────────────────────────
|
|
3892
|
+
MIN_WEAVE ≔ 2; points ≔ ∅; depth ≔ Float64Array(|query|)
|
|
3893
|
+
for cand in the first 2k of ranked:
|
|
3894
|
+
... (alignment as before) ...
|
|
3895
|
+
if |points| < 2: return []
|
|
3896
|
+
# frame gate (weave-local): frame(i) ⇔ depth[i] > MIN_WEAVE ∧ dominates(depth[i], |points|)
|
|
3897
|
+
dominant ≔ points[0]; require dominant ∈ roots
|
|
3898
|
+
require some run outside every recognised site
|
|
3899
|
+
|
|
3900
|
+
results ≔ [] # multi-candidate: each schema records independently
|
|
3901
|
+
runSpans(p) ≔ p's free runs as [qs, qe] pairs
|
|
3902
|
+
|
|
3903
|
+
# ── 1. substitution ──────────────────────────────────────────────
|
|
3904
|
+
... (same detection logic) ...
|
|
3905
|
+
if found:
|
|
3906
|
+
record({ bytes: joinWithBridge(filler, tail) + follow(p.anchor),
|
|
3907
|
+
used: {before, p}, moves: STEP+STEP,
|
|
3908
|
+
accounted: runSpans(before) ∪ runSpans(p),
|
|
3909
|
+
unexplained: query bytes not in those runs })
|
|
3910
|
+
|
|
3911
|
+
# ── 2. redirection ───────────────────────────────────────────────
|
|
3912
|
+
... (same detection logic) ...
|
|
3913
|
+
if found:
|
|
3914
|
+
record({ bytes: g, used: {dominant, last}, moves: STEP,
|
|
3915
|
+
accounted: runSpans(dominant) ∪ runSpans(last),
|
|
3916
|
+
unexplained: query bytes not in those runs })
|
|
3917
|
+
|
|
3918
|
+
# ── 3. comparison ────────────────────────────────────────────────
|
|
3919
|
+
... (same detection logic) ...
|
|
3920
|
+
if found:
|
|
3921
|
+
record({ bytes: joinWithBridge(a, b),
|
|
3922
|
+
used: {dominant, bestAnalog},
|
|
3923
|
+
moves: CONCEPT+STEP+STEP,
|
|
3924
|
+
accounted: runSpans(dominant) ∪ (bestAnalog.point ? runSpans(bestAnalog.point) : []),
|
|
3925
|
+
unexplained: query bytes not in those runs })
|
|
3926
|
+
|
|
3927
|
+
return results # possibly empty — the decider weighs whatever fired
|
|
3928
|
+
```
|
|
3929
|
+
|
|
3930
|
+
### 27.7a Confluence join (§18.5)
|
|
3931
|
+
|
|
3932
|
+
```
|
|
3933
|
+
confluenceJoin(query):
|
|
3934
|
+
if |query| < 2W or N = 0: return ∅
|
|
3935
|
+
(roots, ranked) ≔ climbAttention(query, 2k)
|
|
3936
|
+
if |ranked| < 2: return ∅
|
|
3937
|
+
|
|
3938
|
+
queryWin ≔ windowIds(query) # offset → id, canonical W-window read
|
|
3939
|
+
queryIds ≔ set of queryWin values
|
|
3940
|
+
N ≔ corpusN
|
|
3941
|
+
|
|
3942
|
+
# ── constraint streams ────────────────────────────────────────────
|
|
3943
|
+
streams ≔ ∅
|
|
3944
|
+
for cand in ranked (capped at 2k):
|
|
3945
|
+
ids ≔ set of windowIds(read(cand.anchor)).values
|
|
3946
|
+
cover ≔ ∅ # query spans this anchor holds DISCRIMINATIVE windows of
|
|
3947
|
+
held ≔ ∅ # query spans this anchor holds AT ALL (scaffolding included)
|
|
3948
|
+
for (off, wid) in queryWin where ids has wid:
|
|
3949
|
+
merge off into held
|
|
3950
|
+
if not dominates(reachOf(wid, N), N): # scaffolding never binds
|
|
3951
|
+
merge off into cover
|
|
3952
|
+
if cover ≠ ∅: streams += (cand.anchor, cand.vote, ids, cover, held)
|
|
3953
|
+
if |streams| < 2: return ∅
|
|
3954
|
+
|
|
3955
|
+
# ── find the MEET of two independent streams ─────────────────────
|
|
3956
|
+
disjoint(a, b) ≔ a.cover and b.cover share no query byte
|
|
3957
|
+
for each pair (a, b) where disjoint(a, b):
|
|
3958
|
+
wa ≔ windowIds(read(a.anchor))
|
|
3959
|
+
# window ids in BOTH anchors, ABSENT from the query — merged into
|
|
3960
|
+
# maximal contiguous spans (overlapping windows weave one span)
|
|
3961
|
+
for each contiguous span [s, e) of offsets where
|
|
3962
|
+
wa[off] ∈ b.ids and wa[off] ∉ queryIds:
|
|
3963
|
+
# scaffolding gate: the span's most DISCRIMINATIVE window decides
|
|
3964
|
+
reach ≔ min reachOf(wa[off], N) for each window in [s, e)
|
|
3965
|
+
if not isFinite(reach) or dominates(reach, N): continue
|
|
3966
|
+
# feasible — the entity where the constraints meet
|
|
3967
|
+
met ≔ the one with smallest reach, longest span (tie-break)
|
|
3968
|
+
if met = ∅: return ∅
|
|
3969
|
+
|
|
3970
|
+
return { bytes: read(a.anchor).subarray(met.s, met.e),
|
|
3971
|
+
used: {a.anchor, b.anchor},
|
|
3972
|
+
accounted: a.held ∪ b.held, # ALL matched content
|
|
3973
|
+
moves: 3·STEP } # two matches + one meet
|
|
3974
|
+
```
|
|
3975
|
+
|
|
3976
|
+
### 27.8 Extraction and articulation (§20, §24)
|
|
3977
|
+
|
|
3978
|
+
```
|
|
3979
|
+
extractBySkill(query):
|
|
3980
|
+
ranked ≔ climbAttention(query, 2k).ranked
|
|
3981
|
+
for cand in ranked: # first span-shaped wins
|
|
3982
|
+
ex ≔ skillExemplar(cand.anchor):
|
|
3983
|
+
if hasNext(anchor):
|
|
3984
|
+
(context, answer) ≔ (read(anchor), follow(anchor, guide))
|
|
3985
|
+
else:
|
|
3986
|
+
answer ≔ read(anchor)
|
|
3987
|
+
context ≔ the longest span-shaped context among
|
|
3988
|
+
prevFirst(anchor, hubBound); chooseAmong
|
|
3989
|
+
(the reverse-regime disambiguator) breaks
|
|
3990
|
+
length ties via query-gist resonance
|
|
3991
|
+
require answer is a sparse subsequence of context
|
|
3992
|
+
# OPEN reading: in-order, arbitrary gaps (§20.1)
|
|
3993
|
+
# the subsequent DECOMPOSITION step uses a STRONGER
|
|
3994
|
+
# greedy-longest-run reading; an accepted exemplar
|
|
3995
|
+
# can still fail to decompose → extraction returns ∅
|
|
3996
|
+
if ex ≠ ∅: break
|
|
3997
|
+
if no exemplar: return ∅
|
|
3998
|
+
runs ≔ the answer's pieces, decomposed by greedy longest-run
|
|
3999
|
+
matching inside the context (the STRONG reading); contiguous
|
|
4000
|
+
adjacent runs merged
|
|
4001
|
+
accounted ≔ ∅
|
|
4002
|
+
for each run, with isLast flags:
|
|
4003
|
+
pre ≔ up to W context bytes before the run
|
|
4004
|
+
post ≔ up to W after (or the NEXT run's pre)
|
|
4005
|
+
locate pre then post in the query via locate() — the graded
|
|
4006
|
+
matcher ladder of §14.4:
|
|
4007
|
+
1. exact bytes 2. halo-role match via bestHaloMate above
|
|
4008
|
+
CONCEPT_BAR 3. gist match against query segments above MERGE
|
|
4009
|
+
accounted += the located pre/post frames in the query
|
|
4010
|
+
piece ≔ the query bytes between the located frames
|
|
4011
|
+
# bounded on BOTH sides ⇒ the read span itself is explained
|
|
4012
|
+
if pre-located and post-located: accounted += piece's span
|
|
4013
|
+
return { bytes: concatenation of the pieces,
|
|
4014
|
+
accounted } (∅ if none located)
|
|
4015
|
+
moves ≔ CONCEPT + STEP · |accounted|
|
|
4016
|
+
|
|
4017
|
+
articulate(answer, query):
|
|
4018
|
+
voices ≔ recognised multi-byte forms of the QUERY that bear halos
|
|
4019
|
+
if voices = ∅ or they cover none of the query: return answer
|
|
4020
|
+
subs ≔ ∅
|
|
4021
|
+
for each recognised multi-byte, halo-bearing form f of the ANSWER:
|
|
4022
|
+
v ≔ argmax over voices of cosine(halo(f), halo(v)), ≥ CONCEPT_BAR
|
|
4023
|
+
if v exists and v ≠ f and f is not a fragment of v's own subtree:
|
|
4024
|
+
subs[f] ≔ v.bytes
|
|
4025
|
+
if subs = ∅: return answer
|
|
4026
|
+
solved ≔ the cover search over the ANSWER with subs as the only form
|
|
4027
|
+
rules (each voiced form emits its substitute at cost 0)
|
|
4028
|
+
return solved ? solved.segs composed : answer # unchanged if no cover
|
|
4029
|
+
```
|
|
4030
|
+
|
|
4031
|
+
### 27.9 A worked example, end to end
|
|
4032
|
+
|
|
4033
|
+
The README's demo, traced through the pipeline. Deposits:
|
|
4034
|
+
|
|
4035
|
+
```
|
|
4036
|
+
("The Mona Lisa was painted by Leonardo da Vinci.", "Leonardo da Vinci")
|
|
4037
|
+
("The Starry Night was painted by Vincent van Gogh.", "Vincent van Gogh")
|
|
4038
|
+
("The Night Watch was painted by Rembrandt van Rijn.","Rembrandt van Rijn")
|
|
4039
|
+
("Pablo Picasso", "Pablo Picasso co-founded the Cubist movement")
|
|
4040
|
+
```
|
|
4041
|
+
|
|
4042
|
+
Each pair interns both sides (sharing every repeated span: "was painted by" is
|
|
4043
|
+
one set of nodes across all three sentences), records one continuation edge, and
|
|
4044
|
+
pours halos both ways. The three painter names, having each appeared as an
|
|
4045
|
+
answer following a painting-frame, acquire similar halos; "was painted by …"
|
|
4046
|
+
spans become shared, many-parent interior structure.
|
|
4047
|
+
|
|
4048
|
+
Query: `"The Weeping Woman was painted by Pablo Picasso."`
|
|
4049
|
+
|
|
4050
|
+
1. **Recognise (§15).** Sites include "was painted by" material (shared interior
|
|
4051
|
+
forms), " Pablo Picasso" (a learned context — it has an edge to the Cubism
|
|
4052
|
+
fact), and assorted chunks. "The Weeping Woman" resolves to nothing: never
|
|
4053
|
+
seen.
|
|
4054
|
+
2. **Compute (§16).** No extension claims any span.
|
|
4055
|
+
3. **Grounding decider (§14.1).** Every self-gating mechanism weighs in:
|
|
4056
|
+
- **CAST (§18):** The climb ranks the three painting exemplars and the
|
|
4057
|
+
Picasso context; alignment finds runs, but no substitution seat or
|
|
4058
|
+
redirection shape fits — CAST yields no candidate.
|
|
4059
|
+
- **Confluence (§18.5):** Only one constraint stream (the query asks about
|
|
4060
|
+
one painting, not a conjunction of independent properties) — returns null.
|
|
4061
|
+
- **Cover (§19):** The recognised forms do not compose a cover that lifts an
|
|
4062
|
+
answer clear of the framing (the unseen painting title blocks a clean
|
|
4063
|
+
composition) — returns null. _(On other seeds/corpora this query can also
|
|
4064
|
+
ground via cover; the strategies are redundant by design, and provenance
|
|
4065
|
+
records which one fired.)_
|
|
4066
|
+
- **Extraction (§20):** The climb's ranked anchors include the exemplar "The
|
|
4067
|
+
Mona Lisa was painted by Leonardo da Vinci." — span-shaped. Its frames are
|
|
4068
|
+
located in the query; the analogous span reads out **"Pablo Picasso"**. The
|
|
4069
|
+
candidate's weight: CONCEPT (one skill analogy) + STEP per located frame +
|
|
4070
|
+
PASS per unexplained byte. It is the lightest grounding derivation.
|
|
4071
|
+
- **Recall (§21):** Its best candidate carries the full PASS·|query| plus a
|
|
4072
|
+
STEP — heavier.
|
|
4073
|
+
- **Decider:** Extraction wins — lightest grounding derivation.
|
|
4074
|
+
4. **Reason (§22).** "Pablo Picasso" resolves — and it is a learned context with
|
|
4075
|
+
an unconsumed continuation. Forward absorb follows the edge: **"Pablo Picasso
|
|
4076
|
+
co-founded the Cubist movement"**. The next iteration finds no unconsumed
|
|
4077
|
+
pivot; the chain fixes.
|
|
4078
|
+
5. **Fuse / articulate (§23–24).** One point of attention grounded from the
|
|
4079
|
+
query's text; no halo-sibling substitutions apply. The answer stands.
|
|
4080
|
+
|
|
4081
|
+
Provenance: `extract`. Every step above is present, with spans, node ids, costs,
|
|
4082
|
+
and data-flow edges, in the rationale when one is requested.
|
|
4083
|
+
|
|
4084
|
+
The second demo query, `"a museum charges 12*4 for a family ticket"`: the ALU
|
|
4085
|
+
claims the span `12*4` with result bytes `48`; recognition's sites overlapping
|
|
4086
|
+
that span are masked; the cover search bridges the literal framing (PASS) and
|
|
4087
|
+
the computed span (recognised, STEP + ε), and lifting drops the framing:
|
|
4088
|
+
**"48"**, provenance `cover`.
|
|
4089
|
+
|
|
4090
|
+
### 27.10 Determinism, stated as an invariant
|
|
4091
|
+
|
|
4092
|
+
Every function above is deterministic given (seed, store contents): the alphabet
|
|
4093
|
+
and keyring are seeded; perception is positional; interning is
|
|
4094
|
+
content-addressed; the deduction engine breaks ties by fixed conventions;
|
|
4095
|
+
disambiguation bottoms out in corpus-determined orderings; the ANN index is
|
|
4096
|
+
deterministic for a fixed build. Hence: **same seed + same deposits (in order) +
|
|
4097
|
+
same query ⇒ byte-identical answer.** The only approximation in the system — ANN
|
|
4098
|
+
ranking — affects which _candidates are proposed_, never what any accepted
|
|
4099
|
+
answer _asserts_, and it too is deterministic run to run.
|
|
4100
|
+
|
|
4101
|
+
---
|
|
4102
|
+
|
|
4103
|
+
---
|
|
4104
|
+
|
|
4105
|
+
# Part VI — Reference
|
|
4106
|
+
|
|
4107
|
+
## 28. Glossary
|
|
4108
|
+
|
|
4109
|
+
The one-line inventory of §9 doubles as the glossary; this section adds only the
|
|
4110
|
+
terms of art borrowed from the literature.
|
|
4111
|
+
|
|
4112
|
+
- **A\*LD** — A\* Lightest Derivation: A\* generalized from shortest paths to
|
|
4113
|
+
weighted deduction (Felzenszwalb & McAllester 2007). §5.2.
|
|
4114
|
+
- **Binding / superposition** — the two VSA combination operators:
|
|
4115
|
+
association-forming (order-visible) vs. set-forming (similarity- preserving).
|
|
4116
|
+
§2.1.
|
|
4117
|
+
- **Company signature** — a deterministic unit vector derived from a node's
|
|
4118
|
+
identity (seeded by id), used as the halo-pour unit. Decouples content
|
|
4119
|
+
similarity from distributional similarity. §4, §12.2.
|
|
4120
|
+
- **Concentration of measure** — the high-dimensional phenomenon making random
|
|
4121
|
+
vectors quasi-orthogonal; the statistical basis of every threshold. §2.2.
|
|
4122
|
+
- **Content-addressable memory** — retrieval by content, not location. §3.1.
|
|
4123
|
+
- **Distributional hypothesis** — meaning ≈ distribution of use (Harris 1954).
|
|
4124
|
+
§4.
|
|
4125
|
+
- **Hash-consing** — constructing structures modulo equality so equal
|
|
4126
|
+
substructures are shared. §3.2.
|
|
4127
|
+
- **HNSW** — Hierarchical Navigable Small World graph; sub-linear ANN search
|
|
4128
|
+
(Malkov & Yashunin 2018). §6.1.
|
|
4129
|
+
- **Hilbert curve** — the locality-preserving space-filling curve used to
|
|
4130
|
+
linearize grids. §6.3.
|
|
4131
|
+
- **Hyperdimensional computing** — Kanerva's (2009) umbrella term for computing
|
|
4132
|
+
with high-dimensional random vectors; synonym of VSA as used here. §2.
|
|
4133
|
+
- **IDF** — inverse document frequency; the specificity weighting of the
|
|
4134
|
+
consensus climb (Spärck Jones 1972). §17.3.
|
|
4135
|
+
- **Instance-based learning** — generalization at query time from stored
|
|
4136
|
+
instances. §1.1.
|
|
4137
|
+
- **Merkle DAG** — a graph whose node identities derive from content (Merkle
|
|
4138
|
+
1987). §3.2.
|
|
4139
|
+
- **Non-parametric** — model capacity residing in the data, not a fixed
|
|
4140
|
+
parameter vector. §1.1.
|
|
4141
|
+
- **RaBitQ** — 1-bit quantization with an unbiased similarity estimator (Gao &
|
|
4142
|
+
Long 2024). §6.1.
|
|
4143
|
+
- **Semiring-weighted deduction** — the algebraic generalization of weighted
|
|
4144
|
+
inference (Goodman 1999); Sema uses tropical (min,+) for structure and
|
|
4145
|
+
arithmetic (+,+) for evidence pooling. §5.
|
|
4146
|
+
- **Tropical semiring** — (min, +): the algebra of shortest paths and lightest
|
|
4147
|
+
derivations. §5.3.
|
|
4148
|
+
- **VSA** — Vector Symbolic Architecture (Plate 1995; Gayler 2003). §2.
|
|
4149
|
+
|
|
4150
|
+
## 29. Complexity summary
|
|
4151
|
+
|
|
4152
|
+
n = input/query length; D = dimension; W = fold window; N = learned contexts; k
|
|
4153
|
+
= retrieval breadth. All store lookups are content-addressed O(1) (amortized);
|
|
4154
|
+
all index queries are sub-linear in the collection (empirically ≈ N^0.32
|
|
4155
|
+
distance computations).
|
|
4156
|
+
|
|
4157
|
+
| Operation | Cost | Where |
|
|
4158
|
+
| :------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------- |
|
|
4159
|
+
| Perceive | O(n·D) vector work; O(n) nodes | §10 |
|
|
4160
|
+
| Deposit (intern + windows) | O(n) content-addressed probes | §11 |
|
|
4161
|
+
| Learn a pair | O(1) edge + O(changed) halo pours | §12 |
|
|
4162
|
+
| Recognise | O(n·W) bounded probes | §15 |
|
|
4163
|
+
| Consensus climb | O(regions · k) index queries + expand-until-decided: work bounded by √N per region regardless of corpus size (LIMITed store reads, indexed existence probes) | §17 |
|
|
4164
|
+
| Cover search | output-sensitive A\*LD: proportional to the lightest derivation, not the corpus (§5.2); the dominant per-query index cost is connector pre-resolution, O(sites) queries | §19 |
|
|
4165
|
+
| Recall | O(k) index probes + graded structural checks | §21 |
|
|
4166
|
+
| Reasoning | ≤ K hops, each bounded by the answer's subtree | §22 |
|
|
4167
|
+
| Storage | O(distinct subtrees); vector index over resonance targets only, 1-bit codes (32× compression) | §3, §12.3 |
|
|
4168
|
+
|
|
4169
|
+
Nothing on any per-query path scans the corpus; every fan-out is capped at the
|
|
4170
|
+
hub bound, and the cap is enforced at the _store level_ through LIMITed reads
|
|
4171
|
+
and indexed existence probes — no per-query read materialises a corpus-sized
|
|
4172
|
+
list. That — not hardware — is why the system runs on a CPU and why inference
|
|
4173
|
+
cost stays decoupled from corpus growth.
|
|
4174
|
+
|
|
4175
|
+
## 30. Bibliography
|
|
4176
|
+
|
|
4177
|
+
Foundations cited in this document, in alphabetical order:
|
|
4178
|
+
|
|
4179
|
+
- Felzenszwalb, P. F. & McAllester, D. (2007). _The Generalized A\*
|
|
4180
|
+
Architecture._ Journal of Artificial Intelligence Research 29, 153–190.
|
|
4181
|
+
- Gao, J. & Long, C. (2024). _RaBitQ: Quantizing High-Dimensional Vectors with a
|
|
4182
|
+
Theoretical Error Bound for Approximate Nearest Neighbor Search._ Proc. ACM
|
|
4183
|
+
SIGMOD.
|
|
4184
|
+
- Gayler, R. W. (2003). _Vector Symbolic Architectures Answer Jackendoff's
|
|
4185
|
+
Challenges for Cognitive Neuroscience._ Proc. ICCS/ASCS.
|
|
4186
|
+
- Goodman, J. (1999). _Semiring Parsing._ Computational Linguistics 25(4),
|
|
4187
|
+
573–605.
|
|
4188
|
+
- Harris, Z. S. (1954). _Distributional Structure._ Word 10(2–3), 146–162.
|
|
4189
|
+
- Kanerva, P. (2009). _Hyperdimensional Computing: An Introduction to Computing
|
|
4190
|
+
in Distributed Representation with High-Dimensional Random Vectors._ Cognitive
|
|
4191
|
+
Computation 1, 139–159.
|
|
4192
|
+
- Knuth, D. E. (1977). _A Generalization of Dijkstra's Algorithm._ Information
|
|
4193
|
+
Processing Letters 6(1), 1–5.
|
|
4194
|
+
- Kolodner, J. L. (1992). _An Introduction to Case-Based Reasoning._ Artificial
|
|
4195
|
+
Intelligence Review 6, 3–34.
|
|
4196
|
+
- Malkov, Y. A. & Yashunin, D. A. (2018). _Efficient and Robust Approximate
|
|
4197
|
+
Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs._ IEEE
|
|
4198
|
+
TPAMI 42(4), 824–836.
|
|
4199
|
+
- Merkle, R. C. (1987). _A Digital Signature Based on a Conventional Encryption
|
|
4200
|
+
Function._ Proc. CRYPTO.
|
|
4201
|
+
- Plate, T. A. (1995). _Holographic Reduced Representations._ IEEE Transactions
|
|
4202
|
+
on Neural Networks 6(3), 623–641.
|
|
4203
|
+
- Spärck Jones, K. (1972). _A Statistical Interpretation of Term Specificity and
|
|
4204
|
+
Its Application in Retrieval._ Journal of Documentation 28(1), 11–21.
|
|
4205
|
+
|
|
4206
|
+
---
|
|
4207
|
+
|
|
4208
|
+
_This document describes concepts and algorithms only. For the codebase —
|
|
4209
|
+
layout, build, tests, invariants, and how to extend the system — see
|
|
4210
|
+
[AGENTS.md](AGENTS.md)._
|