@jarenjs/db 0.43.3 → 0.46.4

package/ARCHITECTURE.md

@@ -153,6 +153,14 @@ member the schema types as an array or an object. Those are exact; the
 spatial predicates that only NARROW are rows in the residual table, and
 the implied-conjunct table below carries every proof.
 
+Plus one ORDERING a **`derive: 'vector'`** column makes cheap without
+making it native: the k-nearest composition — `$orderby` on a
+`$similarity` key, descending, `$empty: 'least'`, under a `$subsequence`
+window with a finite limit — over the member the column stores. It is
+never pushed as an `ORDER BY`: the column CUTS the candidate set and the
+engine DECIDES the order ("The k-nearest plan" below), which is a fourth
+mode, `knn`, beside native, row and set.
+
 ### The deliberate-residual table
 
 | construct | reason |
@@ -170,6 +178,9 @@ the implied-conjunct table below carries every proof.
 | a geohash prefix LONGER than the column's precision | a cell-range pre-filter over the derived column's precision is pushed; the longer prefix refines in the engine |
 | a spatial predicate over a member with no matching derived index, or one the schema does not type as an array or an object | nothing is proven; the whole predicate runs in the engine (the deterministic-function hatch may still take it) |
 | an unbounded `$distance` (`>= r`), a circle reaching a pole or crossing the antimeridian, a probe with no bounding box | no conservative box exists — pushing nothing is correct, pushing a wrong box is not |
+| the k-nearest ordering over a `derive: 'vector'` column | the column cuts the candidates; the engine orders them (mode `knn`, "The k-nearest plan" below) — engine work, so `strict: true` refuses it |
+| `$similarity` anywhere else — a threshold in `$where`, a score in `$return`, a second ordering key | no native spelling; runs in the residual over whatever the rest of the document pushed |
+| the k-nearest shape with no finite window, ascending, `$empty: 'greatest'`, a probe that is neither a literal vector nor an external, a literal probe of another width, or a selection not pushed whole (a residual conjunct, an implied one, a `$let` before the where) | nothing is proven; the whole document runs in the engine, and `explain()` names which precondition failed — a k-nearest query never falls to the full scan silently |
 
 ### The type truth table
 
@@ -273,17 +284,31 @@ member as an array or an object **and nothing else** — §8.14 answers
 
 | Jaren predicate | pushed | exact? | why it is implied |
 |---|---|---|---|
-| `$bbox-intersects(<path>, <literal\|external>)` | `w <= L_e AND e >= L_w AND s <= L_n AND n >= L_s` | **exact** | the derived columns ARE `B(row)`, so box overlap is fully decidable. `<=`/`>=`, not `<`/`>`: the kernel counts touching edges as intersecting, and a strict comparison would disagree on every shared edge |
-| `$within(<path>, <literal\|external>)` | the same four comparisons against `B(area)` | implied | the representative position is inside `B(subject)` — a bare position IS the box, and a centroid is a mean of positions, which lies within their min/max — and inside the area's surface implies inside `B(area)`; so the two boxes share at least that position ∎ |
+| `$bbox-intersects(<path>, <literal\|external>)`, `physical: 'columns'` | `w <= L_e AND e >= L_w AND s <= L_n AND n >= L_s` | **exact** | the derived columns ARE `B(row)`, so box overlap is fully decidable. `<=`/`>=`, not `<`/`>`: the kernel counts touching edges as intersecting, and a strict comparison would disagree on every shared edge |
+| the same, `physical: 'rtree'` | the same four comparisons, spelled as `rowid IN (SELECT id FROM <c>_rtree WHERE minx <= L_e AND maxx >= L_w AND miny <= L_n AND maxy >= L_s)` | implied | an R\*Tree stores coordinates as **32-bit floats rounded OUTWARD**, so what it holds is `B(row)` widened — a superset, by ~3 cm in x and ~84 cm in y at 52°N. A superset has no false negatives, which is all a pre-filter needs; it is not a decision, so the exact box test refines and `strict: true` is `JD0010` here where the column mapping was native |
+| `$within(<path>, <literal\|external>)` | the same four comparisons against `B(area)`, in whichever spelling the mapping takes | implied | the representative position is inside `B(subject)` — a bare position IS the box, and a centroid is a mean of positions, which lies within their min/max — and inside the area's surface implies inside `B(area)`; so the two boxes share at least that position ∎ |
 | `{$le\|$lt: [{$distance: [<path>, <literal>]}, r]}` | the same four comparisons against `circleBounds(probe, r)` | implied | every position within `r` metres lies inside the circle's box, which the kernel computes on the same sphere and the same `EARTH_RADIUS` the engine measures with — so the two cannot disagree by model. `$ge`/`$gt` is NOT promoted: no box narrows "farther than r" |
 | `{$starts-with: [{$geohash: [<path>, k]}, "<cell>"]}`, cell length ≤ k | `<c> IN ("<cell>")` or `<c> >= "<cell>" AND <c> < successor` | **exact** | the column HOLDS `$geohash(row, k)`, and geohash is a prefix code, so a prefix test on the expression is the same test on the column |
 | the same, cell length > k | the cell truncated to k | implied | the column can only confirm its own first k characters |
 | `{$exists: {$index-of: [{$geohash-neighbours: "<cell>"}, {$geohash: [<path>, k]}]}}`, cell length = k | `<c> IN (…the nine cells…)` | **exact** | the membership test compares whole strings and the column is exactly one of them. Nine cells, never one: two points ten metres apart can differ in the FIRST character of their cell (D7), so a single prefix is bucketing and only the neighbourhood is proximity |
 
+**The proof is a proof about BOXES, not about SQL**, so the physical
+mapping (MODEL-FORMAT §2.1, `physical`) does not enter it: the same box
+is computed either way and only its spelling changes. The `rtree`
+spelling is a `rowid` subquery — a conjunct on the collection table, so
+the `FROM` clause, the residual machinery and `prefilters` are all
+untouched. It is not a join (which would need join support the emitter
+does not have, for 6 % ) and emphatically not a correlated `EXISTS`,
+which defeats the virtual table's index entirely and measured 85× worse
+than the subquery.
+
 The implied forms carry no `json_type` guard — a derived column IS the
 value — but each is TOTAL through its own `IS NOT NULL`, so a row with
 no box answers `FALSE` rather than SQL's `NULL` and negation composes
-classically. An implied conjunct may not be negated at all: negating a
+classically. The `rtree` form is total for the same reason by a
+different route: a row with no box was never inserted into the virtual
+table (the sync trigger's guard, MODEL-FORMAT §3.2), so it is simply not
+in the list. An implied conjunct may not be negated at all: negating a
 superset is a subset, and that drops rows. That guard also makes the
 leading term a two-sided range, which is what SQLite will actually
 **seek**: with a one-sided range it prefers a table scan, and a scan over
@@ -299,6 +324,100 @@ item). This is the same class as the string-operator and aggregate
 preconditions above, and the same answer: keep `compileSchema` injected
 if that distinction matters to you.
 
+### The k-nearest plan (vector)
+
+A `derive: 'vector'` column (MODEL-FORMAT §2.1) holds each row's member
+as a packed, l2-normalized binary32 vector. Nothing in SQL ranks over
+it, and the reasons are portability and correctness rather than speed:
+none of the `ORDER BY`-over-a-function spellings runs where no function
+can be registered, and an ordering decided in SQL cannot break a tie by
+the document's own secondary keys — which the engine executor has to,
+for the three executors to agree. (On speed the two are close: re-measured
+against the real column with the probe hoisted out of the per-row call,
+`ORDER BY` over a registered function sits at rough parity with fetching
+the column and ranking in the engine — `benchmark/vector.js` publishes
+the band, and the store does not emit it anyway.) The k-nearest
+composition is therefore planned as **a cut the engine finishes**, the
+implied-conjunct pattern applied to an ordering instead of a predicate:
+
+| stage | what | where |
+|---|---|---|
+| narrow | the pushed `$where` conjuncts, exactly as in every other mode | SQL |
+| fetch | `(row identity, packed column)` for every narrowed row — no `ORDER BY`, no `LIMIT`, no similarity call; the probe never binds into the statement | SQL |
+| score | every column unpacked and dotted with the l2-normalized probe (the cosine of the raw vectors, up to binary32 rounding); a `NULL` column scores nothing | engine (`@jarenjs/core/vector`, through `derive.js`) |
+| cut | with `m = offset + limit`: every scored row whose score is within `margin` (`1e-6`) of the m-th best is a candidate; when fewer than `m` rows scored, EVERY row is | engine (`knn.js`) |
+| fetch | the candidates' documents, by identity, in identity order, through the dialect's by-identities statement (batched under every build's parameter cap) | SQL |
+| decide | the ORIGINAL document — its whole `$orderby` (the `$similarity` key over the raw member, every secondary key, `$empty`), its window, its `$return` — as the set residual over exactly those documents | engine |
+
+**Why the cut is exact.** The column's score and the engine's key are
+not the same number: one is a dot product over binary32-normalized
+forms, the other the cosine of the raw doubles, and they differ by up
+to ~1e-8 (measured over the corpus and over thousands of random 768-d
+pairs). A plan that CUT by the column's score alone could therefore
+pick a different m-th row than the engine whenever two true cosines lie
+within that distance. With a margin of at least twice the divergence
+the engine's top `m` is a subset of the candidates: if a row the engine
+ranks inside the window were cut, some candidate the engine ranks
+outside it would have to score higher by the column and lower by the
+engine, which two scores within half the margin of each other cannot
+do. `1e-6` is a hundred times the bound; in practice it admits only true
+ties, and the residual re-ranks a handful of documents — microseconds.
+
+**What the engine's decision buys.** Ties break by the document's OWN
+secondary keys, never by row identity, which the engine executor cannot
+see (row identity serves the fetch, never the order — a stable sort
+over the candidates in identity order sees what it would have seen over
+the whole collection). Offsets, nested windows and every secondary key
+compose for free. And the unrankable tail is right by construction:
+`$empty: 'least'` places a row whose key is empty LAST, so a window
+wider than the scored rows must produce those rows in the engine's own
+secondary order — which is exactly why the cut takes every row when
+fewer than `m` scored (the collection is then no larger than the window)
+rather than dropping what the column cannot rank.
+
+**What can raise, and where** — the spatial ERRORS rule, restated for
+the probe path. §8.15 raises `JQ2001` for a member that is not an array
+(a string, an object, a typed array held as an object); the column is
+`NULL` for such a member (MODEL-FORMAT §3.2), so below the scored cut
+the row is never fetched and never raises, and past it (the full fetch)
+the engine raises as it would have. An ABSENT member is not that case:
+its key is the empty sequence before the operand is checked, the column
+is `NULL`, and both executors place the row in the tail. A store that
+wants the engine's refusal for every row keeps `compileSchema` injected
+— the vector column only exists over a member the schema types `array`
+and nothing else, which is what makes the two agree on every row it
+does fetch.
+
+**The probe.** A literal vector must be the column's width at plan time
+(another width is not recognized, and the reason says both widths). An
+external probe is checked at call time by the binder's own rule: a
+bound value that is not a vector of the column's width — another width,
+a non-finite component, not an array at all — DIVERTS the call to the
+full-collection residual, where the engine answers what it answers
+everywhere (empty keys, so the secondary keys order every row; or its
+own `JQ2001` for a non-array). The plan never raises on the engine's
+behalf.
+
+That diversion is correct and it is expensive: the residual reads every
+document, and `explain()` still reports `knn`, because the plan is the
+shape and the bound value is not part of it. So the fallback is
+COUNTED — `collection.stats().knn.diverted` — which is the only surface
+on which a probe arriving at the wrong width from a model or a form is
+distinguishable from a query that ran the cut. A rising `diverted` beside
+a flat `queries` is a caller embedding through the wrong model.
+
+**Preconditions, each named in `explain()`.** The selection must be
+pushed whole — every `$where` conjunct exact, nothing before it — because
+a conjunct left to the residual could drop a candidate the cut counted,
+and an implied conjunct narrows to a superset the residual then shrinks;
+either could leave the window short. The ordering's first key must be
+the `$similarity`, descending, under `$empty: 'least'`; the window must
+have a finite limit (an unbounded ranking is a full sort, and the engine
+does it over the whole collection, said so). Further keys are the
+engine's business. With `strict: true` the shape is `JD0010` naming the
+rank: the order is engine work, the same honesty as a spatial
+refinement.
+
 ### The two residual modes
 
 - **Row residual** — only the projection is untranslated: predicates,
@@ -315,25 +434,40 @@ if that distinction matters to you.
   does not), and the WHOLE original compiled document runs over the
   materialized candidate array. Re-applying pushed conjuncts is
   idempotent, so pushdown is pure narrowing. Reported as a barrier.
+- **Set residual over a cut** (`knn`) — the same whole-document
+  re-run, over a candidate set an ORDERING chose rather than a
+  predicate (the k-nearest plan above). A barrier too, and
+  `stats().knn` counts the rows scored and the candidates kept per
+  query, so a collection of many exact duplicates is visible rather
+  than merely slow.
 
 ### `explain()`
 
 Extends `compileJsonQuery(...).explain()`'s shape — `{ externals,
-operators, functions, collations, limits }` — with `{ sql, params,
-indexes, prefilters, residual, barriers, scanNarrative }`. `params`
+operators, functions, collations, limits }` — with `{ mode, sql, params,
+indexes, prefilters, rank, residual, barriers, scanNarrative }`. `mode`
+is `'native' | 'row' | 'set' | 'knn'`. `params`
 lists the bound slots in order (external names, literal markers, and
 derived slots naming the external and box axis they compute — values
 are ALWAYS bound, never interpolated). `indexes` names the declared
 indexes whose generated columns the pushed predicates and ordering
 touch, and the `scanNarrative` is the database's own `EXPLAIN QUERY
 PLAN` prose so the claim is checkable against the engine that will run
-it. `prefilters` is the implied conjuncts — `{ construct, columns,
-exact }` each — because whether a declared index is earning its keep is
-not readable from `sql` alone, and because `indexes` says what a
-predicate TOUCHES while the narrative says what the database will
-DO. `estimatedRows` is ABSENT on SQLite drivers — the capability slot
+it. `prefilters` is the implied conjuncts —
+`{ construct, via, columns, exact }` each — because whether a declared
+index is earning its keep is not readable from `sql` alone, and because
+`indexes` says what a predicate TOUCHES while the narrative says what
+the database will DO. `via` is `'columns'` or `'rtree'`: which physical
+realization of a `bbox` column set actually ran, which is not always
+what the model declared — a build without the R\*Tree module falls back
+and this is where it says so (MODEL-FORMAT §4). Under `'rtree'` the
+`columns` member names the virtual table's own columns and `indexes`
+names the virtual table. `rank` is `null` or the k-nearest stage —
+`{ column, dims, probe, limit, offset, margin, decides: 'engine' }` —
+what the fetch reads, the window the cut serves, the margin it keeps,
+and who decides the order (always the engine). `estimatedRows` is ABSENT on SQLite drivers — the capability slot
 is empty and no number is fabricated. `residual` is `null` or
-`{ mode: 'row' | 'set', reasons: [{ construct, reason }] }` with
+`{ mode: 'row' | 'set' | 'knn', reasons: [{ construct, reason }] }` with
 reasons drawn from the deliberate-residual table. With
 `strict: true`, any residual is instead the compile error `JD0010`
 naming the forcing construct.
@@ -457,7 +591,13 @@ delete inside the visible slice answered without re-query), running
 over contributions when the extremum's holder leaves), per-group
 **deltas** (the accumulator machinery once per group), and **re-run**
 for everything else — declared, reported through `live.mode`, never
-silent. Invalidation matches a record by table plus pointer prefix,
+silent. A plan that fell to the set residual ONLY for a spatial
+refinement (a pushed box or cell range with the exact predicate left
+to the engine) is still the rows strategy — the geofence: the fetch is
+index-narrowed and per-row re-evaluation IS the exact test — while an
+ordering or an aggregate beside such a refinement re-runs, the
+refinement never being the reason named. Invalidation matches a record
+by table plus pointer prefix,
 over-approximating toward re-evaluation (a missed update would be a
 correctness bug; an extra one is only slower). Emissions preserve
 reference identity for untouched rows — the O(k) renderer's contract —
@@ -497,4 +637,8 @@ the ENVIRONMENT — OPFS persistence across reloads, the owner topology
 (one context holds the sole connection, tabs are clients over a
 BroadcastChannel), and the second-writer refusal — across Chromium,
 Firefox and WebKit, with the memory fallback stated where OPFS is
-absent.
+absent — and the spatial corpus, run entry by entry through the data
+studio's throwaway-store operation, holds the wasm build to the
+JavaScript engine's recorded answers in every one of those engines,
+OPFS or not, because an entry seeds its own store and needs execution,
+not persistence.

package/README.md

@@ -101,21 +101,75 @@ const adults = await users.execute({
   `$distance` push a bounding box the truth table proves they imply,
   and the exact predicate re-runs over the narrowed candidates —
   `explain().prefilters` says which, over what columns, and whether it
-  decided or merely narrowed.
+  decided or merely narrowed. The worked example, the geofence and the
+  measured numbers are in [Spatial storage](#spatial-storage--the-model-the-plan-the-fence-the-numbers)
+  below.
+- **Migrations are documents.** `planMigration` diffs two models into
+  rendered-DDL + JSLT-transform + assertion steps; a shadow database
+  replays the whole chain before the real store is touched; a
+  checksummed history refuses edited or reordered migrations; a
+  narrowing without an adequate transform is refused against the REAL
+  data, inside the transaction.
+- **The safe profile.** Untrusted query documents run under composed
+  bounds: engine limits on the residual, a mandatory row bound that
+  refuses rather than truncates, reference allow-lists, optional
+  full-scan refusal, and per-collection mandatory predicates no
+  document shape can shed. Read-only stores refuse writes at the
+  driver.
+- **Writes validate** through an injected hook; without one,
+  `store.capabilities.validated` is `false` and the docs say what that
+  costs. The public API is asynchronous (the browser's OPFS story
+  forces it) with a promise-free `store.sync` twin where the driver is
+  synchronous.
+
+## Spatial storage — the model, the plan, the fence, the numbers
+
+A collection stores GeoJSON as it is — a position is an array, a
+geometry is an object, nothing is wrapped — and declares what to index
+over it:
+
+```js
+const store = await openStore({
+  $model: '0.1',
+  collections: {
+    places: {
+      schema: {
+        type: 'object',
+        properties: {
+          id: { type: 'string' },
+          // typed as geography: a spatial predicate is only pushed onto a
+          // member the schema types as an array or an object and nothing else
+          at: { type: ['array', 'object'] },
+        },
+      },
+      key: '/id',
+      indexes: [
+        { name: 'by_box', path: '$.at', derive: 'bbox' },
+        { name: 'by_cell', path: '$.at', derive: 'geohash', precision: 6 },
+      ],
+    },
+  },
+}, { driver: nodeDriver() });
+```
+
+`derive: 'bbox'` materializes the member's bounding box as four
+columns and `derive: 'geohash'` its cell, both computed by
+`@jarenjs/core/geo` — generated columns over registered deterministic
+functions where the driver can index one, stored columns the store
+writes where it cannot (MODEL-FORMAT §§2.1, 3). The query a consumer
+writes then narrows in SQLite and refines in the engine, and
+`explain()` says so:
 
 ```js
-// the collection declares  indexes: [{ name: 'by_box', path: '$.at',
-//   derive: 'bbox' }]  and types that member  at: { type: ['array', 'object'] }
-// — a predicate is only PUSHED onto a member the schema types as geography
 const places = store.collection('places');
-const nearby = {
+const inside = {
   $for: { p: '$[*]' },
   $where: { $within: ['$p.at', '$region'] },
   $return: '$p',
 };
-await places.execute(nearby, { externals: { region } });
+await places.execute(inside, { externals: { region } });
 
-const how = await places.explain(nearby, { externals: { region } });
+const how = await places.explain(inside, { externals: { region } });
 how.prefilters;
 // [{ construct: '$within',
 //    columns: ['gx_at_bbox_w', 'gx_at_bbox_e', 'gx_at_bbox_s', 'gx_at_bbox_n'],
@@ -126,30 +180,261 @@ how.scanNarrative;
 // 'SEARCH places USING INDEX places_by_box (gx_at_bbox_w>? AND gx_at_bbox_w<?); …'
 ```
 
-  The region arrives as a bound parameter: a GeoJSON object is not a
-  value any database can bind, so what binds is one edge of its box per
-  slot, computed at bind time from the same kernel the stored columns
-  came from. A circle that reaches a pole or crosses the antimeridian
-  pushes **nothing** — there is no single box to push — and the answer
-  is the same, reached by reading more rows. `explain()` is what makes
-  that checkable rather than quoted.
-- **Migrations are documents.** `planMigration` diffs two models into
-  rendered-DDL + JSLT-transform + assertion steps; a shadow database
-  replays the whole chain before the real store is touched; a
-  checksummed history refuses edited or reordered migrations; a
-  narrowing without an adequate transform is refused against the REAL
-  data, inside the transaction.
-- **The safe profile.** Untrusted query documents run under composed
-  bounds: engine limits on the residual, a mandatory row bound that
-  refuses rather than truncates, reference allow-lists, optional
-  full-scan refusal, and per-collection mandatory predicates no
-  document shape can shed. Read-only stores refuse writes at the
-  driver.
-- **Writes validate** through an injected hook; without one,
-  `store.capabilities.validated` is `false` and the docs say what that
-  costs. The public API is asynchronous (the browser's OPFS story
-  forces it) with a promise-free `store.sync` twin where the driver is
-  synchronous.
+The region arrives as a bound parameter: a GeoJSON object is not a
+value any database can bind, so what binds is one edge of its box per
+slot, computed at bind time from the same kernel the stored columns
+came from. `$bbox-intersects` and a geohash cell test are exact and
+need no refinement; `$within` and a bounded `$distance` push the box
+they provably imply and re-run the exact predicate over the narrowed
+candidates. A circle that reaches a pole or crosses the antimeridian
+pushes **nothing** — there is no single box to push — and the answer is
+the same, reached by reading more rows. A proximity probe is **nine
+cells** (`$geohash-neighbours`), never one prefix: two points ten
+metres apart can differ in the first character of their cell, so a
+single-cell range is bucketing, not proximity.
+
+**The geofence.** Register the same document as a live query and it is
+maintained as writes arrive — the initial fetch narrows through the
+index, and every touched row is re-evaluated by the engine's *exact*
+predicate:
+
+```js
+const fence = await places.live([{
+  $for: { p: '$[*]' },
+  $where: { $within: ['$p.at', '$region'] },
+  $return: '$p.id',
+}], { externals: { region } });
+fence.mode; // { strategy: 'rows', mode: 'incremental' }
+fence.subscribe(({ patch }) => {
+  // add    when a point enters the region
+  // remove when it leaves
+  // nothing while it moves within (a whole-document return sees a replace)
+});
+```
+
+That is per-row evaluation, not an incremental spatial index: every
+write runs `$within` against the region once, on the store's
+connection, and a large region at a high write rate pays for it on
+every write (LIVE-FORMAT §7 states the cost). An ordering by
+`$distance` or a spatial aggregate re-runs on invalidation with the
+reason in `live.mode` — declared, never silent.
+
+**The numbers, the loss included.** `benchmark/spatial.js` stores <!--bm:spatial.corpus-->50,000 points<!--/bm-->
+over the Netherlands and probes one box at <!--bm:spatial.rows-->258 of 50,000 (0.5 %)<!--/bm--> selectivity,
+asserting every plan case of the committed spatial corpus and every timed shape against the JavaScript
+engine before a single timing is printed. The `$within` a consumer writes went from <!--bm:spatial.scan-->80 ms<!--/bm-->
+as a full scan to <!--bm:spatial.within-->2 ms<!--/bm--> over the `bbox` index (<!--bm:spatial.scanVsIndexed-->40.0<!--/bm-->×);
+`$bbox-intersects` is <!--bm:spatial.bboxIntersects-->1.9 ms<!--/bm-->, a bounded `$distance` <!--bm:spatial.distance-->1.5 ms<!--/bm-->;
+one geohash cell answers in <!--bm:spatial.cellOne-->0.0068 ms for 0 row(s)<!--/bm--> and the honest nine-cell
+probe in <!--bm:spatial.cellNine-->0.026 ms for 2 row(s)<!--/bm-->. The row the store had to win is the same
+`$within` in the in-memory engine over the parsed array, no database at all: <!--bm:spatial.engine-->32 ms<!--/bm-->.
+The indexed store is now <!--bm:spatial.engineVsIndexed-->16.1× faster than<!--/bm--> it — but the un-indexed scan
+is not, and the comparison is not an even one either way: the engine starts from parsed objects where the
+store starts from bytes on a page and pays JSON materialisation for every row it returns. Both rows stay
+published. The deterministic-UDF hatch takes a literal `$within` on a collection with no derived index, and
+its profile is measured on the same rows in MODEL-FORMAT §8.2 (a loss as a sole predicate, a large win
+beside a selective conjunct or a `LIMIT`).
+
+**Two shapes on disk for one declaration.** `derive: 'bbox'` has a
+second physical realization: `physical: 'rtree'` keeps the same four
+derived columns and stores the boxes in a SQLite R\*Tree beside the
+collection, synced by three declared triggers, with no B-tree over the
+columns (MODEL-FORMAT §2.1). The logical model is unchanged — the
+spatial corpus runs every entry under both mappings, in all three
+executors, with no special-cased entry — and the pushed conjunct becomes
+a `rowid` subquery over the virtual table. Through the store the same
+`$within` measures <!--bm:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/bm-->; loading the same rows
+costs <!--bm:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/bm-->, because the R\*Tree is a
+second table written inside every write transaction. Isolated from the
+store on a raw connection the probe is <!--bm:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/bm-->.
+Both halves are published because both are the price. One honest
+difference comes with it: an R\*Tree stores 32-bit floats rounded
+outward, so its box is a superset and `$bbox-intersects` is refined
+rather than exact there — same rows, and `strict: true` says so.
+
+**One document, three executors, proven to agree.** The same spatial
+query document runs in three places — the JavaScript engine
+(`compileJsonQuery`), SQLite through the Node driver, and SQLite
+compiled to wasm in a real browser tab — and one committed corpus holds
+all three to the same answers. `test/json/fixtures/spatial-corpus.json`
+records what the engine answers for every case (generated, never
+hand-typed); `test/db/spatial-oracle.test.js` runs every entry through
+the Node driver under all three mappings — the derived indexes as
+columns, the same indexes as R\*Trees, and none; and
+`packages/website/e2e/spatial-agreement.spec.js` drives the data
+studio's Store pane to run the same entries through the wasm build in
+Chromium, Firefox and WebKit, asserting every answer against the
+fixture on disk and the number of entries run against the corpus. Each
+runner names the executor, the entry and the query when it disagrees.
+That is the whole claim — not faster than anyone, not PostGIS — and the
+browser leg's limit is stated with it: **it proves execution, not
+durability.** Where OPFS is unavailable the tab's store is in-memory,
+which is a property of the host, not of the suite.
+
+**No head-to-head rival, and saying so.** Nothing else in JavaScript
+stores GeoJSON in SQLite from a JSON query document, so the suite
+invents none. The rivals to know about: MongoDB (`$geoWithin`, `$near`,
+a `2dsphere` index) has a GeoJSON-native query document and a real
+spatial index, and runs on a server — no browser execution, and no
+second engine to agree with; DuckDB-wasm with `spatial` runs in a tab
+with a real index and the overlay operations this store refuses to
+build, and its query is SQL, not a document. Neither runs one document
+through three executors proven to agree, and neither validates ring
+closure in a schema.
+
+## Vector storage — the column, the cut, the price, the ceiling
+
+A collection can declare that one member is an embedding, and the store
+keeps it as a packed column beside the document:
+
+```js
+const store = await openStore({
+  $model: '0.1',
+  collections: {
+    memories: {
+      schema: {
+        type: 'object',
+        properties: {
+          id: { type: 'string' },
+          text: { type: 'string' },
+          // typed `array` and nothing else: a column over a member that
+          // may also be a string or null is a column that lies about
+          // some documents. minItems/maxItems make a wrong-width write
+          // a validation error instead of an unrankable row.
+          embedding: { type: 'array', items: { type: 'number' },
+            minItems: 768, maxItems: 768 },
+        },
+      },
+      key: '/id',
+      indexes: [{ name: 'by_vec', path: '$.embedding', derive: 'vector', dims: 768 }],
+    },
+  },
+}, { driver: nodeDriver() });
+```
+
+`derive: 'vector'` materializes the member **l2-normalized and packed as
+little-endian binary32** — `4·dims` bytes, computed by
+`@jarenjs/core/vector` — into one **stored** column on every driver, with
+**no B-tree over it and no registered function** (MODEL-FORMAT §§2.1,
+3.1). Nothing seeks a blob of floats, so the entry names a column rather
+than an index; and because the column is stored rather than generated, a
+plain `SELECT`, a backup or a foreign tool can read the table without
+registering anything — which is also what lets `bun`, whose SQLite
+binding has no function API at all, store and read the same bytes.
+
+**"The k most similar" is not a keyword.** It is the query language's own
+ordering and window (QUERY-FORMAT §8.15) — `$orderby` on a `$similarity`
+key, descending, under a `$subsequence`:
+
+```js
+const memories = store.collection('memories');
+const nearest = {
+  $subsequence: [{
+    $for: { m: '$[*]' },
+    $where: { $eq: ['$m.topic', 'deploys'] },
+    $orderby: [{ $key: { $similarity: ['$m.embedding', '$q'] }, $dir: 'desc', $empty: 'least' },
+      '$m.id'],
+    $return: '$m',
+  }, 0, 10],
+};
+await memories.execute(nearest, { externals: { q: probe } });
+
+const how = await memories.explain(nearest, { externals: { q: probe } });
+how.mode;                       // 'knn'
+how.rank;                       // { column: 'gx_embedding_v768', dims: 768,
+                                //   probe: { external: 'q' }, offset: 0, limit: 10,
+                                //   margin: 1e-6, decides: 'engine' }
+how.sql;                        // SELECT "rowid", "gx_embedding_v768" … WHERE …
+                                // — no ORDER BY, no LIMIT, no similarity call
+memories.stats().knn;           // { queries, rows, candidates, fullFetches, diverted }
+```
+
+The pushed `$where` narrows in SQL exactly as in any other mode; the
+statement then projects `(row identity, packed column)` and nothing else;
+the **engine** unpacks, scores and keeps every row within `1e-6` of the
+`offset + limit`-th best; the candidates' documents are fetched by
+identity, and the ORIGINAL document — its whole `$orderby`, its window,
+its `$return` — runs over exactly those. So **the column cuts and the
+engine decides**: ties break by the document's own secondary keys,
+offsets and nested windows compose for free, and the rows the column
+cannot rank still appear where `$empty: 'least'` puts them. `strict: true`
+refuses the shape with `JD0010` naming the rank, because the ordering is
+engine work — the same honesty the spatial refinement gets. A probe of
+the wrong width, or one that is not an array of numbers, **diverts** to
+the residual, so a query can never quietly become an O(table) scan nobody
+counted (ARCHITECTURE, "The k-nearest plan"). A *literal* probe of the
+wrong width is refused at plan time and `explain()` says why; an
+**external** one is only knowable when it is bound, so the plan stays
+`knn` and the fallback is counted instead — `stats().knn.diverted` is
+that count, and a consumer who binds probes from a model should watch it.
+
+**The numbers, the losses included.** `benchmark/vector.js` measures one
+k-nearest query every physical way it can run — over <!--bm:vector.grid-->10,000 and 50,000 vectors at 384 and 768 dimensions, k = 10, the median of 10 probes<!--/bm--> —
+and asserts that every path returns the identical top-k, ids and order,
+on every probe before a single timing prints. The flagship row is the
+plan a consumer's own document runs, which measures <!--bm:vector.plan-->206 ms at 50,000 × 768<!--/bm-->:
+<!--bm:vector.table-->
+| path (ms) | 10,000 × 384 | 10,000 × 768 | 50,000 × 384 | 50,000 × 768 |
+|---|---:|---:|---:|---:|
+| engine resident sweep (no database) | 3.3 | 6.7 | 17 | 32 |
+| **the k-nearest plan (the store's own)** | 22 | 33 | 139 | 206 |
+| raw fetch + engine sweep (the plan's statement) | 20 | 31 | 130 | 202 |
+| `ORDER BY` over a registered function | 18 | 31 | 121 | 180 |
+| JSON-doc sweep (no vector column) | 249 | 501 | — | — |
+| sqlite-vec | 3.6 | 7.5 | 18 | 38 |
+<!--/bm-->
+
+The row the column exists to beat is the last one that has no column: the
+same query document over a collection that stores the embedding only
+inside the document costs <!--bm:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/bm-->,
+because every row's vector is parsed out of JSON before it can be
+compared. The row the store **cannot** beat is the one with no database
+in it: the same top-k over a resident `Float32Array` is <!--bm:vector.resident-->32 ms, which the plan is 6.4× slower than<!--/bm-->.
+That comparison is not an even one and the direction is the point — the
+sweep starts from decoded floats in RAM and pays nothing for durability,
+for filters that compose with the ranking, or for a process that can
+restart — but it stays published, because a store that is worth its
+price should be able to say what the price is.
+
+**Both halves of the price.** The column costs on the way in as well as
+saving on the way out: writing the same documents with
+the index costs <!--bm:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/bm-->,
+because every write pays a JSON round trip of the member plus a
+normalize and a pack. On disk one vector is <!--bm:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/bm--> —
+smaller, but *added*, since the document still carries the member the
+column is derived from.
+
+**Pushing the rank into SQL, re-measured.** A registered similarity
+function inside an `ORDER BY … LIMIT k` is the obvious alternative, and
+the suite measures it against the real column with the probe hoisted out
+of the per-row call: <!--bm:vector.udf-->180 ms against 202 ms at 50,000 × 768, and 0.87–1.00× the fetch-and-rank across the grid — rough parity on speed<!--/bm-->.
+The plan does not emit it, and after that measurement the reasons are not
+speed: `bun` has no user-function API, so a plan that needed one would
+exclude an executor outright; and an ordering decided in SQL cannot break
+a tie by the document's own secondary keys, which is what the three
+executors have to agree on.
+
+**The rival, and the ceiling.** `sqlite-vec` is the extension built for
+exactly this, and it is measured rather than described: it answers the
+same probes in <!--bm:vector.rival-->38 ms against 206 ms at 50,000 × 768 — 5.4× in sqlite-vec's favour, out of a database 6.6× smaller that holds no documents<!--/bm-->,
+over <!--bm:vector.agreement-->40 probes, no disagreements<!--/bm-->. It is a
+loadable native extension, which is the one thing this store will not
+require — it would exclude the wasm tab and stock `bun`, half the
+execution story — so the comparison is published as what it is: a faster
+engine you may prefer, and a dependency this one does not take. What
+neither of them is, is an approximate index. Exact brute force is linear
+in `n · d`, and the suite states the envelope as arithmetic rather than
+opinion: <!--bm:vector.ceiling-->5.546 ns per vector component — one query reaches 100 ms at about 22,000 vectors of 768 dimensions and one second at about 234,000<!--/bm-->.
+Past that this design is the wrong tool and no margin changes it; what
+lies beyond is an approximate index, and this store does not have one.
+
+**One document, three executors, proven to agree.** As with the spatial
+family, the k-nearest shapes of a committed corpus
+(`test/json/fixtures/vector-corpus.json`) run through the JavaScript
+engine, SQLite through the Node driver, and a real wasm build — indexed
+and unindexed — and every entry must answer identically, including a
+deliberate one-binary32-ulp near-tie and the windows that reach past the
+scored rows into the tail the column cannot rank.
 
 ## What SQLite-only means, frankly
 
@@ -202,8 +487,9 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
 - **Live queries** (LIVE-FORMAT §§7–12): `collection.live(document)`
   maintains a result as writes arrive and emits patches — incremental
   for `where`/`select`/`orderBy`+`limit`/aggregates/single-level
-  `groupBy` (the normative maintenance table), re-run for everything
-  else, **declared, never silent** (`live.mode` names the reason).
+  `groupBy` and a spatial `where` over a derived index (the geofence;
+  the normative maintenance table), re-run for everything else,
+  **declared, never silent** (`live.mode` names the reason).
   Unaffected rows stay reference-identical; a seeded oracle holds the
   maintained result equal to a fresh re-query after every mutation.
 - **Durable runs and the job queue** (JOBS-FORMAT, FLOW-FORMAT §7.6):