@jarenjs/db 0.43.3 → 0.46.4

package/docs/MODEL-FORMAT.md

@@ -71,29 +71,145 @@ The store runs over SQLite — on Node (`@jarenjs/db/node`), on Bun
 An invalid model document is `JD0005` with a `docPath` pointing at the
 offending member. Model checking happens before any database work.
 
-### 2.1 Derived indexes (spatial storage)
+### 2.1 Derived indexes (spatial and vector storage)
 
 A generated column must be a scalar (§3), and a GeoJSON position is an
 array of numbers while a geometry is an object. No path over spatial
 data is therefore indexable as written. `derive` supplies the missing
-vocabulary: it says what indexable scalar is computed from the member.
+vocabulary: it says what is computed from the member — an indexable
+scalar for the spatial kinds, and for an embedding (an array of
+hundreds of numbers, which no B-tree could seek) the packed form a
+fetch-and-rank plan reads whole.
 
 ```jsonc
 "indexes": [
-  { "name": "by_cell", "path": "$.at",       "derive": "geohash", "precision": 7 },
-  { "name": "by_box",  "path": "$.geometry", "derive": "bbox" }
+  { "name": "by_cell", "path": "$.at",        "derive": "geohash", "precision": 7 },
+  { "name": "by_box",  "path": "$.geometry",  "derive": "bbox" },
+  { "name": "by_vec",  "path": "$.embedding", "derive": "vector",  "dims": 768 }
 ]
 ```
 
-| `derive` | `path` selects | `precision` | columns | type |
+| `derive` | `path` selects | `precision` / `dims` | columns | type |
 |---|---|---|---|---|
-| `"geohash"` | a position `[lon, lat]`, a `Point`, or any value with a representative position (the mean of its vertices) | 1..12, **required** | one, `<column>` | `TEXT` |
+| `"geohash"` | a position `[lon, lat]`, a `Point`, or any value with a representative position (the mean of its vertices) | `precision` 1..12, **required** | one, `<column>` | `TEXT` |
 | `"bbox"` | any GeoJSON value | — | four: `<column>_w`, `<column>_s`, `<column>_e`, `<column>_n` | `REAL` |
+| `"vector"` | an array of exactly `dims` finite numbers, typed `array` by the schema | `dims` 1..8192, **required** | one, `<column>_v<dims>` — a COLUMN, not an index (below) | `BLOB` |
 
-`derive` is a CLOSED set of those two values. An open "expression"
+`derive` is a CLOSED set of those three values. An open "expression"
 member would be a second query language inside the model document,
 which this format does not have and will not grow.
 
+**`physical` — the shape a `bbox` index takes on disk.** `derive` says
+what is COMPUTED; `physical` says how it is STORED, and the two are
+separable:
+
+```jsonc
+{ "name": "by_box", "path": "$.geometry", "derive": "bbox", "physical": "rtree" }
+```
+
+| `physical` | shape | index |
+|---|---|---|
+| `"columns"` (the default, and what an absent member means) | the four derived columns | one B-tree over them, in `(w, e, s, n)` order |
+| `"rtree"` | the same four columns, plus an R\*Tree virtual table `<collection>_<column stem>_rtree` and three triggers that keep it in sync | the virtual table; **no B-tree over the columns** |
+
+`physical` is a CLOSED set of those two values and is refused anywhere
+but on a `derive: 'bbox'` index (rule 7 below). The **logical** meaning
+of `derive: 'bbox'` is identical either way — same rows, same answers,
+proven entry by entry by the spatial corpus in all three executors — and
+that is the whole reason it is spelled separately from the derivation.
+
+Three things follow, and none of them is optional:
+
+- **The sync is three DECLARED triggers, not a second write path.** A
+  trigger is inside the writing transaction by construction, no write
+  path can bypass it (`insert`, `upsert`, a translated patch, the patch
+  fallback, a delete and a migration backfill all fire it), and it
+  belongs to the collection table — so an rtree-mapped file opened with
+  a `columns` model reports `JD0002` naming the trigger, and the reverse
+  direction likewise, through the drift check that was already there.
+  The trigger body READS the derived columns, so the box keeps ONE
+  definition and the same trigger text works on the stored-column branch
+  (§3.1) unchanged.
+- **The virtual table is named from the COLUMN STEM, not the index
+  name**, because rule 5 lets two indexes share one column set: there is
+  one R\*Tree per column set, never one per index. Two indexes over one
+  column set asking for two shapes is `JD0004`.
+- **`$bbox-intersects` stops being EXACT under this mapping.** An
+  R\*Tree stores coordinates as 32-bit floats rounded OUTWARD, so the
+  stored box is a superset of the row's — by about 3 cm in longitude and
+  84 cm in latitude at Dutch latitudes. That is safe for a pre-filter (a
+  superset has no false negatives, which is what the implied conjunct
+  needs) and it is not a decision, so the exact box test keeps a
+  refinement and `strict: true` is `JD0010` where the column mapping
+  reported a native plan. `ARCHITECTURE.md`'s truth table carries the
+  per-mapping cell.
+
+**What it costs, both halves** (`benchmark/spatial.js`, the store's own
+rows over 50 000 points): the same `$within` measures <!--bm:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/bm-->, and loading them
+costs <!--bm:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/bm-->. Isolated from the store on a raw
+connection, the same probe is <!--bm:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/bm-->. Read speed bought with write cost and a
+second table: choose it deliberately, per index, which is why it is
+neither automatic nor a store-wide option.
+
+**`vector` — one packed column, never a B-tree.** The column holds the
+member **l2-normalized** and packed as little-endian IEEE 754 binary32:
+`4·dims` bytes, computed by `@jarenjs/core/vector` (`l2Normalize`, then
+`packVector`) — the same kernels the query engine compares with, and
+the one place this package reaches for them. It is normalized because
+over unit vectors the dot product IS the cosine: the engine computes
+`$similarity` (QUERY-FORMAT §8.15) as the cosine of the RAW members,
+and a plan that ranks over the column computes the dot product of the
+stored forms — `cos(raw) ≡ dot(normalized)` is the agreement the two
+must keep, entry by entry, and the reason the column stores the
+normalized form rather than the bytes of the member as written.
+
+Three things follow:
+
+- **No B-tree is created over the column**, and the `indexes` entry
+  therefore names a column, not an index: nothing seeks a blob of
+  floats, and a ranking plan reads the column whole and orders in the
+  engine. The verification set (§3) carries the column and no index.
+- **It is a stored column on every driver** — §3.1 says why, and why
+  that is a deliberate divergence from the spatial kinds.
+- **`dims` is the identity of the column.** Two widths over one path
+  are two columns (`<column>_v768` beside `<column>_v1536`), exactly as
+  two geohash precisions are; and the width is what makes a stored
+  vector comparable at all — a column that accepted any width would be
+  ranking vectors from different models against each other, which
+  produces plausible garbage rather than an error.
+
+**What it buys, and what it costs.** Measured against the same query
+over a collection with no such column — the whole embedding parsed out
+of the stored JSON per row — the column is worth <!--bm:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/bm-->
+on the read, and costs <!--bm:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/bm-->
+on the way in, because every write pays a JSON round trip of the member
+plus the normalize and the pack. On
+disk it is <!--bm:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/bm-->
+per vector — smaller than the member, and *added* to it, since the
+document still carries what the column is derived from. Choose it the
+way `physical: 'rtree'` is chosen: per index, with both halves in view
+(`benchmark/vector.js`; the store's README carries the whole table, the
+losses and the brute-force ceiling).
+
+**What the write path does with a member that is not a vector of
+`dims`** — absent, the wrong width, a non-finite component, not an
+array — is store the document and write `NULL` to the column (§3.2).
+The write is never refused on the column's account: refusing would
+make a schema-valid document unstorable. The declaration that DOES
+refuse a wrong-width vector at the write is the collection's own
+schema, and a model that declares `dims` should constrain the member
+to match:
+
+```jsonc
+"embedding": { "type": "array", "items": { "type": "number" },
+               "minItems": 768, "maxItems": 768 }
+```
+
+With `compileSchema` injected, a document whose embedding has 767
+components is then `JD2003` at the write; without it, the document
+stores and is simply unrankable. Both are honest; only the second is
+silent, and the schema is where the author chooses.
+
 Every rule below is `JD0004` with a `docPath` at the offending member:
 
 1. **`precision` is required for `geohash`, and refused anywhere
@@ -102,7 +218,8 @@ Every rule below is `JD0004` with a `docPath` at the offending member:
    index for city-scale work) and precision 4 is ~20 km. The right
    value follows from the query radius, which the model cannot know.
 2. **A derived index is never `unique`.** Two distinct positions share
-   a cell — and share a box edge — by construction.
+   a cell — and share a box edge — by construction, and a vector column
+   is one nothing seeks.
 3. **The path must still be singular, and there must be exactly one of
    it.** `derive` changes what is computed from the member, never how
    the member is selected: a wildcard path is refused exactly as it is
@@ -119,16 +236,46 @@ Every rule below is `JD0004` with a `docPath` at the offending member:
    a non-geographic operand and a pushed filter would simply not see
    the row. Declaring the type is what turns a derived index from
    storage into a plan.
-5. **Two indexes with the same `(path, derive, precision)` share one
-   column set**, extending §3's rule for undecorated paths. Two
-   `geohash` indexes over one path at DIFFERENT precisions are two
-   column sets, and legitimately so: a coarse bucketing index and a
-   fine proximity one are different indexes.
+5. **Two indexes with the same `(path, derive, precision)` — or
+   `(path, derive, dims)` — share one column set**, extending §3's rule
+   for undecorated paths. Two `geohash` indexes over one path at
+   DIFFERENT precisions are two column sets, and legitimately so: a
+   coarse bucketing index and a fine proximity one are different
+   indexes; two vector widths likewise.
 6. **A `bbox` index covers its four columns in `(w, e, s, n)` order** —
    not the order they are declared in. An intersection test reads
    `w <= ? AND e >= ? AND s <= ? AND n >= ?`, so the two longitude
    bounds sit together at the front of the index where a leading-column
    range can use them; `(a,b)` and `(b,a)` are different indexes (§3).
+   Under `physical: 'rtree'` that B-tree is **not created at all** — the
+   virtual table is the index, and paying for both would be paying
+   twice.
+7. **`physical` belongs to a `bbox` index and to nothing else.** It is
+   refused on a `geohash` index (an R\*Tree carries numbers and a cell
+   is text), on a `vector` index (which has one shape on disk, §3.1),
+   on an undecorated index (which has only one shape), and for any
+   value outside `{"columns", "rtree"}`. It is a property of the
+   COLUMN SET, so two indexes sharing one set must agree about it.
+8. **`dims` is required for `vector`, and refused anywhere else** — on
+   a `geohash` or `bbox` index and on an undecorated one — and must be
+   an integer 1..8192. `precision` is refused on a `vector` index for
+   the mirror reason: a vector has a width, not a cell size.
+9. **A `vector` index needs a member the schema types `array` and
+   nothing else** — `"type": "array"` or `["array"]`; a member the
+   schema does not type, or types `object`, or `["array", "null"]`, is
+   refused. Rule 4's leniency for an untyped spatial member does not
+   carry over: a column over a member the schema lets be a string is a
+   column that lies about some documents, and every refusal here says
+   how to declare the member (`items`, `minItems`, `maxItems`).
+
+*Why the model and not an `openStore` option:* the store must know which
+shape to expect in order to **verify without altering** (§3). With the
+choice in the model, the file and the model agree by construction; with
+it at a call site, a caller who forgets the option gets `JD0002` on a
+database that is perfectly correct. *Why not automatic whenever the
+driver can:* it would change the physical shape of every existing
+spatially-indexed database on a version bump, and the write cost above
+is not something a store may opt a consumer into.
 
 Approximate geohash cell size by precision (the kernel's
 `geohashCellSize` computes the degree figures; the metric ones are at
@@ -158,9 +305,11 @@ no SQL text exists outside a dialect. On SQLite:
 - one **virtual generated column** per distinct indexed path, typed
   from the collection's schema at that path (`string` → `TEXT`,
   `integer` → `INTEGER`, `number` → `REAL`, `boolean` → `INTEGER`,
-  undeclared → `ANY`), and
+  undeclared → `ANY`), one derived column set per spatial `derive`
+  (§3.1), one stored `BLOB` column per `derive: 'vector'` width, and
 - one index per `indexes` entry, named `<collection>_<index name>`,
-  over the generated columns of its paths.
+  over the generated columns of its paths — except a `vector` entry,
+  which names its column and creates no index (§2.1).
 
 Index paths are analyzed through the query engine's published AST: a
 path is indexable exactly when the analysis reports it singular and
@@ -210,10 +359,38 @@ mapping BRANCHES on what the driver declares. That is what the
 capability table (§4) is for: a driver that cannot do a thing says so
 rather than degrading silently.
 
-| `capabilities.deterministicIndexableFunctions` | mapping | drivers |
-|---|---|---|
-| `true` | a **virtual generated column** whose expression calls a deterministic function the store registers at open — `jaren_geohash(<member>, <precision>)`, `jaren_bbox_w(<member>)`, … over `json(jsonb_extract("doc", '<path>'))` | `node`, `wasm` |
-| `false` | a **stored column** the store writes on every insert, upsert and patch, computed in JavaScript from the same kernel call | `bun` |
+| capability | value | mapping | drivers |
+|---|---|---|---|
+| `deterministicIndexableFunctions` | `true` | a **virtual generated column** whose expression calls a deterministic function the store registers at open — `jaren_geohash(<member>, <precision>)`, `jaren_bbox_w(<member>)`, … over `json(jsonb_extract("doc", '<path>'))` | `node`, `wasm` |
+| `deterministicIndexableFunctions` | `false` | a **stored column** the store writes on every insert, upsert and patch, computed in JavaScript from the same kernel call | `bun` |
+| `rtree` | `false` | a `derive: 'bbox'` column set that declared `physical: 'rtree'` (§2.1) is planned, created and verified as the **B-tree over its four columns**, and `explain().prefilters[].via` reports `'columns'` beside `store.capabilities.rtree === false` | any build without `ENABLE_RTREE` |
+
+The two branches are independent: the `physical` mapping composes with
+either derived-column mapping, and the sync triggers read the derived
+columns either way, so their text is identical on both.
+
+**The vector column does not branch.** A `derive: 'vector'` column is
+the stored `BLOB` column under BOTH rows of the table — the store
+writes it on every insert, upsert and patch, on every driver, computed
+in JavaScript through the same seam a migration backfill uses — and
+**no function is ever registered for it**. That is a deliberate
+divergence from the spatial kinds, for three measured reasons:
+
+- a virtual generated column is a host-function call per row, measured
+  at 150× against a stored read in the spatial work — and a vector's
+  call would re-derive a whole array per row, not a scalar;
+- a stored column is readable without any registration, so a plain
+  `SELECT`, a backup and a foreign tool can never hit `unknown
+  function` over it, which the first consequence below shows a virtual
+  column can;
+- `bun` has no function API at all, so one mapping is the only way
+  every driver stores, verifies and reads the same column.
+
+Two things follow that are NOT true of a spatial column: a database
+created under `node` opens under `bun` with no drift on the vector
+column's account (the same declared text on both), and the migration
+planner emits the backfill step for it under either `derived` setting
+(MIGRATION-FORMAT §2.1).
 
 Three consequences, each normative:
 
@@ -231,12 +408,22 @@ Three consequences, each normative:
   `NaN` and no `Infinity`, so the write path computes from the member
   as it will be held rather than from the object handed in. Without
   that the two mappings would answer differently for one document.
-- **The same model document produces two different physical shapes**,
-  and *match* is by declared text (above). A database created under
-  `node` and opened under `bun` therefore reports `JD0002` naming the
-  derived column — correctly: the column really is different. The
-  physical mapping is a property of the driver that CREATED the file,
-  and moving a file between the two is a migration, not an open.
+- **The same model document produces two different physical shapes**
+  for a spatial column, and *match* is by declared text (above). A
+  database created under `node` and opened under `bun` therefore
+  reports `JD0002` naming the derived column — correctly: the column
+  really is different. The physical mapping is a property of the driver
+  that CREATED the file, and moving a file between the two is a
+  migration, not an open.
+- **The R\*Tree fallback is a REPORT, not a degradation.** §4 promises
+  that a model declaring a spatial index is portable across all three
+  drivers and that the physical shape it produces is not, so refusing at
+  open on a build without the module would break a stated property of
+  the format for the sake of tidiness. The answers do not change — the
+  refinement is what makes them identical — and the surface a consumer
+  reads says which shape ran. A store that reported `via: 'rtree'` while
+  running columns, or that said nothing at all, is the one behaviour
+  this format forbids.
 
 ### 3.2 When a derived column is `NULL`
 
@@ -260,12 +447,53 @@ raise. The promotion therefore requires the schema to type the member
 as an array or an object and nothing else; a store that wants the
 engine's refusal instead keeps `compileSchema` injected.
 
+Under `physical: 'rtree'` the same rule is enforced in SQL, by the
+`WHEN <stem>_w IS NOT NULL` guard on the sync triggers: **a document
+with no bounded position is ABSENT from the virtual table**, so the
+pushed subquery simply does not list it — the same answer the column
+mapping's leading `IS NOT NULL` produces. The guard is load-bearing and
+not defensive: an R\*Tree coerces a `NULL` (or a text) coordinate to
+`0.0` without complaining, so without it every unbounded document would
+be indexed at `[0, 0]`.
+
 Traversal and validity stay separate concerns, as they do in the
 kernel: a value that carries SOME positions is bounded by the positions
 it has. A `LineString` whose second vertex did not survive as a
 position is bounded by its first — a document the GeoJSON meta-schema
 refuses in the first place — and both mappings agree about it.
 
+**A `vector` column is `NULL` when the member is not a vector of
+exactly `dims` finite numbers:**
+
+| the stored member | column |
+|---|---|
+| absent | `NULL` |
+| an array of another width (767 where `dims` is 768) | `NULL` |
+| a component that is not finite — JSON has no `NaN`, so it arrives as `null` | `NULL` |
+| not an array: a string, an object, a typed array (which JSON holds as an object), an array of arrays | `NULL` |
+| the zero vector | **stored** — `4·dims` zero bytes; it has no direction and scores 0 against everything, which the query format publishes as a score, not a refusal |
+| an array of `dims` finite numbers | the packed, l2-normalized form |
+
+The document itself stores in every row of that table; the write is
+never refused on the column's account (§2.1). The consequence for the
+k-nearest plan (ARCHITECTURE.md, "The k-nearest plan") is stated here:
+**a row whose vector column is `NULL` scores nothing, so it is never
+among the rows the column's cut keeps** — unrankable, not wrong. That
+agrees with the engine on every row of the table above but one:
+`$similarity` over an absent, wrong-width or non-finite member answers
+the empty sequence (QUERY-FORMAT §8.15), and under `$empty: 'least'`
+such a row sorts LAST, where only a window wider than the scored rows
+reaches it — and there the plan fetches every row and the engine orders
+the tail itself. The one row that differs is the non-array member (a
+string, an object): §8.15 raises `JQ2001` for it where the column holds
+`NULL`, so a window that never reaches the tail never raises for it and
+one that does raises as the engine would. That is why rule 9 (§2.1)
+makes the column exist only over a member the schema types `array` and
+nothing else; a store that wants the refusal for every row keeps
+`compileSchema` injected. What the column cannot do is refuse a
+wrong-width vector at the write — that is the schema's job, and §2.1
+shows the declaration.
+
 ## 4. The driver contract and the synchronous fast path
 
 A driver is `{ name, dialect, open(path, options) }`; `open` returns a
@@ -307,6 +535,14 @@ the derived-column mapping branches on (§3.1), so a model that declares
 a spatial index is portable across all three drivers and the physical
 shape it produces is not.
 
+`rtree` is read from the library's compile options (`ENABLE_RTREE`) and
+is the second mapping branch: a `derive: 'bbox'` index that declares
+`physical: 'rtree'` (§2.1) opens on a build without the module as the
+B-tree over its four columns, and `explain().prefilters[].via` names
+the shape that actually ran. That is this table's posture applied to a
+mapping rather than to a method — the store says which shape it used,
+and never claims one it did not.
+
 A library below SQLite **3.45** fails at open with `JD0001` naming
 the version found.
 
@@ -519,8 +755,15 @@ of the store's collections, no mandatory predicates, no scan refusal.
 2. **The mandatory row bound.** Every non-aggregate fetch carries a
    database-side `LIMIT` of `maxRows + 1`. A fetch that crosses
    `maxRows` — a result set, a residual's candidate set, a diverted
-   full scan — is the coded `JD2007` and the result is refused WHOLE.
-   It is never silently truncated.
+   full scan, or a k-nearest plan's candidate scan — is the coded
+   `JD2007` and the result is refused WHOLE. It is never silently
+   truncated. The k-nearest case is worth stating on its own, because
+   the number bounded is not the one the query asked for: the plan
+   scores every row the pushed `WHERE` admits before it can know which
+   `k` are nearest, so "the two nearest" over an unfiltered collection
+   of ten thousand is a ten-thousand-row fetch and a profile at
+   `maxRows: 1000` refuses it. Narrow it with a predicate, or raise the
+   bound for that query deliberately.
 3. **Reference containment.** The document may reference only the
    externals, host functions and collations the profile declares, and
    only collections the profile allows; an undeclared reference is the
@@ -672,6 +915,34 @@ to build one on — a crude guess would be dishonest. It pushes
 deterministically and this profile is published so the shape of the win
 is known; add a narrowing predicate or a `LIMIT` and the push pays.
 
+**The same profile for a spatial predicate.** A `$within` against a
+LITERAL region takes the hatch only on a collection that declares **no**
+derived spatial index on the member: where one is declared the
+promotion (§2.1, `explain().prefilters`) takes the conjunct first and
+the hatch is consulted only for a refused one, and an external region
+never qualifies — a deterministic function must not close over
+changing state. Measured on the same terms as the `$sqrt` rows
+(`benchmark/spatial.js`: 50 000 stored points, ~0.5 % selectivity,
+mean of 20 executions after a warm one; the residual comparator
+pushes everything BUT the spatial conjunct):
+
+<!--bm:spatial.udfTable-->
+| shape | pushed (ms) | residual (ms) | verdict |
+|---|---|---|---|
+| solo `$within` over a full scan | 94 | 80 | ~even |
+| indexed `$eq` **and** `$within` (~5 % pass the index) | 5.7 | 52 | push **9.1×** |
+| `$within` with `LIMIT 10` | 1.9 | 77 | push **41.0×** |
+<!--/bm-->
+
+So the spatial hatch <!--bm:spatial.udfVerdict-->earns its row: 9.1× beside the selective conjunct and 41.0× under the LIMIT<!--/bm-->,
+by the same rule as `$sqrt`: a sole `$within` over a full scan is a
+loss (the UDF re-parses every row in the callback, and the exact
+containment test is dearer than a square root), a `$within` beside
+something that narrows first is a large win. The solo shape is the one
+a derived index answers — declare `derive: 'bbox'` on the member and
+the same predicate becomes a box seek plus a refinement over the rows
+it returns, which is faster than either column above.
+
 **The honest ceiling.** A `pushable:false` operator (a whole-series
 `$npv`, an `$sma`) is never a UDF — it stays the residual, `explain()`
 lists no `udfs` for it. Aggregate-UDF pushdown (`db.aggregate` step/final

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@jarenjs/db",
   "private": false,
-  "version": "0.43.3",
+  "version": "0.46.4",
   "type": "module",
   "main": "./src/index.js",
   "types": "./types/index.d.ts",
@@ -71,9 +71,9 @@
     "prepack": "npm run build:types"
   },
   "dependencies": {
-    "@jarenjs/core": "^0.43.3",
-    "@jarenjs/json": "^0.43.3",
-    "@jarenjs/validate": "^0.43.3"
+    "@jarenjs/core": "^0.46.4",
+    "@jarenjs/json": "^0.46.4",
+    "@jarenjs/validate": "^0.46.4"
   },
   "bin": {
     "jaren-db": "./src/cli.js"

package/schemas/jaren-migration.draft-07.schema.json

@@ -144,7 +144,7 @@
       "additionalProperties": false
     },
     "deriveStep": {
-      "description": "Recompute named STORED derived index columns from the documents already in a collection. Only the physical mapping that cannot index a registered deterministic function has these columns; the generated mapping populates them itself. The step is idempotent — a derived value is a pure function of the document — so a replay writes what the first run wrote.",
+      "description": "Recompute named STORED derived index columns from the documents already in a collection. A spatial column is stored only under the physical mapping that cannot index a registered deterministic function (the generated mapping populates itself); a vector column is stored under both. The step is idempotent — a derived value is a pure function of the document — so a replay writes what the first run wrote.",
       "type": "object",
       "properties": {
         "kind": {
@@ -168,7 +168,8 @@
               "derive": {
                 "enum": [
                   "geohash",
-                  "bbox"
+                  "bbox",
+                  "vector"
                 ]
               },
               "precision": {
@@ -176,6 +177,11 @@
                 "minimum": 1,
                 "maximum": 12
               },
+              "dims": {
+                "type": "integer",
+                "minimum": 1,
+                "maximum": 8192
+              },
               "component": {
                 "enum": [
                   "w",

package/schemas/jaren-migration.schema.json

@@ -144,7 +144,7 @@
    "additionalProperties": false
   },
   "deriveStep": {
-   "description": "Recompute named STORED derived index columns from the documents already in a collection. Only the physical mapping that cannot index a registered deterministic function has these columns; the generated mapping populates them itself. The step is idempotent — a derived value is a pure function of the document — so a replay writes what the first run wrote.",
+   "description": "Recompute named STORED derived index columns from the documents already in a collection. A spatial column is stored only under the physical mapping that cannot index a registered deterministic function (the generated mapping populates itself); a vector column is stored under both. The step is idempotent — a derived value is a pure function of the document — so a replay writes what the first run wrote.",
    "type": "object",
    "properties": {
     "kind": {
@@ -168,7 +168,8 @@
        "derive": {
         "enum": [
          "geohash",
-         "bbox"
+         "bbox",
+         "vector"
         ]
        },
        "precision": {
@@ -176,6 +177,11 @@
         "minimum": 1,
         "maximum": 12
        },
+       "dims": {
+        "type": "integer",
+        "minimum": 1,
+        "maximum": 8192
+       },
        "component": {
         "enum": [
          "w",

package/schemas/jaren-model.draft-07.schema.json

@@ -110,10 +110,11 @@
      "type": "boolean"
     },
     "derive": {
-     "description": "Derive indexable columns from the selected spatial member instead of indexing it: 'geohash' (one TEXT cell column, precision required) or 'bbox' (four REAL columns — west, south, east, north). A derived index is never unique.",
+     "description": "Derive columns from the selected member instead of indexing it: 'geohash' (one TEXT cell column, precision required), 'bbox' (four REAL columns — west, south, east, north), or 'vector' (one packed BLOB column holding the l2-normalized Float32 form of an array of numbers, dims required; stored on every driver, with no B-tree over it — a fetch-and-rank plan reads it whole). A derived index is never unique.",
      "enum": [
       "geohash",
-      "bbox"
+      "bbox",
+      "vector"
      ]
     },
     "precision": {
@@ -121,6 +122,19 @@
      "type": "integer",
      "minimum": 1,
      "maximum": 12
+    },
+    "dims": {
+     "description": "The width of a derive: 'vector' column in components, 1-8192. Required beside derive: 'vector' and refused anywhere else: the width is the column's identity — two widths over one path are two columns — and a member that is not an array of exactly this many finite numbers stores NULL in the column (the document itself still stores). Constrain the member with minItems/maxItems equal to dims in the collection's schema so a wrong-width vector is refused at the write instead.",
+     "type": "integer",
+     "minimum": 1,
+     "maximum": 8192
+    },
+    "physical": {
+     "description": "The shape a derive: 'bbox' index takes on disk: 'columns' (the default) is four generated columns under one B-tree; 'rtree' keeps the same four columns and adds a SQLite R*Tree virtual table beside the collection, kept in sync by declared triggers, with no B-tree over the columns. The logical meaning of derive: 'bbox' is identical either way. Refused on any other index kind, and on a driver without the module it falls back to 'columns' and says so through explain().prefilters[].via.",
+     "enum": [
+      "columns",
+      "rtree"
+     ]
     }
    },
    "required": [

package/schemas/jaren-model.schema.json

@@ -110,10 +110,11 @@
      "type": "boolean"
     },
     "derive": {
-     "description": "Derive indexable columns from the selected spatial member instead of indexing it: 'geohash' (one TEXT cell column, precision required) or 'bbox' (four REAL columns — west, south, east, north). A derived index is never unique.",
+     "description": "Derive columns from the selected member instead of indexing it: 'geohash' (one TEXT cell column, precision required), 'bbox' (four REAL columns — west, south, east, north), or 'vector' (one packed BLOB column holding the l2-normalized Float32 form of an array of numbers, dims required; stored on every driver, with no B-tree over it — a fetch-and-rank plan reads it whole). A derived index is never unique.",
      "enum": [
       "geohash",
-      "bbox"
+      "bbox",
+      "vector"
      ]
     },
     "precision": {
@@ -121,6 +122,19 @@
      "type": "integer",
      "minimum": 1,
      "maximum": 12
+    },
+    "dims": {
+     "description": "The width of a derive: 'vector' column in components, 1-8192. Required beside derive: 'vector' and refused anywhere else: the width is the column's identity — two widths over one path are two columns — and a member that is not an array of exactly this many finite numbers stores NULL in the column (the document itself still stores). Constrain the member with minItems/maxItems equal to dims in the collection's schema so a wrong-width vector is refused at the write instead.",
+     "type": "integer",
+     "minimum": 1,
+     "maximum": 8192
+    },
+    "physical": {
+     "description": "The shape a derive: 'bbox' index takes on disk: 'columns' (the default) is four generated columns under one B-tree; 'rtree' keeps the same four columns and adds a SQLite R*Tree virtual table beside the collection, kept in sync by declared triggers, with no B-tree over the columns. The logical meaning of derive: 'bbox' is identical either way. Refused on any other index kind, and on a driver without the module it falls back to 'columns' and says so through explain().prefilters[].via.",
+     "enum": [
+      "columns",
+      "rtree"
+     ]
     }
    },
    "required": [

package/src/algebra.js

@@ -10,8 +10,10 @@
  *
  * One plan shape covers this version: a guarded selection over ONE
  * collection with optional ordering, window, aggregate and a
- * whole-document projection. Constructs beyond it are residuals by
- * design (see ARCHITECTURE.md's deliberate-residual table).
+ * whole-document projection — or, instead of an ordering and a window,
+ * a k-nearest RANK the engine finishes over the rows the plan fetches.
+ * Constructs beyond it are residuals by design (see ARCHITECTURE.md's
+ * deliberate-residual table).
  */
 
 /** The plan format version, carried on every plan. */
@@ -55,6 +57,20 @@ export const PLAN_VERSION = 2;
  *
  * @typedef {{ ref: PlanRef, desc: boolean, emptyGreatest: boolean }} PlanOrderTerm
  *
+ * @typedef {{ column: string, dims: number,
+ *   probe: { lit: number[] } | { ext: string },
+ *   offset: number, limit: number, margin: number }} PlanRank
+ *   The k-nearest stage: the packed vector column the ranking reads,
+ *   its declared width, the probe (a plan-time literal vector, or the
+ *   external that carries one at call time), the window the ENGINE
+ *   will apply, and the inclusive score margin of the candidate cut.
+ *   The column cuts — every row whose column score is within `margin`
+ *   of the `offset + limit`-th best is a candidate — and the engine
+ *   decides: the original document, its whole ordering and window
+ *   included, runs over the candidates' documents. A plan carrying a
+ *   rank carries no order and no window of its own: nothing in SQL
+ *   orders or limits the fetch.
+ *
  * @typedef {{
  *   planVersion: number,
  *   alg: 'select',
@@ -62,6 +78,7 @@ export const PLAN_VERSION = 2;
  *   filter: PlanPredicate | null,
  *   order: PlanOrderTerm[] | null,
  *   window: { offset: number, limit: number | null } | null,
+ *   rank: PlanRank | null,
  *   aggregate: { fn: 'count' | 'sum' | 'avg' | 'min' | 'max',
  *     ref: PlanRef | null } | null,
  *   project: 'document',
@@ -81,6 +98,7 @@ export function selectPlan(collection) {
     filter: null,
     order: null,
     window: null,
+    rank: null,
     aggregate: null,
     project: 'document',
   };