@jarenjs/db 0.49.2 → 0.56.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. package/ARCHITECTURE.md +27 -15
  2. package/README.md +141 -41
  3. package/docs/JOBS-FORMAT.md +24 -8
  4. package/docs/LIVE-FORMAT.md +38 -9
  5. package/docs/MIGRATION-FORMAT.md +118 -36
  6. package/docs/MODEL-FORMAT.md +232 -30
  7. package/package.json +4 -5
  8. package/schemas/jaren-migration.draft-07.schema.json +73 -0
  9. package/schemas/jaren-migration.schema.json +73 -0
  10. package/src/capture.js +66 -28
  11. package/src/cli.js +225 -44
  12. package/src/ddl.js +23 -3
  13. package/src/dialects/sqlite.js +2 -1
  14. package/src/driver.js +63 -16
  15. package/src/drivers/wasm.js +1 -0
  16. package/src/emit-model.js +14 -0
  17. package/src/emit.js +10 -3
  18. package/src/entity.js +92 -47
  19. package/src/errors.js +25 -0
  20. package/src/index.js +2 -2
  21. package/src/jobs.js +40 -5
  22. package/src/live-time.js +12 -3
  23. package/src/live.js +11 -1
  24. package/src/migrate.js +397 -191
  25. package/src/model.js +173 -8
  26. package/src/plan.js +135 -38
  27. package/src/query.js +138 -13
  28. package/src/store.js +221 -66
  29. package/src/tracker.js +173 -48
  30. package/types/index.d.ts +152 -10
  31. package/types/node.d.ts +3 -1
  32. package/types/typed.d.ts +58 -2
  33. package/types/wasm.d.ts +7 -0
  34. package/dist/types/algebra.d.ts +0 -230
  35. package/dist/types/app.d.ts +0 -49
  36. package/dist/types/capture.d.ts +0 -85
  37. package/dist/types/cli.d.ts +0 -2
  38. package/dist/types/dag-job.d.ts +0 -40
  39. package/dist/types/ddl.d.ts +0 -229
  40. package/dist/types/derive.d.ts +0 -250
  41. package/dist/types/dialect.d.ts +0 -154
  42. package/dist/types/dialects/sqlite.d.ts +0 -9
  43. package/dist/types/driver.d.ts +0 -110
  44. package/dist/types/drivers/bun.d.ts +0 -47
  45. package/dist/types/drivers/node.d.ts +0 -37
  46. package/dist/types/drivers/wasm.d.ts +0 -65
  47. package/dist/types/emit-model.d.ts +0 -44
  48. package/dist/types/emit.d.ts +0 -75
  49. package/dist/types/entity.d.ts +0 -23
  50. package/dist/types/errors.d.ts +0 -170
  51. package/dist/types/graph.d.ts +0 -28
  52. package/dist/types/index.d.ts +0 -37
  53. package/dist/types/jobs.d.ts +0 -140
  54. package/dist/types/knn.d.ts +0 -69
  55. package/dist/types/live-time.d.ts +0 -141
  56. package/dist/types/live.d.ts +0 -64
  57. package/dist/types/migrate.d.ts +0 -170
  58. package/dist/types/model.d.ts +0 -36
  59. package/dist/types/patch-sql.d.ts +0 -37
  60. package/dist/types/plan.d.ts +0 -142
  61. package/dist/types/profile.d.ts +0 -80
  62. package/dist/types/query.d.ts +0 -112
  63. package/dist/types/residual.d.ts +0 -64
  64. package/dist/types/series.d.ts +0 -227
  65. package/dist/types/store.d.ts +0 -60
  66. package/dist/types/tracker.d.ts +0 -43
  67. package/dist/types/typed.d.ts +0 -15
  68. package/dist/types/types.d.ts +0 -26
  69. package/dist/types/udf.d.ts +0 -75
  70. package/dist/types/window.d.ts +0 -52
package/ARCHITECTURE.md CHANGED
@@ -247,14 +247,21 @@ index-usable spelling and read every row of the collection.
247
247
 
248
248
  The guards make the forms sound for typed AND untyped paths alike —
249
249
  the schema type's job is choosing the generated COLUMN (the index),
250
- never weakening the guard. Two documented preconditions: string
251
- operators, aggregates AND ordering are only promoted on schema-typed
252
- paths because the engine ERRORS on non-conforming operands where SQL
253
- would coerce or sort a stored `null` under an ordered key, a
254
- non-string under a string operator — so on an unvalidated store, rows
255
- violating the collection schema can make the engine throw where the
256
- database answers; keep `compileSchema` injected if that distinction
257
- matters to you.
250
+ never weakening the guard. Two documented preconditions. String
251
+ operators, aggregates and ordering are only promoted on schema-typed
252
+ paths that cannot hold `null` and are not boolean, because the engine
253
+ ERRORS on non-conforming operands where SQL would coerce or sort a
254
+ `null` under an ordered key, a non-string under a string operator —
255
+ so a path whose schema admits `null` is a named residual for those
256
+ forms and the two engines keep answering alike over conforming data.
257
+ And the guards protect the JSON TYPE, not the VALUE: a typed
258
+ generated column carries SQLite affinity, so on an unvalidated store a
259
+ row that violates the schema — the string `'020'` under an `integer`
260
+ path — reads as the integer `20` in the column while `json_type` still
261
+ says text, and a pushed comparison answers rows the engine (comparing
262
+ the JSON value) does not. Keep `compileSchema` injected: with the hook
263
+ no such row is ever stored, and without it the pushdown is exact only
264
+ over documents that happen to conform.
258
265
 
259
266
  **Bind-time diversion.** SQLite cannot bind a boolean, and a
260
267
  `null`-valued external needs Jaren's null semantics, not SQL's. At
@@ -571,8 +578,9 @@ collection binding, so the planner passes that binding's **name** in:
571
578
  wrapped under any other name every reference would read as an external,
572
579
  the determinism rule would reject the fragment, and the hatch would
573
580
  silently not engage — no error and no reason in `explain()`.
574
- Registration is keyed by `contentKey(fragment)`
575
- so identical fragments share one registration, and the planner MUST
581
+ Registration is keyed by `semanticKey(fragment)` (the
582
+ order-insensitive identity from `@jarenjs/core/object`) so identical
583
+ fragments share one registration, and the planner MUST
576
584
  produce a correct plan with the capability disabled (tested that way).
577
585
  Preference order: native SQL → deterministic function → residual, and
578
586
  `explain()` names the choice. **Index-form UDFs are deliberately not
@@ -585,11 +593,15 @@ UDF-expression indexes.
585
593
  ### The statement cache
586
594
 
587
595
  A caller of the core primitives, not an eighth implementation:
588
- `createBoundedCache` keyed by `contentKey(document)` plus collection,
589
- dialect and strictness. `contentKey` is the memo-grade key
590
- (`hashContent(stableStringify(x) ?? '')` drops `undefined` members,
591
- no cycle guard; both properties acceptable for a cache key), never
592
- `canonicalizeJson` (signature-grade, throws on `undefined`).
596
+ `createSemanticCache` keyed by the whole discriminating tuple — the
597
+ document plus collection, dialect, strictness, the pushdown switch and
598
+ the profilewhere the identity is the tuple's COMPLETE
599
+ serialization, never a fingerprint of it: a 32-bit content hash
600
+ collides after tens of thousands of documents, and a collision here
601
+ answers one query with another query's plan and rows (the
602
+ cache-identity test exists to keep that key abolished).
603
+ `store.stats()` exposes hits, misses and evictions so the cache is
604
+ proven rather than assumed.
593
605
  `store.stats()` exposes hits, misses and evictions, so the cache is
594
606
  proven rather than assumed.
595
607
 
package/README.md CHANGED
@@ -48,7 +48,8 @@ const store = await openStore({
48
48
  const users = store.collection('users');
49
49
  await users.insert({ id: 'u1', email: 'ada@example.test', age: 36 });
50
50
 
51
- // a query document — here written by hand; linq writes the same thing
51
+ // a query document — here written by hand; linq writes the same one,
52
+ // its source wrapped as ['$[*]'] so an item that is an array stays one item
52
53
  const adults = await users.execute({
53
54
  $for: { it: '$[*]' },
54
55
  $where: { $ge: ['$it.age', 21] },
@@ -56,6 +57,78 @@ const adults = await users.execute({
56
57
  });
57
58
  ```
58
59
 
60
+ Or, by code — the same document, written by the model pen
61
+ ([MODEL-PEN.md](../linq/docs/MODEL-PEN.md)) and typed without a generate
62
+ step (`InferMeta<typeof model>` binds the typed store):
63
+
64
+ ```js
65
+ import * as m from '@jarenjs/linq/model';
66
+
67
+ const model = m.defineModel({
68
+ collections: {
69
+ users: m.collection(
70
+ m.object({ id: m.string(), email: m.string().email(), age: m.integer().optional() }).open(),
71
+ { key: (u) => u.id, indexes: [m.index((u) => u.age)] },
72
+ ),
73
+ },
74
+ });
75
+ const store = await openStore(model, { driver: nodeDriver(), path: 'app.db' });
76
+ ```
77
+
78
+ Over an entities model, an entity set is a provider too (MODEL-FORMAT
79
+ §10.1): a chain binds through the set's root and the store runs the
80
+ document whole — the translator for a bare-binding selection or
81
+ equijoin, the declared residual for a projection — while the store
82
+ itself, serving several roots, is refused by name (`JL0007`):
83
+
84
+ ```js
85
+ import { from, fromAsync } from '@jarenjs/linq';
86
+
87
+ const starred = from(store.sync.entity('Post')).where((p) => p.stars.ge(3));
88
+ starred.toDocument(); // { $for: { it: '$.Post[*]' }, $where: { $ge: ['$it.stars', 3] }, $return: '$it' }
89
+ starred.toArray(); // the entity translator, one statement
90
+ await fromAsync(store.entity('Post'))
91
+ .join(fromAsync(store.entity('User')), (p) => p.authorId, (u) => u.id, (p) => p)
92
+ .toArray(); // a two-root equijoin, one statement
93
+ ```
94
+
95
+ A declared relation navigates on the chain, and the document still
96
+ carries no relation name: the chain reads the set's relation table
97
+ (`store.entity('Post').relations`, MODEL-FORMAT §10.1) and lowers
98
+ `p.author.email` to the correlated phrase the engine and the store both
99
+ run. The store answers it as the residual it is — `explain()` says so,
100
+ `strict` refuses it — over the two fetched roots, never a statement per
101
+ row:
102
+
103
+ ```js
104
+ const byAuthor = from(store.sync.entity('Post'))
105
+ .where((p) => p.stars.ge(3))
106
+ .select((p) => ({ title: p.title, by: p.author.email }));
107
+
108
+ byAuthor.toDocument();
109
+ // { $for: { it: '$.Post[*]' },
110
+ // $where: { $ge: ['$it.stars', 3] },
111
+ // $return: { title: '$it.title',
112
+ // by: { $for: { r1: '$.User[*]' },
113
+ // $where: { $eq: ['$r1.id', '$it.authorId'] },
114
+ // $return: '$r1.email' } } }
115
+ byAuthor.explain().hops; // [{ member: 'author', kind: 'oneToOne', binding: 'r1' }]
116
+ store.sync.explain(byAuthor.toDocument());
117
+ // { mode: 'set', referenced: ['Post', 'User'], sql: null,
118
+ // reasons: [{ construct: '$return',
119
+ // reason: 'entity queries return one bare binding natively; projections run in the engine' }], … }
120
+ byAuthor.toArray(); // the rows, two fetches — one per referenced root
121
+
122
+ from(store.sync.entity('User')).where((u) => u.posts.all().count().ge(2));
123
+ // … $where: { $ge: [{ $count: { $for: { r1: '$.Post[*]' },
124
+ // $where: { $eq: ['$r1.authorId', '$it.id'] },
125
+ // $return: '$r1' } }, 2] } …
126
+ ```
127
+
128
+ A many-to-many member (`u.labels`) is refused at build time (`JL0105`)
129
+ naming the join table: it is not a queryable root in this version, so
130
+ `load({ include: { labels: true } })` is how the memberships are read.
131
+
59
132
  `execute` answers in the ENGINE's result shape (QUERY-FORMAT §1,
60
133
  "singleton ≡ item"): `undefined` for no rows, the document itself for
61
134
  exactly one, an array for more — typed `SequenceResult<R>`, with `R`
@@ -71,9 +144,12 @@ shape binds at `store.collection<User>('users')`.
71
144
  equivalent runs as a real compiled Jaren query (the residual), and
72
145
  `explain()` always says which is which — the SQL, the bound
73
146
  parameters, the indexes used (verified against the database's own
74
- plan output), and the residual's named reasons. A 418-run
75
- differential oracle keeps both paths agreeing. `strict: true` turns
76
- any residual into a compile error.
147
+ plan output), and the residual's named reasons. A differential
148
+ oracle a committed corpus and a seeded generator, every case run
149
+ in both modes keeps both paths agreeing, with the one arithmetic
150
+ deviation declared rather than hidden (MODEL-FORMAT §10.6: SQLite's
151
+ compensated `SUM` and the engine's naive one differ in the last
152
+ bit). `strict: true` turns any residual into a compile error.
77
153
  - **Registered operators, correct in the residual, pushed where it
78
154
  pays.** Open with a registry (`operators:
79
155
  createJsltRegistry().use(mathPack).use(financePack)`) and a query may
@@ -129,7 +205,13 @@ shape binds at `store.collection<User>('users')`.
129
205
  replays the whole chain before the real store is touched; a
130
206
  checksummed history refuses edited or reordered migrations; a
131
207
  narrowing without an adequate transform is refused against the REAL
132
- data, inside the transaction.
208
+ data, inside the transaction. And by code: `@jarenjs/linq/migration`
209
+ ([MIGRATION-PEN.md](../linq/docs/MIGRATION-PEN.md))
210
+ writes the same document with the data transform typed old row → new
211
+ row, `jaren-db` loads model and migration MODULES beside JSON, plans
212
+ from the committed `model.snapshot.json`, refuses a module that is
213
+ not pure, and `jaren-db check` fails CI on a model that moved without
214
+ a plan (MIGRATION-FORMAT §11).
133
215
  - **The safe profile.** Untrusted query documents run under composed
134
216
  bounds: engine limits on the residual, a mandatory row bound that
135
217
  refuses rather than truncates, reference allow-lists, optional
@@ -239,16 +321,16 @@ every write (LIVE-FORMAT §7 states the cost). An ordering by
239
321
  `$distance` or a spatial aggregate re-runs on invalidation with the
240
322
  reason in `live.mode` — declared, never silent.
241
323
 
242
- **The numbers, the loss included.** `benchmark/spatial.js` stores <!--bm:spatial.corpus-->50,000 points<!--/bm-->
243
- over the Netherlands and probes one box at <!--bm:spatial.rows-->258 of 50,000 (0.5 %)<!--/bm--> selectivity,
324
+ **The numbers, the loss included.** `benchmark/spatial.js` stores <!--fact:spatial.corpus-->50,000 points<!--/fact-->
325
+ over the Netherlands and probes one box at <!--fact:spatial.rows-->258 of 50,000 (0.5 %)<!--/fact--> selectivity,
244
326
  asserting every plan case of the committed spatial corpus and every timed shape against the JavaScript
245
- engine before a single timing is printed. The `$within` a consumer writes went from <!--bm:spatial.scan-->80 ms<!--/bm-->
246
- as a full scan to <!--bm:spatial.within-->2 ms<!--/bm--> over the `bbox` index (<!--bm:spatial.scanVsIndexed-->40.0<!--/bm-->×);
247
- `$bbox-intersects` is <!--bm:spatial.bboxIntersects-->1.9 ms<!--/bm-->, a bounded `$distance` <!--bm:spatial.distance-->1.5 ms<!--/bm-->;
248
- one geohash cell answers in <!--bm:spatial.cellOne-->0.0068 ms for 0 row(s)<!--/bm--> and the honest nine-cell
249
- probe in <!--bm:spatial.cellNine-->0.026 ms for 2 row(s)<!--/bm-->. The row the store had to win is the same
250
- `$within` in the in-memory engine over the parsed array, no database at all: <!--bm:spatial.engine-->32 ms<!--/bm-->.
251
- The indexed store is now <!--bm:spatial.engineVsIndexed-->16.1× faster than<!--/bm--> it — but the un-indexed scan
327
+ engine before a single timing is printed. The `$within` a consumer writes went from <!--fact:spatial.scan-->80 ms<!--/fact-->
328
+ as a full scan to <!--fact:spatial.within-->2 ms<!--/fact--> over the `bbox` index (<!--fact:spatial.scanVsIndexed-->40.0<!--/fact-->×);
329
+ `$bbox-intersects` is <!--fact:spatial.bboxIntersects-->1.9 ms<!--/fact-->, a bounded `$distance` <!--fact:spatial.distance-->1.5 ms<!--/fact-->;
330
+ one geohash cell answers in <!--fact:spatial.cellOne-->0.0068 ms for 0 row(s)<!--/fact--> and the honest nine-cell
331
+ probe in <!--fact:spatial.cellNine-->0.026 ms for 2 row(s)<!--/fact-->. The row the store had to win is the same
332
+ `$within` in the in-memory engine over the parsed array, no database at all: <!--fact:spatial.engine-->32 ms<!--/fact-->.
333
+ The indexed store is now <!--fact:spatial.engineVsIndexed-->16.1× faster than<!--/fact--> it — but the un-indexed scan
252
334
  is not, and the comparison is not an even one either way: the engine starts from parsed objects where the
253
335
  store starts from bytes on a page and pays JSON materialisation for every row it returns. Both rows stay
254
336
  published. The deterministic-UDF hatch takes a literal `$within` on a collection with no derived index, and
@@ -263,10 +345,10 @@ columns (MODEL-FORMAT §2.1). The logical model is unchanged — the
263
345
  spatial corpus runs every entry under both mappings, in all three
264
346
  executors, with no special-cased entry — and the pushed conjunct becomes
265
347
  a `rowid` subquery over the virtual table. Through the store the same
266
- `$within` measures <!--bm:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/bm-->; loading the same rows
267
- costs <!--bm:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/bm-->, because the R\*Tree is a
348
+ `$within` measures <!--fact:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/fact-->; loading the same rows
349
+ costs <!--fact:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/fact-->, because the R\*Tree is a
268
350
  second table written inside every write transaction. Isolated from the
269
- store on a raw connection the probe is <!--bm:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/bm-->.
351
+ store on a raw connection the probe is <!--fact:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/fact-->.
270
352
  Both halves are published because both are the price. One honest
271
353
  difference comes with it: an R\*Tree stores 32-bit floats rounded
272
354
  outward, so its box is a superset and `$bbox-intersects` is refined
@@ -389,11 +471,11 @@ wrong width is refused at plan time and `explain()` says why; an
389
471
  that count, and a consumer who binds probes from a model should watch it.
390
472
 
391
473
  **The numbers, the losses included.** `benchmark/vector.js` measures one
392
- k-nearest query every physical way it can run — over <!--bm:vector.grid-->10,000 and 50,000 vectors at 384 and 768 dimensions, k = 10, the median of 10 probes<!--/bm--> —
474
+ k-nearest query every physical way it can run — over <!--fact:vector.grid-->10,000 and 50,000 vectors at 384 and 768 dimensions, k = 10, the median of 10 probes<!--/fact--> —
393
475
  and asserts that every path returns the identical top-k, ids and order,
394
476
  on every probe before a single timing prints. The flagship row is the
395
- plan a consumer's own document runs, which measures <!--bm:vector.plan-->206 ms at 50,000 × 768<!--/bm-->:
396
- <!--bm:vector.table-->
477
+ plan a consumer's own document runs, which measures <!--fact:vector.plan-->206 ms at 50,000 × 768<!--/fact-->:
478
+ <!--fact:vector.table-->
397
479
  | path (ms) | 10,000 × 384 | 10,000 × 768 | 50,000 × 384 | 50,000 × 768 |
398
480
  |---|---:|---:|---:|---:|
399
481
  | engine resident sweep (no database) | 3.3 | 6.7 | 17 | 32 |
@@ -402,14 +484,14 @@ plan a consumer's own document runs, which measures <!--bm:vector.plan-->206 ms
402
484
  | `ORDER BY` over a registered function | 18 | 31 | 121 | 180 |
403
485
  | JSON-doc sweep (no vector column) | 249 | 501 | — | — |
404
486
  | sqlite-vec | 3.6 | 7.5 | 18 | 38 |
405
- <!--/bm-->
487
+ <!--/fact-->
406
488
 
407
489
  The row the column exists to beat is the last one that has no column: the
408
490
  same query document over a collection that stores the embedding only
409
- inside the document costs <!--bm:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/bm-->,
491
+ inside the document costs <!--fact:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/fact-->,
410
492
  because every row's vector is parsed out of JSON before it can be
411
493
  compared. The row the store **cannot** beat is the one with no database
412
- in it: the same top-k over a resident `Float32Array` is <!--bm:vector.resident-->32 ms, which the plan is 6.4× slower than<!--/bm-->.
494
+ in it: the same top-k over a resident `Float32Array` is <!--fact:vector.resident-->32 ms, which the plan is 6.4× slower than<!--/fact-->.
413
495
  That comparison is not an even one and the direction is the point — the
414
496
  sweep starts from decoded floats in RAM and pays nothing for durability,
415
497
  for filters that compose with the ranking, or for a process that can
@@ -418,16 +500,16 @@ price should be able to say what the price is.
418
500
 
419
501
  **Both halves of the price.** The column costs on the way in as well as
420
502
  saving on the way out: writing the same documents with
421
- the index costs <!--bm:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/bm-->,
503
+ the index costs <!--fact:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/fact-->,
422
504
  because every write pays a JSON round trip of the member plus a
423
- normalize and a pack. On disk one vector is <!--bm:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/bm--> —
505
+ normalize and a pack. On disk one vector is <!--fact:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/fact--> —
424
506
  smaller, but *added*, since the document still carries the member the
425
507
  column is derived from.
426
508
 
427
509
  **Pushing the rank into SQL, re-measured.** A registered similarity
428
510
  function inside an `ORDER BY … LIMIT k` is the obvious alternative, and
429
511
  the suite measures it against the real column with the probe hoisted out
430
- of the per-row call: <!--bm:vector.udf-->180 ms against 202 ms at 50,000 × 768, and 0.87–1.00× the fetch-and-rank across the grid — rough parity on speed<!--/bm-->.
512
+ of the per-row call: <!--fact:vector.udf-->180 ms against 202 ms at 50,000 × 768, and 0.87–1.00× the fetch-and-rank across the grid — rough parity on speed<!--/fact-->.
431
513
  The plan does not emit it, and after that measurement the reasons are not
432
514
  speed: `bun` has no user-function API, so a plan that needed one would
433
515
  exclude an executor outright; and an ordering decided in SQL cannot break
@@ -436,15 +518,15 @@ executors have to agree on.
436
518
 
437
519
  **The rival, and the ceiling.** `sqlite-vec` is the extension built for
438
520
  exactly this, and it is measured rather than described: it answers the
439
- same probes in <!--bm:vector.rival-->38 ms against 206 ms at 50,000 × 768 — 5.4× in sqlite-vec's favour, out of a database 6.6× smaller that holds no documents<!--/bm-->,
440
- over <!--bm:vector.agreement-->40 probes, no disagreements<!--/bm-->. It is a
521
+ same probes in <!--fact:vector.rival-->38 ms against 206 ms at 50,000 × 768 — 5.4× in sqlite-vec's favour, out of a database 6.6× smaller that holds no documents<!--/fact-->,
522
+ over <!--fact:vector.agreement-->40 probes, no disagreements<!--/fact-->. It is a
441
523
  loadable native extension, which is the one thing this store will not
442
524
  require — it would exclude the wasm tab and stock `bun`, half the
443
525
  execution story — so the comparison is published as what it is: a faster
444
526
  engine you may prefer, and a dependency this one does not take. What
445
527
  neither of them is, is an approximate index. Exact brute force is linear
446
528
  in `n · d`, and the suite states the envelope as arithmetic rather than
447
- opinion: <!--bm:vector.ceiling-->5.546 ns per vector component — one query reaches 100 ms at about 22,000 vectors of 768 dimensions and one second at about 234,000<!--/bm-->.
529
+ opinion: <!--fact:vector.ceiling-->5.546 ns per vector component — one query reaches 100 ms at about 22,000 vectors of 768 dimensions and one second at about 234,000<!--/fact-->.
448
530
  Past that this design is the wrong tool and no margin changes it; what
449
531
  lies beyond is an approximate index, and this store does not have one.
450
532
 
@@ -501,13 +583,13 @@ same range, the same buckets, the same rolling window and the same
501
583
  as-of join over one seeded corpus by plain references, by the temporal
502
584
  kernel, by a generic query document, by hand-written SQL and by the
503
585
  store — every route checked against the others before a timing is
504
- taken. At <!--bm:series.corpus-->100,000 samples at 1-second spacing, Node v24.19.0<!--/bm-->,
505
- the store is measured three ways at once — <!--bm:series.storeShapes-->the planned range costs 4.2× the hand-written statement and 1535.5× the resident cut, and the pushed bucket ladder 2.5× the hand-written GROUP BY, 1.6× FASTER than the generic query route, and 155.1× the one-pass loop<!--/bm-->.
586
+ taken. At <!--fact:series.corpus-->100,000 samples at 1-second spacing, Node v24.19.0<!--/fact-->,
587
+ the store is measured three ways at once — <!--fact:series.storeShapes-->the planned range costs 4.2× the hand-written statement and 1535.5× the resident cut, and the pushed bucket ladder 2.5× the hand-written GROUP BY, 1.6× FASTER than the generic query route, and 155.1× the one-pass loop<!--/fact-->.
506
588
  The range row is not the planner's price: the statement selects two
507
589
  COLUMNS where the store renders and parses a whole JSON document per
508
590
  row, which is what storing documents costs.
509
591
 
510
- And what a refinement costs, with the loss in it: <!--bm:series.storeRefinement-->A window measured in time is not pushed: the store answers it at 18.5× the kernel over an array already in memory, over 100,000 candidates the index bounded. The batched as-of join reads 99,129 rows in 1 statement and costs 1424.3× fifty-one separate index reads — a bound is what it buys, not a speed-up, and without a tolerance a backward join can only be bounded above.<!--/bm-->
592
+ And what a refinement costs, with the loss in it: <!--fact:series.storeRefinement-->A window measured in time is not pushed: the store answers it at 18.5× the kernel over an array already in memory, over 100,000 candidates the index bounded. The batched as-of join reads 99,129 rows in 1 statement and costs 1424.3× fifty-one separate index reads — a bound is what it buys, not a speed-up, and without a tolerance a backward join can only be bounded above.<!--/fact-->
511
593
 
512
594
  **A refinement is named, never quiet.** `explain().series` reports
513
595
  `mode` — `native`, `hybrid` or `engine` — the declared index the fetch
@@ -515,7 +597,8 @@ seeks through, the instant bounds it used, which kernel finished the
515
597
  answer, and a reason code for every thing the database could not do:
516
598
  `fill-policy`, `calendar-width`, `named-zone`, `rolling-refinement`,
517
599
  `asof-refinement`, `unsupported-aggregate`, `nonliteral-spec`,
518
- `instant-not-integer`, `missing-series-prefix`. `strict: true` refuses
600
+ `instant-not-integer`, `missing-series-prefix`, `row-selector`,
601
+ `value-not-numeric`, `nonnative-grouping`, `invalid-spec`. `strict: true` refuses
519
602
  every one of them before a statement runs, and the counts `explain()`
520
603
  prints are the LAST ACTUAL execution's — `null` until the document has
521
604
  run, because an estimate wearing a count's name is worse than no
@@ -575,6 +658,18 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
575
658
  `typedStore` (from `@jarenjs/db/typed`) types every read, checks
576
659
  every write, and widens `load` results by their include
577
660
  specification.
661
+ - **Membership** (§11.7): `link(own, member, target)` and `unlink` attach
662
+ and detach one many-to-many membership at a time through the unit of
663
+ work — written against the join table as it stands at save time, so a
664
+ repeated save changes nothing.
665
+ - **The front door**: `@jarenjs/linq/db` opens this store behind a
666
+ client typed from the model pen — `db.entities.Post.where((p) =>
667
+ p.stars.ge(3))` is the chain over the entity set, pushed down;
668
+ `db.entities.User.include((u) => u.posts, { where: (p) => p.stars.ge(3),
669
+ take: 2 }).toArray()` emits exactly the `load` spec above and runs in
670
+ the same one statement; `link`/`unlink` reach §11.7 and `live` the
671
+ registration below. It imports this package as an optional peer; this
672
+ package never imports it.
578
673
  - **Relational migrations and the `jaren-db` CLI** (MIGRATION-FORMAT
579
674
  §§9–12): the strategy-table diff, the documented twelve-step table
580
675
  rebuild with `foreign_key_check` inside the transaction, shape
@@ -611,7 +706,12 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
611
706
  queries, the same live updates run on the official SQLite wasm build
612
707
  over the header-free OPFS SAH-pool VFS — one tab owns the
613
708
  connection, others are clients. Proven in the `#/data` studio across
614
- Chromium, Firefox and WebKit.
709
+ Chromium, Firefox and WebKit. The subpath exports the two helpers
710
+ that studio is built on: `sqlite3Handle(sqlite3, { DbClass })` builds
711
+ the injected handle from a loaded wasm module and the database class
712
+ the host picks (`sqlite3.oo1.DB` in memory, the SAH-pool
713
+ `OpfsSAHPoolDb` for OPFS), and `adaptOo1Database(sqlite3, db)` wraps
714
+ an oo1 database the host already opened.
615
715
 
616
716
  ### What an event-time view costs
617
717
 
@@ -623,16 +723,16 @@ before a single timing is printed — a fast live view with the wrong
623
723
  answer is not a fast live view — and the run exits non-zero if they
624
724
  disagree.
625
725
 
626
- <!--bm:live.eventTimeTable-->
726
+ <!--fact:live.eventTimeTable-->
627
727
  | view | maintained | re-run | ratio |
628
728
  |---|---:|---:|---:|
629
729
  | bucket (60 s ladder, mean), 1000 rows | 119 µs | 1.09 ms | 9.2× |
630
730
  | rolling (5 min window, mean), 1000 rows | 407 µs | 3.86 ms | 9.5× |
631
731
  | bucket (60 s ladder, mean), 10000 rows | 136 µs | 10.9 ms | 80.2× |
632
732
  | rolling (5 min window, mean), 10000 rows | 13.7 ms | 62.7 ms | 4.6× |
633
- <!--/bm-->
733
+ <!--/fact-->
634
734
 
635
- The gain is <!--bm:live.eventTimeBand-->80.2× for the bucket and 4.6× for the rolling at 10,000 readings<!--/bm-->. A bucket
735
+ The gain is <!--fact:live.eventTimeBand-->80.2× for the bucket and 4.6× for the rolling at 10,000 readings<!--/fact-->. A bucket
636
736
  view is nearly flat in the series length, because a write folds one
637
737
  bucket again and the rest of the ladder is untouched. A rolling view is
638
738
  not, and the table says so: its answer is one row per reading, so the
@@ -677,10 +777,10 @@ replication on these primitives is a roadmap item, not a hint.
677
777
  - **The wasm build journals** (its session extension is not yet
678
778
  adapted); OPFS needs a secure context, and where it is absent the
679
779
  store runs in memory with the durability difference stated.
680
- - **Named future work, not silent gaps**: `$groupby` pushdown,
681
- relation-name query sugar, a many-to-many membership API, incremental
682
- joins, other SQL dialects, replication, database introspection
683
- (MODEL-FORMAT §10.6, the roadmap).
780
+ - **Named future work, not silent gaps**: `$groupby` pushdown beyond
781
+ the `$time-bucket` ladder, a many-to-many hop on the chain, membership
782
+ on an auto-keyed pending insert, incremental joins, other SQL dialects,
783
+ replication, database introspection (MODEL-FORMAT §10.6, the roadmap).
684
784
 
685
785
  The normative formats are
686
786
  [docs/MODEL-FORMAT.md](docs/MODEL-FORMAT.md) (storage §§1–7, safe
@@ -49,7 +49,10 @@ restarting.
49
49
  `_jaren_job_checkpoints` holds `(run_id, node_id, value)` rows — the
50
50
  flow checkpoint store of §7, keyed by the JOB id (the run id IS the
51
51
  job id). Both tables are created on open when `jobs` is requested;
52
- neither appears in the model.
52
+ neither appears in the model. `enqueue`'s options are validated as
53
+ they are stored: `runAt` must be a finite epoch in milliseconds and
54
+ `maxAttempts` a positive integer (`TypeError`) — a `NaN` eligibility
55
+ was once stored, and that job was pending forever.
53
56
 
54
57
  ## 3. Leasing and exactly-once execution
55
58
 
@@ -113,22 +116,31 @@ reclaimed — size `leaseMs` to the slowest honest handler.
113
116
  ## 6. Workers and concurrency
114
117
 
115
118
  `createWorker({ handlers, concurrency, pollInterval, leaseMs, owner,
116
- maxAttempts, backoff, random })` returns `{ start(), stop(),
119
+ backoffBase, backoffCap, stopGraceMs })` returns `{ start(), stop(),
117
120
  stats() }`:
118
121
 
119
- - `concurrency` (default 1) independent claim-execute loops share one
120
- worker registration;
122
+ - `concurrency` (default 1, a positive integer `TypeError`
123
+ otherwise, because a worker with zero loops would `start()` and never
124
+ claim) independent claim-execute loops share one worker registration;
121
125
  - an idle loop sleeps `pollInterval` (default 500 ms — at most
122
126
  2 claims/s of idle cost per loop, stated). An `enqueue` on the SAME
123
127
  store wakes every idle local loop immediately, so same-process
124
128
  latency is not poll-bound; **cross-process wake-up is polling**,
125
129
  plainly (§8);
130
+ - retry backoff is `min(backoffCap, backoffBase × 2^(attempts − 1))`,
131
+ jittered to between half and all of itself (defaults 1 s and 60 s);
132
+ `maxAttempts` belongs to the JOB (`enqueue(kind, payload,
133
+ { maxAttempts })`, default 5), not to the worker;
126
134
  - `stop({ graceMs })` aborts in-flight handlers and resolves once they
127
- settle **or** the grace period expires (default 5 s), answering
135
+ settle **or** the grace period expires (default `stopGraceMs`, 5 s),
136
+ answering
128
137
  `{ drained, inFlight }`; a loop the grace period could not drain is
129
138
  **cancelled**, not left running — see §6.1;
130
- - `stats()` reports claims, completions, failures, wakes, polls and the
131
- in-flight handler count.
139
+ - `stats()` reports claims, completions, failures, wakes, polls, the
140
+ in-flight handler count and `claimErrors` — a claim statement the
141
+ database refused (a read-only file, a closed store), counted rather
142
+ than swallowed, so a worker that can never claim is visible instead
143
+ of silently idle.
132
144
 
133
145
  ### 6.1 A handler cannot break the loop, and cannot hold shutdown
134
146
 
@@ -224,6 +236,10 @@ await store.jobs.enqueue('sync-report', { input: { day: '2026-08-05' } });
224
236
  This format adds NO codes. API misuse (a malformed handler map, a
225
237
  non-string kind, a worker started twice) is a `TypeError` at the
226
238
  call, matching the capture and live precedents; storage failures ride
227
- the existing `JD2005` wrap; a job's own failure is DATA — recorded in
239
+ the existing `JD2005` wrap and a coded error passes through it
240
+ unchanged (`JD2063` after `close()`); `store.jobs` on a read-only
241
+ store is `JD0002` at first use, because the queue tables cannot be
242
+ created — named there, not a raw `SQLITE_READONLY` at the first
243
+ enqueue; a job's own failure is DATA — recorded in
228
244
  `last_error` and the state machine of §4 — because a queue that
229
245
  throws away its failure story has failed twice.
@@ -92,7 +92,10 @@ them).
92
92
  already knows what it wrote. **Less complete, stated plainly**: it
93
93
  cannot see writes made through raw SQL, triggers, or another
94
94
  connection; a journal-mode delete of a row the store never read
95
- emits its `remove` without having seen the old document; and of the
95
+ emits its `remove` without having seen the old document; a keyed
96
+ `put` reads the stored document first, so it emits the `replace`
97
+ session mode emits and a `put` that changes nothing emits nothing;
98
+ and of the
96
99
  database's own `ON DELETE` side effects it reconstructs exactly ONE
97
100
  — join-table membership dying with its entity (read before the
98
101
  delete) — while cascades into CHILD rows (`onDelete: 'cascade'` /
@@ -114,8 +117,14 @@ reads forward:
114
117
  const records = await store.changesSince(lastSeq); // JD2051 when no log
115
118
  ```
116
119
 
117
- `seq` is monotonic; with the log enabled it continues across reopens
118
- (seeded from `MAX(seq)`), without it it is per-process. Retention is
120
+ `seq` is monotonic; with the log enabled the DATABASE allocates it —
121
+ each record's `seq` is `MAX(seq) + 1` computed inside the insert
122
+ statement and read back through `RETURNING` — so two stores over one
123
+ file never collide on the log's key and each sees the other's
124
+ sequence continue; without the log it is per-process. `changesSince`
125
+ answers records in the shape observers receive, `collections`
126
+ included; a cursor that is not a number is a `TypeError`, as is a
127
+ `retention` that is not a positive integer. Retention is
119
128
  a bounded count (`retention`, default 1000): older rows are pruned in
120
129
  the same transaction. The log is an ordered, replayable stream —
121
130
  which is what makes a late-joining consumer possible. **Replication
@@ -161,6 +170,14 @@ multi-entity shape of MODEL-FORMAT §10) the same way. Live queries
161
170
  REQUIRE change capture — the patch stream is the invalidation source —
162
171
  and registering on a store opened without `capture` is `JD0050`.
163
172
 
173
+ A producer may hand a registration a CHAIN instead of a document: the
174
+ `@jarenjs/linq/db` client's `live(chain, options)` passes the chain's
175
+ `toDocument()` and its `explain().bindings` as the externals to exactly
176
+ these two registrations (`store.live` for an entity-root chain,
177
+ `collection.live` for a collection's), so the strategy, the reason and
178
+ the maintenance are this table's — an entity chain re-runs, declared —
179
+ and this document stays the only place they are decided.
180
+
164
181
  **This table is normative.** Every row is implemented and tested;
165
182
  nothing outside it is attempted. Classification reads the compiled
166
183
  PLAN (never the raw document), so "extractable" below means exactly
@@ -260,7 +277,11 @@ state. `externals` are fixed at registration — a query whose inputs
260
277
  change is a new registration.
261
278
 
262
279
  Maintenance runs synchronously inside patch delivery, in commit
263
- order, on the store's own connection. Writes from ANOTHER connection
280
+ order, on the store's own connection. Delivery is never re-entered: a
281
+ write made from inside an observer or a subscriber commits at once,
282
+ but its record is queued and delivered after the current record has
283
+ reached every consumer, so sibling live views see commits in commit
284
+ order rather than in call-stack order. Writes from ANOTHER connection
264
285
  are invisible to capture (§6) and therefore to live queries; the
265
286
  coarse `dataVersion()` signal and the §11 topology are the honest
266
287
  answers, and re-registering re-reads.
@@ -375,9 +396,11 @@ ERRORING rather than degrading (the D14 rule — the bound is printed):
375
396
 
376
397
  Non-claims, in one place: no incremental joins (re-run is the declared
377
398
  strategy), no cross-connection invalidation (§6's `data_version` is
378
- the signal), no maintenance over asynchronous connections in this
379
- version (every current driver is synchronous; the browser driver's
380
- order owns that story), no replication, and no ordering guarantee for
399
+ the signal), no maintenance over asynchronous connections
400
+ `capabilities.live` is `false` there and a registration is `JD0051`
401
+ naming the reason, because maintenance point-reads rows synchronously
402
+ inside delivery (the wasm driver's oo1 API is synchronous, which is
403
+ why the browser has live queries at all) — no replication, and no ordering guarantee for
381
404
  unordered queries beyond §9's determinism.
382
405
 
383
406
  ## 13. Event time
@@ -404,7 +427,8 @@ live.stats().watermark; // what it is now
404
427
  `eventTime` is a **closed** member set: `path`, `watermark`,
405
428
  `allowedLateness` (default 0) and `retention`. Anything else — a
406
429
  misspelling, a non-finite epoch, a negative lateness, a `path` that is
407
- not a singular row selector — is `JD0053` at registration, not a member
430
+ not a singular row selector — is `JD0053` at registration (its
431
+ `docPath` names the collection, `/collections/<name>`), not a member
408
432
  quietly ignored. `advance()` refuses a value that is not finite or that
409
433
  goes backwards (a `TypeError`), and it is absent on every view
410
434
  registered without an `eventTime`. An entity document has no collection
@@ -415,7 +439,12 @@ to place rows in and re-runs, so an `eventTime` on `store.live` is
415
439
 
416
440
  Two documents, and only these two shapes: `$resample` and `$rolling`
417
441
  whose series operand is the collection (`"$[*]"`, or a FLWOR over it
418
- whose `$where` narrows and whose `$return` is the bare binding).
442
+ whose `$where` narrows and whose `$return` is the bare binding). A
443
+ spec that spells `at` and `value` explicitly — the spelling the query
444
+ language accepts — is maintained: the view folds with the kernel
445
+ reading the declared instant member and the `value` member the spec
446
+ names, and a `value` selector the view cannot follow re-runs with the
447
+ reason named, never a maintained view that dies on its first fold.
419
448
 
420
449
  - **A bucket view keeps its rows by bucket.** A write touches one bucket
421
450
  — two, when it moves a reading across a boundary — and exactly those