@jarenjs/db 0.49.2 → 0.56.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +27 -15
- package/README.md +141 -41
- package/docs/JOBS-FORMAT.md +24 -8
- package/docs/LIVE-FORMAT.md +38 -9
- package/docs/MIGRATION-FORMAT.md +118 -36
- package/docs/MODEL-FORMAT.md +232 -30
- package/package.json +4 -5
- package/schemas/jaren-migration.draft-07.schema.json +73 -0
- package/schemas/jaren-migration.schema.json +73 -0
- package/src/capture.js +66 -28
- package/src/cli.js +225 -44
- package/src/ddl.js +23 -3
- package/src/dialects/sqlite.js +2 -1
- package/src/driver.js +63 -16
- package/src/drivers/wasm.js +1 -0
- package/src/emit-model.js +14 -0
- package/src/emit.js +10 -3
- package/src/entity.js +92 -47
- package/src/errors.js +25 -0
- package/src/index.js +2 -2
- package/src/jobs.js +40 -5
- package/src/live-time.js +12 -3
- package/src/live.js +11 -1
- package/src/migrate.js +397 -191
- package/src/model.js +173 -8
- package/src/plan.js +135 -38
- package/src/query.js +138 -13
- package/src/store.js +221 -66
- package/src/tracker.js +173 -48
- package/types/index.d.ts +152 -10
- package/types/node.d.ts +3 -1
- package/types/typed.d.ts +58 -2
- package/types/wasm.d.ts +7 -0
- package/dist/types/algebra.d.ts +0 -230
- package/dist/types/app.d.ts +0 -49
- package/dist/types/capture.d.ts +0 -85
- package/dist/types/cli.d.ts +0 -2
- package/dist/types/dag-job.d.ts +0 -40
- package/dist/types/ddl.d.ts +0 -229
- package/dist/types/derive.d.ts +0 -250
- package/dist/types/dialect.d.ts +0 -154
- package/dist/types/dialects/sqlite.d.ts +0 -9
- package/dist/types/driver.d.ts +0 -110
- package/dist/types/drivers/bun.d.ts +0 -47
- package/dist/types/drivers/node.d.ts +0 -37
- package/dist/types/drivers/wasm.d.ts +0 -65
- package/dist/types/emit-model.d.ts +0 -44
- package/dist/types/emit.d.ts +0 -75
- package/dist/types/entity.d.ts +0 -23
- package/dist/types/errors.d.ts +0 -170
- package/dist/types/graph.d.ts +0 -28
- package/dist/types/index.d.ts +0 -37
- package/dist/types/jobs.d.ts +0 -140
- package/dist/types/knn.d.ts +0 -69
- package/dist/types/live-time.d.ts +0 -141
- package/dist/types/live.d.ts +0 -64
- package/dist/types/migrate.d.ts +0 -170
- package/dist/types/model.d.ts +0 -36
- package/dist/types/patch-sql.d.ts +0 -37
- package/dist/types/plan.d.ts +0 -142
- package/dist/types/profile.d.ts +0 -80
- package/dist/types/query.d.ts +0 -112
- package/dist/types/residual.d.ts +0 -64
- package/dist/types/series.d.ts +0 -227
- package/dist/types/store.d.ts +0 -60
- package/dist/types/tracker.d.ts +0 -43
- package/dist/types/typed.d.ts +0 -15
- package/dist/types/types.d.ts +0 -26
- package/dist/types/udf.d.ts +0 -75
- package/dist/types/window.d.ts +0 -52
package/ARCHITECTURE.md
CHANGED
|
@@ -247,14 +247,21 @@ index-usable spelling and read every row of the collection.
|
|
|
247
247
|
|
|
248
248
|
The guards make the forms sound for typed AND untyped paths alike —
|
|
249
249
|
the schema type's job is choosing the generated COLUMN (the index),
|
|
250
|
-
never weakening the guard. Two documented preconditions
|
|
251
|
-
operators, aggregates
|
|
252
|
-
paths
|
|
253
|
-
|
|
254
|
-
non-string under a string operator —
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
250
|
+
never weakening the guard. Two documented preconditions. String
|
|
251
|
+
operators, aggregates and ordering are only promoted on schema-typed
|
|
252
|
+
paths that cannot hold `null` and are not boolean, because the engine
|
|
253
|
+
ERRORS on non-conforming operands where SQL would coerce or sort — a
|
|
254
|
+
`null` under an ordered key, a non-string under a string operator —
|
|
255
|
+
so a path whose schema admits `null` is a named residual for those
|
|
256
|
+
forms and the two engines keep answering alike over conforming data.
|
|
257
|
+
And the guards protect the JSON TYPE, not the VALUE: a typed
|
|
258
|
+
generated column carries SQLite affinity, so on an unvalidated store a
|
|
259
|
+
row that violates the schema — the string `'020'` under an `integer`
|
|
260
|
+
path — reads as the integer `20` in the column while `json_type` still
|
|
261
|
+
says text, and a pushed comparison answers rows the engine (comparing
|
|
262
|
+
the JSON value) does not. Keep `compileSchema` injected: with the hook
|
|
263
|
+
no such row is ever stored, and without it the pushdown is exact only
|
|
264
|
+
over documents that happen to conform.
|
|
258
265
|
|
|
259
266
|
**Bind-time diversion.** SQLite cannot bind a boolean, and a
|
|
260
267
|
`null`-valued external needs Jaren's null semantics, not SQL's. At
|
|
@@ -571,8 +578,9 @@ collection binding, so the planner passes that binding's **name** in:
|
|
|
571
578
|
wrapped under any other name every reference would read as an external,
|
|
572
579
|
the determinism rule would reject the fragment, and the hatch would
|
|
573
580
|
silently not engage — no error and no reason in `explain()`.
|
|
574
|
-
Registration is keyed by `
|
|
575
|
-
|
|
581
|
+
Registration is keyed by `semanticKey(fragment)` (the
|
|
582
|
+
order-insensitive identity from `@jarenjs/core/object`) so identical
|
|
583
|
+
fragments share one registration, and the planner MUST
|
|
576
584
|
produce a correct plan with the capability disabled (tested that way).
|
|
577
585
|
Preference order: native SQL → deterministic function → residual, and
|
|
578
586
|
`explain()` names the choice. **Index-form UDFs are deliberately not
|
|
@@ -585,11 +593,15 @@ UDF-expression indexes.
|
|
|
585
593
|
### The statement cache
|
|
586
594
|
|
|
587
595
|
A caller of the core primitives, not an eighth implementation:
|
|
588
|
-
`
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
596
|
+
`createSemanticCache` keyed by the whole discriminating tuple — the
|
|
597
|
+
document plus collection, dialect, strictness, the pushdown switch and
|
|
598
|
+
the profile — where the identity is the tuple's COMPLETE
|
|
599
|
+
serialization, never a fingerprint of it: a 32-bit content hash
|
|
600
|
+
collides after tens of thousands of documents, and a collision here
|
|
601
|
+
answers one query with another query's plan and rows (the
|
|
602
|
+
cache-identity test exists to keep that key abolished).
|
|
603
|
+
`store.stats()` exposes hits, misses and evictions so the cache is
|
|
604
|
+
proven rather than assumed.
|
|
593
605
|
`store.stats()` exposes hits, misses and evictions, so the cache is
|
|
594
606
|
proven rather than assumed.
|
|
595
607
|
|
package/README.md
CHANGED
|
@@ -48,7 +48,8 @@ const store = await openStore({
|
|
|
48
48
|
const users = store.collection('users');
|
|
49
49
|
await users.insert({ id: 'u1', email: 'ada@example.test', age: 36 });
|
|
50
50
|
|
|
51
|
-
// a query document — here written by hand; linq writes the same
|
|
51
|
+
// a query document — here written by hand; linq writes the same one,
|
|
52
|
+
// its source wrapped as ['$[*]'] so an item that is an array stays one item
|
|
52
53
|
const adults = await users.execute({
|
|
53
54
|
$for: { it: '$[*]' },
|
|
54
55
|
$where: { $ge: ['$it.age', 21] },
|
|
@@ -56,6 +57,78 @@ const adults = await users.execute({
|
|
|
56
57
|
});
|
|
57
58
|
```
|
|
58
59
|
|
|
60
|
+
Or, by code — the same document, written by the model pen
|
|
61
|
+
([MODEL-PEN.md](../linq/docs/MODEL-PEN.md)) and typed without a generate
|
|
62
|
+
step (`InferMeta<typeof model>` binds the typed store):
|
|
63
|
+
|
|
64
|
+
```js
|
|
65
|
+
import * as m from '@jarenjs/linq/model';
|
|
66
|
+
|
|
67
|
+
const model = m.defineModel({
|
|
68
|
+
collections: {
|
|
69
|
+
users: m.collection(
|
|
70
|
+
m.object({ id: m.string(), email: m.string().email(), age: m.integer().optional() }).open(),
|
|
71
|
+
{ key: (u) => u.id, indexes: [m.index((u) => u.age)] },
|
|
72
|
+
),
|
|
73
|
+
},
|
|
74
|
+
});
|
|
75
|
+
const store = await openStore(model, { driver: nodeDriver(), path: 'app.db' });
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Over an entities model, an entity set is a provider too (MODEL-FORMAT
|
|
79
|
+
§10.1): a chain binds through the set's root and the store runs the
|
|
80
|
+
document whole — the translator for a bare-binding selection or
|
|
81
|
+
equijoin, the declared residual for a projection — while the store
|
|
82
|
+
itself, serving several roots, is refused by name (`JL0007`):
|
|
83
|
+
|
|
84
|
+
```js
|
|
85
|
+
import { from, fromAsync } from '@jarenjs/linq';
|
|
86
|
+
|
|
87
|
+
const starred = from(store.sync.entity('Post')).where((p) => p.stars.ge(3));
|
|
88
|
+
starred.toDocument(); // { $for: { it: '$.Post[*]' }, $where: { $ge: ['$it.stars', 3] }, $return: '$it' }
|
|
89
|
+
starred.toArray(); // the entity translator, one statement
|
|
90
|
+
await fromAsync(store.entity('Post'))
|
|
91
|
+
.join(fromAsync(store.entity('User')), (p) => p.authorId, (u) => u.id, (p) => p)
|
|
92
|
+
.toArray(); // a two-root equijoin, one statement
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
A declared relation navigates on the chain, and the document still
|
|
96
|
+
carries no relation name: the chain reads the set's relation table
|
|
97
|
+
(`store.entity('Post').relations`, MODEL-FORMAT §10.1) and lowers
|
|
98
|
+
`p.author.email` to the correlated phrase the engine and the store both
|
|
99
|
+
run. The store answers it as the residual it is — `explain()` says so,
|
|
100
|
+
`strict` refuses it — over the two fetched roots, never a statement per
|
|
101
|
+
row:
|
|
102
|
+
|
|
103
|
+
```js
|
|
104
|
+
const byAuthor = from(store.sync.entity('Post'))
|
|
105
|
+
.where((p) => p.stars.ge(3))
|
|
106
|
+
.select((p) => ({ title: p.title, by: p.author.email }));
|
|
107
|
+
|
|
108
|
+
byAuthor.toDocument();
|
|
109
|
+
// { $for: { it: '$.Post[*]' },
|
|
110
|
+
// $where: { $ge: ['$it.stars', 3] },
|
|
111
|
+
// $return: { title: '$it.title',
|
|
112
|
+
// by: { $for: { r1: '$.User[*]' },
|
|
113
|
+
// $where: { $eq: ['$r1.id', '$it.authorId'] },
|
|
114
|
+
// $return: '$r1.email' } } }
|
|
115
|
+
byAuthor.explain().hops; // [{ member: 'author', kind: 'oneToOne', binding: 'r1' }]
|
|
116
|
+
store.sync.explain(byAuthor.toDocument());
|
|
117
|
+
// { mode: 'set', referenced: ['Post', 'User'], sql: null,
|
|
118
|
+
// reasons: [{ construct: '$return',
|
|
119
|
+
// reason: 'entity queries return one bare binding natively; projections run in the engine' }], … }
|
|
120
|
+
byAuthor.toArray(); // the rows, two fetches — one per referenced root
|
|
121
|
+
|
|
122
|
+
from(store.sync.entity('User')).where((u) => u.posts.all().count().ge(2));
|
|
123
|
+
// … $where: { $ge: [{ $count: { $for: { r1: '$.Post[*]' },
|
|
124
|
+
// $where: { $eq: ['$r1.authorId', '$it.id'] },
|
|
125
|
+
// $return: '$r1' } }, 2] } …
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
A many-to-many member (`u.labels`) is refused at build time (`JL0105`)
|
|
129
|
+
naming the join table: it is not a queryable root in this version, so
|
|
130
|
+
`load({ include: { labels: true } })` is how the memberships are read.
|
|
131
|
+
|
|
59
132
|
`execute` answers in the ENGINE's result shape (QUERY-FORMAT §1,
|
|
60
133
|
"singleton ≡ item"): `undefined` for no rows, the document itself for
|
|
61
134
|
exactly one, an array for more — typed `SequenceResult<R>`, with `R`
|
|
@@ -71,9 +144,12 @@ shape binds at `store.collection<User>('users')`.
|
|
|
71
144
|
equivalent runs as a real compiled Jaren query (the residual), and
|
|
72
145
|
`explain()` always says which is which — the SQL, the bound
|
|
73
146
|
parameters, the indexes used (verified against the database's own
|
|
74
|
-
plan output), and the residual's named reasons. A
|
|
75
|
-
|
|
76
|
-
|
|
147
|
+
plan output), and the residual's named reasons. A differential
|
|
148
|
+
oracle — a committed corpus and a seeded generator, every case run
|
|
149
|
+
in both modes — keeps both paths agreeing, with the one arithmetic
|
|
150
|
+
deviation declared rather than hidden (MODEL-FORMAT §10.6: SQLite's
|
|
151
|
+
compensated `SUM` and the engine's naive one differ in the last
|
|
152
|
+
bit). `strict: true` turns any residual into a compile error.
|
|
77
153
|
- **Registered operators, correct in the residual, pushed where it
|
|
78
154
|
pays.** Open with a registry (`operators:
|
|
79
155
|
createJsltRegistry().use(mathPack).use(financePack)`) and a query may
|
|
@@ -129,7 +205,13 @@ shape binds at `store.collection<User>('users')`.
|
|
|
129
205
|
replays the whole chain before the real store is touched; a
|
|
130
206
|
checksummed history refuses edited or reordered migrations; a
|
|
131
207
|
narrowing without an adequate transform is refused against the REAL
|
|
132
|
-
data, inside the transaction.
|
|
208
|
+
data, inside the transaction. And by code: `@jarenjs/linq/migration`
|
|
209
|
+
([MIGRATION-PEN.md](../linq/docs/MIGRATION-PEN.md))
|
|
210
|
+
writes the same document with the data transform typed old row → new
|
|
211
|
+
row, `jaren-db` loads model and migration MODULES beside JSON, plans
|
|
212
|
+
from the committed `model.snapshot.json`, refuses a module that is
|
|
213
|
+
not pure, and `jaren-db check` fails CI on a model that moved without
|
|
214
|
+
a plan (MIGRATION-FORMAT §11).
|
|
133
215
|
- **The safe profile.** Untrusted query documents run under composed
|
|
134
216
|
bounds: engine limits on the residual, a mandatory row bound that
|
|
135
217
|
refuses rather than truncates, reference allow-lists, optional
|
|
@@ -239,16 +321,16 @@ every write (LIVE-FORMAT §7 states the cost). An ordering by
|
|
|
239
321
|
`$distance` or a spatial aggregate re-runs on invalidation with the
|
|
240
322
|
reason in `live.mode` — declared, never silent.
|
|
241
323
|
|
|
242
|
-
**The numbers, the loss included.** `benchmark/spatial.js` stores <!--
|
|
243
|
-
over the Netherlands and probes one box at <!--
|
|
324
|
+
**The numbers, the loss included.** `benchmark/spatial.js` stores <!--fact:spatial.corpus-->50,000 points<!--/fact-->
|
|
325
|
+
over the Netherlands and probes one box at <!--fact:spatial.rows-->258 of 50,000 (0.5 %)<!--/fact--> selectivity,
|
|
244
326
|
asserting every plan case of the committed spatial corpus and every timed shape against the JavaScript
|
|
245
|
-
engine before a single timing is printed. The `$within` a consumer writes went from <!--
|
|
246
|
-
as a full scan to <!--
|
|
247
|
-
`$bbox-intersects` is <!--
|
|
248
|
-
one geohash cell answers in <!--
|
|
249
|
-
probe in <!--
|
|
250
|
-
`$within` in the in-memory engine over the parsed array, no database at all: <!--
|
|
251
|
-
The indexed store is now <!--
|
|
327
|
+
engine before a single timing is printed. The `$within` a consumer writes went from <!--fact:spatial.scan-->80 ms<!--/fact-->
|
|
328
|
+
as a full scan to <!--fact:spatial.within-->2 ms<!--/fact--> over the `bbox` index (<!--fact:spatial.scanVsIndexed-->40.0<!--/fact-->×);
|
|
329
|
+
`$bbox-intersects` is <!--fact:spatial.bboxIntersects-->1.9 ms<!--/fact-->, a bounded `$distance` <!--fact:spatial.distance-->1.5 ms<!--/fact-->;
|
|
330
|
+
one geohash cell answers in <!--fact:spatial.cellOne-->0.0068 ms for 0 row(s)<!--/fact--> and the honest nine-cell
|
|
331
|
+
probe in <!--fact:spatial.cellNine-->0.026 ms for 2 row(s)<!--/fact-->. The row the store had to win is the same
|
|
332
|
+
`$within` in the in-memory engine over the parsed array, no database at all: <!--fact:spatial.engine-->32 ms<!--/fact-->.
|
|
333
|
+
The indexed store is now <!--fact:spatial.engineVsIndexed-->16.1× faster than<!--/fact--> it — but the un-indexed scan
|
|
252
334
|
is not, and the comparison is not an even one either way: the engine starts from parsed objects where the
|
|
253
335
|
store starts from bytes on a page and pays JSON materialisation for every row it returns. Both rows stay
|
|
254
336
|
published. The deterministic-UDF hatch takes a literal `$within` on a collection with no derived index, and
|
|
@@ -263,10 +345,10 @@ columns (MODEL-FORMAT §2.1). The logical model is unchanged — the
|
|
|
263
345
|
spatial corpus runs every entry under both mappings, in all three
|
|
264
346
|
executors, with no special-cased entry — and the pushed conjunct becomes
|
|
265
347
|
a `rowid` subquery over the virtual table. Through the store the same
|
|
266
|
-
`$within` measures <!--
|
|
267
|
-
costs <!--
|
|
348
|
+
`$within` measures <!--fact:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/fact-->; loading the same rows
|
|
349
|
+
costs <!--fact:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/fact-->, because the R\*Tree is a
|
|
268
350
|
second table written inside every write transaction. Isolated from the
|
|
269
|
-
store on a raw connection the probe is <!--
|
|
351
|
+
store on a raw connection the probe is <!--fact:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/fact-->.
|
|
270
352
|
Both halves are published because both are the price. One honest
|
|
271
353
|
difference comes with it: an R\*Tree stores 32-bit floats rounded
|
|
272
354
|
outward, so its box is a superset and `$bbox-intersects` is refined
|
|
@@ -389,11 +471,11 @@ wrong width is refused at plan time and `explain()` says why; an
|
|
|
389
471
|
that count, and a consumer who binds probes from a model should watch it.
|
|
390
472
|
|
|
391
473
|
**The numbers, the losses included.** `benchmark/vector.js` measures one
|
|
392
|
-
k-nearest query every physical way it can run — over <!--
|
|
474
|
+
k-nearest query every physical way it can run — over <!--fact:vector.grid-->10,000 and 50,000 vectors at 384 and 768 dimensions, k = 10, the median of 10 probes<!--/fact--> —
|
|
393
475
|
and asserts that every path returns the identical top-k, ids and order,
|
|
394
476
|
on every probe before a single timing prints. The flagship row is the
|
|
395
|
-
plan a consumer's own document runs, which measures <!--
|
|
396
|
-
<!--
|
|
477
|
+
plan a consumer's own document runs, which measures <!--fact:vector.plan-->206 ms at 50,000 × 768<!--/fact-->:
|
|
478
|
+
<!--fact:vector.table-->
|
|
397
479
|
| path (ms) | 10,000 × 384 | 10,000 × 768 | 50,000 × 384 | 50,000 × 768 |
|
|
398
480
|
|---|---:|---:|---:|---:|
|
|
399
481
|
| engine resident sweep (no database) | 3.3 | 6.7 | 17 | 32 |
|
|
@@ -402,14 +484,14 @@ plan a consumer's own document runs, which measures <!--bm:vector.plan-->206 ms
|
|
|
402
484
|
| `ORDER BY` over a registered function | 18 | 31 | 121 | 180 |
|
|
403
485
|
| JSON-doc sweep (no vector column) | 249 | 501 | — | — |
|
|
404
486
|
| sqlite-vec | 3.6 | 7.5 | 18 | 38 |
|
|
405
|
-
<!--/
|
|
487
|
+
<!--/fact-->
|
|
406
488
|
|
|
407
489
|
The row the column exists to beat is the last one that has no column: the
|
|
408
490
|
same query document over a collection that stores the embedding only
|
|
409
|
-
inside the document costs <!--
|
|
491
|
+
inside the document costs <!--fact:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/fact-->,
|
|
410
492
|
because every row's vector is parsed out of JSON before it can be
|
|
411
493
|
compared. The row the store **cannot** beat is the one with no database
|
|
412
|
-
in it: the same top-k over a resident `Float32Array` is <!--
|
|
494
|
+
in it: the same top-k over a resident `Float32Array` is <!--fact:vector.resident-->32 ms, which the plan is 6.4× slower than<!--/fact-->.
|
|
413
495
|
That comparison is not an even one and the direction is the point — the
|
|
414
496
|
sweep starts from decoded floats in RAM and pays nothing for durability,
|
|
415
497
|
for filters that compose with the ranking, or for a process that can
|
|
@@ -418,16 +500,16 @@ price should be able to say what the price is.
|
|
|
418
500
|
|
|
419
501
|
**Both halves of the price.** The column costs on the way in as well as
|
|
420
502
|
saving on the way out: writing the same documents with
|
|
421
|
-
the index costs <!--
|
|
503
|
+
the index costs <!--fact:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/fact-->,
|
|
422
504
|
because every write pays a JSON round trip of the member plus a
|
|
423
|
-
normalize and a pack. On disk one vector is <!--
|
|
505
|
+
normalize and a pack. On disk one vector is <!--fact:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/fact--> —
|
|
424
506
|
smaller, but *added*, since the document still carries the member the
|
|
425
507
|
column is derived from.
|
|
426
508
|
|
|
427
509
|
**Pushing the rank into SQL, re-measured.** A registered similarity
|
|
428
510
|
function inside an `ORDER BY … LIMIT k` is the obvious alternative, and
|
|
429
511
|
the suite measures it against the real column with the probe hoisted out
|
|
430
|
-
of the per-row call: <!--
|
|
512
|
+
of the per-row call: <!--fact:vector.udf-->180 ms against 202 ms at 50,000 × 768, and 0.87–1.00× the fetch-and-rank across the grid — rough parity on speed<!--/fact-->.
|
|
431
513
|
The plan does not emit it, and after that measurement the reasons are not
|
|
432
514
|
speed: `bun` has no user-function API, so a plan that needed one would
|
|
433
515
|
exclude an executor outright; and an ordering decided in SQL cannot break
|
|
@@ -436,15 +518,15 @@ executors have to agree on.
|
|
|
436
518
|
|
|
437
519
|
**The rival, and the ceiling.** `sqlite-vec` is the extension built for
|
|
438
520
|
exactly this, and it is measured rather than described: it answers the
|
|
439
|
-
same probes in <!--
|
|
440
|
-
over <!--
|
|
521
|
+
same probes in <!--fact:vector.rival-->38 ms against 206 ms at 50,000 × 768 — 5.4× in sqlite-vec's favour, out of a database 6.6× smaller that holds no documents<!--/fact-->,
|
|
522
|
+
over <!--fact:vector.agreement-->40 probes, no disagreements<!--/fact-->. It is a
|
|
441
523
|
loadable native extension, which is the one thing this store will not
|
|
442
524
|
require — it would exclude the wasm tab and stock `bun`, half the
|
|
443
525
|
execution story — so the comparison is published as what it is: a faster
|
|
444
526
|
engine you may prefer, and a dependency this one does not take. What
|
|
445
527
|
neither of them is, is an approximate index. Exact brute force is linear
|
|
446
528
|
in `n · d`, and the suite states the envelope as arithmetic rather than
|
|
447
|
-
opinion: <!--
|
|
529
|
+
opinion: <!--fact:vector.ceiling-->5.546 ns per vector component — one query reaches 100 ms at about 22,000 vectors of 768 dimensions and one second at about 234,000<!--/fact-->.
|
|
448
530
|
Past that this design is the wrong tool and no margin changes it; what
|
|
449
531
|
lies beyond is an approximate index, and this store does not have one.
|
|
450
532
|
|
|
@@ -501,13 +583,13 @@ same range, the same buckets, the same rolling window and the same
|
|
|
501
583
|
as-of join over one seeded corpus by plain references, by the temporal
|
|
502
584
|
kernel, by a generic query document, by hand-written SQL and by the
|
|
503
585
|
store — every route checked against the others before a timing is
|
|
504
|
-
taken. At <!--
|
|
505
|
-
the store is measured three ways at once — <!--
|
|
586
|
+
taken. At <!--fact:series.corpus-->100,000 samples at 1-second spacing, Node v24.19.0<!--/fact-->,
|
|
587
|
+
the store is measured three ways at once — <!--fact:series.storeShapes-->the planned range costs 4.2× the hand-written statement and 1535.5× the resident cut, and the pushed bucket ladder 2.5× the hand-written GROUP BY, 1.6× FASTER than the generic query route, and 155.1× the one-pass loop<!--/fact-->.
|
|
506
588
|
The range row is not the planner's price: the statement selects two
|
|
507
589
|
COLUMNS where the store renders and parses a whole JSON document per
|
|
508
590
|
row, which is what storing documents costs.
|
|
509
591
|
|
|
510
|
-
And what a refinement costs, with the loss in it: <!--
|
|
592
|
+
And what a refinement costs, with the loss in it: <!--fact:series.storeRefinement-->A window measured in time is not pushed: the store answers it at 18.5× the kernel over an array already in memory, over 100,000 candidates the index bounded. The batched as-of join reads 99,129 rows in 1 statement and costs 1424.3× fifty-one separate index reads — a bound is what it buys, not a speed-up, and without a tolerance a backward join can only be bounded above.<!--/fact-->
|
|
511
593
|
|
|
512
594
|
**A refinement is named, never quiet.** `explain().series` reports
|
|
513
595
|
`mode` — `native`, `hybrid` or `engine` — the declared index the fetch
|
|
@@ -515,7 +597,8 @@ seeks through, the instant bounds it used, which kernel finished the
|
|
|
515
597
|
answer, and a reason code for every thing the database could not do:
|
|
516
598
|
`fill-policy`, `calendar-width`, `named-zone`, `rolling-refinement`,
|
|
517
599
|
`asof-refinement`, `unsupported-aggregate`, `nonliteral-spec`,
|
|
518
|
-
`instant-not-integer`, `missing-series-prefix
|
|
600
|
+
`instant-not-integer`, `missing-series-prefix`, `row-selector`,
|
|
601
|
+
`value-not-numeric`, `nonnative-grouping`, `invalid-spec`. `strict: true` refuses
|
|
519
602
|
every one of them before a statement runs, and the counts `explain()`
|
|
520
603
|
prints are the LAST ACTUAL execution's — `null` until the document has
|
|
521
604
|
run, because an estimate wearing a count's name is worse than no
|
|
@@ -575,6 +658,18 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
|
|
|
575
658
|
`typedStore` (from `@jarenjs/db/typed`) types every read, checks
|
|
576
659
|
every write, and widens `load` results by their include
|
|
577
660
|
specification.
|
|
661
|
+
- **Membership** (§11.7): `link(own, member, target)` and `unlink` attach
|
|
662
|
+
and detach one many-to-many membership at a time through the unit of
|
|
663
|
+
work — written against the join table as it stands at save time, so a
|
|
664
|
+
repeated save changes nothing.
|
|
665
|
+
- **The front door**: `@jarenjs/linq/db` opens this store behind a
|
|
666
|
+
client typed from the model pen — `db.entities.Post.where((p) =>
|
|
667
|
+
p.stars.ge(3))` is the chain over the entity set, pushed down;
|
|
668
|
+
`db.entities.User.include((u) => u.posts, { where: (p) => p.stars.ge(3),
|
|
669
|
+
take: 2 }).toArray()` emits exactly the `load` spec above and runs in
|
|
670
|
+
the same one statement; `link`/`unlink` reach §11.7 and `live` the
|
|
671
|
+
registration below. It imports this package as an optional peer; this
|
|
672
|
+
package never imports it.
|
|
578
673
|
- **Relational migrations and the `jaren-db` CLI** (MIGRATION-FORMAT
|
|
579
674
|
§§9–12): the strategy-table diff, the documented twelve-step table
|
|
580
675
|
rebuild with `foreign_key_check` inside the transaction, shape
|
|
@@ -611,7 +706,12 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
|
|
|
611
706
|
queries, the same live updates run on the official SQLite wasm build
|
|
612
707
|
over the header-free OPFS SAH-pool VFS — one tab owns the
|
|
613
708
|
connection, others are clients. Proven in the `#/data` studio across
|
|
614
|
-
Chromium, Firefox and WebKit.
|
|
709
|
+
Chromium, Firefox and WebKit. The subpath exports the two helpers
|
|
710
|
+
that studio is built on: `sqlite3Handle(sqlite3, { DbClass })` builds
|
|
711
|
+
the injected handle from a loaded wasm module and the database class
|
|
712
|
+
the host picks (`sqlite3.oo1.DB` in memory, the SAH-pool
|
|
713
|
+
`OpfsSAHPoolDb` for OPFS), and `adaptOo1Database(sqlite3, db)` wraps
|
|
714
|
+
an oo1 database the host already opened.
|
|
615
715
|
|
|
616
716
|
### What an event-time view costs
|
|
617
717
|
|
|
@@ -623,16 +723,16 @@ before a single timing is printed — a fast live view with the wrong
|
|
|
623
723
|
answer is not a fast live view — and the run exits non-zero if they
|
|
624
724
|
disagree.
|
|
625
725
|
|
|
626
|
-
<!--
|
|
726
|
+
<!--fact:live.eventTimeTable-->
|
|
627
727
|
| view | maintained | re-run | ratio |
|
|
628
728
|
|---|---:|---:|---:|
|
|
629
729
|
| bucket (60 s ladder, mean), 1000 rows | 119 µs | 1.09 ms | 9.2× |
|
|
630
730
|
| rolling (5 min window, mean), 1000 rows | 407 µs | 3.86 ms | 9.5× |
|
|
631
731
|
| bucket (60 s ladder, mean), 10000 rows | 136 µs | 10.9 ms | 80.2× |
|
|
632
732
|
| rolling (5 min window, mean), 10000 rows | 13.7 ms | 62.7 ms | 4.6× |
|
|
633
|
-
<!--/
|
|
733
|
+
<!--/fact-->
|
|
634
734
|
|
|
635
|
-
The gain is <!--
|
|
735
|
+
The gain is <!--fact:live.eventTimeBand-->80.2× for the bucket and 4.6× for the rolling at 10,000 readings<!--/fact-->. A bucket
|
|
636
736
|
view is nearly flat in the series length, because a write folds one
|
|
637
737
|
bucket again and the rest of the ladder is untouched. A rolling view is
|
|
638
738
|
not, and the table says so: its answer is one row per reading, so the
|
|
@@ -677,10 +777,10 @@ replication on these primitives is a roadmap item, not a hint.
|
|
|
677
777
|
- **The wasm build journals** (its session extension is not yet
|
|
678
778
|
adapted); OPFS needs a secure context, and where it is absent the
|
|
679
779
|
store runs in memory with the durability difference stated.
|
|
680
|
-
- **Named future work, not silent gaps**: `$groupby` pushdown
|
|
681
|
-
|
|
682
|
-
joins, other SQL dialects,
|
|
683
|
-
(MODEL-FORMAT §10.6, the roadmap).
|
|
780
|
+
- **Named future work, not silent gaps**: `$groupby` pushdown beyond
|
|
781
|
+
the `$time-bucket` ladder, a many-to-many hop on the chain, membership
|
|
782
|
+
on an auto-keyed pending insert, incremental joins, other SQL dialects,
|
|
783
|
+
replication, database introspection (MODEL-FORMAT §10.6, the roadmap).
|
|
684
784
|
|
|
685
785
|
The normative formats are
|
|
686
786
|
[docs/MODEL-FORMAT.md](docs/MODEL-FORMAT.md) (storage §§1–7, safe
|
package/docs/JOBS-FORMAT.md
CHANGED
|
@@ -49,7 +49,10 @@ restarting.
|
|
|
49
49
|
`_jaren_job_checkpoints` holds `(run_id, node_id, value)` rows — the
|
|
50
50
|
flow checkpoint store of §7, keyed by the JOB id (the run id IS the
|
|
51
51
|
job id). Both tables are created on open when `jobs` is requested;
|
|
52
|
-
neither appears in the model.
|
|
52
|
+
neither appears in the model. `enqueue`'s options are validated as
|
|
53
|
+
they are stored: `runAt` must be a finite epoch in milliseconds and
|
|
54
|
+
`maxAttempts` a positive integer (`TypeError`) — a `NaN` eligibility
|
|
55
|
+
was once stored, and that job was pending forever.
|
|
53
56
|
|
|
54
57
|
## 3. Leasing and exactly-once execution
|
|
55
58
|
|
|
@@ -113,22 +116,31 @@ reclaimed — size `leaseMs` to the slowest honest handler.
|
|
|
113
116
|
## 6. Workers and concurrency
|
|
114
117
|
|
|
115
118
|
`createWorker({ handlers, concurrency, pollInterval, leaseMs, owner,
|
|
116
|
-
|
|
119
|
+
backoffBase, backoffCap, stopGraceMs })` returns `{ start(), stop(),
|
|
117
120
|
stats() }`:
|
|
118
121
|
|
|
119
|
-
- `concurrency` (default 1
|
|
120
|
-
worker
|
|
122
|
+
- `concurrency` (default 1, a positive integer — `TypeError`
|
|
123
|
+
otherwise, because a worker with zero loops would `start()` and never
|
|
124
|
+
claim) independent claim-execute loops share one worker registration;
|
|
121
125
|
- an idle loop sleeps `pollInterval` (default 500 ms — at most
|
|
122
126
|
2 claims/s of idle cost per loop, stated). An `enqueue` on the SAME
|
|
123
127
|
store wakes every idle local loop immediately, so same-process
|
|
124
128
|
latency is not poll-bound; **cross-process wake-up is polling**,
|
|
125
129
|
plainly (§8);
|
|
130
|
+
- retry backoff is `min(backoffCap, backoffBase × 2^(attempts − 1))`,
|
|
131
|
+
jittered to between half and all of itself (defaults 1 s and 60 s);
|
|
132
|
+
`maxAttempts` belongs to the JOB (`enqueue(kind, payload,
|
|
133
|
+
{ maxAttempts })`, default 5), not to the worker;
|
|
126
134
|
- `stop({ graceMs })` aborts in-flight handlers and resolves once they
|
|
127
|
-
settle **or** the grace period expires (default 5 s),
|
|
135
|
+
settle **or** the grace period expires (default `stopGraceMs`, 5 s),
|
|
136
|
+
answering
|
|
128
137
|
`{ drained, inFlight }`; a loop the grace period could not drain is
|
|
129
138
|
**cancelled**, not left running — see §6.1;
|
|
130
|
-
- `stats()` reports claims, completions, failures, wakes, polls
|
|
131
|
-
in-flight handler count
|
|
139
|
+
- `stats()` reports claims, completions, failures, wakes, polls, the
|
|
140
|
+
in-flight handler count and `claimErrors` — a claim statement the
|
|
141
|
+
database refused (a read-only file, a closed store), counted rather
|
|
142
|
+
than swallowed, so a worker that can never claim is visible instead
|
|
143
|
+
of silently idle.
|
|
132
144
|
|
|
133
145
|
### 6.1 A handler cannot break the loop, and cannot hold shutdown
|
|
134
146
|
|
|
@@ -224,6 +236,10 @@ await store.jobs.enqueue('sync-report', { input: { day: '2026-08-05' } });
|
|
|
224
236
|
This format adds NO codes. API misuse (a malformed handler map, a
|
|
225
237
|
non-string kind, a worker started twice) is a `TypeError` at the
|
|
226
238
|
call, matching the capture and live precedents; storage failures ride
|
|
227
|
-
the existing `JD2005` wrap
|
|
239
|
+
the existing `JD2005` wrap and a coded error passes through it
|
|
240
|
+
unchanged (`JD2063` after `close()`); `store.jobs` on a read-only
|
|
241
|
+
store is `JD0002` at first use, because the queue tables cannot be
|
|
242
|
+
created — named there, not a raw `SQLITE_READONLY` at the first
|
|
243
|
+
enqueue; a job's own failure is DATA — recorded in
|
|
228
244
|
`last_error` and the state machine of §4 — because a queue that
|
|
229
245
|
throws away its failure story has failed twice.
|
package/docs/LIVE-FORMAT.md
CHANGED
|
@@ -92,7 +92,10 @@ them).
|
|
|
92
92
|
already knows what it wrote. **Less complete, stated plainly**: it
|
|
93
93
|
cannot see writes made through raw SQL, triggers, or another
|
|
94
94
|
connection; a journal-mode delete of a row the store never read
|
|
95
|
-
emits its `remove` without having seen the old document;
|
|
95
|
+
emits its `remove` without having seen the old document; a keyed
|
|
96
|
+
`put` reads the stored document first, so it emits the `replace`
|
|
97
|
+
session mode emits and a `put` that changes nothing emits nothing;
|
|
98
|
+
and of the
|
|
96
99
|
database's own `ON DELETE` side effects it reconstructs exactly ONE
|
|
97
100
|
— join-table membership dying with its entity (read before the
|
|
98
101
|
delete) — while cascades into CHILD rows (`onDelete: 'cascade'` /
|
|
@@ -114,8 +117,14 @@ reads forward:
|
|
|
114
117
|
const records = await store.changesSince(lastSeq); // JD2051 when no log
|
|
115
118
|
```
|
|
116
119
|
|
|
117
|
-
`seq` is monotonic; with the log enabled
|
|
118
|
-
|
|
120
|
+
`seq` is monotonic; with the log enabled the DATABASE allocates it —
|
|
121
|
+
each record's `seq` is `MAX(seq) + 1` computed inside the insert
|
|
122
|
+
statement and read back through `RETURNING` — so two stores over one
|
|
123
|
+
file never collide on the log's key and each sees the other's
|
|
124
|
+
sequence continue; without the log it is per-process. `changesSince`
|
|
125
|
+
answers records in the shape observers receive, `collections`
|
|
126
|
+
included; a cursor that is not a number is a `TypeError`, as is a
|
|
127
|
+
`retention` that is not a positive integer. Retention is
|
|
119
128
|
a bounded count (`retention`, default 1000): older rows are pruned in
|
|
120
129
|
the same transaction. The log is an ordered, replayable stream —
|
|
121
130
|
which is what makes a late-joining consumer possible. **Replication
|
|
@@ -161,6 +170,14 @@ multi-entity shape of MODEL-FORMAT §10) the same way. Live queries
|
|
|
161
170
|
REQUIRE change capture — the patch stream is the invalidation source —
|
|
162
171
|
and registering on a store opened without `capture` is `JD0050`.
|
|
163
172
|
|
|
173
|
+
A producer may hand a registration a CHAIN instead of a document: the
|
|
174
|
+
`@jarenjs/linq/db` client's `live(chain, options)` passes the chain's
|
|
175
|
+
`toDocument()` and its `explain().bindings` as the externals to exactly
|
|
176
|
+
these two registrations (`store.live` for an entity-root chain,
|
|
177
|
+
`collection.live` for a collection's), so the strategy, the reason and
|
|
178
|
+
the maintenance are this table's — an entity chain re-runs, declared —
|
|
179
|
+
and this document stays the only place they are decided.
|
|
180
|
+
|
|
164
181
|
**This table is normative.** Every row is implemented and tested;
|
|
165
182
|
nothing outside it is attempted. Classification reads the compiled
|
|
166
183
|
PLAN (never the raw document), so "extractable" below means exactly
|
|
@@ -260,7 +277,11 @@ state. `externals` are fixed at registration — a query whose inputs
|
|
|
260
277
|
change is a new registration.
|
|
261
278
|
|
|
262
279
|
Maintenance runs synchronously inside patch delivery, in commit
|
|
263
|
-
order, on the store's own connection.
|
|
280
|
+
order, on the store's own connection. Delivery is never re-entered: a
|
|
281
|
+
write made from inside an observer or a subscriber commits at once,
|
|
282
|
+
but its record is queued and delivered after the current record has
|
|
283
|
+
reached every consumer, so sibling live views see commits in commit
|
|
284
|
+
order rather than in call-stack order. Writes from ANOTHER connection
|
|
264
285
|
are invisible to capture (§6) and therefore to live queries; the
|
|
265
286
|
coarse `dataVersion()` signal and the §11 topology are the honest
|
|
266
287
|
answers, and re-registering re-reads.
|
|
@@ -375,9 +396,11 @@ ERRORING rather than degrading (the D14 rule — the bound is printed):
|
|
|
375
396
|
|
|
376
397
|
Non-claims, in one place: no incremental joins (re-run is the declared
|
|
377
398
|
strategy), no cross-connection invalidation (§6's `data_version` is
|
|
378
|
-
the signal), no maintenance over asynchronous connections
|
|
379
|
-
|
|
380
|
-
|
|
399
|
+
the signal), no maintenance over asynchronous connections —
|
|
400
|
+
`capabilities.live` is `false` there and a registration is `JD0051`
|
|
401
|
+
naming the reason, because maintenance point-reads rows synchronously
|
|
402
|
+
inside delivery (the wasm driver's oo1 API is synchronous, which is
|
|
403
|
+
why the browser has live queries at all) — no replication, and no ordering guarantee for
|
|
381
404
|
unordered queries beyond §9's determinism.
|
|
382
405
|
|
|
383
406
|
## 13. Event time
|
|
@@ -404,7 +427,8 @@ live.stats().watermark; // what it is now
|
|
|
404
427
|
`eventTime` is a **closed** member set: `path`, `watermark`,
|
|
405
428
|
`allowedLateness` (default 0) and `retention`. Anything else — a
|
|
406
429
|
misspelling, a non-finite epoch, a negative lateness, a `path` that is
|
|
407
|
-
not a singular row selector — is `JD0053` at registration
|
|
430
|
+
not a singular row selector — is `JD0053` at registration (its
|
|
431
|
+
`docPath` names the collection, `/collections/<name>`), not a member
|
|
408
432
|
quietly ignored. `advance()` refuses a value that is not finite or that
|
|
409
433
|
goes backwards (a `TypeError`), and it is absent on every view
|
|
410
434
|
registered without an `eventTime`. An entity document has no collection
|
|
@@ -415,7 +439,12 @@ to place rows in and re-runs, so an `eventTime` on `store.live` is
|
|
|
415
439
|
|
|
416
440
|
Two documents, and only these two shapes: `$resample` and `$rolling`
|
|
417
441
|
whose series operand is the collection (`"$[*]"`, or a FLWOR over it
|
|
418
|
-
whose `$where` narrows and whose `$return` is the bare binding).
|
|
442
|
+
whose `$where` narrows and whose `$return` is the bare binding). A
|
|
443
|
+
spec that spells `at` and `value` explicitly — the spelling the query
|
|
444
|
+
language accepts — is maintained: the view folds with the kernel
|
|
445
|
+
reading the declared instant member and the `value` member the spec
|
|
446
|
+
names, and a `value` selector the view cannot follow re-runs with the
|
|
447
|
+
reason named, never a maintained view that dies on its first fold.
|
|
419
448
|
|
|
420
449
|
- **A bucket view keeps its rows by bucket.** A write touches one bucket
|
|
421
450
|
— two, when it moves a reading across a boundary — and exactly those
|