@jarenjs/db 0.46.5 → 0.49.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +106 -2
- package/README.md +136 -2
- package/dist/types/algebra.d.ts +34 -3
- package/dist/types/dialect.d.ts +5 -0
- package/dist/types/errors.d.ts +3 -0
- package/dist/types/live-time.d.ts +141 -0
- package/dist/types/live.d.ts +3 -1
- package/dist/types/plan.d.ts +2 -0
- package/dist/types/query.d.ts +2 -1
- package/dist/types/residual.d.ts +5 -2
- package/dist/types/series.d.ts +227 -0
- package/dist/types/store.d.ts +8 -1
- package/docs/LIVE-FORMAT.md +103 -0
- package/docs/MODEL-FORMAT.md +19 -0
- package/package.json +4 -4
- package/src/algebra.js +22 -3
- package/src/dialect.js +13 -0
- package/src/dialects/sqlite.js +19 -0
- package/src/emit.js +32 -6
- package/src/errors.js +3 -0
- package/src/live-time.js +596 -0
- package/src/live.js +41 -8
- package/src/plan.js +706 -16
- package/src/query.js +160 -11
- package/src/residual.js +15 -6
- package/src/series.js +349 -0
- package/src/store.js +22 -3
- package/types/index.d.ts +54 -2
package/ARCHITECTURE.md
CHANGED
|
@@ -153,6 +153,17 @@ member the schema types as an array or an object. Those are exact; the
|
|
|
153
153
|
spatial predicates that only NARROW are rows in the residual table, and
|
|
154
154
|
the implied-conjunct table below carries every proof.
|
|
155
155
|
|
|
156
|
+
Plus one GROUPING a composite instant index makes exact: the
|
|
157
|
+
fixed-width temporal bucket. A `$groupby` whose single key is
|
|
158
|
+
`$time-bucket` over a schema-typed integer epoch path with a literal
|
|
159
|
+
width and origin, projected to the group key plus `$count` of the
|
|
160
|
+
binding and `$sum`/`$avg`/`$min`/`$max` over singular schema-typed
|
|
161
|
+
numeric paths, becomes a `GROUP BY` over integer arithmetic. So does a
|
|
162
|
+
`$resample` whose frozen spec asks for nothing that ladder cannot do —
|
|
163
|
+
a fixed width, `fill: 'omit'`, and one of `sum|mean|min|max|count`.
|
|
164
|
+
Everything else about time is a NAMED refinement ("The temporal plan"
|
|
165
|
+
below).
|
|
166
|
+
|
|
156
167
|
Plus one ORDERING a **`derive: 'vector'`** column makes cheap without
|
|
157
168
|
making it native: the k-nearest composition — `$orderby` on a
|
|
158
169
|
`$similarity` key, descending, `$empty: 'least'`, under a `$subsequence`
|
|
@@ -181,6 +192,7 @@ mode, `knn`, beside native, row and set.
|
|
|
181
192
|
| the k-nearest ordering over a `derive: 'vector'` column | the column cuts the candidates; the engine orders them (mode `knn`, "The k-nearest plan" below) — engine work, so `strict: true` refuses it |
|
|
182
193
|
| `$similarity` anywhere else — a threshold in `$where`, a score in `$return`, a second ordering key | no native spelling; runs in the residual over whatever the rest of the document pushed |
|
|
183
194
|
| the k-nearest shape with no finite window, ascending, `$empty: 'greatest'`, a probe that is neither a literal vector nor an external, a literal probe of another width, or a selection not pushed whole (a residual conjunct, an implied one, a `$let` before the where) | nothing is proven; the whole document runs in the engine, and `explain()` names which precondition failed — a k-nearest query never falls to the full scan silently |
|
|
195
|
+
| `$resample` with a fill policy other than `omit`, `$rolling`, `$asof`, `first`/`last`, a calendar width, a named zone, a `$time-bucket` whose width or origin is an EXPRESSION, or an instant the schema does not type as an integer | the index bounds the fetch and `@jarenjs/core/series` decides over what comes back; `explain().series.reasons` carries the CODE — `fill-policy`, `rolling-refinement`, `asof-refinement`, `unsupported-aggregate`, `calendar-width`, `named-zone`, `nonliteral-spec`, `instant-not-integer` — and `strict: true` refuses every one of them |
|
|
184
196
|
|
|
185
197
|
### The type truth table
|
|
186
198
|
|
|
@@ -418,6 +430,79 @@ engine's business. With `strict: true` the shape is `JD0010` naming the
|
|
|
418
430
|
rank: the order is engine work, the same honesty as a spatial
|
|
419
431
|
refinement.
|
|
420
432
|
|
|
433
|
+
### The temporal plan (series)
|
|
434
|
+
|
|
435
|
+
Time needs no storage kind here. The physical declaration is the
|
|
436
|
+
composite JSONPath index a model already has —
|
|
437
|
+
`{ "name": "by_series_at", "path": ["$.series", "$.at"] }` — over a
|
|
438
|
+
numeric epoch member, and the whole of `src/series.js` is deciding
|
|
439
|
+
which questions that index answers. There is no `derive: 'series'`, no
|
|
440
|
+
column type, no host function and no extension.
|
|
441
|
+
|
|
442
|
+
A B-tree seeks exactly as far as its leading columns are decided: a run
|
|
443
|
+
of equalities, then at most one range. So the index that a temporal
|
|
444
|
+
plan reports is the declared one with the longest leading run of
|
|
445
|
+
columns an equality PINNED whose next column is the instant the query
|
|
446
|
+
ranges over. Three shapes follow from that, and they are closed:
|
|
447
|
+
|
|
448
|
+
| shape | the document | the plan |
|
|
449
|
+
|---|---|---|
|
|
450
|
+
| **range** | every leading column pinned, a half-open range on the instant, ordered by it | native — `SEARCH … (series=? AND at>? AND at<?)` |
|
|
451
|
+
| **as-of** | the same prefix with ONE instant bound, ordered by the instant, under a finite window | native — the index read backwards, one row |
|
|
452
|
+
| **bucket** | a `$groupby` over `$time-bucket`, or a `$resample` whose spec asks for nothing more | native — `GROUP BY` over `at - (((at - origin) % every + every) % every)` |
|
|
453
|
+
|
|
454
|
+
The ladder is integer arithmetic all the way down, and the non-negative
|
|
455
|
+
remainder is why: a truncating division puts an instant before 1970 in
|
|
456
|
+
the bucket AFTER its own. The width and the anchor are read through the
|
|
457
|
+
kernel's own `compileBuckets`, so `'PT1H'` and `3600000` are the same
|
|
458
|
+
ladder and the default anchor is the kernel's rather than a second
|
|
459
|
+
guess at it; an `{ offset }` calendar context folds into the anchor,
|
|
460
|
+
because a constant number of minutes east is arithmetic. A named zone
|
|
461
|
+
never does: its clock is host code the database does not have.
|
|
462
|
+
|
|
463
|
+
Everything else is a **named refinement** — the implied-conjunct
|
|
464
|
+
pattern again, applied to a whole operator. The planner narrows the
|
|
465
|
+
fetch by whatever the frozen spec makes provable and the residual, which
|
|
466
|
+
is the ENGINE running the caller's own document, decides:
|
|
467
|
+
|
|
468
|
+
| operator | what bounds the fetch |
|
|
469
|
+
|---|---|
|
|
470
|
+
| `$resample` (fill, calendar, `first`/`last`) | the spec's own `start`/`end`, plus the operand's `$where` |
|
|
471
|
+
| `$rolling` | the operand's `$where` alone — a window measured in time answers once per input instant |
|
|
472
|
+
| `$asof` with the collection on the RIGHT | the probes' own span (`at <= max` backward, `at >= min` forward, both sides under a `tolerance`) and a membership test over the probes' `by` keys |
|
|
473
|
+
| `$asof` with the collection on the LEFT | nothing — a join answers once per LEFT row, so every left row is needed |
|
|
474
|
+
|
|
475
|
+
The as-of bound is one STATEMENT, whatever the probes number, which is
|
|
476
|
+
what `test/db/statement-count.test.js` pins: the failure mode a batch
|
|
477
|
+
exists to refuse is one seek per left row. Without a `tolerance` a
|
|
478
|
+
backward join can only be bounded ABOVE, so that one statement can read
|
|
479
|
+
most of a long history — `benchmark/series.js` publishes the candidate
|
|
480
|
+
count beside the timing rather than netting it out.
|
|
481
|
+
|
|
482
|
+
**A group with no instant.** A row whose instant member is missing or
|
|
483
|
+
is not a number groups under SQL `NULL`; the kernel REFUSES such a row
|
|
484
|
+
(`JQ2001`). SQL cannot refuse, so a native bucket that meets one hands
|
|
485
|
+
the whole question back — the full-collection residual, where the engine
|
|
486
|
+
answers what it answers everywhere — and the fallback is COUNTED as
|
|
487
|
+
`collection.stats().series.diverted`, the same honesty the k-nearest
|
|
488
|
+
divert has.
|
|
489
|
+
|
|
490
|
+
**`explain().series`** is `null` unless the document asked a temporal
|
|
491
|
+
question, which it does by naming a §8.16 operator or by being the
|
|
492
|
+
closed range/as-of shape over a COMPOSITE instant index. (A singular
|
|
493
|
+
index over an ordinary member cannot make a query temporal: nothing in
|
|
494
|
+
a column says "instant", and inventing one would make every `age > 21`
|
|
495
|
+
a temporal plan.) It carries `{ mode, operation, index, prefix, range,
|
|
496
|
+
ladder, aggregates, refinement, reasons, counts }`. `mode` is
|
|
497
|
+
`'native'` (the statement alone answers), `'hybrid'` (the database
|
|
498
|
+
narrows and a kernel decides) or `'engine'` (the database narrowed
|
|
499
|
+
nothing). `reasons` is `{ code, reason }` where the code is the first
|
|
500
|
+
word of the sentence, so the machine-readable code and the sentence
|
|
501
|
+
`strict: true` prints cannot drift apart. `counts` is the LAST ACTUAL
|
|
502
|
+
execution's `{ statements, candidates, results }` and is `null` before
|
|
503
|
+
the document has run once — an estimate mislabelled as a count is
|
|
504
|
+
exactly what an honest explain may not print.
|
|
505
|
+
|
|
421
506
|
### The two residual modes
|
|
422
507
|
|
|
423
508
|
- **Row residual** — only the projection is untranslated: predicates,
|
|
@@ -465,7 +550,9 @@ and this is where it says so (MODEL-FORMAT §4). Under `'rtree'` the
|
|
|
465
550
|
names the virtual table. `rank` is `null` or the k-nearest stage —
|
|
466
551
|
`{ column, dims, probe, limit, offset, margin, decides: 'engine' }` —
|
|
467
552
|
what the fetch reads, the window the cut serves, the margin it keeps,
|
|
468
|
-
and who decides the order (always the engine). `
|
|
553
|
+
and who decides the order (always the engine). `series` is `null` or
|
|
554
|
+
the temporal record ("The temporal plan" above), whose `counts` are the
|
|
555
|
+
last ACTUAL run's rather than an estimate. `estimatedRows` is ABSENT on SQLite drivers — the capability slot
|
|
469
556
|
is empty and no number is fabricated. `residual` is `null` or
|
|
470
557
|
`{ mode: 'row' | 'set' | 'knn', reasons: [{ construct, reason }] }` with
|
|
471
558
|
reasons drawn from the deliberate-residual table. With
|
|
@@ -578,7 +665,7 @@ rollback truncates the journal buffer to its checkpoint. Overhead is
|
|
|
578
665
|
measured and published: capture off ~6µs, journal ~17µs, session ~69µs
|
|
579
666
|
per single-op commit, amortizing across a transaction.
|
|
580
667
|
|
|
581
|
-
## Live queries (`src/live.js`, `src/window.js`)
|
|
668
|
+
## Live queries (`src/live.js`, `src/live-time.js`, `src/window.js`)
|
|
582
669
|
|
|
583
670
|
A live query classifies its document against the normative maintenance
|
|
584
671
|
table by reading the COMPILED PLAN — translated filters, order terms
|
|
@@ -605,6 +692,23 @@ proven by a seeded oracle that holds the maintained result equal to a
|
|
|
605
692
|
fresh re-query after every mutation. Incremental beats re-run 13× at
|
|
606
693
|
1k rows, 51× at 10k.
|
|
607
694
|
|
|
695
|
+
**Event time** (`src/live-time.js`) adds two more strategies for the
|
|
696
|
+
one document shape whose answer is a function of instants rather than
|
|
697
|
+
of rows: a `$resample` bucket state and a `$rolling` window state, both
|
|
698
|
+
over a FIXED width. There is no clock under either — the watermark is a
|
|
699
|
+
finite epoch the host supplies and `advance()` is the only way it moves
|
|
700
|
+
— and both fold through `@jarenjs/core/series` itself rather than
|
|
701
|
+
carrying a second aggregate, so a maintained answer cannot drift from
|
|
702
|
+
what a fresh query gives. A write touches one bucket (two when it
|
|
703
|
+
crosses a boundary) or the windows ending in `[t, t + width)`, and
|
|
704
|
+
exactly those are recomputed. Everything a per-key state cannot place
|
|
705
|
+
exactly re-runs with its member named: a calendar ladder, a named zone,
|
|
706
|
+
a `locf`/`linear` fill, a `first`/`last` aggregate, a retention under
|
|
707
|
+
`width + allowedLateness`. A reading behind the lateness boundary is
|
|
708
|
+
never folded in silently — the view re-reads and the emission carries a
|
|
709
|
+
`lateData` record — which is the one place this layer spends a full
|
|
710
|
+
re-query to keep a promise rather than a number.
|
|
711
|
+
|
|
608
712
|
## Durable runs and the job queue (`src/jobs.js`, `src/dag-job.js`)
|
|
609
713
|
|
|
610
714
|
The queue's whole correctness story is one guarded statement: the
|
package/README.md
CHANGED
|
@@ -113,6 +113,17 @@ shape binds at `store.collection<User>('users')`.
|
|
|
113
113
|
decided or merely narrowed. The worked example, the geofence and the
|
|
114
114
|
measured numbers are in [Spatial storage](#spatial-storage--the-model-the-plan-the-fence-the-numbers)
|
|
115
115
|
below.
|
|
116
|
+
- **A time series is a composite index, not a storage kind.** Declare
|
|
117
|
+
`{ "path": ["$.series", "$.at"] }` over a numeric epoch member and the
|
|
118
|
+
planner recognizes three shapes over it: a half-open range under a
|
|
119
|
+
series equality, an as-of lookup (the index read backwards, one row),
|
|
120
|
+
and a fixed-width bucket ladder — a `$groupby` over `$time-bucket`, or
|
|
121
|
+
a `$resample` whose spec asks for nothing a `GROUP BY` cannot do — as
|
|
122
|
+
integer arithmetic in SQL. A fill policy, a calendar width, a rolling
|
|
123
|
+
window and an as-of JOIN are **named refinements**: the index bounds
|
|
124
|
+
the fetch and `@jarenjs/core/series` decides, with `explain().series`
|
|
125
|
+
carrying the reason code and the last run's actual candidate and
|
|
126
|
+
result counts. See [Time series](#time-series--the-index-the-ladder-the-refinement).
|
|
116
127
|
- **Migrations are documents.** `planMigration` diffs two models into
|
|
117
128
|
rendered-DDL + JSLT-transform + assertion steps; a shadow database
|
|
118
129
|
replays the whole chain before the real store is touched; a
|
|
@@ -445,6 +456,91 @@ and unindexed — and every entry must answer identically, including a
|
|
|
445
456
|
deliberate one-binary32-ulp near-tie and the windows that reach past the
|
|
446
457
|
scored rows into the tail the column cannot rank.
|
|
447
458
|
|
|
459
|
+
## Time series — the index, the ladder, the refinement
|
|
460
|
+
|
|
461
|
+
The physical declaration is one a model already has:
|
|
462
|
+
|
|
463
|
+
```json
|
|
464
|
+
{ "name": "by_series_at", "path": ["$.series", "$.at"] }
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
No `derive` kind, no column type, no host function, no extension. What
|
|
468
|
+
the planner adds is the reading of that index — a B-tree seeks as far
|
|
469
|
+
as its leading columns are decided, so a query that pins `series` with
|
|
470
|
+
an equality and ranges over `at` is a SEARCH, and one that only bounds
|
|
471
|
+
`at` is a scan the plan says so about.
|
|
472
|
+
|
|
473
|
+
```js
|
|
474
|
+
// native: SEARCH sample USING INDEX sample_by_series_at (gx_series=? AND gx_at>? AND gx_at<?)
|
|
475
|
+
await sample.execute({
|
|
476
|
+
$for: { s: '$[*]' },
|
|
477
|
+
$where: { $and: [
|
|
478
|
+
{ $eq: ['$s.series', 'sensor-a'] },
|
|
479
|
+
{ $ge: ['$s.at', from] },
|
|
480
|
+
{ $lt: ['$s.at', to] },
|
|
481
|
+
] },
|
|
482
|
+
$orderby: [{ $key: '$s.at' }],
|
|
483
|
+
$return: '$s',
|
|
484
|
+
});
|
|
485
|
+
|
|
486
|
+
// native: the same index, GROUP BY over integer bucket arithmetic
|
|
487
|
+
await sample.execute({ $resample: [
|
|
488
|
+
{ $for: { s: '$[*]' }, $where: { $eq: ['$s.series', 'sensor-a'] }, $return: '$s' },
|
|
489
|
+
{ every: 'PT1M', aggregate: 'mean' },
|
|
490
|
+
] });
|
|
491
|
+
|
|
492
|
+
// hybrid: the index bounds the fetch, rollingSeries decides
|
|
493
|
+
await sample.execute({ $rolling: [
|
|
494
|
+
{ $for: { s: '$[*]' }, $where: { $eq: ['$s.series', 'sensor-a'] }, $return: '$s' },
|
|
495
|
+
{ width: 'PT1M', aggregate: 'mean', minPeriods: 30 },
|
|
496
|
+
] });
|
|
497
|
+
```
|
|
498
|
+
|
|
499
|
+
**The numbers, the loss included.** `benchmark/series.js` answers the
|
|
500
|
+
same range, the same buckets, the same rolling window and the same
|
|
501
|
+
as-of join over one seeded corpus by plain references, by the temporal
|
|
502
|
+
kernel, by a generic query document, by hand-written SQL and by the
|
|
503
|
+
store — every route checked against the others before a timing is
|
|
504
|
+
taken. At <!--bm:series.corpus-->100,000 samples at 1-second spacing, Node v24.19.0<!--/bm-->,
|
|
505
|
+
the store is measured three ways at once — <!--bm:series.storeShapes-->the planned range costs 4.2× the hand-written statement and 1535.5× the resident cut, and the pushed bucket ladder 2.5× the hand-written GROUP BY, 1.6× FASTER than the generic query route, and 155.1× the one-pass loop<!--/bm-->.
|
|
506
|
+
The range row is not the planner's price: the statement selects two
|
|
507
|
+
COLUMNS where the store renders and parses a whole JSON document per
|
|
508
|
+
row, which is what storing documents costs.
|
|
509
|
+
|
|
510
|
+
And what a refinement costs, with the loss in it: <!--bm:series.storeRefinement-->A window measured in time is not pushed: the store answers it at 18.5× the kernel over an array already in memory, over 100,000 candidates the index bounded. The batched as-of join reads 99,129 rows in 1 statement and costs 1424.3× fifty-one separate index reads — a bound is what it buys, not a speed-up, and without a tolerance a backward join can only be bounded above.<!--/bm-->
|
|
511
|
+
|
|
512
|
+
**A refinement is named, never quiet.** `explain().series` reports
|
|
513
|
+
`mode` — `native`, `hybrid` or `engine` — the declared index the fetch
|
|
514
|
+
seeks through, the instant bounds it used, which kernel finished the
|
|
515
|
+
answer, and a reason code for every thing the database could not do:
|
|
516
|
+
`fill-policy`, `calendar-width`, `named-zone`, `rolling-refinement`,
|
|
517
|
+
`asof-refinement`, `unsupported-aggregate`, `nonliteral-spec`,
|
|
518
|
+
`instant-not-integer`, `missing-series-prefix`. `strict: true` refuses
|
|
519
|
+
every one of them before a statement runs, and the counts `explain()`
|
|
520
|
+
prints are the LAST ACTUAL execution's — `null` until the document has
|
|
521
|
+
run, because an estimate wearing a count's name is worse than no
|
|
522
|
+
number.
|
|
523
|
+
|
|
524
|
+
**The as-of join is bounded, and the bound is the claim.** `$asof` with
|
|
525
|
+
the collection on the right reads the probes it was given, bounds the
|
|
526
|
+
fetch by their own span and by a membership test over their `by` keys,
|
|
527
|
+
and issues exactly ONE statement whatever the probes number — the
|
|
528
|
+
failure mode a batch exists to refuse is one seek per left row, and
|
|
529
|
+
`test/db/statement-count.test.js` pins it at 1, 10 and 200 probes.
|
|
530
|
+
Without a `tolerance` a backward join can only be bounded ABOVE, so
|
|
531
|
+
that one statement can read most of a long history: the benchmark
|
|
532
|
+
publishes the candidate count beside the timing rather than netting it
|
|
533
|
+
out, and at fifty-one probes over a hundred thousand rows the batch
|
|
534
|
+
LOSES to fifty-one separate index reads. Few questions of a large
|
|
535
|
+
series belong to a batch; a join of two series does.
|
|
536
|
+
|
|
537
|
+
**One corpus, five executors, proven to agree.** The committed temporal
|
|
538
|
+
corpus (`test/json/fixtures/series-corpus.json`) runs through the plain
|
|
539
|
+
references, the query vocabulary, `node:sqlite`, a real wasm build and
|
|
540
|
+
both drivers again with pushdown forced off — indexed and unindexed —
|
|
541
|
+
and every case must answer identically, plan mode and reason codes
|
|
542
|
+
included.
|
|
543
|
+
|
|
448
544
|
## What SQLite-only means, frankly
|
|
449
545
|
|
|
450
546
|
SQLite is the supported backend — 3.45 or newer, on `node:sqlite`,
|
|
@@ -493,7 +589,7 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
|
|
|
493
589
|
journal where it does not (`bun:sqlite`, the wasm build). One diff
|
|
494
590
|
format runs store → patch → live query → O(k) render. Capture is
|
|
495
591
|
opt-in; the overhead is published, not waved away.
|
|
496
|
-
- **Live queries** (LIVE-FORMAT §§7–
|
|
592
|
+
- **Live queries** (LIVE-FORMAT §§7–13): `collection.live(document)`
|
|
497
593
|
maintains a result as writes arrive and emits patches — incremental
|
|
498
594
|
for `where`/`select`/`orderBy`+`limit`/aggregates/single-level
|
|
499
595
|
`groupBy` and a spatial `where` over a derived index (the geofence;
|
|
@@ -501,6 +597,11 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
|
|
|
501
597
|
**declared, never silent** (`live.mode` names the reason).
|
|
502
598
|
Unaffected rows stay reference-identical; a seeded oracle holds the
|
|
503
599
|
maintained result equal to a fresh re-query after every mutation.
|
|
600
|
+
- **Event time** (LIVE-FORMAT §13): a `$resample` or `$rolling` view
|
|
601
|
+
over a fixed width maintains exact event-time buckets and windows
|
|
602
|
+
against a watermark the HOST supplies — never a clock — and a reading
|
|
603
|
+
behind the declared lateness re-reads and emits a `lateData` record
|
|
604
|
+
rather than being folded in as though it had arrived on time.
|
|
504
605
|
- **Durable runs and the job queue** (JOBS-FORMAT, FLOW-FORMAT §7.6):
|
|
505
606
|
a `@jarenjs/flow` DAG run checkpoints declared nodes and RESUMES
|
|
506
607
|
after a crash; `store.jobs` leases work in one guarded statement
|
|
@@ -512,6 +613,35 @@ SQLite's own story (WAL plus a busy timeout, both set and visible on
|
|
|
512
613
|
connection, others are clients. Proven in the `#/data` studio across
|
|
513
614
|
Chromium, Firefox and WebKit.
|
|
514
615
|
|
|
616
|
+
### What an event-time view costs
|
|
617
|
+
|
|
618
|
+
`benchmark/live.js` maintains a 60 s bucket ladder and a 5 minute
|
|
619
|
+
rolling window over a seeded series and rewrites one reading per commit,
|
|
620
|
+
inside the lateness the view allows. The maintained rows are checked
|
|
621
|
+
against `resampleSeries` / `rollingSeries` over the **whole** collection
|
|
622
|
+
before a single timing is printed — a fast live view with the wrong
|
|
623
|
+
answer is not a fast live view — and the run exits non-zero if they
|
|
624
|
+
disagree.
|
|
625
|
+
|
|
626
|
+
<!--bm:live.eventTimeTable-->
|
|
627
|
+
| view | maintained | re-run | ratio |
|
|
628
|
+
|---|---:|---:|---:|
|
|
629
|
+
| bucket (60 s ladder, mean), 1000 rows | 119 µs | 1.09 ms | 9.2× |
|
|
630
|
+
| rolling (5 min window, mean), 1000 rows | 407 µs | 3.86 ms | 9.5× |
|
|
631
|
+
| bucket (60 s ladder, mean), 10000 rows | 136 µs | 10.9 ms | 80.2× |
|
|
632
|
+
| rolling (5 min window, mean), 10000 rows | 13.7 ms | 62.7 ms | 4.6× |
|
|
633
|
+
<!--/bm-->
|
|
634
|
+
|
|
635
|
+
The gain is <!--bm:live.eventTimeBand-->80.2× for the bucket and 4.6× for the rolling at 10,000 readings<!--/bm-->. A bucket
|
|
636
|
+
view is nearly flat in the series length, because a write folds one
|
|
637
|
+
bucket again and the rest of the ladder is untouched. A rolling view is
|
|
638
|
+
not, and the table says so: its answer is one row per reading, so the
|
|
639
|
+
emitted diff walks every one of them whatever changed. A bucket view
|
|
640
|
+
also holds one maintained entry per reading *plus* one per bucket, which
|
|
641
|
+
is over §12's default `maxMaintained` at ten thousand readings — the
|
|
642
|
+
bound errors rather than degrading, and raising it is a decision
|
|
643
|
+
somebody makes.
|
|
644
|
+
|
|
515
645
|
## Sync-readiness — what exists and what does not
|
|
516
646
|
|
|
517
647
|
The change stream is an ordered log of RFC 6902 patches with a
|
|
@@ -540,6 +670,10 @@ replication on these primitives is a roadmap item, not a hint.
|
|
|
540
670
|
priority classes, no cron, no workflow compensation.
|
|
541
671
|
- **Live-query maintenance is limited to the declared table** (§7);
|
|
542
672
|
joins, entity queries and non-canonical shapes re-run, reported.
|
|
673
|
+
- **`eventTime.retention` bounds repair work, not memory.** It is the
|
|
674
|
+
horizon a view claims and is checked against the window it maintains;
|
|
675
|
+
the maintained state is still bounded by `live.maxMaintained`, and no
|
|
676
|
+
version of this compacts a bucket's rows away.
|
|
543
677
|
- **The wasm build journals** (its session extension is not yet
|
|
544
678
|
adapted); OPFS needs a secure context, and where it is absent the
|
|
545
679
|
store runs in memory with the durability difference stated.
|
|
@@ -554,7 +688,7 @@ profile §8, entities §9, relational translation §10, the unit of work
|
|
|
554
688
|
§11), [docs/MIGRATION-FORMAT.md](docs/MIGRATION-FORMAT.md) (documents
|
|
555
689
|
§§1–8, relational changes §§9–12),
|
|
556
690
|
[docs/LIVE-FORMAT.md](docs/LIVE-FORMAT.md) (capture §§1–6, live queries
|
|
557
|
-
§§7–12) and [docs/JOBS-FORMAT.md](docs/JOBS-FORMAT.md) (the durable
|
|
691
|
+
§§7–12, event time §13) and [docs/JOBS-FORMAT.md](docs/JOBS-FORMAT.md) (the durable
|
|
558
692
|
queue §§1–9); the seams, the pushdown contract and every engine are in
|
|
559
693
|
[ARCHITECTURE.md](ARCHITECTURE.md); the benchmark methodology is in
|
|
560
694
|
[benchmark/README.md](../../benchmark/README.md).
|
package/dist/types/algebra.d.ts
CHANGED
|
@@ -10,9 +10,10 @@
|
|
|
10
10
|
* One plan shape covers this version: a guarded selection over ONE
|
|
11
11
|
* collection with optional ordering, window, aggregate and a
|
|
12
12
|
* whole-document projection — or, instead of an ordering and a window,
|
|
13
|
-
* a k-nearest RANK the engine finishes over the rows the plan fetches
|
|
14
|
-
*
|
|
15
|
-
*
|
|
13
|
+
* a k-nearest RANK the engine finishes over the rows the plan fetches,
|
|
14
|
+
* or, instead of a projection, a fixed-width temporal BUCKET the plan
|
|
15
|
+
* groups and aggregates itself. Constructs beyond it are residuals by
|
|
16
|
+
* design (see ARCHITECTURE.md's deliberate-residual table).
|
|
16
17
|
*/
|
|
17
18
|
/** The plan format version, carried on every plan. */
|
|
18
19
|
export declare const PLAN_VERSION = 2;
|
|
@@ -85,6 +86,19 @@ export type PlanOrderTerm = {
|
|
|
85
86
|
desc: boolean;
|
|
86
87
|
emptyGreatest: boolean;
|
|
87
88
|
};
|
|
89
|
+
export type PlanBucket = {
|
|
90
|
+
ref: PlanRef;
|
|
91
|
+
every: number;
|
|
92
|
+
origin: number;
|
|
93
|
+
as: string;
|
|
94
|
+
order: 'asc' | 'desc' | 'first-seen';
|
|
95
|
+
aggregates: {
|
|
96
|
+
fn: 'rows' | 'sum' | 'avg' | 'min' | 'max';
|
|
97
|
+
ref: PlanRef | null;
|
|
98
|
+
as: string;
|
|
99
|
+
empty: 'null' | 'zero' | 'omit';
|
|
100
|
+
}[];
|
|
101
|
+
};
|
|
88
102
|
export type PlanRank = {
|
|
89
103
|
column: string;
|
|
90
104
|
dims: number;
|
|
@@ -108,6 +122,7 @@ export type Plan = {
|
|
|
108
122
|
limit: number | null;
|
|
109
123
|
} | null;
|
|
110
124
|
rank: PlanRank | null;
|
|
125
|
+
bucket: PlanBucket | null;
|
|
111
126
|
aggregate: {
|
|
112
127
|
fn: 'count' | 'sum' | 'avg' | 'min' | 'max';
|
|
113
128
|
ref: PlanRef | null;
|
|
@@ -152,6 +167,21 @@ export type Plan = {
|
|
|
152
167
|
*
|
|
153
168
|
* @typedef {{ ref: PlanRef, desc: boolean, emptyGreatest: boolean }} PlanOrderTerm
|
|
154
169
|
*
|
|
170
|
+
* @typedef {{ ref: PlanRef, every: number, origin: number, as: string,
|
|
171
|
+
* order: 'asc' | 'desc' | 'first-seen',
|
|
172
|
+
* aggregates: { fn: 'rows' | 'sum' | 'avg' | 'min' | 'max',
|
|
173
|
+
* ref: PlanRef | null, as: string,
|
|
174
|
+
* empty: 'null' | 'zero' | 'omit' }[] }} PlanBucket
|
|
175
|
+
* The fixed-width temporal GROUP BY: the instant column, the ladder's
|
|
176
|
+
* width and anchor in epoch milliseconds, the name the bucket's start
|
|
177
|
+
* is answered under, how the groups are ordered, and one aggregate
|
|
178
|
+
* per answered member. `rows` is `COUNT(*)` — the D5 count of SOURCE
|
|
179
|
+
* rows, duplicates and measured gaps included — and the four value
|
|
180
|
+
* aggregates skip a `NULL` reading exactly as the kernel skips a
|
|
181
|
+
* `null` one. `first-seen` order is the group's earliest row identity,
|
|
182
|
+
* which is the engine's own "order of first appearance" (§6.5).
|
|
183
|
+
* A plan carrying a bucket carries no `aggregate` and no `rank`.
|
|
184
|
+
*
|
|
155
185
|
* @typedef {{ column: string, dims: number,
|
|
156
186
|
* probe: { lit: number[] } | { ext: string },
|
|
157
187
|
* offset: number, limit: number, margin: number }} PlanRank
|
|
@@ -174,6 +204,7 @@ export type Plan = {
|
|
|
174
204
|
* order: PlanOrderTerm[] | null,
|
|
175
205
|
* window: { offset: number, limit: number | null } | null,
|
|
176
206
|
* rank: PlanRank | null,
|
|
207
|
+
* bucket: PlanBucket | null,
|
|
177
208
|
* aggregate: { fn: 'count' | 'sum' | 'avg' | 'min' | 'max',
|
|
178
209
|
* ref: PlanRef | null } | null,
|
|
179
210
|
* project: 'document',
|
package/dist/types/dialect.d.ts
CHANGED
|
@@ -60,6 +60,9 @@ export type JsonPathSegment = {
|
|
|
60
60
|
* strEndsWith: (valueSql: string, patternA: string, patternB: string, patternC: string) => string,
|
|
61
61
|
* strContains: (valueSql: string, patternSql: string) => string,
|
|
62
62
|
* orderNulls: (nullsFirst: boolean) => string,
|
|
63
|
+
* timeBucket: (instantSql: string, originSql: string, everyA: string,
|
|
64
|
+
* everyB: string, everyC: string) => string,
|
|
65
|
+
* groupAggregate: (fn: string, valueSql: string | null) => string,
|
|
63
66
|
* rowIdentity: () => string,
|
|
64
67
|
* identityIn: (identitySql: string, paramSqls: string[]) => string,
|
|
65
68
|
* rtree?: { module: string, columns: readonly string[] },
|
|
@@ -114,6 +117,8 @@ export declare function createDialect(spec: {
|
|
|
114
117
|
strEndsWith: (valueSql: string, patternA: string, patternB: string, patternC: string) => string;
|
|
115
118
|
strContains: (valueSql: string, patternSql: string) => string;
|
|
116
119
|
orderNulls: (nullsFirst: boolean) => string;
|
|
120
|
+
timeBucket: (instantSql: string, originSql: string, everyA: string, everyB: string, everyC: string) => string;
|
|
121
|
+
groupAggregate: (fn: string, valueSql: string | null) => string;
|
|
117
122
|
rowIdentity: () => string;
|
|
118
123
|
identityIn: (identitySql: string, paramSqls: string[]) => string;
|
|
119
124
|
rtree?: {
|
package/dist/types/errors.d.ts
CHANGED
|
@@ -31,6 +31,7 @@ export declare const DB_CODES: Readonly<{
|
|
|
31
31
|
JD0050: "live queries require change capture";
|
|
32
32
|
JD0051: "the demanded live mode is unavailable";
|
|
33
33
|
JD0052: "the live-query bound was reached";
|
|
34
|
+
JD0053: "the live event-time declaration is invalid";
|
|
34
35
|
JD0020: "the migration's from-shape does not match the database";
|
|
35
36
|
JD0021: "the migration is missing a required data transform";
|
|
36
37
|
JD0022: "an applied migration disagrees with the history record";
|
|
@@ -90,6 +91,8 @@ export declare const DB_CODES: Readonly<{
|
|
|
90
91
|
* classifies as re-run; the reason names the forcing construct
|
|
91
92
|
* - `JD0052` — registering would exceed the store's `live.maxQueries`
|
|
92
93
|
* bound; the bound is printed, never silent
|
|
94
|
+
* - `JD0053` — a live query's `eventTime` names a member it does not
|
|
95
|
+
* admit, or a watermark/retention that is not a finite span
|
|
93
96
|
* - `JD0020` — a migration's `from` hash does not match the
|
|
94
97
|
* database's recorded shape; running it would corrupt
|
|
95
98
|
* - `JD0021` — a draft transform was not filled in, or a document no
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file Event-time live views (LIVE-FORMAT §13): a `$resample` or
|
|
3
|
+
* `$rolling` document over a collection, maintained against an
|
|
4
|
+
* explicit watermark.
|
|
5
|
+
*
|
|
6
|
+
* There is no clock in this file, and there is none anywhere under it.
|
|
7
|
+
* A live view over time needs to know what "now" is — which bucket is
|
|
8
|
+
* still open, which reading counts as late — and the only honest
|
|
9
|
+
* source of that is the host, because the machine's clock is a
|
|
10
|
+
* different quantity from the instant a reading carries. So the
|
|
11
|
+
* watermark ARRIVES: it is a finite epoch supplied at registration and
|
|
12
|
+
* moved forward by `advance()`, it never goes backwards, and a test can
|
|
13
|
+
* put it wherever the story needs it without waiting for a timer.
|
|
14
|
+
*
|
|
15
|
+
* What the maintenance actually does:
|
|
16
|
+
*
|
|
17
|
+
* - **A bucket view keeps its rows by bucket.** A write touches one
|
|
18
|
+
* bucket (two, when it moves a reading across a boundary), and only
|
|
19
|
+
* those are folded again — through `resampleSeries` itself, over that
|
|
20
|
+
* bucket's own rows, so the aggregate is the kernel's and cannot
|
|
21
|
+
* drift from what a fresh query would answer.
|
|
22
|
+
* - **A rolling view keeps its rows in instant order.** A write at `t`
|
|
23
|
+
* can only change the windows ending in `[t, t + width)`, so exactly
|
|
24
|
+
* that stretch is recomputed — again by the kernel, over the slice
|
|
25
|
+
* that stretch can see.
|
|
26
|
+
*
|
|
27
|
+
* And what it refuses. A calendar ladder walks a wall clock, a named
|
|
28
|
+
* zone needs host code, `locf`/`linear` couple every bucket to its
|
|
29
|
+
* neighbours, and `first`/`last` name a row by a position a maintained
|
|
30
|
+
* map does not preserve. Each of those re-runs with its own reason
|
|
31
|
+
* rather than being approximated. So does a reading older than the
|
|
32
|
+
* declared lateness: the view re-runs, the emission carries a
|
|
33
|
+
* `lateData` record naming the instant and the boundary, and the row is
|
|
34
|
+
* never quietly folded into a bucket its reader already believed
|
|
35
|
+
* closed.
|
|
36
|
+
*
|
|
37
|
+
* `retention` is the horizon this view claims to work over. It is
|
|
38
|
+
* checked, not assumed: it must cover a whole window plus the lateness
|
|
39
|
+
* the caller allows, which is the span a single repair can read. It is
|
|
40
|
+
* NOT a compaction policy — the maintained state is bounded by
|
|
41
|
+
* `live.maxMaintained` exactly as every other strategy's is, and this
|
|
42
|
+
* file drops nothing that an answer still depends on.
|
|
43
|
+
*/
|
|
44
|
+
/**
|
|
45
|
+
* Validate the `eventTime` option into the record the classifier and
|
|
46
|
+
* the strategies read, or `null` when the caller declared none.
|
|
47
|
+
* @param {any} options - the live options
|
|
48
|
+
* @param {string} collection - for the error's `collection` property
|
|
49
|
+
* @returns {null | { member: string, watermark: number,
|
|
50
|
+
* allowedLateness: number, retention: number }}
|
|
51
|
+
* @throws {DbCompileError} `JD0053` for any member this does not admit
|
|
52
|
+
*/
|
|
53
|
+
export declare function normalizeEventTime(options: any, collection: string): null | {
|
|
54
|
+
member: string;
|
|
55
|
+
watermark: number;
|
|
56
|
+
allowedLateness: number;
|
|
57
|
+
retention: number;
|
|
58
|
+
};
|
|
59
|
+
/**
|
|
60
|
+
* Classify a document as an event-time view, or say why it is not one.
|
|
61
|
+
*
|
|
62
|
+
* Returns `null` when the document does not name `$resample` or
|
|
63
|
+
* `$rolling` over this collection at all — the caller then goes on to
|
|
64
|
+
* §7's ordinary table. Every other outcome is a decision: a maintained
|
|
65
|
+
* description, or `{ strategy: 'rerun', reason }`.
|
|
66
|
+
* @param {any} inner - the unwrapped document
|
|
67
|
+
* @param {boolean} windowed - whether a `$subsequence` wrapped it
|
|
68
|
+
* @param {boolean} keyed
|
|
69
|
+
* @param {null | { member: string, watermark: number,
|
|
70
|
+
* allowedLateness: number, retention: number }} eventTime
|
|
71
|
+
* @returns {any}
|
|
72
|
+
*/
|
|
73
|
+
export declare function classifyEventTime(inner: any, windowed: boolean, keyed: boolean, eventTime: null | {
|
|
74
|
+
member: string;
|
|
75
|
+
watermark: number;
|
|
76
|
+
allowedLateness: number;
|
|
77
|
+
retention: number;
|
|
78
|
+
}): any;
|
|
79
|
+
/**
|
|
80
|
+
* `$resample` over a fixed ladder: one maintained fold per bucket.
|
|
81
|
+
* @param {any} description
|
|
82
|
+
* @param {any} context
|
|
83
|
+
*/
|
|
84
|
+
export declare function bucketStrategy(description: any, context: any): {
|
|
85
|
+
advance: (next: any) => void;
|
|
86
|
+
stats: () => {
|
|
87
|
+
lateData: number;
|
|
88
|
+
reruns: number;
|
|
89
|
+
recomputes: number;
|
|
90
|
+
watermark: any;
|
|
91
|
+
};
|
|
92
|
+
entries: () => number;
|
|
93
|
+
init: () => any;
|
|
94
|
+
apply(record: any): {
|
|
95
|
+
rebuild: boolean;
|
|
96
|
+
late: any;
|
|
97
|
+
rows?: undefined;
|
|
98
|
+
} | {
|
|
99
|
+
rebuild?: undefined;
|
|
100
|
+
late?: undefined;
|
|
101
|
+
rows: {
|
|
102
|
+
at: any;
|
|
103
|
+
value: number | null;
|
|
104
|
+
count: number;
|
|
105
|
+
}[] | ({
|
|
106
|
+
at: number;
|
|
107
|
+
value: number | null;
|
|
108
|
+
count: number;
|
|
109
|
+
} | undefined)[];
|
|
110
|
+
} | null;
|
|
111
|
+
/** A re-run rebuilds the whole state from the store — the only
|
|
112
|
+
* answer to a reading the maintained state cannot place. */
|
|
113
|
+
rebuild(): any;
|
|
114
|
+
};
|
|
115
|
+
/**
|
|
116
|
+
* `$rolling` over a fixed width: one output per input instant, with
|
|
117
|
+
* only the stretch a write can reach recomputed.
|
|
118
|
+
* @param {any} description
|
|
119
|
+
* @param {any} context
|
|
120
|
+
*/
|
|
121
|
+
export declare function rollingStrategy(description: any, context: any): {
|
|
122
|
+
advance: (next: any) => void;
|
|
123
|
+
stats: () => {
|
|
124
|
+
lateData: number;
|
|
125
|
+
reruns: number;
|
|
126
|
+
recomputes: number;
|
|
127
|
+
watermark: any;
|
|
128
|
+
};
|
|
129
|
+
entries: () => number;
|
|
130
|
+
init: () => any;
|
|
131
|
+
apply(record: any): {
|
|
132
|
+
rows?: undefined;
|
|
133
|
+
rebuild: boolean;
|
|
134
|
+
late: any;
|
|
135
|
+
} | {
|
|
136
|
+
rebuild?: undefined;
|
|
137
|
+
late?: undefined;
|
|
138
|
+
rows: any[];
|
|
139
|
+
} | null;
|
|
140
|
+
rebuild(): any;
|
|
141
|
+
};
|
package/dist/types/live.d.ts
CHANGED
|
@@ -31,9 +31,11 @@ export declare const LIVE_DEFAULTS: Readonly<{
|
|
|
31
31
|
* columnByCanonical)
|
|
32
32
|
* @param {boolean} keyed - whether documents carry their key (a
|
|
33
33
|
* declared key pointer); unkeyed rows cannot be tracked by key
|
|
34
|
+
* @param {any} [eventTime] - the normalized `eventTime` option
|
|
35
|
+
* (`live-time.js`), or null when the caller declared none
|
|
34
36
|
* @returns {any}
|
|
35
37
|
*/
|
|
36
|
-
export declare function classifyLiveQuery(document: any, queryShape: any, keyed: boolean): any;
|
|
38
|
+
export declare function classifyLiveQuery(document: any, queryShape: any, keyed: boolean, eventTime?: any): any;
|
|
37
39
|
/**
|
|
38
40
|
* Diff two row arrays into sequential add/remove/replace ops under
|
|
39
41
|
* `/rows`, relying on REFERENCE identity for unchanged rows (the §9
|
package/dist/types/plan.d.ts
CHANGED
|
@@ -60,6 +60,7 @@ export declare function assertDecidedKind(node: any): void;
|
|
|
60
60
|
* udfs: string[],
|
|
61
61
|
* prefilters: { construct: string, via: 'columns' | 'rtree',
|
|
62
62
|
* columns: string[], exact: boolean }[],
|
|
63
|
+
* series: any,
|
|
63
64
|
* }}
|
|
64
65
|
*/
|
|
65
66
|
export declare function planQuery(document: any, shape: any, options?: {
|
|
@@ -83,6 +84,7 @@ export declare function planQuery(document: any, shape: any, options?: {
|
|
|
83
84
|
columns: string[];
|
|
84
85
|
exact: boolean;
|
|
85
86
|
}[];
|
|
87
|
+
series: any;
|
|
86
88
|
};
|
|
87
89
|
/**
|
|
88
90
|
* Build the planner shape for one entity: canonical top-level paths
|
package/dist/types/query.d.ts
CHANGED
|
@@ -37,12 +37,13 @@
|
|
|
37
37
|
* residual.
|
|
38
38
|
* @param {number} [bound]
|
|
39
39
|
* @param {{ functions?: any, extensions?: any } | null} [operators]
|
|
40
|
+
* @param {any} [zoneProvider] - D7's injected clock, or absent
|
|
40
41
|
* @returns {any}
|
|
41
42
|
*/
|
|
42
43
|
export declare function createQueryState(bound?: number, operators?: {
|
|
43
44
|
functions?: any;
|
|
44
45
|
extensions?: any;
|
|
45
|
-
} | null): any;
|
|
46
|
+
} | null, zoneProvider?: any): any;
|
|
46
47
|
/**
|
|
47
48
|
* The query engine for one collection.
|
|
48
49
|
* @param {{ connection: any, state: any, collection: any,
|
package/dist/types/residual.d.ts
CHANGED
|
@@ -29,12 +29,14 @@
|
|
|
29
29
|
* @param {any} [limits]
|
|
30
30
|
* @param {{ functions?: any, extensions?: any } | null} [operators] -
|
|
31
31
|
* the store's registered operators, so the residual can evaluate them
|
|
32
|
+
* @param {any} [zoneProvider] - D7's injected clock, so a calendar
|
|
33
|
+
* ladder on a named zone compiles rather than being refused
|
|
32
34
|
* @returns {(candidates: any[], externals: any) => any}
|
|
33
35
|
*/
|
|
34
36
|
export declare function compileSetResidual(document: any, limits?: any, operators?: {
|
|
35
37
|
functions?: any;
|
|
36
38
|
extensions?: any;
|
|
37
|
-
} | null): (candidates: any[], externals: any) => any;
|
|
39
|
+
} | null, zoneProvider?: any): (candidates: any[], externals: any) => any;
|
|
38
40
|
/**
|
|
39
41
|
* Compile the per-row projection for row-mode evaluation.
|
|
40
42
|
* @param {any} rowDocument - The planner's complete one-row document
|
|
@@ -44,12 +46,13 @@ export declare function compileSetResidual(document: any, limits?: any, operator
|
|
|
44
46
|
* the only place that knows the name.
|
|
45
47
|
* @param {any} [limits]
|
|
46
48
|
* @param {{ functions?: any, extensions?: any } | null} [operators]
|
|
49
|
+
* @param {any} [zoneProvider] - D7's injected clock
|
|
47
50
|
* @returns {(row: any, externals: any) => any[]} the row's items
|
|
48
51
|
*/
|
|
49
52
|
export declare function compileRowResidual(rowDocument: any, limits?: any, operators?: {
|
|
50
53
|
functions?: any;
|
|
51
54
|
extensions?: any;
|
|
52
|
-
} | null): (row: any, externals: any) => any[];
|
|
55
|
+
} | null, zoneProvider?: any): (row: any, externals: any) => any[];
|
|
53
56
|
/**
|
|
54
57
|
* Map a flat item array onto the engine's result shape: an empty
|
|
55
58
|
* sequence is `undefined`, a singleton is the item, anything longer is
|