@jarenjs/db 0.66.1 → 0.72.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +19 -0
- package/README.md +44 -4
- package/docs/JOBS-FORMAT.md +14 -1
- package/docs/LIVE-FORMAT.md +47 -6
- package/docs/MIGRATION-FORMAT.md +45 -26
- package/docs/MODEL-FORMAT.md +8 -0
- package/docs/REPLICATION-FORMAT.md +208 -0
- package/package.json +4 -4
- package/schemas/jaren-replication-snapshot.draft-07.schema.json +83 -0
- package/schemas/jaren-replication-snapshot.schema.json +83 -0
- package/schemas/jaren-replication.draft-07.schema.json +82 -0
- package/schemas/jaren-replication.schema.json +82 -0
- package/src/capture.js +13 -3
- package/src/cli.js +89 -68
- package/src/cursor.js +11 -5
- package/src/dag-job.js +17 -12
- package/src/dialect.js +1 -1
- package/src/dialects/sqlite.js +1 -0
- package/src/document-files.js +76 -20
- package/src/document-steps.js +94 -120
- package/src/documents.js +24 -6
- package/src/drivers/node.js +1 -1
- package/src/errors.js +8 -0
- package/src/index.js +2 -0
- package/src/jobs.js +42 -0
- package/src/live-join.js +250 -0
- package/src/live-nested.js +120 -0
- package/src/live.js +18 -4
- package/src/logical-rows.js +90 -0
- package/src/migrate.js +44 -5
- package/src/replication-format.js +115 -0
- package/src/replication.js +332 -0
- package/src/store.js +73 -14
- package/types/index.d.ts +94 -10
- package/types/node.d.ts +9 -3
- package/types/typed.d.ts +3 -2
package/ARCHITECTURE.md
CHANGED
|
@@ -1095,3 +1095,22 @@ processes creating one fresh file never meet the deferred-upgrade
|
|
|
1095
1095
|
`SQLITE_BUSY` the busy handler cannot retry. The job queue's
|
|
1096
1096
|
administration (`page`, `cancel`, `requeue`, `sweep`) lives in
|
|
1097
1097
|
`src/jobs.js` beside the fence it authorises through (JOBS-FORMAT §10).
|
|
1098
|
+
|
|
1099
|
+
|
|
1100
|
+
## Replication and bounded dependency maintenance
|
|
1101
|
+
|
|
1102
|
+
`replication-format.js` owns JSON normalization; `replication.js` persists replica
|
|
1103
|
+
identity, frontiers, canonical receipts, before-images, outbox and conflict
|
|
1104
|
+
evidence through the capture transaction's `beforeCommit` seam. `logical-rows.js`
|
|
1105
|
+
shares mapped, validated row access between replication and live dependencies.
|
|
1106
|
+
Remote application suppresses only local envelope allocation: its data still
|
|
1107
|
+
invalidates capture subscribers after commit. Snapshot reset carries complete
|
|
1108
|
+
receipt evidence and refuses histories beyond its explicit credits.
|
|
1109
|
+
|
|
1110
|
+
`live-join.js` compiles indexed dependency descriptors from the entity planner
|
|
1111
|
+
and mapping. The same cache strategy maintains inner/left joins and eligible
|
|
1112
|
+
nested graph projections. `live-nested.js` maintains explicit two-level groups
|
|
1113
|
+
by refreshing affected parents from bounded leaves. Source and result payloads
|
|
1114
|
+
consume row and byte credits; unsupported shapes keep named reruns. See
|
|
1115
|
+
[the replication contract](docs/REPLICATION-FORMAT.md) and
|
|
1116
|
+
[the strategy matrix](docs/LIVE-FORMAT.md).
|
package/README.md
CHANGED
|
@@ -688,6 +688,15 @@ opinion: <!--fact:vector.ceiling-->5.142 ns per vector component — one query r
|
|
|
688
688
|
Past that this design is the wrong tool and no margin changes it; what
|
|
689
689
|
lies beyond is an approximate index, and this store does not have one.
|
|
690
690
|
|
|
691
|
+
The [labelled retrieval comparison](../../benchmark/README.md#labelled-recall-and-repeated-refinement)
|
|
692
|
+
also scores an optional storage adapter with approximate candidates against
|
|
693
|
+
exact cosine on cached live embeddings. The AI ledger reports the adapter's
|
|
694
|
+
algorithm, exhaustive flag and returned candidate count, validates candidate
|
|
695
|
+
identity/vector shape, and re-scores candidates itself. The DB ledger recipe
|
|
696
|
+
continues to normalize as an exact adapter. No ANN runtime dependency is added.
|
|
697
|
+
|
|
698
|
+
Index decision: <!--fact:recall.annDecision-->0/6 contender rows cleared all bars; retain exact. Required exact-top-10 recall ≥ 0.95, p95 speedup ≥ 2×, and a measured exact p95 ≥ 100 ms. The largest reference corpus contains 5183 documents; scale beyond it remains unmeasured.<!--/fact-->
|
|
699
|
+
|
|
691
700
|
**One document, three executors, proven to agree.** As with the spatial
|
|
692
701
|
family, the k-nearest shapes of a committed corpus
|
|
693
702
|
(`test/json/fixtures/vector-corpus.json`) run through the JavaScript
|
|
@@ -990,11 +999,17 @@ shutdown limits, the browser persistence matrix and measured latency/memory loss
|
|
|
990
999
|
`sessionReason`. One diff
|
|
991
1000
|
format runs store → patch → live query → O(k) render. Capture is
|
|
992
1001
|
opt-in; the overhead is published, not waved away.
|
|
1002
|
+
- **Portable replication** ([REPLICATION-FORMAT](docs/REPLICATION-FORMAT.md)):
|
|
1003
|
+
opt-in replica identities, causal frontiers, bounded logical envelopes,
|
|
1004
|
+
durable replay receipts, explicit conflict evidence and snapshot resets.
|
|
1005
|
+
Data and acknowledgements commit together. Hosts supply transport and any
|
|
1006
|
+
conflict resolver; the default preserves both contenders and rejects the write.
|
|
993
1007
|
- **Live queries** (LIVE-FORMAT §§7–13): `collection.live(document)`
|
|
994
1008
|
maintains a result as writes arrive and emits patches — incremental
|
|
995
1009
|
for `where`/`select`/`orderBy`+`limit`/aggregates/single-level
|
|
996
1010
|
`groupBy` and a spatial `where` over a derived index (the geofence;
|
|
997
|
-
the normative maintenance table),
|
|
1011
|
+
the normative maintenance table), indexed inner/left entity joins, bounded
|
|
1012
|
+
graph projections and explicit two-level groups; re-run for other shapes,
|
|
998
1013
|
**declared, never silent** (`live.mode` names the reason).
|
|
999
1014
|
Unaffected rows stay reference-identical; a seeded oracle holds the
|
|
1000
1015
|
maintained result equal to a fresh re-query after every mutation.
|
|
@@ -1022,6 +1037,10 @@ shutdown limits, the browser persistence matrix and measured latency/memory loss
|
|
|
1022
1037
|
and `worker.stop()` quiesces every claim, renewal, checkpoint and
|
|
1023
1038
|
settlement before it resolves, so closing the store releases the
|
|
1024
1039
|
database file deterministically.
|
|
1040
|
+
DAG jobs inspect persisted workflow/input/task identity before reading
|
|
1041
|
+
checkpoint values. `jobs.reset(id, { expectedGeneration })` explicitly
|
|
1042
|
+
discards an inactive run's checkpoints and restarts it under a new fence;
|
|
1043
|
+
live leases and stale observations refuse (JOBS-FORMAT §10).
|
|
1025
1044
|
- **The browser** (`@jarenjs/db/wasm`): the same store, the same
|
|
1026
1045
|
queries, the same live updates run on the official SQLite wasm build
|
|
1027
1046
|
over a probed persistence ladder: isolated SharedArrayBuffer OPFS,
|
|
@@ -1276,7 +1295,8 @@ its side-effect-free status read are in
|
|
|
1276
1295
|
there. Same-host processes over WAL are the supported topology. No
|
|
1277
1296
|
priority classes, no cron, no workflow compensation.
|
|
1278
1297
|
- **Live-query maintenance is limited to the declared table** (§7);
|
|
1279
|
-
joins
|
|
1298
|
+
indexed joins and graph projections require bounded dependencies. Offset
|
|
1299
|
+
windows, unindexed joins, load-spec graphs and non-canonical shapes re-run, reported.
|
|
1280
1300
|
- **`eventTime.retention` bounds repair work, not memory.** It is the
|
|
1281
1301
|
horizon a view claims and is checked against the window it maintains;
|
|
1282
1302
|
the maintained state is still bounded by `live.maxMaintained`, and no
|
|
@@ -1287,8 +1307,8 @@ its side-effect-free status read are in
|
|
|
1287
1307
|
durability differences; see [execution hosts](docs/HOSTS.md).
|
|
1288
1308
|
- **Named future work, not silent gaps**: `$groupby` pushdown beyond
|
|
1289
1309
|
the `$time-bucket` ladder, a many-to-many hop on the chain, membership
|
|
1290
|
-
on an auto-keyed pending insert,
|
|
1291
|
-
|
|
1310
|
+
on an auto-keyed pending insert, additional join/group shapes, other SQL dialects,
|
|
1311
|
+
transport policy, database introspection (MODEL-FORMAT §10.6, the roadmap).
|
|
1292
1312
|
|
|
1293
1313
|
The normative formats are
|
|
1294
1314
|
[docs/MODEL-FORMAT.md](docs/MODEL-FORMAT.md) (storage §§1–7, safe
|
|
@@ -1320,7 +1340,27 @@ Every subpath a consumer can import, derived from the manifest by
|
|
|
1320
1340
|
| `@jarenjs/db/schemas/jaren-migration.schema.json` | schema | — |
|
|
1321
1341
|
| `@jarenjs/db/schemas/jaren-model.draft-07.schema.json` | schema | — |
|
|
1322
1342
|
| `@jarenjs/db/schemas/jaren-model.schema.json` | schema | — |
|
|
1343
|
+
| `@jarenjs/db/schemas/jaren-replication-snapshot.draft-07.schema.json` | schema | — |
|
|
1344
|
+
| `@jarenjs/db/schemas/jaren-replication-snapshot.schema.json` | schema | — |
|
|
1345
|
+
| `@jarenjs/db/schemas/jaren-replication.draft-07.schema.json` | schema | — |
|
|
1346
|
+
| `@jarenjs/db/schemas/jaren-replication.schema.json` | schema | — |
|
|
1323
1347
|
| `@jarenjs/db/package.json` | metadata | — |
|
|
1324
1348
|
| `@jarenjs/db/node-worker` | JavaScript | declared |
|
|
1325
1349
|
| `@jarenjs/db/node-pool` | JavaScript | declared |
|
|
1326
1350
|
<!--/fact-->
|
|
1351
|
+
|
|
1352
|
+
|
|
1353
|
+
The [durable AI ledger recipe](../ai/README.md#a-durable-ledger-over-jarenjsdb)
|
|
1354
|
+
exposes optional atomic namespace mutation as well as its base storage and rank
|
|
1355
|
+
capabilities. It reads and publishes through one immediate transaction, using the
|
|
1356
|
+
transaction's synchronous collection facade so another SQLite connection on the
|
|
1357
|
+
same event loop cannot block an awaiting writer. A rejected callback changes
|
|
1358
|
+
neither records nor counters; ledger id minting and guarded multi-record updates
|
|
1359
|
+
use that boundary.
|
|
1360
|
+
|
|
1361
|
+
Document-only migrations also accept named files:
|
|
1362
|
+
`jaren-db documents --migrations migrations --in users=users.json --in events=events.jsonl --out migrated.json`.
|
|
1363
|
+
The output is one atomically published collection bundle; read it with
|
|
1364
|
+
`--format collections`. Assertion plans expose provider, ordered-fold and
|
|
1365
|
+
bounded materialization strategies through `onAssertionPlan`.
|
|
1366
|
+
See [MIGRATION-FORMAT §6 and §11](docs/MIGRATION-FORMAT.md).
|
package/docs/JOBS-FORMAT.md
CHANGED
|
@@ -390,7 +390,10 @@ await store.jobs.enqueue('sync-report', { input: { day: '2026-08-05' } });
|
|
|
390
390
|
|
|
391
391
|
**The legacy rule is deterministic, and never reads unknown as equal.**
|
|
392
392
|
A run checkpointed by a release that did not record task identity has
|
|
393
|
-
no `taskVersionsHash`.
|
|
393
|
+
no `taskVersionsHash`. Identity inspection reads only the reserved metadata
|
|
394
|
+
value and probes whether other rows exist; it does not load or parse node
|
|
395
|
+
values. Missing metadata with saved values refuses `JD2069`. Refused
|
|
396
|
+
legacy upgrades leave the original identity untouched. If it recorded no node value yet, there is
|
|
394
397
|
nothing that could be replayed wrongly, so the identity is upgraded in
|
|
395
398
|
place and the run proceeds. If it DID record values, the implementation
|
|
396
399
|
that produced them cannot be confirmed and the resume is refused,
|
|
@@ -471,6 +474,16 @@ a cancellation policy, a retry policy are the host's.
|
|
|
471
474
|
and counts it under `stats().cancellations`; an attempt held by
|
|
472
475
|
another process meets the fence at its next settling call and records
|
|
473
476
|
a loss, as any superseded attempt does.
|
|
477
|
+
- **`reset(id, { expectedGeneration })`** — explicitly discard one inactive
|
|
478
|
+
run's checkpoints and restart its attempts at zero. Read the job's
|
|
479
|
+
`leaseGeneration` first and pass that observation. One transaction
|
|
480
|
+
conditionally updates the job, increments its fence generation and deletes
|
|
481
|
+
its checkpoint rows. A concurrent generation change refuses `JD2066`, a
|
|
482
|
+
live lease `JD2068`, an unknown or completed job `JD2065`; no refusal
|
|
483
|
+
discards values. The next claim recomputes the workflow under current
|
|
484
|
+
identity. Reset does not undo external task effects; their idempotency
|
|
485
|
+
keys remain the host's responsibility. It is root-only administration,
|
|
486
|
+
with the same signal/deadline options as requeue.
|
|
474
487
|
- **`requeue(id)`** — returns a failed, dead, cancelled or lease-expired
|
|
475
488
|
job to `pending` at the clock's instant (the queue's own order). The
|
|
476
489
|
attempt history is KEPT: `attempts` counts every claim the job ever
|
package/docs/LIVE-FORMAT.md
CHANGED
|
@@ -273,7 +273,10 @@ what the pushdown planner already means by it.
|
|
|
273
273
|
| a spatial predicate the planner **refused** (no `derive` index on the member, an untyped member, an unbounded probe) | **re-run on invalidation**, the refusal named — it never translated, so nothing narrows the fetch | the previous result, for diffing |
|
|
274
274
|
| a `$resample` or `$rolling` document over the collection, with an explicit `eventTime` and a fixed width (§13) | **event-time bucket / rolling state**: rows kept by bucket, or in instant order; only what a write can reach is folded again, through `@jarenjs/core/series` itself | the contributing rows, plus one fold per bucket |
|
|
275
275
|
| the same document with no `eventTime`, a calendar width, a named zone, a `locf`/`linear` fill, a `first`/`last` aggregate, or a retention that does not cover the window | **re-run on invalidation**, the member that stopped it named (§13.2) | the previous result, for diffing |
|
|
276
|
-
| joins
|
|
276
|
+
| indexed inner equi-joins and canonical allowing-empty left joins over mapped entity roots | **join dependency maintenance**; point-read changed keys and reevaluate their bounded outer owners | source rows, key indexes and projected tuples, bounded by `maxMaintained` and `maxBytes` |
|
|
277
|
+
| nested entity graph projections with indexed equality edges and unique binding names | **graph dependency maintenance**; a child change refreshes its bounded owners | source rows, reverse key indexes and graph outputs |
|
|
278
|
+
| explicit two-level collection groups with a singular parent key and bounded nested input | **nested-group maintenance**; recompute affected parents through the query engine | source leaves and parent outputs |
|
|
279
|
+
| unindexed/non-equi joins, self joins, explicit entity ordering/windows, object-root documents and load-spec graphs | **re-run on invalidation**, with the dependency or planner reason | previous result for diffing |
|
|
277
280
|
| anything else: non-translatable predicates, `limit` without `orderBy`, `offset` > 0, windowed aggregates, `@jarenjs/linq`'s nested two-level `groupBy` emission, non-canonical group returns | **re-run on invalidation**, the reason named | the previous result, for diffing |
|
|
278
281
|
|
|
279
282
|
The **physical mapping** of a `derive: 'bbox'` index (MODEL-FORMAT §2.1,
|
|
@@ -310,8 +313,8 @@ refuses at registration (`JD0051`), the same shape as capture's
|
|
|
310
313
|
demanded session — an application that needs the property can refuse
|
|
311
314
|
to start.
|
|
312
315
|
|
|
313
|
-
The **canonical group form** the classifier recognises (
|
|
314
|
-
|
|
316
|
+
The **canonical group form** the classifier recognises (the linq chain's group-of-groups emission still re-runs; explicit
|
|
317
|
+
nested groups have a separate bounded strategy below):
|
|
315
318
|
|
|
316
319
|
```json
|
|
317
320
|
{ "$for": { "it": "$[*]" },
|
|
@@ -502,13 +505,14 @@ ERRORING rather than degrading (the D14 rule — the bound is printed):
|
|
|
502
505
|
delete-correctness, and a count over a table larger than the bound
|
|
503
506
|
is a conscious `maxMaintained` raise, not a silent one.
|
|
504
507
|
|
|
505
|
-
Non-claims, in one place: no
|
|
506
|
-
|
|
508
|
+
Non-claims, in one place: no maintenance of unindexed or non-equality joins,
|
|
509
|
+
no cross-connection invalidation (§6's `data_version` is
|
|
507
510
|
the signal), no maintenance over asynchronous connections —
|
|
508
511
|
`capabilities.live` is `false` there and a registration is `JD0051`
|
|
509
512
|
naming the reason, because maintenance point-reads rows synchronously
|
|
510
513
|
inside delivery (the wasm driver's oo1 API is synchronous, which is
|
|
511
|
-
why the browser has live queries at all)
|
|
514
|
+
why the browser has live queries at all). Replication is specified separately in
|
|
515
|
+
[REPLICATION-FORMAT](REPLICATION-FORMAT.md). There is no ordering guarantee for
|
|
512
516
|
unordered queries beyond §9's determinism.
|
|
513
517
|
|
|
514
518
|
## 13. Event time
|
|
@@ -616,3 +620,40 @@ holds a shuffled stream of inserts, in-place updates, instant moves and
|
|
|
616
620
|
deletes against `resampleSeries` / `rollingSeries` over the whole
|
|
617
621
|
collection after each one, which is the only oracle that cannot drift
|
|
618
622
|
with the implementation.
|
|
623
|
+
|
|
624
|
+
|
|
625
|
+
## Bounded joins, graph projections and nested groups
|
|
626
|
+
|
|
627
|
+
`join`, `graph` and `nested-group` strategies charge their source rows and
|
|
628
|
+
result entries to `live.maxMaintained`, and serialized input/output payloads to
|
|
629
|
+
`live.maxBytes` (default 4 MiB). These credits bound cached payloads rather than
|
|
630
|
+
claiming to measure JavaScript heap overhead. Initialization uses a limited
|
|
631
|
+
source read; updates read changed keys, then visit cached indexed dependencies.
|
|
632
|
+
Bounds are checked while caches grow. Overflow is `JD2060`, emits one error,
|
|
633
|
+
closes the subscription and releases dependency caches. It never relabels an
|
|
634
|
+
unbounded query as incremental.
|
|
635
|
+
|
|
636
|
+
Entity equality columns need a primary-key prefix, declared index or mapped
|
|
637
|
+
foreign-key index. Every binding needs a key and a distinct root. The projected
|
|
638
|
+
identity is the tuple of source identities, so duplicate projected values remain
|
|
639
|
+
distinct. Default entity result order follows physical row insertion order;
|
|
640
|
+
point reads preserve that order even when a key is deleted and reinserted. A
|
|
641
|
+
left join uses a canonical `$allowing-empty` binding over an equality-filtered
|
|
642
|
+
inner subquery. Its absent child can be defaulted to null. Graph projections
|
|
643
|
+
embed equality-filtered child queries in a single outer row's return object.
|
|
644
|
+
Global-root reads outside those bindings re-run because changing one row can
|
|
645
|
+
change every projected graph.
|
|
646
|
+
|
|
647
|
+
`dependencyReads`, `refreshedRoots` and `refreshedGroups` expose the work done.
|
|
648
|
+
`dependencyReads` counts logical changed-row reads; a row-position lookup is an
|
|
649
|
+
additional statement. No full query reruns occur under these strategy labels.
|
|
650
|
+
Materializing and diffing the final bounded output still costs work proportional
|
|
651
|
+
to its size. See the equal-correctness [measurements](REPLICATION-FORMAT.md#measurements)
|
|
652
|
+
for startup and high-fan-out losses beside selective wins.
|
|
653
|
+
|
|
654
|
+
Two-level grouping currently accepts an explicit parent `$groupby` over a
|
|
655
|
+
singular member, with one nested group over that parent's bound row sequence.
|
|
656
|
+
Count, sum, average, minimum and maximum recompute from only the affected
|
|
657
|
+
parent's bounded leaves. An offset, an unsupported operator, a global input to
|
|
658
|
+
the nested group, or a group-of-groups LINQ emission remains a named rerun.
|
|
659
|
+
Replicated writes enter the same committed capture stream as local writes.
|
package/docs/MIGRATION-FORMAT.md
CHANGED
|
@@ -235,25 +235,34 @@ hash of the `baseline` model when no migration has run.
|
|
|
235
235
|
|
|
236
236
|
| strategy | which assertions | what it costs |
|
|
237
237
|
|---|---|---|
|
|
238
|
-
|
|
|
239
|
-
|
|
|
240
|
-
|
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
238
|
+
| provider | a collection `$count` that the existing query planner proves native without a typed intermediate schema | one aggregate result, with no assertion document fetch |
|
|
239
|
+
| per-document | an independent, unwindowed FLWOR over `$[*]`, with the default empty-sequence expectation | one keyset batch; fails at the first violation |
|
|
240
|
+
| fold | `$count`, `$sum`, `$avg`, `$min`, `$max` over a partition-independent operand; sequence EBV of an independent FLWOR | one batch plus fixed accumulator state |
|
|
241
|
+
| fold with bounds | `$distinct` over such an operand with an explicit positive `maxDistinct` | one batch plus unique items under cardinality and byte bounds |
|
|
242
|
+
| materialize | global or positional operands, nested scans, unsupported shapes, or unbounded distinct | the collection under `assertionBounds` |
|
|
243
|
+
|
|
244
|
+
Folds feed the operand's sequence items, in original order, into the
|
|
245
|
+
query engine's shared accumulator. Floating-point addition is **not
|
|
246
|
+
associative**: combining batch totals changes answers. The ordered state
|
|
247
|
+
preserves the whole-query result, including singleton arrays, negative
|
|
248
|
+
zero, NaN, mixed-type refusals and Unicode code-point ordering. Sequence
|
|
249
|
+
EBV is checked once across all batches, including an empty source.
|
|
250
|
+
`$[0]`, root-dependent filters and positional bindings never qualify as
|
|
251
|
+
independent merely because they occur inside an aggregate.
|
|
252
|
+
|
|
253
|
+
`classifyAssertion(query, { expect, maxDistinct })` exposes the portable
|
|
254
|
+
strategy. `onAssertionPlan(plan)` reports migration, step, collection,
|
|
255
|
+
strategy, shape, reason and bounds before the assertion reads documents;
|
|
256
|
+
Store dry-run query statements also expose strategy and reason. Provider
|
|
257
|
+
promotion reuses the query planner with no assumed typed columns. Typed
|
|
258
|
+
SQL aggregates still need a trustworthy intermediate schema and proof
|
|
259
|
+
that their numeric/order semantics match; otherwise the ordered fold runs.
|
|
251
260
|
|
|
252
261
|
**A materializing assertion is bounded.** `options.assertionBounds`
|
|
253
262
|
defaults to `{ maxRows: 100000, maxBytes: 67108864 }` and is crossed
|
|
254
263
|
BEFORE the excess is held — the walk stops at the row that would break
|
|
255
264
|
it, refusing `JD2007` (rows) or `JD2076` (bytes) and naming the two
|
|
256
|
-
assertion
|
|
265
|
+
assertion strategies that are answered in batches instead. `null` on either
|
|
257
266
|
member removes that bound, which a caller must ask for: an unbounded
|
|
258
267
|
read nobody declared is exactly what this classification removes. This
|
|
259
268
|
is a deliberate behavior change — a migration that used to read a very
|
|
@@ -265,14 +274,11 @@ hash of the `baseline` model when no migration has run.
|
|
|
265
274
|
transformed | derived | asserted }`, one event per batch), and never
|
|
266
275
|
hold the whole collection in memory. A PER-DOCUMENT assertion — a
|
|
267
276
|
FLWOR over `$[*]` whose `$where` and `$return` read only the binding
|
|
268
|
-
— walks the same batches and fails
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
early, before the data grows. A cross-document assertion that is one
|
|
274
|
-
associative aggregate no longer costs that read at all — see the
|
|
275
|
-
classification table above.
|
|
277
|
+
— with the default empty expectation walks the same batches and fails
|
|
278
|
+
fast. Other assertions follow the classification table above.
|
|
279
|
+
`batchSize` must be a positive safe integer. `maxDistinct` opts into a
|
|
280
|
+
distinct fold and bounds retained unique items; `maxBytes` applies to
|
|
281
|
+
those items. Duplicates do not consume additional cardinality.
|
|
276
282
|
- A transform MUST NOT change a caller-keyed document's key member —
|
|
277
283
|
the key column would go stale; the run refuses (`JD0023`).
|
|
278
284
|
|
|
@@ -510,10 +516,23 @@ jaren-db documents --migrations <dir> --in <file|-> (--out <file|-> | --in-place
|
|
|
510
516
|
line-delimited, everything else one JSON array) unless `--format` /
|
|
511
517
|
`--out-format` says otherwise, and stdio defaults to JSONL. Input and
|
|
512
518
|
output encodings are independent, so this is also the converter.
|
|
513
|
-
- **
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
519
|
+
- **Named sources share one collection bundle.** Repeat
|
|
520
|
+
`--in users=users.json --in events=events.jsonl --out migrated.json`.
|
|
521
|
+
Output defaults to `--out-format collections`: a standard JSON object
|
|
522
|
+
mapping collection names to arrays, including empty collections.
|
|
523
|
+
Publication is one atomic rename after every collection succeeds.
|
|
524
|
+
Multiple source files cannot use `--in-place`, because separate
|
|
525
|
+
renames cannot give this guarantee. Single-source `--collection`
|
|
526
|
+
still asserts the collection name and refuses a mismatch.
|
|
527
|
+
- **A bundle can be read explicitly** with `--format collections`, then
|
|
528
|
+
replaced using `--in-place --yes`. Bundle input is materialized under
|
|
529
|
+
a whole-file `maxBytes` ceiling before parsing and per-collection
|
|
530
|
+
`maxRows` validation after parsing. JSON and JSONL single-collection
|
|
531
|
+
sources remain streaming. `--max-rows`, `--max-bytes`, and
|
|
532
|
+
`--max-distinct` configure assertion admission too; the first two
|
|
533
|
+
accept `none` to remove their limit deliberately. A materializing
|
|
534
|
+
file run admits rows before retaining them. Physical-step and query
|
|
535
|
+
compilation refusals precede every input read.
|
|
517
536
|
- **`--out` writes elsewhere; `--in-place` replaces the input and
|
|
518
537
|
needs `--yes`.** Either way the documents land in a sibling
|
|
519
538
|
temporary that is renamed over the target only once every document
|
package/docs/MODEL-FORMAT.md
CHANGED
|
@@ -1078,6 +1078,14 @@ error.
|
|
|
1078
1078
|
| `JD2092` | a worker row, compatibility result or remote identity count exceeds its declared bound |
|
|
1079
1079
|
| `JD2093` | malformed worker protocol request |
|
|
1080
1080
|
| `JD2094` | invalid or uncommitted durable snapshot; reopen the last committed version |
|
|
1081
|
+
| `JD0060` | a replication envelope or snapshot is invalid |
|
|
1082
|
+
| `JD2100` | a replica sequence or causal dependency has a gap |
|
|
1083
|
+
| `JD2101` | an envelope identity names different content or an unknown local origin |
|
|
1084
|
+
| `JD2102` | the replica identity or model revision disagrees |
|
|
1085
|
+
| `JD2103` | a resolver fails its synchronous decision contract |
|
|
1086
|
+
| `JD2104` | a logical row or snapshot disagrees with its causal history |
|
|
1087
|
+
| `JD2105` | an explicit snapshot reset is required, or reset would discard acknowledged history |
|
|
1088
|
+
| `JD2106` | a replication operation or snapshot exceeds its configured bound |
|
|
1081
1089
|
|
|
1082
1090
|
The table above is proven in sync with the runtime `DB_CODES` table by
|
|
1083
1091
|
a test.
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
# Replication format and runtime
|
|
2
|
+
|
|
3
|
+
`jaren-replication` 0.1 is a transport-neutral logical transaction. Open an empty
|
|
4
|
+
store with `replication: { replica: 'host-issued-id' }`; capture is enabled when
|
|
5
|
+
omitted. Existing replication stores must reopen with the same identity and
|
|
6
|
+
model. Every writer of that file must enable replication. External SQL writes
|
|
7
|
+
and opening a replicated file without replication are outside the history
|
|
8
|
+
contract; a detected before-image disagreement refuses with `JD2104`.
|
|
9
|
+
|
|
10
|
+
The host owns network transport, authentication, replica identity allocation and
|
|
11
|
+
resolver policy. A replica id must remain unique to its writer lineage. The
|
|
12
|
+
runtime works on SQLite Node, worker and wasm hosts through session or journal
|
|
13
|
+
capture. Drivers without `changeCapture` refuse at open; PostgreSQL capture is
|
|
14
|
+
not implemented. Journal replication refuses models with cascading or set-null
|
|
15
|
+
child foreign keys (`JD0051`), because the journal cannot observe those effects.
|
|
16
|
+
Session capture includes them. Both modes support membership cascades and
|
|
17
|
+
roll back an envelope whose side effects are absent from its logical operations.
|
|
18
|
+
|
|
19
|
+
## Envelope
|
|
20
|
+
|
|
21
|
+
```json
|
|
22
|
+
{
|
|
23
|
+
"$replication": "0.1",
|
|
24
|
+
"replica": "site-a",
|
|
25
|
+
"seq": 1,
|
|
26
|
+
"frontier": {},
|
|
27
|
+
"model": "application-model-revision",
|
|
28
|
+
"operations": [
|
|
29
|
+
{ "table": "notes", "key": "note/one", "before": null,
|
|
30
|
+
"after": { "id": "note/one", "body": "hello" } }
|
|
31
|
+
]
|
|
32
|
+
}
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
`model` is the store's `shapeHash(model)` revision. `seq` is a positive safe
|
|
36
|
+
integer, allocated from the replica's durable frontier in the data transaction.
|
|
37
|
+
The identity is `replicationIdentity(replica, seq)`, the JSON encoding of their
|
|
38
|
+
pair, so separators inside host ids cannot collide. A frontier maps host ids to
|
|
39
|
+
non-negative safe sequences. The sender's frontier entry must equal `seq - 1`,
|
|
40
|
+
with an absent entry meaning zero. Property names such as `__proto__` and
|
|
41
|
+
`constructor` remain ordinary replica ids.
|
|
42
|
+
|
|
43
|
+
Each operation is one net row transition. `before: null` inserts, `after: null`
|
|
44
|
+
deletes, and two objects replace the complete logical row. A table/key occurs
|
|
45
|
+
once per envelope. Composite keys use the capture format's JSON-array token;
|
|
46
|
+
single keys use their scalar text. Entity documents include mapped scalar and
|
|
47
|
+
foreign-key fields, omit relation projections and retain authored version and
|
|
48
|
+
timestamp values. Membership tables carry their two keys. Application performs
|
|
49
|
+
schema validation through the store's configured validator and reuses physical
|
|
50
|
+
column plans; it does not generate fresh defaults or version stamps.
|
|
51
|
+
|
|
52
|
+
`normalizeReplication` validates and detaches the document. `encodeReplication`
|
|
53
|
+
produces deterministic JSON. Object members are canonically ordered; operation
|
|
54
|
+
array order remains part of identity. Capture sorts net rows by table/key, making
|
|
55
|
+
physical session and journal output equal. Envelope order preserves transaction
|
|
56
|
+
order. Physical insertion order inside a netted transaction or snapshot is not a
|
|
57
|
+
portable query ordering: use an explicit query order when comparing replicas.
|
|
58
|
+
Receipts retain the entire canonical payload, avoiding hash-collision ambiguity.
|
|
59
|
+
|
|
60
|
+
Unknown members and versions, empty ids, unsafe sequences, non-JSON values,
|
|
61
|
+
cycles, duplicate net rows and unchanged operations are `JD0060`. Nesting is
|
|
62
|
+
limited to 128 object/array ancestors. JSON Schema artifacts describe structural
|
|
63
|
+
validation; cross-field sequence, uniqueness and causal checks are runtime
|
|
64
|
+
constraints. Current and draft-07 artifacts ship for envelopes and snapshots:
|
|
65
|
+
|
|
66
|
+
- `@jarenjs/db/schemas/jaren-replication.schema.json`
|
|
67
|
+
- `@jarenjs/db/schemas/jaren-replication.draft-07.schema.json`
|
|
68
|
+
- `@jarenjs/db/schemas/jaren-replication-snapshot.schema.json`
|
|
69
|
+
- `@jarenjs/db/schemas/jaren-replication-snapshot.draft-07.schema.json`
|
|
70
|
+
|
|
71
|
+
## Applying and exporting
|
|
72
|
+
|
|
73
|
+
```js
|
|
74
|
+
const source = await openStore(model, {
|
|
75
|
+
driver, replication: { replica: 'site-a', retention: 1000 }
|
|
76
|
+
});
|
|
77
|
+
const page = await source.replication.page({ after: 0, limit: 100, maxBytes: 1048576 });
|
|
78
|
+
for (const envelope of page.items) {
|
|
79
|
+
const result = await target.replication.apply(envelope);
|
|
80
|
+
if (result.status === 'conflict') hostReport(result.conflicts);
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
`apply` returns `applied`, `duplicate` or `conflict`, together with the committed
|
|
85
|
+
frontier and conflict records. Successful duplicate delivery performs no row
|
|
86
|
+
write or live emission. An acknowledged identity with different bytes is
|
|
87
|
+
`JD2101`. A sequence or causal gap is `JD2100`: no suffix is applied. Deliver
|
|
88
|
+
missing envelopes first, then retry, or perform an explicit reset. Model mismatch
|
|
89
|
+
is `JD2102`. Imported envelopes keep their original identity and produce no local
|
|
90
|
+
outbox entry, preventing echoes. A host forwarding between peers must preserve
|
|
91
|
+
the original envelopes; `page()` exports this replica's own commits only.
|
|
92
|
+
|
|
93
|
+
Data, receipt, per-row causal metadata, capture record and frontier settle in
|
|
94
|
+
the same transaction. Replication is available only on the root Store; transaction
|
|
95
|
+
views expose no replication API. Failed validation, cancellation, constraint checks or
|
|
96
|
+
commit roll everything back. Subscribers run through the existing committed
|
|
97
|
+
capture delivery path. Tests kill a process immediately before and after commit
|
|
98
|
+
and verify both data and frontier after reopening.
|
|
99
|
+
|
|
100
|
+
Pages use the shared cursor and byte-credit drain. `limit` counts whole envelopes;
|
|
101
|
+
`maxBytes` counts complete canonical envelope bytes. An indivisible oversize
|
|
102
|
+
envelope is `JD2074`. Retention loss returns `resetRequired: true`, no items and
|
|
103
|
+
no continuation. `earliestAvailable`, `highWatermark`, `next` and `hasMore` describe
|
|
104
|
+
the local outbox. Pages and snapshots take a consistent store transaction.
|
|
105
|
+
`signal` and `deadline` are honored through existing cancellation contracts:
|
|
106
|
+
queue cancellation is `JD2064`, operation-boundary cancellation `JD2072`, and an
|
|
107
|
+
expired deadline `JD2075`.
|
|
108
|
+
|
|
109
|
+
Defaults are `retention: 1000`, `maxOperations: 10000`, `maxBytes: 4194304`.
|
|
110
|
+
Operation overflow is `JD2106`; all credits are positive safe integers. Retention
|
|
111
|
+
prunes only outgoing envelope history. Replay receipts, row tombstones and
|
|
112
|
+
conflict evidence remain durable. They are deliberately not claimed to be a
|
|
113
|
+
bounded total database size. Internal tables use the reserved `_jaren_replica`
|
|
114
|
+
prefix. No schema migration or revision rewrite of this protocol is implicit.
|
|
115
|
+
|
|
116
|
+
## Conflicts
|
|
117
|
+
|
|
118
|
+
Concurrent incompatible whole-row writes reject by default. A `conflict` result
|
|
119
|
+
does not acknowledge the remote envelope and leaves the local row intact.
|
|
120
|
+
`replication.conflicts({ limit, maxBytes, signal, deadline })` reads persisted evidence: the common base,
|
|
121
|
+
local value and frontier, remote value and envelope position, resolver identity
|
|
122
|
+
and decision. A rejected envelope reserves its canonical identity too, so retries
|
|
123
|
+
cannot replace its contender with different content.
|
|
124
|
+
|
|
125
|
+
An optional `replication.resolver` has `{ id, resolve }`. Its synchronous pure
|
|
126
|
+
function receives frozen evidence and returns `{ action: 'local' }`,
|
|
127
|
+
`{ action: 'remote' }`, or `{ action: 'merged', value }`; a merged value may be
|
|
128
|
+
null for deletion. Invalid decisions refuse with `JD2103`. Both contenders and
|
|
129
|
+
the resolver id are retained for every resolution. No resolver is bundled.
|
|
130
|
+
Determinism and convergence of a custom policy are the host's responsibility:
|
|
131
|
+
choosing the receiver's local value on each replica is deterministic but does
|
|
132
|
+
not make replicas agree. Independent non-conflicting deliveries converge without
|
|
133
|
+
resolver policy. Concurrent changes to different fields of the same row are
|
|
134
|
+
conservatively whole-row conflicts in this version.
|
|
135
|
+
|
|
136
|
+
## Reset handshake
|
|
137
|
+
|
|
138
|
+
`source.replication.snapshot()` returns a `jaren-replication-snapshot` 0.1
|
|
139
|
+
document containing `model`, `frontier`, logical `rows` with their causal
|
|
140
|
+
frontiers, and complete acknowledged `receipts`.
|
|
141
|
+
`target.replication.reset(snapshot)` validates and atomically installs it while
|
|
142
|
+
emitting ordinary live invalidations. The snapshot must dominate all target
|
|
143
|
+
acknowledgements and preserve every known receipt's bytes. It cannot discard
|
|
144
|
+
unsynchronized local history (`JD2105`) or rewrite identities (`JD2101`).
|
|
145
|
+
Snapshot receipt sequences must completely cover their frontier, and row
|
|
146
|
+
frontiers cannot be ahead of it. Subsequent duplicate deliveries remain
|
|
147
|
+
verifiable because the receipts travel with the snapshot.
|
|
148
|
+
|
|
149
|
+
Resets are intentionally bounded single documents: rows plus receipts must fit
|
|
150
|
+
`maxOperations`, and the complete encoding must fit `maxBytes`. Snapshot and
|
|
151
|
+
conflict readers pull through the shared cursor and stop before accumulating
|
|
152
|
+
beyond the configured credits. Resolver outputs and conflict evidence also
|
|
153
|
+
obey `maxBytes`. Very large
|
|
154
|
+
histories require a separately designed paged snapshot/receipt compaction
|
|
155
|
+
protocol; this implementation refuses them rather than splicing partial state.
|
|
156
|
+
A reset preserves the receiver's own replica identity and clears its outgoing
|
|
157
|
+
history. Future writes continue its sequence from the imported frontier.
|
|
158
|
+
|
|
159
|
+
## Authoring
|
|
160
|
+
|
|
161
|
+
`@jarenjs/linq/db` exports a deterministic pen using the same validator:
|
|
162
|
+
|
|
163
|
+
```js
|
|
164
|
+
const envelope = defineReplication({ replica: 'site-a', seq: 1,
|
|
165
|
+
frontier: {}, model: revision })
|
|
166
|
+
.change('notes', 'one', null, { id: 'one', body: 'hello' })
|
|
167
|
+
.toDocument();
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
The pen snapshots its input and emits the same canonical bytes as hand-authored
|
|
171
|
+
JSON. Normal applications export committed Store envelopes; constructing an
|
|
172
|
+
envelope does not allocate or acknowledge a Store sequence.
|
|
173
|
+
|
|
174
|
+
## Measurements
|
|
175
|
+
|
|
176
|
+
Run `npm run benchmark:changeflow`. Every timed mutation is followed by a fresh
|
|
177
|
+
query and a patch-only consumer comparison; every measured mutation changes its
|
|
178
|
+
visible result. Graph measurements mutate a child of one owner. Replication deliveries compare
|
|
179
|
+
complete database state, including relations. Conflict timings retry one
|
|
180
|
+
rejected contender and assert unchanged data/frontier plus durable evidence. Timing includes capture and live
|
|
181
|
+
delivery; oracle time is reported separately. Heap deltas are uncollected
|
|
182
|
+
allocations observed after initialization, not precise retained-heap sizes.
|
|
183
|
+
|
|
184
|
+
<!--fact:db.changeflow-->
|
|
185
|
+
|
|
186
|
+
Measured 2026-09-08, v24.19.0, AMD Ryzen 9 5900HX with Radeon Graphics; 15 mutations per case.
|
|
187
|
+
|
|
188
|
+
| Shape | Strategy | Initialize ms | Mutation p50 ms | Mutation p95 ms | Initialization heap bytes |
|
|
189
|
+
|---|---|---:|---:|---:|---:|
|
|
190
|
+
| selective join | join | 16.663 | 0.323 | 2.119 | 4878848 |
|
|
191
|
+
| selective join | rerun | 0.899 | 0.746 | 1.622 | 347160 |
|
|
192
|
+
| high fan-out join | join | 7.314 | 1.901 | 3.179 | 4851232 |
|
|
193
|
+
| high fan-out join | rerun | 0.860 | 0.850 | 1.188 | 340328 |
|
|
194
|
+
| graph | graph | 7.462 | 0.459 | 0.750 | 4811712 |
|
|
195
|
+
| graph | rerun | 4.539 | 4.042 | 4.979 | 1417928 |
|
|
196
|
+
| nested groups | nested-group | 6.775 | 0.252 | 0.795 | 2057832 |
|
|
197
|
+
| nested groups | rerun | 0.897 | 0.472 | 0.685 | 295808 |
|
|
198
|
+
| offset groups | rerun | 1.519 | 0.459 | 0.547 | 383648 |
|
|
199
|
+
| offset groups | rerun | 0.884 | 0.426 | 0.585 | 282280 |
|
|
200
|
+
|
|
201
|
+
| Capture | Envelopes | Operations | Bytes | Apply p50 ms | Replay p50 ms | Conflict p50 ms |
|
|
202
|
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
|
203
|
+
| session | 15 | 15 | 3169 | 0.458 | 0.118 | 0.204 |
|
|
204
|
+
| journal | 15 | 15 | 3169 | 0.294 | 0.092 | 0.161 |
|
|
205
|
+
|
|
206
|
+
Selective maintenance avoids repeated full SQL evaluation. Initialization and high-fan-out maintenance can cost more than rerunning; the table includes both. Offset groups remain rerun in both requested modes.
|
|
207
|
+
|
|
208
|
+
<!--/fact-->
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jarenjs/db",
|
|
3
3
|
"private": false,
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.72.0",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/index.js",
|
|
7
7
|
"types": "./types/index.d.ts",
|
|
@@ -84,9 +84,9 @@
|
|
|
84
84
|
"prepack": "npm run build:types"
|
|
85
85
|
},
|
|
86
86
|
"dependencies": {
|
|
87
|
-
"@jarenjs/core": "^0.
|
|
88
|
-
"@jarenjs/json": "^0.
|
|
89
|
-
"@jarenjs/validate": "^0.
|
|
87
|
+
"@jarenjs/core": "^0.72.0",
|
|
88
|
+
"@jarenjs/json": "^0.72.0",
|
|
89
|
+
"@jarenjs/validate": "^0.72.0"
|
|
90
90
|
},
|
|
91
91
|
"bin": {
|
|
92
92
|
"jaren-db": "./src/cli.js"
|