@jarenjs/db 0.66.1 → 0.72.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/ARCHITECTURE.md CHANGED
@@ -1095,3 +1095,22 @@ processes creating one fresh file never meet the deferred-upgrade
1095
1095
  `SQLITE_BUSY` the busy handler cannot retry. The job queue's
1096
1096
  administration (`page`, `cancel`, `requeue`, `sweep`) lives in
1097
1097
  `src/jobs.js` beside the fence it authorises through (JOBS-FORMAT §10).
1098
+
1099
+
1100
+ ## Replication and bounded dependency maintenance
1101
+
1102
+ `replication-format.js` owns JSON normalization; `replication.js` persists replica
1103
+ identity, frontiers, canonical receipts, before-images, outbox and conflict
1104
+ evidence through the capture transaction's `beforeCommit` seam. `logical-rows.js`
1105
+ shares mapped, validated row access between replication and live dependencies.
1106
+ Remote application suppresses only local envelope allocation: its data still
1107
+ invalidates capture subscribers after commit. Snapshot reset carries complete
1108
+ receipt evidence and refuses histories beyond its explicit credits.
1109
+
1110
+ `live-join.js` compiles indexed dependency descriptors from the entity planner
1111
+ and mapping. The same cache strategy maintains inner/left joins and eligible
1112
+ nested graph projections. `live-nested.js` maintains explicit two-level groups
1113
+ by refreshing affected parents from bounded leaves. Source and result payloads
1114
+ consume row and byte credits; unsupported shapes keep named reruns. See
1115
+ [the replication contract](docs/REPLICATION-FORMAT.md) and
1116
+ [the strategy matrix](docs/LIVE-FORMAT.md).
package/README.md CHANGED
@@ -688,6 +688,15 @@ opinion: <!--fact:vector.ceiling-->5.142 ns per vector component — one query r
688
688
  Past that this design is the wrong tool and no margin changes it; what
689
689
  lies beyond is an approximate index, and this store does not have one.
690
690
 
691
+ The [labelled retrieval comparison](../../benchmark/README.md#labelled-recall-and-repeated-refinement)
692
+ also scores an optional storage adapter with approximate candidates against
693
+ exact cosine on cached live embeddings. The AI ledger reports the adapter's
694
+ algorithm, exhaustive flag and returned candidate count, validates candidate
695
+ identity/vector shape, and re-scores candidates itself. The DB ledger recipe
696
+ continues to normalize as an exact adapter. No ANN runtime dependency is added.
697
+
698
+ Index decision: <!--fact:recall.annDecision-->0/6 contender rows cleared all bars; retain exact. Required exact-top-10 recall ≥ 0.95, p95 speedup ≥ 2×, and a measured exact p95 ≥ 100 ms. The largest reference corpus contains 5183 documents; scale beyond it remains unmeasured.<!--/fact-->
699
+
691
700
  **One document, three executors, proven to agree.** As with the spatial
692
701
  family, the k-nearest shapes of a committed corpus
693
702
  (`test/json/fixtures/vector-corpus.json`) run through the JavaScript
@@ -990,11 +999,17 @@ shutdown limits, the browser persistence matrix and measured latency/memory loss
990
999
  `sessionReason`. One diff
991
1000
  format runs store → patch → live query → O(k) render. Capture is
992
1001
  opt-in; the overhead is published, not waved away.
1002
+ - **Portable replication** ([REPLICATION-FORMAT](docs/REPLICATION-FORMAT.md)):
1003
+ opt-in replica identities, causal frontiers, bounded logical envelopes,
1004
+ durable replay receipts, explicit conflict evidence and snapshot resets.
1005
+ Data and acknowledgements commit together. Hosts supply transport and any
1006
+ conflict resolver; the default preserves both contenders and rejects the write.
993
1007
  - **Live queries** (LIVE-FORMAT §§7–13): `collection.live(document)`
994
1008
  maintains a result as writes arrive and emits patches — incremental
995
1009
  for `where`/`select`/`orderBy`+`limit`/aggregates/single-level
996
1010
  `groupBy` and a spatial `where` over a derived index (the geofence;
997
- the normative maintenance table), re-run for everything else,
1011
+ the normative maintenance table), indexed inner/left entity joins, bounded
1012
+ graph projections and explicit two-level groups; re-run for other shapes,
998
1013
  **declared, never silent** (`live.mode` names the reason).
999
1014
  Unaffected rows stay reference-identical; a seeded oracle holds the
1000
1015
  maintained result equal to a fresh re-query after every mutation.
@@ -1022,6 +1037,10 @@ shutdown limits, the browser persistence matrix and measured latency/memory loss
1022
1037
  and `worker.stop()` quiesces every claim, renewal, checkpoint and
1023
1038
  settlement before it resolves, so closing the store releases the
1024
1039
  database file deterministically.
1040
+ DAG jobs inspect persisted workflow/input/task identity before reading
1041
+ checkpoint values. `jobs.reset(id, { expectedGeneration })` explicitly
1042
+ discards an inactive run's checkpoints and restarts it under a new fence;
1043
+ live leases and stale observations refuse (JOBS-FORMAT §10).
1025
1044
  - **The browser** (`@jarenjs/db/wasm`): the same store, the same
1026
1045
  queries, the same live updates run on the official SQLite wasm build
1027
1046
  over a probed persistence ladder: isolated SharedArrayBuffer OPFS,
@@ -1276,7 +1295,8 @@ its side-effect-free status read are in
1276
1295
  there. Same-host processes over WAL are the supported topology. No
1277
1296
  priority classes, no cron, no workflow compensation.
1278
1297
  - **Live-query maintenance is limited to the declared table** (§7);
1279
- joins, entity queries and non-canonical shapes re-run, reported.
1298
+ indexed joins and graph projections require bounded dependencies. Offset
1299
+ windows, unindexed joins, load-spec graphs and non-canonical shapes re-run, reported.
1280
1300
  - **`eventTime.retention` bounds repair work, not memory.** It is the
1281
1301
  horizon a view claims and is checked against the window it maintains;
1282
1302
  the maintained state is still bounded by `live.maxMaintained`, and no
@@ -1287,8 +1307,8 @@ its side-effect-free status read are in
1287
1307
  durability differences; see [execution hosts](docs/HOSTS.md).
1288
1308
  - **Named future work, not silent gaps**: `$groupby` pushdown beyond
1289
1309
  the `$time-bucket` ladder, a many-to-many hop on the chain, membership
1290
- on an auto-keyed pending insert, incremental joins, other SQL dialects,
1291
- replication, database introspection (MODEL-FORMAT §10.6, the roadmap).
1310
+ on an auto-keyed pending insert, additional join/group shapes, other SQL dialects,
1311
+ transport policy, database introspection (MODEL-FORMAT §10.6, the roadmap).
1292
1312
 
1293
1313
  The normative formats are
1294
1314
  [docs/MODEL-FORMAT.md](docs/MODEL-FORMAT.md) (storage §§1–7, safe
@@ -1320,7 +1340,27 @@ Every subpath a consumer can import, derived from the manifest by
1320
1340
  | `@jarenjs/db/schemas/jaren-migration.schema.json` | schema | — |
1321
1341
  | `@jarenjs/db/schemas/jaren-model.draft-07.schema.json` | schema | — |
1322
1342
  | `@jarenjs/db/schemas/jaren-model.schema.json` | schema | — |
1343
+ | `@jarenjs/db/schemas/jaren-replication-snapshot.draft-07.schema.json` | schema | — |
1344
+ | `@jarenjs/db/schemas/jaren-replication-snapshot.schema.json` | schema | — |
1345
+ | `@jarenjs/db/schemas/jaren-replication.draft-07.schema.json` | schema | — |
1346
+ | `@jarenjs/db/schemas/jaren-replication.schema.json` | schema | — |
1323
1347
  | `@jarenjs/db/package.json` | metadata | — |
1324
1348
  | `@jarenjs/db/node-worker` | JavaScript | declared |
1325
1349
  | `@jarenjs/db/node-pool` | JavaScript | declared |
1326
1350
  <!--/fact-->
1351
+
1352
+
1353
+ The [durable AI ledger recipe](../ai/README.md#a-durable-ledger-over-jarenjsdb)
1354
+ exposes optional atomic namespace mutation as well as its base storage and rank
1355
+ capabilities. It reads and publishes through one immediate transaction, using the
1356
+ transaction's synchronous collection facade so another SQLite connection on the
1357
+ same event loop cannot block an awaiting writer. A rejected callback changes
1358
+ neither records nor counters; ledger id minting and guarded multi-record updates
1359
+ use that boundary.
1360
+
1361
+ Document-only migrations also accept named files:
1362
+ `jaren-db documents --migrations migrations --in users=users.json --in events=events.jsonl --out migrated.json`.
1363
+ The output is one atomically published collection bundle; read it with
1364
+ `--format collections`. Assertion plans expose provider, ordered-fold and
1365
+ bounded materialization strategies through `onAssertionPlan`.
1366
+ See [MIGRATION-FORMAT §6 and §11](docs/MIGRATION-FORMAT.md).
@@ -390,7 +390,10 @@ await store.jobs.enqueue('sync-report', { input: { day: '2026-08-05' } });
390
390
 
391
391
  **The legacy rule is deterministic, and never reads unknown as equal.**
392
392
  A run checkpointed by a release that did not record task identity has
393
- no `taskVersionsHash`. If it recorded no node value yet, there is
393
+ no `taskVersionsHash`. Identity inspection reads only the reserved metadata
394
+ value and probes whether other rows exist; it does not load or parse node
395
+ values. Missing metadata with saved values refuses `JD2069`. Refused
396
+ legacy upgrades leave the original identity untouched. If it recorded no node value yet, there is
394
397
  nothing that could be replayed wrongly, so the identity is upgraded in
395
398
  place and the run proceeds. If it DID record values, the implementation
396
399
  that produced them cannot be confirmed and the resume is refused,
@@ -471,6 +474,16 @@ a cancellation policy, a retry policy are the host's.
471
474
  and counts it under `stats().cancellations`; an attempt held by
472
475
  another process meets the fence at its next settling call and records
473
476
  a loss, as any superseded attempt does.
477
+ - **`reset(id, { expectedGeneration })`** — explicitly discard one inactive
478
+ run's checkpoints and restart its attempts at zero. Read the job's
479
+ `leaseGeneration` first and pass that observation. One transaction
480
+ conditionally updates the job, increments its fence generation and deletes
481
+ its checkpoint rows. A concurrent generation change refuses `JD2066`, a
482
+ live lease `JD2068`, an unknown or completed job `JD2065`; no refusal
483
+ discards values. The next claim recomputes the workflow under current
484
+ identity. Reset does not undo external task effects; their idempotency
485
+ keys remain the host's responsibility. It is root-only administration,
486
+ with the same signal/deadline options as requeue.
474
487
  - **`requeue(id)`** — returns a failed, dead, cancelled or lease-expired
475
488
  job to `pending` at the clock's instant (the queue's own order). The
476
489
  attempt history is KEPT: `attempts` counts every claim the job ever
@@ -273,7 +273,10 @@ what the pushdown planner already means by it.
273
273
  | a spatial predicate the planner **refused** (no `derive` index on the member, an untyped member, an unbounded probe) | **re-run on invalidation**, the refusal named — it never translated, so nothing narrows the fetch | the previous result, for diffing |
274
274
  | a `$resample` or `$rolling` document over the collection, with an explicit `eventTime` and a fixed width (§13) | **event-time bucket / rolling state**: rows kept by bucket, or in instant order; only what a write can reach is folded again, through `@jarenjs/core/series` itself | the contributing rows, plus one fold per bucket |
275
275
  | the same document with no `eventTime`, a calendar width, a named zone, a `locf`/`linear` fill, a `first`/`last` aggregate, or a retention that does not cover the window | **re-run on invalidation**, the member that stopped it named (§13.2) | the previous result, for diffing |
276
- | joins, multi-entity roots, graph loads, every entity query | **re-run on invalidation declared, not attempted** in this version | the previous result, for diffing |
276
+ | indexed inner equi-joins and canonical allowing-empty left joins over mapped entity roots | **join dependency maintenance**; point-read changed keys and reevaluate their bounded outer owners | source rows, key indexes and projected tuples, bounded by `maxMaintained` and `maxBytes` |
277
+ | nested entity graph projections with indexed equality edges and unique binding names | **graph dependency maintenance**; a child change refreshes its bounded owners | source rows, reverse key indexes and graph outputs |
278
+ | explicit two-level collection groups with a singular parent key and bounded nested input | **nested-group maintenance**; recompute affected parents through the query engine | source leaves and parent outputs |
279
+ | unindexed/non-equi joins, self joins, explicit entity ordering/windows, object-root documents and load-spec graphs | **re-run on invalidation**, with the dependency or planner reason | previous result for diffing |
277
280
  | anything else: non-translatable predicates, `limit` without `orderBy`, `offset` > 0, windowed aggregates, `@jarenjs/linq`'s nested two-level `groupBy` emission, non-canonical group returns | **re-run on invalidation**, the reason named | the previous result, for diffing |
278
281
 
279
282
  The **physical mapping** of a `derive: 'bbox'` index (MODEL-FORMAT §2.1,
@@ -310,8 +313,8 @@ refuses at registration (`JD0051`), the same shape as capture's
310
313
  demanded session — an application that needs the property can refuse
311
314
  to start.
312
315
 
313
- The **canonical group form** the classifier recognises (and the only
314
- one the linq chain's nested emission re-runs, stated plainly):
316
+ The **canonical group form** the classifier recognises (the linq chain's group-of-groups emission still re-runs; explicit
317
+ nested groups have a separate bounded strategy below):
315
318
 
316
319
  ```json
317
320
  { "$for": { "it": "$[*]" },
@@ -502,13 +505,14 @@ ERRORING rather than degrading (the D14 rule — the bound is printed):
502
505
  delete-correctness, and a count over a table larger than the bound
503
506
  is a conscious `maxMaintained` raise, not a silent one.
504
507
 
505
- Non-claims, in one place: no incremental joins (re-run is the declared
506
- strategy), no cross-connection invalidation (§6's `data_version` is
508
+ Non-claims, in one place: no maintenance of unindexed or non-equality joins,
509
+ no cross-connection invalidation (§6's `data_version` is
507
510
  the signal), no maintenance over asynchronous connections —
508
511
  `capabilities.live` is `false` there and a registration is `JD0051`
509
512
  naming the reason, because maintenance point-reads rows synchronously
510
513
  inside delivery (the wasm driver's oo1 API is synchronous, which is
511
- why the browser has live queries at all) no replication, and no ordering guarantee for
514
+ why the browser has live queries at all). Replication is specified separately in
515
+ [REPLICATION-FORMAT](REPLICATION-FORMAT.md). There is no ordering guarantee for
512
516
  unordered queries beyond §9's determinism.
513
517
 
514
518
  ## 13. Event time
@@ -616,3 +620,40 @@ holds a shuffled stream of inserts, in-place updates, instant moves and
616
620
  deletes against `resampleSeries` / `rollingSeries` over the whole
617
621
  collection after each one, which is the only oracle that cannot drift
618
622
  with the implementation.
623
+
624
+
625
+ ## Bounded joins, graph projections and nested groups
626
+
627
+ `join`, `graph` and `nested-group` strategies charge their source rows and
628
+ result entries to `live.maxMaintained`, and serialized input/output payloads to
629
+ `live.maxBytes` (default 4 MiB). These credits bound cached payloads rather than
630
+ claiming to measure JavaScript heap overhead. Initialization uses a limited
631
+ source read; updates read changed keys, then visit cached indexed dependencies.
632
+ Bounds are checked while caches grow. Overflow is `JD2060`, emits one error,
633
+ closes the subscription and releases dependency caches. It never relabels an
634
+ unbounded query as incremental.
635
+
636
+ Entity equality columns need a primary-key prefix, declared index or mapped
637
+ foreign-key index. Every binding needs a key and a distinct root. The projected
638
+ identity is the tuple of source identities, so duplicate projected values remain
639
+ distinct. Default entity result order follows physical row insertion order;
640
+ point reads preserve that order even when a key is deleted and reinserted. A
641
+ left join uses a canonical `$allowing-empty` binding over an equality-filtered
642
+ inner subquery. Its absent child can be defaulted to null. Graph projections
643
+ embed equality-filtered child queries in a single outer row's return object.
644
+ Global-root reads outside those bindings re-run because changing one row can
645
+ change every projected graph.
646
+
647
+ `dependencyReads`, `refreshedRoots` and `refreshedGroups` expose the work done.
648
+ `dependencyReads` counts logical changed-row reads; a row-position lookup is an
649
+ additional statement. No full query reruns occur under these strategy labels.
650
+ Materializing and diffing the final bounded output still costs work proportional
651
+ to its size. See the equal-correctness [measurements](REPLICATION-FORMAT.md#measurements)
652
+ for startup and high-fan-out losses beside selective wins.
653
+
654
+ Two-level grouping currently accepts an explicit parent `$groupby` over a
655
+ singular member, with one nested group over that parent's bound row sequence.
656
+ Count, sum, average, minimum and maximum recompute from only the affected
657
+ parent's bounded leaves. An offset, an unsupported operator, a global input to
658
+ the nested group, or a group-of-groups LINQ emission remains a named rerun.
659
+ Replicated writes enter the same committed capture stream as local writes.
@@ -235,25 +235,34 @@ hash of the `baseline` model when no migration has run.
235
235
 
236
236
  | strategy | which assertions | what it costs |
237
237
  |---|---|---|
238
- | per-document | a FLWOR over `$[*]` whose `$where`/`$return` read only the binding | one keyset batch at a time; fails fast at the first batch that violates |
239
- | fold | exactly one of `$count`, `$sum`, `$min`, `$max` over the root | one batch at a time; each batch is answered by the ENGINE and the partial answers combine |
240
- | materialize | everything else (`$let`, `$distinct`, a nested `$for`, two aggregates) | every document at once, under `assertionBounds` |
241
-
242
- A fold is sound because the operator is associative: the answer over a
243
- collection is the combination of the answers over any partition of it.
244
- Nothing reimplements an operator each batch is evaluated by the same
245
- compiled query the whole-collection path would use, and only the
246
- COMBINE step is written here, so null handling, empty-sequence answers
247
- and type coercions are the engine's. A suite runs every fold shape both
248
- ways, over ten corpora and six partitions, and requires the value and
249
- the verdict to be indistinguishable; a shape that cannot pass it is not
250
- in the set.
238
+ | provider | a collection `$count` that the existing query planner proves native without a typed intermediate schema | one aggregate result, with no assertion document fetch |
239
+ | per-document | an independent, unwindowed FLWOR over `$[*]`, with the default empty-sequence expectation | one keyset batch; fails at the first violation |
240
+ | fold | `$count`, `$sum`, `$avg`, `$min`, `$max` over a partition-independent operand; sequence EBV of an independent FLWOR | one batch plus fixed accumulator state |
241
+ | fold with bounds | `$distinct` over such an operand with an explicit positive `maxDistinct` | one batch plus unique items under cardinality and byte bounds |
242
+ | materialize | global or positional operands, nested scans, unsupported shapes, or unbounded distinct | the collection under `assertionBounds` |
243
+
244
+ Folds feed the operand's sequence items, in original order, into the
245
+ query engine's shared accumulator. Floating-point addition is **not
246
+ associative**: combining batch totals changes answers. The ordered state
247
+ preserves the whole-query result, including singleton arrays, negative
248
+ zero, NaN, mixed-type refusals and Unicode code-point ordering. Sequence
249
+ EBV is checked once across all batches, including an empty source.
250
+ `$[0]`, root-dependent filters and positional bindings never qualify as
251
+ independent merely because they occur inside an aggregate.
252
+
253
+ `classifyAssertion(query, { expect, maxDistinct })` exposes the portable
254
+ strategy. `onAssertionPlan(plan)` reports migration, step, collection,
255
+ strategy, shape, reason and bounds before the assertion reads documents;
256
+ Store dry-run query statements also expose strategy and reason. Provider
257
+ promotion reuses the query planner with no assumed typed columns. Typed
258
+ SQL aggregates still need a trustworthy intermediate schema and proof
259
+ that their numeric/order semantics match; otherwise the ordered fold runs.
251
260
 
252
261
  **A materializing assertion is bounded.** `options.assertionBounds`
253
262
  defaults to `{ maxRows: 100000, maxBytes: 67108864 }` and is crossed
254
263
  BEFORE the excess is held — the walk stops at the row that would break
255
264
  it, refusing `JD2007` (rows) or `JD2076` (bytes) and naming the two
256
- assertion shapes that are answered in batches instead. `null` on either
265
+ assertion strategies that are answered in batches instead. `null` on either
257
266
  member removes that bound, which a caller must ask for: an unbounded
258
267
  read nobody declared is exactly what this classification removes. This
259
268
  is a deliberate behavior change — a migration that used to read a very
@@ -265,14 +274,11 @@ hash of the `baseline` model when no migration has run.
265
274
  transformed | derived | asserted }`, one event per batch), and never
266
275
  hold the whole collection in memory. A PER-DOCUMENT assertion — a
267
276
  FLWOR over `$[*]` whose `$where` and `$return` read only the binding
268
- — walks the same batches and fails fast at the first batch that
269
- violates, because its answer over each batch is its answer over the
270
- whole. A cross-document assertion (one that reads the root: `$count:
271
- '$[*]'`, a `$let`, a `$distinct`, a nested `$for`) reads the whole
272
- collection into one array a stated cost; keep such assertions
273
- early, before the data grows. A cross-document assertion that is one
274
- associative aggregate no longer costs that read at all — see the
275
- classification table above.
277
+ with the default empty expectation walks the same batches and fails
278
+ fast. Other assertions follow the classification table above.
279
+ `batchSize` must be a positive safe integer. `maxDistinct` opts into a
280
+ distinct fold and bounds retained unique items; `maxBytes` applies to
281
+ those items. Duplicates do not consume additional cardinality.
276
282
  - A transform MUST NOT change a caller-keyed document's key member —
277
283
  the key column would go stale; the run refuses (`JD0023`).
278
284
 
@@ -510,10 +516,23 @@ jaren-db documents --migrations <dir> --in <file|-> (--out <file|-> | --in-place
510
516
  line-delimited, everything else one JSON array) unless `--format` /
511
517
  `--out-format` says otherwise, and stdio defaults to JSONL. Input and
512
518
  output encodings are independent, so this is also the converter.
513
- - **A file holds ONE collection.** The migrations name it; a chain
514
- whose document steps touch more than one cannot be applied to a
515
- file, and is refused rather than partly run. `--collection` asserts
516
- which collection the file holds and refuses a mismatch.
519
+ - **Named sources share one collection bundle.** Repeat
520
+ `--in users=users.json --in events=events.jsonl --out migrated.json`.
521
+ Output defaults to `--out-format collections`: a standard JSON object
522
+ mapping collection names to arrays, including empty collections.
523
+ Publication is one atomic rename after every collection succeeds.
524
+ Multiple source files cannot use `--in-place`, because separate
525
+ renames cannot give this guarantee. Single-source `--collection`
526
+ still asserts the collection name and refuses a mismatch.
527
+ - **A bundle can be read explicitly** with `--format collections`, then
528
+ replaced using `--in-place --yes`. Bundle input is materialized under
529
+ a whole-file `maxBytes` ceiling before parsing and per-collection
530
+ `maxRows` validation after parsing. JSON and JSONL single-collection
531
+ sources remain streaming. `--max-rows`, `--max-bytes`, and
532
+ `--max-distinct` configure assertion admission too; the first two
533
+ accept `none` to remove their limit deliberately. A materializing
534
+ file run admits rows before retaining them. Physical-step and query
535
+ compilation refusals precede every input read.
517
536
  - **`--out` writes elsewhere; `--in-place` replaces the input and
518
537
  needs `--yes`.** Either way the documents land in a sibling
519
538
  temporary that is renamed over the target only once every document
@@ -1078,6 +1078,14 @@ error.
1078
1078
  | `JD2092` | a worker row, compatibility result or remote identity count exceeds its declared bound |
1079
1079
  | `JD2093` | malformed worker protocol request |
1080
1080
  | `JD2094` | invalid or uncommitted durable snapshot; reopen the last committed version |
1081
+ | `JD0060` | a replication envelope or snapshot is invalid |
1082
+ | `JD2100` | a replica sequence or causal dependency has a gap |
1083
+ | `JD2101` | an envelope identity names different content or an unknown local origin |
1084
+ | `JD2102` | the replica identity or model revision disagrees |
1085
+ | `JD2103` | a resolver fails its synchronous decision contract |
1086
+ | `JD2104` | a logical row or snapshot disagrees with its causal history |
1087
+ | `JD2105` | an explicit snapshot reset is required, or reset would discard acknowledged history |
1088
+ | `JD2106` | a replication operation or snapshot exceeds its configured bound |
1081
1089
 
1082
1090
  The table above is proven in sync with the runtime `DB_CODES` table by
1083
1091
  a test.
@@ -0,0 +1,208 @@
1
+ # Replication format and runtime
2
+
3
+ `jaren-replication` 0.1 is a transport-neutral logical transaction. Open an empty
4
+ store with `replication: { replica: 'host-issued-id' }`; capture is enabled when
5
+ omitted. Existing replication stores must reopen with the same identity and
6
+ model. Every writer of that file must enable replication. External SQL writes
7
+ and opening a replicated file without replication are outside the history
8
+ contract; a detected before-image disagreement refuses with `JD2104`.
9
+
10
+ The host owns network transport, authentication, replica identity allocation and
11
+ resolver policy. A replica id must remain unique to its writer lineage. The
12
+ runtime works on SQLite Node, worker and wasm hosts through session or journal
13
+ capture. Drivers without `changeCapture` refuse at open; PostgreSQL capture is
14
+ not implemented. Journal replication refuses models with cascading or set-null
15
+ child foreign keys (`JD0051`), because the journal cannot observe those effects.
16
+ Session capture includes them. Both modes support membership cascades and
17
+ roll back an envelope whose side effects are absent from its logical operations.
18
+
19
+ ## Envelope
20
+
21
+ ```json
22
+ {
23
+ "$replication": "0.1",
24
+ "replica": "site-a",
25
+ "seq": 1,
26
+ "frontier": {},
27
+ "model": "application-model-revision",
28
+ "operations": [
29
+ { "table": "notes", "key": "note/one", "before": null,
30
+ "after": { "id": "note/one", "body": "hello" } }
31
+ ]
32
+ }
33
+ ```
34
+
35
+ `model` is the store's `shapeHash(model)` revision. `seq` is a positive safe
36
+ integer, allocated from the replica's durable frontier in the data transaction.
37
+ The identity is `replicationIdentity(replica, seq)`, the JSON encoding of their
38
+ pair, so separators inside host ids cannot collide. A frontier maps host ids to
39
+ non-negative safe sequences. The sender's frontier entry must equal `seq - 1`,
40
+ with an absent entry meaning zero. Property names such as `__proto__` and
41
+ `constructor` remain ordinary replica ids.
42
+
43
+ Each operation is one net row transition. `before: null` inserts, `after: null`
44
+ deletes, and two objects replace the complete logical row. A table/key occurs
45
+ once per envelope. Composite keys use the capture format's JSON-array token;
46
+ single keys use their scalar text. Entity documents include mapped scalar and
47
+ foreign-key fields, omit relation projections and retain authored version and
48
+ timestamp values. Membership tables carry their two keys. Application performs
49
+ schema validation through the store's configured validator and reuses physical
50
+ column plans; it does not generate fresh defaults or version stamps.
51
+
52
+ `normalizeReplication` validates and detaches the document. `encodeReplication`
53
+ produces deterministic JSON. Object members are canonically ordered; operation
54
+ array order remains part of identity. Capture sorts net rows by table/key, making
55
+ physical session and journal output equal. Envelope order preserves transaction
56
+ order. Physical insertion order inside a netted transaction or snapshot is not a
57
+ portable query ordering: use an explicit query order when comparing replicas.
58
+ Receipts retain the entire canonical payload, avoiding hash-collision ambiguity.
59
+
60
+ Unknown members and versions, empty ids, unsafe sequences, non-JSON values,
61
+ cycles, duplicate net rows and unchanged operations are `JD0060`. Nesting is
62
+ limited to 128 object/array ancestors. JSON Schema artifacts describe structural
63
+ validation; cross-field sequence, uniqueness and causal checks are runtime
64
+ constraints. Current and draft-07 artifacts ship for envelopes and snapshots:
65
+
66
+ - `@jarenjs/db/schemas/jaren-replication.schema.json`
67
+ - `@jarenjs/db/schemas/jaren-replication.draft-07.schema.json`
68
+ - `@jarenjs/db/schemas/jaren-replication-snapshot.schema.json`
69
+ - `@jarenjs/db/schemas/jaren-replication-snapshot.draft-07.schema.json`
70
+
71
+ ## Applying and exporting
72
+
73
+ ```js
74
+ const source = await openStore(model, {
75
+ driver, replication: { replica: 'site-a', retention: 1000 }
76
+ });
77
+ const page = await source.replication.page({ after: 0, limit: 100, maxBytes: 1048576 });
78
+ for (const envelope of page.items) {
79
+ const result = await target.replication.apply(envelope);
80
+ if (result.status === 'conflict') hostReport(result.conflicts);
81
+ }
82
+ ```
83
+
84
+ `apply` returns `applied`, `duplicate` or `conflict`, together with the committed
85
+ frontier and conflict records. Successful duplicate delivery performs no row
86
+ write or live emission. An acknowledged identity with different bytes is
87
+ `JD2101`. A sequence or causal gap is `JD2100`: no suffix is applied. Deliver
88
+ missing envelopes first, then retry, or perform an explicit reset. Model mismatch
89
+ is `JD2102`. Imported envelopes keep their original identity and produce no local
90
+ outbox entry, preventing echoes. A host forwarding between peers must preserve
91
+ the original envelopes; `page()` exports this replica's own commits only.
92
+
93
+ Data, receipt, per-row causal metadata, capture record and frontier settle in
94
+ the same transaction. Replication is available only on the root Store; transaction
95
+ views expose no replication API. Failed validation, cancellation, constraint checks or
96
+ commit roll everything back. Subscribers run through the existing committed
97
+ capture delivery path. Tests kill a process immediately before and after commit
98
+ and verify both data and frontier after reopening.
99
+
100
+ Pages use the shared cursor and byte-credit drain. `limit` counts whole envelopes;
101
+ `maxBytes` counts complete canonical envelope bytes. An indivisible oversize
102
+ envelope is `JD2074`. Retention loss returns `resetRequired: true`, no items and
103
+ no continuation. `earliestAvailable`, `highWatermark`, `next` and `hasMore` describe
104
+ the local outbox. Pages and snapshots take a consistent store transaction.
105
+ `signal` and `deadline` are honored through existing cancellation contracts:
106
+ queue cancellation is `JD2064`, operation-boundary cancellation `JD2072`, and an
107
+ expired deadline `JD2075`.
108
+
109
+ Defaults are `retention: 1000`, `maxOperations: 10000`, `maxBytes: 4194304`.
110
+ Operation overflow is `JD2106`; all credits are positive safe integers. Retention
111
+ prunes only outgoing envelope history. Replay receipts, row tombstones and
112
+ conflict evidence remain durable. They are deliberately not claimed to be a
113
+ bounded total database size. Internal tables use the reserved `_jaren_replica`
114
+ prefix. No schema migration or revision rewrite of this protocol is implicit.
115
+
116
+ ## Conflicts
117
+
118
+ Concurrent incompatible whole-row writes reject by default. A `conflict` result
119
+ does not acknowledge the remote envelope and leaves the local row intact.
120
+ `replication.conflicts({ limit, maxBytes, signal, deadline })` reads persisted evidence: the common base,
121
+ local value and frontier, remote value and envelope position, resolver identity
122
+ and decision. A rejected envelope reserves its canonical identity too, so retries
123
+ cannot replace its contender with different content.
124
+
125
+ An optional `replication.resolver` has `{ id, resolve }`. Its synchronous pure
126
+ function receives frozen evidence and returns `{ action: 'local' }`,
127
+ `{ action: 'remote' }`, or `{ action: 'merged', value }`; a merged value may be
128
+ null for deletion. Invalid decisions refuse with `JD2103`. Both contenders and
129
+ the resolver id are retained for every resolution. No resolver is bundled.
130
+ Determinism and convergence of a custom policy are the host's responsibility:
131
+ choosing the receiver's local value on each replica is deterministic but does
132
+ not make replicas agree. Independent non-conflicting deliveries converge without
133
+ resolver policy. Concurrent changes to different fields of the same row are
134
+ conservatively whole-row conflicts in this version.
135
+
136
+ ## Reset handshake
137
+
138
+ `source.replication.snapshot()` returns a `jaren-replication-snapshot` 0.1
139
+ document containing `model`, `frontier`, logical `rows` with their causal
140
+ frontiers, and complete acknowledged `receipts`.
141
+ `target.replication.reset(snapshot)` validates and atomically installs it while
142
+ emitting ordinary live invalidations. The snapshot must dominate all target
143
+ acknowledgements and preserve every known receipt's bytes. It cannot discard
144
+ unsynchronized local history (`JD2105`) or rewrite identities (`JD2101`).
145
+ Snapshot receipt sequences must completely cover their frontier, and row
146
+ frontiers cannot be ahead of it. Subsequent duplicate deliveries remain
147
+ verifiable because the receipts travel with the snapshot.
148
+
149
+ Resets are intentionally bounded single documents: rows plus receipts must fit
150
+ `maxOperations`, and the complete encoding must fit `maxBytes`. Snapshot and
151
+ conflict readers pull through the shared cursor and stop before accumulating
152
+ beyond the configured credits. Resolver outputs and conflict evidence also
153
+ obey `maxBytes`. Very large
154
+ histories require a separately designed paged snapshot/receipt compaction
155
+ protocol; this implementation refuses them rather than splicing partial state.
156
+ A reset preserves the receiver's own replica identity and clears its outgoing
157
+ history. Future writes continue its sequence from the imported frontier.
158
+
159
+ ## Authoring
160
+
161
+ `@jarenjs/linq/db` exports a deterministic pen using the same validator:
162
+
163
+ ```js
164
+ const envelope = defineReplication({ replica: 'site-a', seq: 1,
165
+ frontier: {}, model: revision })
166
+ .change('notes', 'one', null, { id: 'one', body: 'hello' })
167
+ .toDocument();
168
+ ```
169
+
170
+ The pen snapshots its input and emits the same canonical bytes as hand-authored
171
+ JSON. Normal applications export committed Store envelopes; constructing an
172
+ envelope does not allocate or acknowledge a Store sequence.
173
+
174
+ ## Measurements
175
+
176
+ Run `npm run benchmark:changeflow`. Every timed mutation is followed by a fresh
177
+ query and a patch-only consumer comparison; every measured mutation changes its
178
+ visible result. Graph measurements mutate a child of one owner. Replication deliveries compare
179
+ complete database state, including relations. Conflict timings retry one
180
+ rejected contender and assert unchanged data/frontier plus durable evidence. Timing includes capture and live
181
+ delivery; oracle time is reported separately. Heap deltas are uncollected
182
+ allocations observed after initialization, not precise retained-heap sizes.
183
+
184
+ <!--fact:db.changeflow-->
185
+
186
+ Measured 2026-09-08, v24.19.0, AMD Ryzen 9 5900HX with Radeon Graphics; 15 mutations per case.
187
+
188
+ | Shape | Strategy | Initialize ms | Mutation p50 ms | Mutation p95 ms | Initialization heap bytes |
189
+ |---|---|---:|---:|---:|---:|
190
+ | selective join | join | 16.663 | 0.323 | 2.119 | 4878848 |
191
+ | selective join | rerun | 0.899 | 0.746 | 1.622 | 347160 |
192
+ | high fan-out join | join | 7.314 | 1.901 | 3.179 | 4851232 |
193
+ | high fan-out join | rerun | 0.860 | 0.850 | 1.188 | 340328 |
194
+ | graph | graph | 7.462 | 0.459 | 0.750 | 4811712 |
195
+ | graph | rerun | 4.539 | 4.042 | 4.979 | 1417928 |
196
+ | nested groups | nested-group | 6.775 | 0.252 | 0.795 | 2057832 |
197
+ | nested groups | rerun | 0.897 | 0.472 | 0.685 | 295808 |
198
+ | offset groups | rerun | 1.519 | 0.459 | 0.547 | 383648 |
199
+ | offset groups | rerun | 0.884 | 0.426 | 0.585 | 282280 |
200
+
201
+ | Capture | Envelopes | Operations | Bytes | Apply p50 ms | Replay p50 ms | Conflict p50 ms |
202
+ |---|---:|---:|---:|---:|---:|---:|---:|
203
+ | session | 15 | 15 | 3169 | 0.458 | 0.118 | 0.204 |
204
+ | journal | 15 | 15 | 3169 | 0.294 | 0.092 | 0.161 |
205
+
206
+ Selective maintenance avoids repeated full SQL evaluation. Initialization and high-fan-out maintenance can cost more than rerunning; the table includes both. Offset groups remain rerun in both requested modes.
207
+
208
+ <!--/fact-->
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@jarenjs/db",
3
3
  "private": false,
4
- "version": "0.66.1",
4
+ "version": "0.72.0",
5
5
  "type": "module",
6
6
  "main": "./src/index.js",
7
7
  "types": "./types/index.d.ts",
@@ -84,9 +84,9 @@
84
84
  "prepack": "npm run build:types"
85
85
  },
86
86
  "dependencies": {
87
- "@jarenjs/core": "^0.66.1",
88
- "@jarenjs/json": "^0.66.1",
89
- "@jarenjs/validate": "^0.66.1"
87
+ "@jarenjs/core": "^0.72.0",
88
+ "@jarenjs/json": "^0.72.0",
89
+ "@jarenjs/validate": "^0.72.0"
90
90
  },
91
91
  "bin": {
92
92
  "jaren-db": "./src/cli.js"