@rljson/db 0.0.13 → 0.0.14

package/README.architecture.md

@@ -777,6 +777,68 @@ describe('Tree WHERE clause fix', () => {
 });
 ```
 
+## Connector (Sync Protocol)
+
+The `Connector` class implements the client side of the RLJSON sync protocol. It sits between a local `Db` and a socket, enriching outgoing refs with protocol metadata and processing incoming refs with safety guarantees.
+
+### Responsibilities
+
+```
+┌────────────────────────────────────────────────┐
+│                  Connector                     │
+│                                                │
+│  Outgoing path (Db → socket):                  │
+│    1. Db.notify fires with InsertHistoryRow    │
+│    2. Auto-populate predecessors from          │
+│       InsertHistoryRow.previous                │
+│    3. Enrich with seq, c, t, p (per config)    │
+│    4. Add to sent-refs dedup set               │
+│    5. Emit ConnectorPayload on socket          │
+│                                                │
+│  Incoming path (socket → callbacks):           │
+│    1. Receive ConnectorPayload from socket      │
+│    2. Reject self-echo (origin === own origin)  │
+│    3. Reject duplicate (ref already received)   │
+│    4. Detect sequence gaps → request gap-fill   │
+│    5. Emit client ACK (if requireAck)           │
+│    6. Invoke listen() callbacks                │
+└────────────────────────────────────────────────┘
+```
+
+### Bounded dedup strategy
+
+The Connector tracks recently sent and received refs to prevent duplicates. Both sets use **two-generation eviction**:
+
+1. A `current` Set accumulates refs.
+2. When `current.size ≥ maxDedupSetSize`, it rotates: `previous = current`, `current = new Set()`.
+3. Lookups check **both** `current` and `previous`.
+4. This caps memory at ≈ 2 × `maxDedupSetSize` entries (default 10 000 per generation).
+
+The eviction is O(1) — no per-entry bookkeeping — and avoids the overhead of a full LRU cache.
+
+### Auto-predecessor from InsertHistory
+
+When `causalOrdering` is enabled, the Connector's Db observer reads the `previous` field from each `InsertHistoryRow` emitted by `Db.notify`. These `InsertHistoryTimeId` values become the `p` (predecessors) array in the outgoing `ConnectorPayload`. This eliminates the need for manual `setPredecessors()` calls in normal database-driven workflows.
+
+### listen()
+
+`listen()` registers callbacks that receive incoming refs through the full sync pipeline: origin filtering, dedup, gap detection, and ACK. All protocol safeguards are applied before callbacks are invoked.
+
+### sendWithAck ordering
+
+The `sendWithAck()` method registers the ACK listener **before** emitting the ref on the socket. This prevents a race condition with synchronous transports (e.g., in-memory sockets used in tests) where the ACK fires during `send()` and would be lost if the listener were registered after.
+
+### Bootstrap handler
+
+The Connector registers a listener on `events.bootstrap` in `_init()` via `_registerBootstrapHandler()`. When the server sends a bootstrap message (latest ref on connect, or periodic heartbeat), the handler feeds the `ConnectorPayload` directly into `_processIncoming()`. This means:
+
+- **Dedup**: Refs already received via multicast are filtered out automatically
+- **Gap detection**: If `causalOrdering` is enabled, bootstrap refs participate in sequence tracking
+- **Callbacks**: `listen()` callbacks fire for genuinely new refs
+- **ACK**: If `requireAck` is enabled, client ACKs are sent back
+
+The `tearDown()` method cleans up the bootstrap listener alongside all other socket listeners.
+
 ## Future Enhancements
 
 ### Planned Features

package/README.public.md

@@ -906,6 +906,79 @@ const db = new Db(multi);
 // Priority: io1 > io2 > io3
 ```
 
+## Connector (sync protocol)
+
+The `Connector` bridges a local `Db` with a remote server via socket events. It enriches outgoing refs with protocol metadata and processes incoming refs with dedup, origin filtering, and gap detection.
+
+### Creating a Connector
+
+```typescript
+import { Connector } from '@rljson/db';
+import { Route } from '@rljson/rljson';
+import type { SyncConfig } from '@rljson/rljson';
+
+const route = Route.fromFlat('/sharedTree');
+
+// Minimal — no enrichment
+const connector = new Connector(db, route, socket);
+
+// With sync config — enables enriched payloads
+const syncConfig: SyncConfig = {
+  causalOrdering: true,
+  requireAck: true,
+  ackTimeoutMs: 5_000,
+  includeClientIdentity: true,
+  maxDedupSetSize: 10_000,
+};
+const connector = new Connector(db, route, socket, { syncConfig });
+```
+
+### Sending refs
+
+```typescript
+// Fire-and-forget
+connector.send(ref);
+
+// Wait for server ACK (requires requireAck: true)
+const ack = await connector.sendWithAck(ref);
+// ack: { r, ok, receivedBy, totalClients }
+```
+
+### Receiving refs
+
+```typescript
+// Safe callback with dedup, origin filtering, gap detection
+connector.listen(async (ref) => {
+  console.log('New ref:', ref);
+});
+```
+
+### Predecessors
+
+When `causalOrdering` is enabled, the Connector automatically populates the `p` (predecessors) field from the `InsertHistoryRow.previous` array whenever the local Db notifies about a new insert. No manual call to `setPredecessors()` is needed for standard database-driven sends.
+
+```typescript
+// Manual override (advanced) — sets predecessors for the NEXT send only
+connector.setPredecessors(['1700000000000:AbCd']);
+```
+
+### Bounded dedup
+
+The Connector tracks recently sent and received refs to prevent duplicates. The dedup sets use **two-generation eviction**: when the current set reaches `maxDedupSetSize` (default 10 000), it rotates to previous and a new current set starts. Lookups check both generations. This caps memory usage at ≈ 2 × `maxDedupSetSize` entries.
+
+### Bootstrap handling
+
+The Connector automatically listens for `${route}:bootstrap` events from the server. When a client joins after data has already been sent, the server pushes the latest ref via this event. The Connector feeds it into `_processIncoming()`, so dedup, gap detection, and `listen()` callbacks all work identically to regular multicast refs.
+
+No additional setup is needed — the bootstrap handler is registered in `_init()` and cleaned up in `tearDown()`.
+
+### Cleanup
+
+```typescript
+connector.tearDown();
+// Removes all socket listeners and clears internal state
+```
+
 ## Examples
 
 See [src/example.ts](src/example.ts) for a complete working example demonstrating:

package/dist/README.architecture.md

@@ -777,6 +777,68 @@ describe('Tree WHERE clause fix', () => {
 });
 ```
 
+## Connector (Sync Protocol)
+
+The `Connector` class implements the client side of the RLJSON sync protocol. It sits between a local `Db` and a socket, enriching outgoing refs with protocol metadata and processing incoming refs with safety guarantees.
+
+### Responsibilities
+
+```
+┌────────────────────────────────────────────────┐
+│                  Connector                     │
+│                                                │
+│  Outgoing path (Db → socket):                  │
+│    1. Db.notify fires with InsertHistoryRow    │
+│    2. Auto-populate predecessors from          │
+│       InsertHistoryRow.previous                │
+│    3. Enrich with seq, c, t, p (per config)    │
+│    4. Add to sent-refs dedup set               │
+│    5. Emit ConnectorPayload on socket          │
+│                                                │
+│  Incoming path (socket → callbacks):           │
+│    1. Receive ConnectorPayload from socket      │
+│    2. Reject self-echo (origin === own origin)  │
+│    3. Reject duplicate (ref already received)   │
+│    4. Detect sequence gaps → request gap-fill   │
+│    5. Emit client ACK (if requireAck)           │
+│    6. Invoke listen() callbacks                │
+└────────────────────────────────────────────────┘
+```
+
+### Bounded dedup strategy
+
+The Connector tracks recently sent and received refs to prevent duplicates. Both sets use **two-generation eviction**:
+
+1. A `current` Set accumulates refs.
+2. When `current.size ≥ maxDedupSetSize`, it rotates: `previous = current`, `current = new Set()`.
+3. Lookups check **both** `current` and `previous`.
+4. This caps memory at ≈ 2 × `maxDedupSetSize` entries (default 10 000 per generation).
+
+The eviction is O(1) — no per-entry bookkeeping — and avoids the overhead of a full LRU cache.
+
+### Auto-predecessor from InsertHistory
+
+When `causalOrdering` is enabled, the Connector's Db observer reads the `previous` field from each `InsertHistoryRow` emitted by `Db.notify`. These `InsertHistoryTimeId` values become the `p` (predecessors) array in the outgoing `ConnectorPayload`. This eliminates the need for manual `setPredecessors()` calls in normal database-driven workflows.
+
+### listen()
+
+`listen()` registers callbacks that receive incoming refs through the full sync pipeline: origin filtering, dedup, gap detection, and ACK. All protocol safeguards are applied before callbacks are invoked.
+
+### sendWithAck ordering
+
+The `sendWithAck()` method registers the ACK listener **before** emitting the ref on the socket. This prevents a race condition with synchronous transports (e.g., in-memory sockets used in tests) where the ACK fires during `send()` and would be lost if the listener were registered after.
+
+### Bootstrap handler
+
+The Connector registers a listener on `events.bootstrap` in `_init()` via `_registerBootstrapHandler()`. When the server sends a bootstrap message (latest ref on connect, or periodic heartbeat), the handler feeds the `ConnectorPayload` directly into `_processIncoming()`. This means:
+
+- **Dedup**: Refs already received via multicast are filtered out automatically
+- **Gap detection**: If `causalOrdering` is enabled, bootstrap refs participate in sequence tracking
+- **Callbacks**: `listen()` callbacks fire for genuinely new refs
+- **ACK**: If `requireAck` is enabled, client ACKs are sent back
+
+The `tearDown()` method cleans up the bootstrap listener alongside all other socket listeners.
+
 ## Future Enhancements
 
 ### Planned Features

package/dist/README.public.md

@@ -906,6 +906,79 @@ const db = new Db(multi);
 // Priority: io1 > io2 > io3
 ```
 
+## Connector (sync protocol)
+
+The `Connector` bridges a local `Db` with a remote server via socket events. It enriches outgoing refs with protocol metadata and processes incoming refs with dedup, origin filtering, and gap detection.
+
+### Creating a Connector
+
+```typescript
+import { Connector } from '@rljson/db';
+import { Route } from '@rljson/rljson';
+import type { SyncConfig } from '@rljson/rljson';
+
+const route = Route.fromFlat('/sharedTree');
+
+// Minimal — no enrichment
+const connector = new Connector(db, route, socket);
+
+// With sync config — enables enriched payloads
+const syncConfig: SyncConfig = {
+  causalOrdering: true,
+  requireAck: true,
+  ackTimeoutMs: 5_000,
+  includeClientIdentity: true,
+  maxDedupSetSize: 10_000,
+};
+const connector = new Connector(db, route, socket, { syncConfig });
+```
+
+### Sending refs
+
+```typescript
+// Fire-and-forget
+connector.send(ref);
+
+// Wait for server ACK (requires requireAck: true)
+const ack = await connector.sendWithAck(ref);
+// ack: { r, ok, receivedBy, totalClients }
+```
+
+### Receiving refs
+
+```typescript
+// Safe callback with dedup, origin filtering, gap detection
+connector.listen(async (ref) => {
+  console.log('New ref:', ref);
+});
+```
+
+### Predecessors
+
+When `causalOrdering` is enabled, the Connector automatically populates the `p` (predecessors) field from the `InsertHistoryRow.previous` array whenever the local Db notifies about a new insert. No manual call to `setPredecessors()` is needed for standard database-driven sends.
+
+```typescript
+// Manual override (advanced) — sets predecessors for the NEXT send only
+connector.setPredecessors(['1700000000000:AbCd']);
+```
+
+### Bounded dedup
+
+The Connector tracks recently sent and received refs to prevent duplicates. The dedup sets use **two-generation eviction**: when the current set reaches `maxDedupSetSize` (default 10 000), it rotates to previous and a new current set starts. Lookups check both generations. This caps memory usage at ≈ 2 × `maxDedupSetSize` entries.
+
+### Bootstrap handling
+
+The Connector automatically listens for `${route}:bootstrap` events from the server. When a client joins after data has already been sent, the server pushes the latest ref via this event. The Connector feeds it into `_processIncoming()`, so dedup, gap detection, and `listen()` callbacks all work identically to regular multicast refs.
+
+No additional setup is needed — the bootstrap handler is registered in `_init()` and cleaned up in `tearDown()`.
+
+### Cleanup
+
+```typescript
+connector.tearDown();
+// Removes all socket listeners and clears internal state
+```
+
 ## Examples
 
 See [src/example.ts](src/example.ts) for a complete working example demonstrating:

package/dist/connector/connector.d.ts

@@ -1,10 +1,7 @@
 import { Socket } from '@rljson/io';
-import { Route } from '@rljson/rljson';
+import { AckPayload, ClientId, InsertHistoryTimeId, Route, SyncConfig, SyncEventNames } from '@rljson/rljson';
 import { Db } from '../db.ts';
-export type ConnectorPayload = {
-    o: string;
-    r: string;
-};
+export type { ConnectorPayload } from '@rljson/rljson';
 export type ConnectorCallback = (ref: string) => Promise<any>;
 export declare class Connector {
     private readonly _db;
@@ -13,15 +10,79 @@ export declare class Connector {
     private _origin;
     private _callbacks;
     private _isListening;
-    private _sentRefs;
-    private _receivedRefs;
-    constructor(_db: Db, _route: Route, _socket: Socket);
+    private _sentRefsCurrent;
+    private _sentRefsPrevious;
+    private _receivedRefsCurrent;
+    private _receivedRefsPrevious;
+    private readonly _maxDedup;
+    private readonly _syncConfig;
+    private readonly _clientId;
+    private readonly _events;
+    private _seq;
+    private _lastPredecessors;
+    private _peerSeqs;
+    constructor(_db: Db, _route: Route, _socket: Socket, syncConfig?: SyncConfig, clientIdentity?: ClientId);
+    /**
+     * Sends a ref to the server via the socket.
+     * Enriches the payload based on SyncConfig flags.
+     * @param ref - The ref to send
+     */
     send(ref: string): void;
-    listen(callback: (editHistoryRef: string) => Promise<void>): void;
+    /**
+     * Sends a ref and waits for server acknowledgment.
+     * Only meaningful when `syncConfig.requireAck` is `true`.
+     * @param ref - The ref to send
+     * @returns A promise that resolves with the AckPayload
+     */
+    sendWithAck(ref: string): Promise<AckPayload>;
+    /**
+     * Sets the causal predecessors for the next send.
+     * @param predecessors - The InsertHistory timeIds of causal predecessors
+     */
+    setPredecessors(predecessors: InsertHistoryTimeId[]): void;
+    /**
+     * Registers a callback for incoming refs on this route.
+     *
+     * Incoming refs are processed through the full sync pipeline:
+     * origin filtering, dedup, gap detection, and ACK.
+     *
+     * @param callback - The callback to invoke with each deduplicated incoming ref
+     */
+    listen(callback: ConnectorCallback): void;
+    /**
+     * Returns the current sequence number.
+     * Only meaningful when `causalOrdering` is enabled.
+     */
+    get seq(): number;
+    /**
+     * Returns the stable client identity.
+     * Only available when `includeClientIdentity` is enabled.
+     */
+    get clientIdentity(): ClientId | undefined;
+    /**
+     * Returns the sync configuration, if any.
+     */
+    get syncConfig(): SyncConfig | undefined;
+    /**
+     * Returns the typed event names for this connector's route.
+     */
+    get events(): SyncEventNames;
     private _init;
-    teardown(): void;
+    tearDown(): void;
+    private _hasSentRef;
+    private _addSentRef;
+    private _hasReceivedRef;
+    private _addReceivedRef;
     private _notifyCallbacks;
+    private _processIncoming;
     private _registerSocketObserver;
+    private _registerGapFillHandler;
+    /**
+     * Listens for bootstrap messages from the server.
+     * The server sends the latest ref on connect and optionally via heartbeat.
+     * _processIncoming handles dedup so already-seen refs are filtered out.
+     */
+    private _registerBootstrapHandler;
     private _registerDbObserver;
     get socket(): Socket;
     get route(): Route;

package/dist/db.d.ts

@@ -1,6 +1,6 @@
 import { Io } from '@rljson/io';
 import { Json, JsonValue } from '@rljson/json';
-import { Edit, EditHistory, InsertHistoryRow, InsertHistoryTimeId, MultiEdit, Ref, Rljson, Route, SliceId, TableType } from '@rljson/rljson';
+import { Edit, EditHistory, InsertHistoryRow, InsertHistoryTimeId, MultiEdit, Ref, Rljson, Route, SliceId, TableType, Tree } from '@rljson/rljson';
 import { Controller, ControllerChildProperty, ControllerRefs } from './controller/controller.ts';
 import { Core } from './core.ts';
 import { Join } from './join/join.ts';
@@ -90,6 +90,28 @@ export declare class Db {
         skipNotification?: boolean;
         skipHistory?: boolean;
     }): Promise<InsertHistoryRow<any>[]>;
+    /**
+     * Insert pre-decomposed tree nodes into a tree table.
+     *
+     * Unlike `insert()`, which expects a plain nested object and decomposes it
+     * via `treeFromObject()`, this method accepts an array of already-decomposed
+     * `Tree` nodes (e.g. from FsScanner). The **root node must be the last
+     * element** in the array.
+     *
+     * The method goes through the full insert pipeline:
+     * 1. Writes each node via TreeController
+     * 2. Creates an InsertHistoryRow automatically
+     * 3. Calls `notify.notify()` so Connector observers fire
+     *
+     * @param treeKey - The tree table key (must end with "Tree")
+     * @param trees - Pre-decomposed Tree nodes, root LAST
+     * @param options - Optional: skip notification or history
+     * @returns The InsertHistoryRow for the root node
+     */
+    insertTrees(treeKey: string, trees: Tree[], options?: {
+        skipNotification?: boolean;
+        skipHistory?: boolean;
+    }): Promise<InsertHistoryRow<any>[]>;
     /**
      * Recursively runs controllers based on the route of the Insert
      * @param insert - The Insert to run