@automerge/automerge-repo-network-websocket 1.1.0-alpha.7 → 1.1.1

package/README.md

@@ -17,9 +17,7 @@ is the server.
 
 ### Overview
 
-The websocket wire protocol consists of a handshake where each peer tells the
-other what their peer ID is followed by the sync loop where each peer can send
-the other sync messages and ephemeral messages.
+The websocket wire protocol consists of a handshake where each peer tells the other what their peer ID - and some other metadata - is followed by the sync loop where each peer can send the other sync messages and ephemeral messages.
 
 ### Handshake
 
@@ -54,6 +52,10 @@ Handshake is the following steps:
     peer sends an [error](#error) message and terminates the connection
   * it begins the sync phase
 
+#### Peer IDs and storage IDs
+
+The peer ID is an ephemeral ID which is assumed to only live for the lifetime of the process which advertises the given ID (e.g. a browser tab). Peers may optionally advertise a storage ID in the `join` and `peer` messages, this is an ID which is assumed to be tied to a persistent storage of some kind (e.g. an IndexedDB in a browser). Many peer IDs can advertise the same storage ID (as in the case of many browser tabs). The use of a storage ID allows other peers to know whether to save and reload sync states for a given peer (if the peer advertises a storage ID, then save and reload the sync state attached to that storage ID).
+
 
 ### Sync Phase
 
@@ -65,6 +67,22 @@ and receiving is emitting the [corresponding
 event](https://automerge.org/automerge-repo/interfaces/_automerge_automerge_repo.NetworkAdapterEvents.html)
 from the `NetworkAdapter` on receipt.
 
+#### Remote heads gossiping
+
+In some cases peers wish to know about the state of peers who are separated from them by several intermediate peers. For example, a tab running a text editor may wish to show whether the contents of the editor are up to date with respect to a tab running in a browser on another users device. This is achieved by gossiping remote heads across intermediate nodes. The logic for this is the following:
+
+* For a given connection each peer maintains a list of the storage IDs the remote peer is interested in (note this is storage IDs, not peer IDs)
+* Any peer can send a [`remote-subscription-changed`](#remote-subscription-changed) message to change the set of storage IDs they want the recipient to watch on the sender's behalf
+* Any time a peer receives a sync message it checks:
+  * Is the sync message from a peer with a storage ID which some other remote peer has registered interest in
+  * Is the remote peer permitted access to the document which the message pertains to (i.e. either the `sharePolicy` return `true` or the local peer is already syncing the document with the remote)
+* The local peer sends a [`remote-heads-changed`](#remote-heads-changed) message to each remote peer who passes these checks
+* Additionally, whenever the local peer receives a `remote-heads-changed` message it performs the same checks and additionally checks if the timestamp on the `remote-heads-changed` message is greater than the last timestamp for the same storage ID/document combination and if so it forwards it.
+
+In the `browser <-> sync server <-> browser` text editor example above each browser tab would send a `remote-subscription-changed` message to the sync server adding the other browsers storage ID (presumably communicated out of band) to their subscriptions with the sync server. The sync server will then send `remote-heads-changed` messages to each tab when their heads change.
+
+In a more elaborate example such as `browser <-> sync server <-> sync server <-> browser` the intermediate sync servers could be configured to have their `sharePolicy` return `true` for every document when syncing with each other so that `remote-heads-changed` messages are forwarded between them unconditionally, allowing the browser tabs to still learn of each others heads.
+
 ### Message Types
 
 All messages are encoded using CBOR and are described in this document using
@@ -77,12 +95,21 @@ These type definitions are used in every message type
 ```cddl
 ; The base64 encoded bytes of a Peer ID
 peer_id = str
+; The base64 encoded bytes of a Storage ID
+storage_id = str
 ; The possible protocol versions (currently always the string "1")
 protocol_version = "1"
 ; The bytes of an automerge sync message
 sync_message = bstr
 ; The base58check encoded bytes of a document ID
 document_id = str
+; Metadata sent in either the join or peer message types
+peer_metadata = {
+    ; The storage ID of this peer
+    ? storageId: storage_id, 
+    ; Whether the sender expects to connect again with this storage ID
+    isEphemeral: bool
+}
 ```
 
 #### Join
@@ -94,6 +121,7 @@ Sent by the initiating peer in the [handshake](#handshake) phase.
     type: "join",
     senderId: peer_id,
     supportedProtocolVersions: protocol_version
+    ? metadata: peer_metadata,
 }
 ```
 
@@ -108,6 +136,7 @@ Sent by the receiving peer in response to the join message in the
     senderId: peer_id,
     selectedProtocolVersion: protocol_version,
     targetId: peer_id,
+    ? metadata: peer_metadata,
 }
 ```
 
@@ -195,3 +224,48 @@ connection will close
     message: str,
 }
 ```
+
+#### Remote subscription changed
+
+Sent when the sender wishes to change the set of storage IDs they wish to be notified of when the given remotes heads change.
+
+```cddl
+{
+  type: "remote-subscription-change"
+  senderId: peer_id
+  targetId: peer_id
+
+  ; The storage IDs to add to the subscription
+  ? add: [* storage_id]
+
+  ; The storage IDs to remove from the subscription
+  remove: [* storage_id]
+}
+```
+
+#### Remote heads changed
+
+Sent when the sender wishes to inform the receiver that a peer with a storage ID in the receivers remote heads subscription has changed heads. This is either sent when the local peer receives a new sync message directly from the listened-to peer, or when the local peer receives a `remote-heads-changed` message relating to the listened-to peer from another peer.
+
+```cddl
+{
+  type: "remote-heads-changed"
+  senderId: peer_id
+  targetId: peer_id
+
+  ; The document ID of the document that has changed
+  documentId: document_id
+
+  ; A map from storage ID to the heads advertised for a given storage ID
+  newHeads: { 
+    * storage_id => {
+      ; The heads of the new document for the given storage ID as 
+      ; a list of base64 encoded SHA2 hashes
+      heads: [* string]
+      ; The local time on the node which initially sent the remote-heads-changed
+      ; message as milliseconds since the unix epoch
+      timestamp: uint
+    }
+  }
+}
+```

package/dist/BrowserWebSocketClientAdapter.d.ts

@@ -1,5 +1,5 @@
 /// <reference types="ws" />
-import { NetworkAdapter, type PeerId, type PeerMetadata } from "@automerge/automerge-repo";
+import { NetworkAdapter, PeerId, PeerMetadata } from "@automerge/automerge-repo";
 import WebSocket from "isomorphic-ws";
 import { FromClientMessage } from "./messages.js";
 declare abstract class WebSocketNetworkAdapter extends NetworkAdapter {
@@ -7,19 +7,20 @@ declare abstract class WebSocketNetworkAdapter extends NetworkAdapter {
 }
 export declare class BrowserWebSocketClientAdapter extends WebSocketNetworkAdapter {
     #private;
-    timerId?: ReturnType<typeof setTimeout>;
+    readonly url: string;
+    readonly retryInterval: number;
     remotePeerId?: PeerId;
-    url: string;
-    constructor(url: string);
-    connect(peerId: PeerId, peerMetadata: PeerMetadata): void;
+    constructor(url: string, retryInterval?: number);
+    connect(peerId: PeerId, peerMetadata?: PeerMetadata): void;
     onOpen: () => void;
     onClose: () => void;
     onMessage: (event: WebSocket.MessageEvent) => void;
+    onError: (event: WebSocket.ErrorEvent) => void;
     join(): void;
     disconnect(): void;
     send(message: FromClientMessage): void;
-    announceConnection(peerId: PeerId, peerMetadata: PeerMetadata): void;
-    receiveMessage(message: Uint8Array): void;
+    peerCandidate(remotePeerId: PeerId, peerMetadata: PeerMetadata): void;
+    receiveMessage(messageBytes: Uint8Array): void;
 }
 export {};
 //# sourceMappingURL=BrowserWebSocketClientAdapter.d.ts.map

package/dist/BrowserWebSocketClientAdapter.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"BrowserWebSocketClientAdapter.d.ts","sourceRoot":"","sources":["../src/BrowserWebSocketClientAdapter.ts"],"names":[],"mappings":";AAAA,OAAO,EACL,cAAc,EACd,KAAK,MAAM,EACX,KAAK,YAAY,EAElB,MAAM,2BAA2B,CAAA;AAClC,OAAO,SAAS,MAAM,eAAe,CAAA;AAIrC,OAAO,EACL,iBAAiB,EAIlB,MAAM,eAAe,CAAA;AAKtB,uBAAe,uBAAwB,SAAQ,cAAc;IAC3D,MAAM,CAAC,EAAE,SAAS,CAAA;CACnB;AAED,qBAAa,6BAA8B,SAAQ,uBAAuB;;IAGxE,OAAO,CAAC,EAAE,UAAU,CAAC,OAAO,UAAU,CAAC,CAAA;IACvC,YAAY,CAAC,EAAE,MAAM,CAAA;IAGrB,GAAG,EAAE,MAAM,CAAA;gBAEC,GAAG,EAAE,MAAM;IAKvB,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,YAAY,EAAE,YAAY;IAmClD,MAAM,aAKL;IAGD,OAAO,aAYN;IAED,SAAS,UAAW,sBAAsB,UAEzC;IAED,IAAI;IAWJ,UAAU;IAOV,IAAI,CAAC,OAAO,EAAE,iBAAiB;IAwB/B,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,YAAY,EAAE,YAAY;IAc7D,cAAc,CAAC,OAAO,EAAE,UAAU;CA4BnC"}
+{"version":3,"file":"BrowserWebSocketClientAdapter.d.ts","sourceRoot":"","sources":["../src/BrowserWebSocketClientAdapter.ts"],"names":[],"mappings":";AAAA,OAAO,EACL,cAAc,EACd,MAAM,EACN,YAAY,EAEb,MAAM,2BAA2B,CAAA;AAClC,OAAO,SAAS,MAAM,eAAe,CAAA;AAIrC,OAAO,EACL,iBAAiB,EAKlB,MAAM,eAAe,CAAA;AAKtB,uBAAe,uBAAwB,SAAQ,cAAc;IAC3D,MAAM,CAAC,EAAE,SAAS,CAAA;CACnB;AAED,qBAAa,6BAA8B,SAAQ,uBAAuB;;aAQtD,GAAG,EAAE,MAAM;aACX,aAAa;IAJ/B,YAAY,CAAC,EAAE,MAAM,CAAA;gBAGH,GAAG,EAAE,MAAM,EACX,aAAa,SAAO;IAMtC,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,YAAY,CAAC,EAAE,YAAY;IAqCnD,MAAM,aAKL;IAGD,OAAO,aAWN;IAED,SAAS,UAAW,sBAAsB,UAEzC;IAED,OAAO,UAAW,oBAAoB,UAQrC;IAQD,IAAI;IAUJ,UAAU;IAMV,IAAI,CAAC,OAAO,EAAE,iBAAiB;IAY/B,aAAa,CAAC,YAAY,EAAE,MAAM,EAAE,YAAY,EAAE,YAAY;IAU9D,cAAc,CAAC,YAAY,EAAE,UAAU;CAiBxC"}

package/dist/BrowserWebSocketClientAdapter.js

@@ -1,140 +1,145 @@
 import { NetworkAdapter, cbor, } from "@automerge/automerge-repo";
 import WebSocket from "isomorphic-ws";
 import debug from "debug";
+import { isErrorMessage, isPeerMessage, } from "./messages.js";
 import { ProtocolV1 } from "./protocolVersion.js";
-const log = debug("WebsocketClient");
+import { assert } from "./assert.js";
+import { toArrayBuffer } from "./toArrayBuffer.js";
 class WebSocketNetworkAdapter extends NetworkAdapter {
     socket;
 }
 export class BrowserWebSocketClientAdapter extends WebSocketNetworkAdapter {
-    // Type trickery required for platform independence,
-    // see https://stackoverflow.com/questions/45802988/typescript-use-correct-version-of-settimeout-node-vs-window
-    timerId;
-    remotePeerId; // this adapter only connects to one remote client at a time
-    #startupComplete = false;
     url;
-    constructor(url) {
+    retryInterval;
+    #isReady = false;
+    #retryIntervalId;
+    #log = debug("automerge-repo:websocket:browser");
+    remotePeerId; // this adapter only connects to one remote client at a time
+    constructor(url, retryInterval = 5000) {
         super();
         this.url = url;
+        this.retryInterval = retryInterval;
+        this.#log = this.#log.extend(url);
     }
     connect(peerId, peerMetadata) {
-        // If we're reconnecting  make sure we remove the old event listeners
-        // before creating a new connection.
-        if (this.socket) {
+        if (!this.socket || !this.peerId) {
+            // first time connecting
+            this.#log("connecting");
+            this.peerId = peerId;
+            this.peerMetadata = peerMetadata ?? {};
+        }
+        else {
+            this.#log("reconnecting");
+            assert(peerId === this.peerId);
+            // Remove the old event listeners before creating a new connection.
             this.socket.removeEventListener("open", this.onOpen);
             this.socket.removeEventListener("close", this.onClose);
             this.socket.removeEventListener("message", this.onMessage);
+            this.socket.removeEventListener("error", this.onError);
         }
-        if (!this.timerId) {
-            this.timerId = setInterval(() => this.connect(peerId, peerMetadata), 5000);
-        }
-        this.peerId = peerId;
-        this.peerMetadata = peerMetadata;
+        // Wire up retries
+        if (!this.#retryIntervalId)
+            this.#retryIntervalId = setInterval(() => {
+                this.connect(peerId, peerMetadata);
+            }, this.retryInterval);
         this.socket = new WebSocket(this.url);
         this.socket.binaryType = "arraybuffer";
         this.socket.addEventListener("open", this.onOpen);
         this.socket.addEventListener("close", this.onClose);
         this.socket.addEventListener("message", this.onMessage);
-        // mark this adapter as ready if we haven't received an ack in 1 second.
+        this.socket.addEventListener("error", this.onError);
+        // Mark this adapter as ready if we haven't received an ack in 1 second.
         // We might hear back from the other end at some point but we shouldn't
         // hold up marking things as unavailable for any longer
-        setTimeout(() => {
-            if (!this.#startupComplete) {
-                this.#startupComplete = true;
-                this.emit("ready", { network: this });
-            }
-        }, 1000);
+        setTimeout(() => this.#ready(), 1000);
         this.join();
     }
     onOpen = () => {
-        log(`@ ${this.url}: open`);
-        clearInterval(this.timerId);
-        this.timerId = undefined;
-        this.send(joinMessage(this.peerId, this.peerMetadata));
+        this.#log("open");
+        clearInterval(this.#retryIntervalId);
+        this.#retryIntervalId = undefined;
+        this.join();
     };
     // When a socket closes, or disconnects, remove it from the array.
     onClose = () => {
-        log(`${this.url}: close`);
-        if (this.remotePeerId) {
+        this.#log("close");
+        if (this.remotePeerId)
             this.emit("peer-disconnected", { peerId: this.remotePeerId });
-        }
-        if (!this.timerId) {
-            if (this.peerId) {
-                this.connect(this.peerId, this.peerMetadata);
-            }
-        }
+        if (this.retryInterval > 0 && !this.#retryIntervalId)
+            // try to reconnect
+            setTimeout(() => {
+                assert(this.peerId);
+                return this.connect(this.peerId, this.peerMetadata);
+            }, this.retryInterval);
     };
     onMessage = (event) => {
         this.receiveMessage(event.data);
     };
-    join() {
-        if (!this.socket) {
-            throw new Error("WTF, get a socket");
+    onError = (event) => {
+        const { code } = event.error;
+        if (code === "ECONNREFUSED") {
+            this.#log("Connection refused, retrying...");
         }
+        else {
+            /* c8 ignore next */
+            throw event.error;
+        }
+    };
+    #ready() {
+        if (this.#isReady)
+            return;
+        this.#isReady = true;
+        this.emit("ready", { network: this });
+    }
+    join() {
+        assert(this.peerId);
+        assert(this.socket);
         if (this.socket.readyState === WebSocket.OPEN) {
             this.send(joinMessage(this.peerId, this.peerMetadata));
         }
         else {
-            // The onOpen handler automatically sends a join message
+            // We'll try again in the `onOpen` handler
         }
     }
     disconnect() {
-        if (!this.socket) {
-            throw new Error("WTF, get a socket");
-        }
+        assert(this.peerId);
+        assert(this.socket);
         this.send({ type: "leave", senderId: this.peerId });
     }
     send(message) {
-        if ("data" in message && message.data.byteLength === 0) {
-            throw new Error("tried to send a zero-length message");
-        }
-        if (!this.peerId) {
-            throw new Error("Why don't we have a PeerID?");
-        }
-        if (!this.socket || this.socket.readyState !== WebSocket.OPEN) {
-            throw new Error("Websocket Socket not ready!");
-        }
+        if ("data" in message && message.data?.byteLength === 0)
+            throw new Error("Tried to send a zero-length message");
+        assert(this.peerId);
+        assert(this.socket);
+        if (this.socket.readyState !== WebSocket.OPEN)
+            throw new Error(`Websocket not ready (${this.socket.readyState})`);
         const encoded = cbor.encode(message);
-        // This incantation deals with websocket sending the whole
-        // underlying buffer even if we just have a uint8array view on it
-        const arrayBuf = encoded.buffer.slice(encoded.byteOffset, encoded.byteOffset + encoded.byteLength);
-        this.socket?.send(arrayBuf);
+        this.socket.send(toArrayBuffer(encoded));
     }
-    announceConnection(peerId, peerMetadata) {
-        // return a peer object
-        const myPeerId = this.peerId;
-        if (!myPeerId) {
-            throw new Error("we should have a peer ID by now");
-        }
-        if (!this.#startupComplete) {
-            this.#startupComplete = true;
-            this.emit("ready", { network: this });
-        }
-        this.remotePeerId = peerId;
-        this.emit("peer-candidate", { peerId, peerMetadata });
+    peerCandidate(remotePeerId, peerMetadata) {
+        assert(this.socket);
+        this.#ready();
+        this.remotePeerId = remotePeerId;
+        this.emit("peer-candidate", {
+            peerId: remotePeerId,
+            peerMetadata,
+        });
     }
-    receiveMessage(message) {
-        const decoded = cbor.decode(new Uint8Array(message));
-        const { type, senderId } = decoded;
-        const socket = this.socket;
-        if (!socket) {
-            throw new Error("Missing socket at receiveMessage");
-        }
-        if (message.byteLength === 0) {
+    receiveMessage(messageBytes) {
+        const message = cbor.decode(new Uint8Array(messageBytes));
+        assert(this.socket);
+        if (messageBytes.byteLength === 0)
             throw new Error("received a zero-length message");
+        if (isPeerMessage(message)) {
+            const { peerMetadata } = message;
+            this.#log(`peer: ${message.senderId}`);
+            this.peerCandidate(message.senderId, peerMetadata);
         }
-        switch (type) {
-            case "peer": {
-                const { peerMetadata } = decoded;
-                log(`peer: ${senderId}`);
-                this.announceConnection(senderId, peerMetadata);
-                break;
-            }
-            case "error":
-                log(`error: ${decoded.message}`);
-                break;
-            default:
-                this.emit("message", decoded);
+        else if (isErrorMessage(message)) {
+            this.#log(`error: ${message.message}`);
+        }
+        else {
+            this.emit("message", message);
         }
     }
 }

package/dist/NodeWSServerAdapter.d.ts

@@ -4,14 +4,16 @@ import { type WebSocketServer } from "isomorphic-ws";
 import { NetworkAdapter, type PeerMetadata, type PeerId } from "@automerge/automerge-repo";
 import { FromServerMessage } from "./messages.js";
 export declare class NodeWSServerAdapter extends NetworkAdapter {
-    server: WebSocketServer;
+    #private;
+    private server;
+    private keepAliveInterval;
     sockets: {
         [peerId: PeerId]: WebSocket;
     };
-    constructor(server: WebSocketServer);
-    connect(peerId: PeerId, peerMetadata: PeerMetadata): void;
+    constructor(server: WebSocketServer, keepAliveInterval?: number);
+    connect(peerId: PeerId, peerMetadata?: PeerMetadata): void;
     disconnect(): void;
     send(message: FromServerMessage): void;
-    receiveMessage(message: Uint8Array, socket: WebSocket): void;
+    receiveMessage(messageBytes: Uint8Array, socket: WebSocket): void;
 }
 //# sourceMappingURL=NodeWSServerAdapter.d.ts.map

package/dist/NodeWSServerAdapter.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"NodeWSServerAdapter.d.ts","sourceRoot":"","sources":["../src/NodeWSServerAdapter.ts"],"names":[],"mappings":";AAAA,OAAO,SAAS,MAAM,eAAe,CAAA;AACrC,OAAO,EAAE,KAAK,eAAe,EAAE,MAAM,eAAe,CAAA;AAKpD,OAAO,EAEL,cAAc,EACd,KAAK,YAAY,EACjB,KAAK,MAAM,EACZ,MAAM,2BAA2B,CAAA;AAClC,OAAO,EAAqB,iBAAiB,EAAE,MAAM,eAAe,CAAA;AASpE,qBAAa,mBAAoB,SAAQ,cAAc;IACrD,MAAM,EAAE,eAAe,CAAA;IACvB,OAAO,EAAE;QAAE,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAA;KAAE,CAAK;gBAEjC,MAAM,EAAE,eAAe;IAKnC,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,YAAY,EAAE,YAAY;IAqDlD,UAAU;IAIV,IAAI,CAAC,OAAO,EAAE,iBAAiB;IAyB/B,cAAc,CAAC,OAAO,EAAE,UAAU,EAAE,MAAM,EAAE,SAAS;CAsEtD"}
+{"version":3,"file":"NodeWSServerAdapter.d.ts","sourceRoot":"","sources":["../src/NodeWSServerAdapter.ts"],"names":[],"mappings":";AAAA,OAAO,SAAS,MAAM,eAAe,CAAA;AACrC,OAAO,EAAE,KAAK,eAAe,EAAE,MAAM,eAAe,CAAA;AAKpD,OAAO,EAEL,cAAc,EACd,KAAK,YAAY,EACjB,KAAK,MAAM,EACZ,MAAM,2BAA2B,CAAA;AAClC,OAAO,EAEL,iBAAiB,EAGlB,MAAM,eAAe,CAAA;AAOtB,qBAAa,mBAAoB,SAAQ,cAAc;;IAInD,OAAO,CAAC,MAAM;IACd,OAAO,CAAC,iBAAiB;IAJ3B,OAAO,EAAE;QAAE,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAA;KAAE,CAAK;gBAGnC,MAAM,EAAE,eAAe,EACvB,iBAAiB,SAAO;IAKlC,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,YAAY,CAAC,EAAE,YAAY;IAyCnD,UAAU;IAQV,IAAI,CAAC,OAAO,EAAE,iBAAiB;IAqB/B,cAAc,CAAC,YAAY,EAAE,UAAU,EAAE,MAAM,EAAE,SAAS;CA0E3D"}