y-reticulum 0.1.0

package/README.md

@@ -0,0 +1,106 @@
+# Reticulum connector for [Yjs](https://github.com/yjs/yjs)
+
+Propagates document updates over [Reticulum](https://reticulum.network/) mesh network.
+
+* Public key encryption and authorization using [Reticulum Identities](https://reticulum.network/manual/zen.html#identity-and-nomadism)
+* Flexible network topology and multiple interfaces ranging from TCP to LoRa and HF radio links
+* Very little setup needed with Reticulum announce and discovery mechanisms
+* Sync and awareness traffic rides a reliable, in-order, windowed Link Channel
+  (retransmitted on lossy hops) for performant CRDT synchronization
+* Larger CRDT updates are automatically transported as bz2 compressed Resources
+
+Built on [reticulum-js](https://reticulum.js.org/) with aim to support browsers, Node.js, and Deno. For browsers, please read the [browser connectivity](https://reticulum.js.org/documents/Browser_Connectivity.html) notes.
+
+## Status
+
+Just getting started
+
+## Install
+
+```sh
+npm i y-reticulum
+```
+
+## Usage
+
+Clients connected to the same room name share document updates. In addition to
+a `Y.Doc`, you pass a configured [@reticulum/core](https://reticulum.js.org/)
+instance — the provider does not open network interfaces itself.
+
+```js
+import * as Y from "yjs"
+import { Identity, Reticulum } from "@reticulum/core"
+import { TCPClientInterface } from "@reticulum/node"
+import { ReticulumProvider } from "y-reticulum"
+
+// 1. Connect to the Reticulum mesh. Prefer the local shared instance (e.g. a
+//    running `rnsd`); fall back to a direct TCP interface when there is none.
+const rns = new Reticulum()
+const shared = await rns.connectToSharedInstance()
+if (!shared) {
+  const tcp = new TCPClientInterface({ host: "127.0.0.1", port: 42424 })
+  await tcp.connect()
+  rns.addInterface(tcp, true)
+}
+
+// 2. An identity for this peer (persist it between runs in real apps so your
+//    Reticulum address stays stable).
+const identity = await Identity.generate()
+
+// 3. Create the Yjs document and the provider.
+const ydoc = new Y.Doc()
+const provider = new ReticulumProvider("your-room-name", ydoc, {
+  reticulum: rns,
+  identity,
+})
+
+provider.on("status", ({ connected }) => console.log("connected:", connected))
+provider.on("synced", ({ synced }) => console.log("synced:", synced))
+provider.on("peers", ({ added, removed }) =>
+  console.log("peers added:", added, "removed:", removed),
+)
+
+await provider.connect()
+
+const yarray = ydoc.getArray("array")
+```
+
+## API
+
+```js
+new ReticulumProvider(roomName, ydoc[, opts])
+```
+
+`opts` accepts the following (all optional except `reticulum`):
+
+```js
+{
+  // A configured Reticulum instance with at least one (default) interface
+  // attached. Required — the provider does not open interfaces itself.
+  reticulum,
+  // Identity for this peer's room destination. Generated (non-persistent) if
+  // omitted; supply your own to keep a stable address across restarts.
+  identity,
+  // Reuse an existing Awareness instance - see https://github.com/yjs/y-protocols
+  awareness: new awarenessProtocol.Awareness(ydoc),
+  // Upper bound on simultaneous peer Links. Mirrors y-webrtc's `maxConns`.
+  maxConns: 20,
+  // Cadence (ms) at which the room destination is re-announced for discovery.
+  // Delegated to @reticulum/core's Destination.startAnnouncing, which clamps
+  // to the 60s floor from the Reticulum spec (sub-minute intervals trigger
+  // ingress rate limiting).
+  announceIntervalMs: 60_000,
+}
+```
+
+The provider extends `ObservableV2` and emits:
+
+| Event | Payload | When |
+| --- | --- | --- |
+| `status` | `{ connected: boolean }` | the provider (dis)connects from the mesh |
+| `synced` | `{ synced: boolean }` | sync state with the peer mesh changes |
+| `peers` | `{ added: string[], removed: string[] }` | peers are discovered or drop off |
+
+## License
+
+Licensed under the [EUPL 1.2](https://interoperable-europe.ec.europa.eu/collection/eupl/eupl-text-eupl-12).

package/SPEC.md

@@ -0,0 +1,229 @@
+# y-reticulum — Reticulum provider for Yjs
+
+A Yjs provider that synchronizes documents over the [Reticulum Network System
+(RNS)](https://reticulum.network/) mesh. The goal is to reach feature parity
+with the [y-webrtc](https://github.com/yjs/y-webrtc) provider, substituting
+Reticulum's transport and discovery primitives for WebRTC + signaling servers.
+
+## Goals
+
+- Synchronize `Y.Doc` state and `Awareness` between peers over Reticulum.
+- Work anywhere `reticulum-js` runs (Node.js, Deno, browsers).
+- Provide an API and event surface familiar to anyone who has used
+  `WebrtcProvider` (`status`, `synced`, `peers`).
+- Be roughly on the same feature level as y-webrtc.
+
+## Non-goals (for now)
+
+- Acting as a Reticulum *transport* (routing) node. We are a leaf node, same as
+  the rest of `reticulum-js`.
+- E2E encryption of the room beyond what Reticulum's Links already provide.
+- A `BroadcastChannel` same-origin/tab shortcut. Reticulum is the single
+  transport.
+
+## Architecture
+
+### How y-webrtc works (reference)
+
+- A `WebrtcProvider` wraps a `Y.Doc` and opens a `Room` (one per room name).
+- Peer discovery happens through one or more **signaling servers** (WebSocket):
+  clients `announce` their `peerId` and relay WebRTC `offer`/`answer`/`signal`
+  messages through the server.
+- Each pair of peers opens a **WebRTC data channel** (`simple-peer`).
+- Doc/awareness updates are encoded with `lib0` and framed with a 1-byte
+  message-type tag, then sent over every peer channel.
+- An optional `BroadcastChannel` path shortcuts same-origin tabs.
+
+The message wire protocol (reused verbatim here):
+
+| Tag | Meaning |
+|---|---|
+| `0` | sync (carries `messageYjsSyncStep1` / `Step2` / `update`) |
+| `1` | awareness update |
+| `3` | query awareness |
+| `4` | broadcastchannel peer-id add/remove — **not needed** (no BC) |
+
+### Reticulum primitives we build on
+
+- **`Destination`** (IN/OUT, SINGLE/GROUP/PLAIN) + **`Announce`** for
+  authenticated, signed peer discovery. Each peer's announce carries its public
+  identity, which others recall to open Links.
+- **`Link`** — an ephemeral, encrypted channel between two destinations,
+  established via a `LINKREQUEST`/`LRPROOF` handshake. The base for everything
+  below.
+- **`Channel`** — a reliable, in-order, windowed typed-message layer over a
+  Link (`link.getChannel()`). Adds automatic retries (retransmit on a missing
+  proof), send-window flow control, and dedup — so a sync update or awareness
+  change dropped on a lossy hop is retransmitted rather than lost. Carries the
+  bulk of Yjs traffic.
+- **`Resource`** — chunked, hash-verified large-payload transport over a Link,
+  with automatic bz2 compression. Used for oversized sync payloads (initial
+  state, big `syncStep2`) that exceed the channel MDU.
+- **`request()`/`response()`** RPC built on Links — *not* used directly; we run
+  our own framing so the message-type tag matches y-webrtc's semantics.
+
+### Concept mapping
+
+| y-webrtc | y-reticulum |
+|---|---|
+| Signaling server `announce`/`publish` | Reticulum `Announce` to a deterministic destination |
+| WebRTC data channel | `Link` + `Channel` |
+| Raw peer `.send(bytes)` | reliable `Channel` message (small) — `ContextType.CHANNEL` DATA packets with retries + send window |
+| Large peer payloads (none) | `Resource` (bz2-compressed) |
+| BroadcastChannel (same-origin) | not applicable |
+| `WebrtcProvider` (`ObservableV2`: `status`/`synced`/`peers`) | `ReticulumProvider` with the same events |
+| Message tags 0/1/3 | reused verbatim |
+
+### Discovery model (chosen: deterministic destination per room)
+
+Each peer creates a **`SINGLE` IN destination** whose full app name is derived
+from the room name, e.g.:
+
+```
+y-reticulum.sync.<hex(hash(roomName))>
+```
+
+The peer announces this destination. Other peers running the same room name
+learn the announcer's identity from the announce and open a `Link` to it.
+Because every peer both announces and listens, the topology is a full mesh of
+pairwise Links (bounded by `maxConns`, same as y-webrtc).
+
+> A pure group-destination broadcast (one destination, no Links) was considered
+> and rejected: it loses Link-level reliability/encryption/ordering and makes
+> `peers`/`maxConns` semantics awkward. We keep the "announce → connect to peers
+> individually" model that mirrors y-webrtc.
+
+### Room identity and the destination hash
+
+- The room name is hashed to form the destination aspect so that two peers that
+  type the same room name arrive at the same destination namespace without
+  leaking the cleartext room name in announce app data.
+- Each peer generates (and persists, when a storage adapter is available) its
+  own `Identity`. The announce carries a small `app_data` blob identifying this
+  peer (peer id + provider version) for diagnostics.
+
+### Wire protocol on a Link
+
+Identical framing to y-webrtc, minus tag `4`:
+
+```
+<1-byte tag><payload encoded with lib0>
+```
+
+- tag `0` sync → `syncProtocol.readSyncMessage` / `writeSyncStep1/2` / `writeUpdate`
+- tag `1` awareness → `awarenessProtocol.encodeAwarenessUpdate` / `applyAwarenessUpdate`
+- tag `3` query awareness → reply with tag `1`
+
+Small messages travel as reliable `Channel` messages (a `MessageBase` whose
+body is the raw y-webrtc frame), giving automatic retries, in-order delivery,
+and send-window flow control over the Link. Messages exceeding the channel MDU
+(link MDU minus the 6-byte channel envelope) cannot fit in a single channel
+message and are transported via a `Resource` (bz2-compressed), reassembled on
+the receiver before being handed to the same `readMessage` path.
+
+## Public API (target)
+
+```js
+import * as Y from "yjs";
+import { ReticulumProvider } from "y-reticulum";
+
+const doc = new Y.Doc();
+const provider = new ReticulumProvider("my-room", doc, {
+  identity,        // optional; generated/persisted if omitted
+  reticulum,       // optional pre-configured Reticulum instance
+  awareness,       // optional; created if omitted
+  maxConns,        // optional; default 20-ish like y-webrtc
+  announceInterval,// optional
+});
+
+provider.on("status", ({ connected }) => { /* ... */ });
+provider.on("synced", ({ synced }) => { /* ... */ });
+provider.on("peers",  ({ added, removed }) => { /* ... */ });
+```
+
+Methods mirror y-webrtc: `connect()`, `disconnect()`, `destroy()`.
+
+## Project layout
+
+```
+src/
+  index.js                 # public exports
+  provider.js              # ReticulumProvider
+  room.js                  # Room abstraction (announces, tracks peer Links)
+  peer-conn.js             # one peer-to-peer Link wrapper
+  messages.js              # message tags + readMessage/broadcast helpers
+  destination.js           # room-name → deterministic destination name helpers
+test/
+  *.smoke.js               # smoketests per layer
+examples/                  # demo clients (later)
+```
+
+## Type safety
+
+All source is plain JavaScript (`.js`) with **JSDoc type annotations verified by
+the TypeScript checker** — no hand-written `.ts` source files. This matches the
+conventions of both `y-webrtc` and `reticulum-js`.
+
+- `tsconfig.json` enables `allowJs: true` and `checkJs: true` (plus
+  `declaration` / `emitDeclarationOnly: true` so a `.d.ts` bundle is produced).
+- Every function, method, and constructor gets `@param {Type} name` /
+  `@returns {Type}` annotations; module-level `@typedef`s describe option objects
+  and event payloads; `@import` (or `import` in `@type`) references types from
+  `yjs`, `y-protocols`, and `@reticulum/core`.
+- `npm run types` (`tsc`) **must pass after every change** — this is enforced by
+  `AGENTS.md`. Treat type errors as build failures, not warnings.
+- `lib0` types (`encoding.Encoder`, `decoding.Decoder`, `observable.ObservableV2`,
+  etc.) are referenced the same way `y-webrtc` references them. Note `lib0` is a
+  transitive dependency of `y-protocols`; if it is not resolvable it must be added
+  as a direct dependency (ask first, per `AGENTS.md`).
+- Emitted `.d.ts` files are excluded from version control (already in
+  `.gitignore`).
+
+## Implementation phases
+
+### Phase 0 — Scaffolding
+- Add `tsconfig.json` (`allowJs` + `checkJs` + `declaration` +
+  `emitDeclarationOnly`) so `npm run types` passes against `src/`.
+- Empty, fully JSDoc-annotated `src/index.js` re-exporting the (upcoming)
+  provider.
+- Confirm `npm run types` and `npm run format` are green.
+
+### Phase 1 — Transport smoketest (foundation)
+- A smoketest that spins up two in-process `Reticulum` instances (loopback
+  interface), derives the same room destination name on each, announces, and
+  establishes a `Link`, then exchanges raw bytes both ways.
+- Validates discovery + Link transport before sync semantics land.
+
+### Phase 2 — Provider skeleton
+- `ReticulumProvider` constructor wiring (`Y.Doc`, `Awareness`, identity,
+  Reticulum connect), `connect()`/`disconnect()`, `status` events.
+- `Room` that announces and listens for announces/links; `peers` emission.
+- No Yjs sync yet — just connection lifecycle.
+
+### Phase 3 — Sync protocol layer
+- `readMessage` / broadcast over peer Links.
+- Doc `update` handler → broadcast sync `update`.
+- Awareness update/query handlers.
+- `synced` tracking across peers (mirror `checkIsSynced`).
+- Smoketest: two providers, one edits, the other observes the change.
+
+### Phase 4 — Resource-backed large transfers
+- Route oversized payloads through `Resource`; reassemble and feed into the
+  same `readMessage` path.
+- Smoketest: large initial-doc sync.
+
+### Phase 5 — Hardening & parity
+- `maxConns` enforcement, reconnection, keepalive/timeout behavior, clean
+  teardown (`destroy`), graceful announce removal on disconnect.
+- Parity checklist against y-webrtc features.
+
+## Open questions
+
+- Default announce cadence: now delegated to `@reticulum/core`'s
+  `Destination.startAnnouncing`, which enforces the §9.7 60 s floor; y-reticulum
+  defaults to that floor and forwards any user override (clamped). Open:
+  whether to send a path request up front to accelerate first-peer discovery
+  on a fresh mesh.
+- Whether/how to expose the configured Reticulum interfaces (auto vs. explicit
+  TCP/WebSocket) or always prefer `connectToSharedInstance()` with a fallback.
+- `maxConns` semantics: do we cap total Links, or per-room Links?

package/package.json

@@ -0,0 +1,42 @@
+{
+  "name": "y-reticulum",
+  "version": "0.1.0",
+  "description": "Reticulum provider for Yjs",
+  "keywords": [
+    "reticulum",
+    "rns",
+    "yjs"
+  ],
+  "homepage": "https://github.com/bergie/y-reticulum#readme",
+  "bugs": {
+    "url": "https://github.com/bergie/y-reticulum/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git://github.com/bergie/y-reticulum.git"
+  },
+  "license": "EUPL-1.2",
+  "author": "Henri Bergius <henri.bergius@iki.fi>",
+  "type": "module",
+  "main": "src/index.js",
+  "scripts": {
+    "lint": "npx @biomejs/biome check --use-editorconfig=true src/",
+    "format": "npx @biomejs/biome check --use-editorconfig=true --write src/ test/",
+    "types": "npx tsc",
+    "test": "node --test --test-force-exit --test-timeout 5000 test/*.js test/**/*.js",
+    "test:deno": "deno test --no-check --allow-net --allow-env --allow-read=node_modules test/*.js",
+    "test:bun": "bun test"
+  },
+  "dependencies": {
+    "@digitaldefiance/bzip2-wasm": "^1.1.1",
+    "@reticulum/core": "^0.5.0",
+    "@reticulum/node": "^0.5.0",
+    "lib0": "^0.2.117",
+    "y-protocols": "^1.0.7"
+  },
+  "devDependencies": {
+    "@types/node": "^26.1.2",
+    "typescript": "^6.0.3",
+    "yjs": "^13.6.31"
+  }
+}

package/src/compression.js

@@ -0,0 +1,36 @@
+/**
+ * @file compression.js
+ * @description Shared bzip2 provider for compressing Reticulum Resources.
+ *
+ * `@digitaldefiance/bzip2-wasm` is a hard dependency of y-reticulum, so both
+ * peers can always compress/decompress large sync payloads (an initial doc
+ * state or a big update). The WASM module needs a one-time async `init()`; this
+ * module exposes a shared, lazily-initialized instance. If init ever fails we
+ * resolve to `null` and sync transparently falls back to uncompressed Resources.
+ */
+import BZip2 from "@digitaldefiance/bzip2-wasm";
+
+/** @type {Promise<import("@digitaldefiance/bzip2-wasm").default | null> | null} */
+let initPromise = null;
+
+/**
+ * Returns a shared, initialized BZip2 instance, or `null` if the WASM module
+ * failed to load. Safe to call repeatedly — initialization runs only once.
+ *
+ * @returns {Promise<import("@digitaldefiance/bzip2-wasm").default | null>}
+ */
+export function getCompressionProvider() {
+  if (!initPromise) {
+    initPromise = (async () => {
+      try {
+        const bz2 = new BZip2();
+        await bz2.init();
+        return bz2;
+      } catch {
+        // WASM unavailable / failed to load — Resources will go uncompressed.
+        return null;
+      }
+    })();
+  }
+  return initPromise;
+}

package/src/destination.js

@@ -0,0 +1,41 @@
+/**
+ * @file destination.js
+ * @description Helpers mapping a Yjs room name to a Reticulum destination.
+ *
+ * Two peers that pass the same room name must arrive at the same Reticulum
+ * "aspect" so they can discover each other via the Announce mechanism. We hash
+ * the room name into the aspect so the cleartext name is not leaked on the wire,
+ * and so the resulting 10-byte `nameHash` doubles as the room-membership filter
+ * when comparing inbound announces (see SPEC.md → Discovery model).
+ */
+
+/**
+ * App-name prefix shared by every y-reticulum sync destination. The trailing
+ * segment is a hex digest of the room name (see {@link roomDestinationName}).
+ */
+export const DESTINATION_APP_PREFIX = "y-reticulum.sync";
+
+/**
+ * Derives the deterministic Reticulum destination app-name for a Yjs room.
+ *
+ * The room name is hashed (first 8 bytes of its SHA-256, rendered as 16 hex
+ * chars) so the on-wire aspect does not leak the cleartext room name. Two peers
+ * that pass the same `roomName` arrive at the same app-name — and therefore the
+ * same 10-byte `nameHash` — which is exactly what room peer-discovery filters on
+ * when comparing inbound announces.
+ *
+ * @param {string} roomName
+ * @returns {Promise<string>} app-name like `y-reticulum.sync.<16 hex chars>`
+ */
+export async function roomDestinationName(roomName) {
+  const digest = await crypto.subtle.digest(
+    "SHA-256",
+    new TextEncoder().encode(roomName),
+  );
+  const bytes = new Uint8Array(digest);
+  let hex = "";
+  for (let i = 0; i < 8; i++) {
+    hex += bytes[i].toString(16).padStart(2, "0");
+  }
+  return `${DESTINATION_APP_PREFIX}.${hex}`;
+}

package/src/index.js

@@ -0,0 +1,11 @@
+/**
+ * @file index.js
+ * @description Public entry point for the `y-reticulum` package.
+ */
+
+export { getCompressionProvider } from "./compression.js";
+export { roomDestinationName } from "./destination.js";
+export { messageAwareness, messageSync, readMessage } from "./messages.js";
+export { PeerConn } from "./peer-conn.js";
+export { ReticulumProvider } from "./provider.js";
+export { Room } from "./room.js";

package/src/messages.js

@@ -0,0 +1,90 @@
+/**
+ * @file messages.js
+ * @description The Yjs sync wire protocol used over a peer Link.
+ *
+ * This is y-webrtc's framing, minus its BroadcastChannel peer-id message
+ * (tag 4) which has no Reticulum equivalent. Each message is a 1-byte tag
+ * followed by a lib0-encoded body:
+ *
+ *   0  sync        — carries syncStep1 / syncStep2 / update (y-protocols/sync)
+ *   1  awareness   — an awareness update (y-protocols/awareness)
+ *   3  queryAwareness — request the peer's full awareness state
+ *
+ * Bytes flow through {@link PeerConn}; this module only knows how to decode
+ * them and apply them to a Doc / Awareness.
+ */
+import * as decoding from "lib0/decoding";
+import * as encoding from "lib0/encoding";
+import * as awarenessProtocol from "y-protocols/awareness";
+import * as syncProtocol from "y-protocols/sync";
+
+/** @type {0} */
+export const messageSync = 0;
+/** @type {1} */
+export const messageAwareness = 1;
+/** @type {3} */
+export const messageQueryAwareness = 3;
+
+/**
+ * Decodes one inbound framed message, applying it to the doc / awareness, and
+ * returns the bytes of a reply to send back to the same peer (or `null`).
+ *
+ * Mirrors y-webrtc's `readMessage`: a `syncStep1` requests our state and so
+ * produces a `syncStep2` reply; a `syncStep2` delivers the peer's state and
+ * marks the room synced (once, via `onSynced`); `queryAwareness` produces an
+ * awareness reply.
+ *
+ * @param {import("yjs").Doc} doc
+ * @param {awarenessProtocol.Awareness} awareness
+ * @param {Uint8Array} buf
+ * @param {any} origin - transactionOrigin for any updates this applies.
+ * @param {boolean} roomSynced - whether the room is already synced (gates the
+ *   one-shot `onSynced` callback, matching y-webrtc).
+ * @param {() => void} onSynced - invoked once when a syncStep2 first arrives.
+ * @returns {Uint8Array | null} reply bytes, or `null` when no reply is needed.
+ */
+export function readMessage(doc, awareness, buf, origin, roomSynced, onSynced) {
+  const decoder = decoding.createDecoder(buf);
+  const encoder = encoding.createEncoder();
+  const messageType = decoding.readVarUint(decoder);
+  let sendReply = false;
+  switch (messageType) {
+    case messageSync: {
+      encoding.writeVarUint(encoder, messageSync);
+      const syncMessageType = syncProtocol.readSyncMessage(
+        decoder,
+        encoder,
+        doc,
+        origin,
+      );
+      if (syncMessageType === syncProtocol.messageYjsSyncStep2 && !roomSynced) {
+        onSynced();
+      }
+      if (syncMessageType === syncProtocol.messageYjsSyncStep1) {
+        sendReply = true;
+      }
+      break;
+    }
+    case messageQueryAwareness:
+      encoding.writeVarUint(encoder, messageAwareness);
+      encoding.writeVarUint8Array(
+        encoder,
+        awarenessProtocol.encodeAwarenessUpdate(
+          awareness,
+          Array.from(awareness.getStates().keys()),
+        ),
+      );
+      sendReply = true;
+      break;
+    case messageAwareness:
+      awarenessProtocol.applyAwarenessUpdate(
+        awareness,
+        decoding.readVarUint8Array(decoder),
+        origin,
+      );
+      break;
+    default:
+      return null;
+  }
+  return sendReply ? encoding.toUint8Array(encoder) : null;
+}