@le-space/orbitdb-storage-bridge 0.14.0 → 0.15.0

package/README.md

@@ -1,331 +1,109 @@
 # OrbitDB Storage Bridge
 
-> **OrbitDB database backup, restoration and replication through pluggable storage backends, with hash and identity preservation**
-
+> Back up, restore and replicate OrbitDB databases through pluggable storage backends, with hash and identity preservation.
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Node.js](https://img.shields.io/badge/Node.js-22+-green.svg)](https://nodejs.org/)
 [![CI/CD Pipeline](https://github.com/NiKrause/orbitdb-storage-bridge/actions/workflows/ci.yml/badge.svg)](https://github.com/NiKrause/orbitdb-storage-bridge/actions/workflows/ci.yml)
-[![ESLint](https://img.shields.io/badge/ESLint-passing-brightgreen.svg)](https://github.com/NiKrause/orbitdb-storage-bridge/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@le-space/orbitdb-storage-bridge.svg)](https://www.npmjs.com/package/@le-space/orbitdb-storage-bridge)
 
 > [!NOTE]
-> **Renamed from `orbitdb-storacha-bridge`.** Up to 0.6.0 this package was published as
-> `orbitdb-storacha-bridge`. From 0.7.0 it is `@le-space/orbitdb-storage-bridge`, because Storacha is no
-> longer the only backend it bridges to. The rename changes no API — swap the dependency and the
-> import specifiers:
->
-> ```bash
-> npm uninstall orbitdb-storacha-bridge
-> npm install @le-space/orbitdb-storage-bridge
-> ```
->
-> `orbitdb-storacha-bridge/courier-sync` becomes `@le-space/orbitdb-storage-bridge/courier-sync`, and so on
-> for every entry point. Names that refer to Storacha itself stay as they are:
-> `OrbitDBStorachaBridge`, `StorachaIntegration.svelte`, `backends/storacha`, the `storacha_*`
-> localStorage keys that hold saved logins — and so does the debug namespace
-> `libp2p:orbitdb-storacha:*`.
-
-
-> [!IMPORTANT]
-> **The Storacha upload service is gone.** Writes were switched off in May 2026 and the
-> service has since been decommissioned. Backup no longer works, and restore only reaches
-> blocks that something other than Storacha still holds. See
-> [Status: Storacha sunset](#status-storacha-sunset-may-2026) and
-> [docs/STORAGE-BACKENDS.md](docs/STORAGE-BACKENDS.md) for where to go instead.
-
-
-## Table of Contents
-
-- [OrbitDB Storage Bridge](#@le-space/orbitdb-storage-bridge)
-  - [Table of Contents](#table-of-contents)
-  - [Status: Storacha sunset (May 2026)](#status-storacha-sunset-may-2026)
-  - [What we want to accomplish](#what-we-want-to-accomplish)
-    - [The Challenge of Distributed Data Persistence](#the-challenge-of-distributed-data-persistence)
-    - [Architectural Considerations for Local-First Applications](#architectural-considerations-for-local-first-applications)
-    - [Use Cases](#use-cases)
-    - [Architecture Notes](#architecture-notes)
-  - [What This Does](#what-this-does)
-  - [Roadmap](#roadmap)
-  - [Installation](#installation)
-  - [Environment Setup](#environment-setup)
-  - [Demo](#demo)
-    - [NodeJS Demo Scripts (full backup with Manifest, Identity and AccessController and entries blocks)](#nodejs-demo-scripts-full-backup-with-manifest-identity-and-accesscontroller-and-entries-blocks)
-    - [Svelte Components](#svelte-components)
-  - [How It Works](#how-it-works)
-  - [Restore Mechanism](#restore-mechanism)
-  - [Logging](#logging)
-  - [Testing](#testing)
-  - [Contributing](#contributing)
-  - [License](#license)
-
-
-## Does this work from a browser? Measure it
-
-`examples/browser/storage-probe/` is a static page that asks the services this package talks to,
-from a real browser, with no server: do Aleph, Pinata and Lighthouse answer a page at all, does
-their CORS survive a refusal as well as a success, and does anybody on IPFS hold a given CID at an
-address a browser can dial. With your own key it does a real upload and reads it back. The key is
-kept in that browser and sent only to the service it belongs to.
-
-It is published from this repository's Pages, and it opens from a checkout just as well — there is
-no build step, which is the point: what it measures is a browser talking to a service, with
-nothing in between.
-
-## Status: Storacha sunset (May 2026)
-
-Storacha switched off user writes in **May 2026** and has since decommissioned the service.
-Verified on 2026-09-05:
-
-| Check | Result |
-| --- | --- |
-| `up.storacha.network`, `console.storacha.network`, `indexer.storacha.network`, `forge.storacha.network` | no DNS record — upload, console and indexing endpoints are gone |
-| `storacha.network`, `docs.storacha.network` | `301` → `fil.one`, the team's new S3-compatible product |
-| `storacha.link`, `w3s.link` | `301` → `dweb.link`; the gateways only forward to the public IPFS gateway now |
-| the widget demo CID linked in the roadmap below | `504` on `w3s.link`, `dweb.link`, `ipfs.io` and `trustless-gateway.link` |
-| `@storacha/client` on npm | last release `2.1.4`, 2026-05-15, not marked deprecated |
-
-The gateway those redirects pointed at is gone too. On **2026-09-21** Protocol Labs retired
-`ipfs.io` and `dweb.link`: both answer `429` with an RFC 8594 `Sunset` header and a link to
-[gatewaychanges.ipfs.io](https://gatewaychanges.ipfs.io/), so `storacha.link` and `w3s.link`
-now redirect to a closed door. Retrieval defaults here are `ipfs.aleph.cloud` — the one free
-path gateway measured still serving arbitrary CIDs on 2026-09-23 — with
-`trustless-gateway.link` available for verifiable single-block requests
-(`Accept: application/vnd.ipld.raw`). One host is not a fallback chain, which is why `peer-fetch.js` fetches from the providers that
-hold the blocks instead — bitswap over libp2p, measured from a real page at 0.73 s to dial and
-0.26 s for the block, with no credential in the path. See
-[docs/RECOVERY-ON-A-SECOND-DEVICE.md](docs/RECOVERY-ON-A-SECOND-DEVICE.md).
-
-The shutdown is traceable in the open:
-[`upload-service#708`](https://github.com/storacha/upload-service/pull/708) added a `writesDisabled`
-kill switch that makes the eight user-initiated write capabilities
-(`space/blob/{add,remove,replicate}`, `space/index/add`, `upload/{add,remove}`, `store/{add,remove}`)
-return `ServiceUnavailable`, and [`w3infra#636`](https://github.com/storacha/w3infra/pull/636) wired
-`WRITES_DISABLED=true` into the production stack — both merged 2026-05-15. Five days later Storacha
-shipped `storacha space migrate`
-([`@storacha/filecoin-pin-migration`](https://www.npmjs.com/package/@storacha/filecoin-pin-migration)),
-a migration path from Storacha spaces to Filecoin Onchain Cloud. That tool reads spaces through
-endpoints that no longer resolve, so the official migration window has closed.
-
-We found no announcement page: the Storacha blog now redirects to `fil.one/blog`, which carries a
-single post ("Introducing Fil One", 2026-08-12). The dates above come from the code and the DNS,
-not from a press release.
-
-**What this means for this library**
-
-- **Backup does not work.** Every write path ends in `client.uploadFile()` against `up.storacha.network`.
-- **Restore only reaches what someone else still holds.** The p2p-first path still finds blocks that a
-  peer or another pinning service pins; the gateway fallback and the `capability.upload.list`
-  discovery step cannot reach Storacha any more.
-- **The OrbitDB half is unaffected.** Block extraction, CID bridging, CAR packing, identity
-  preservation, UCAN signing and courier-sync are backend-agnostic — only the handful of Storacha
-  client methods mapped in [docs/STORAGE-BACKENDS.md](docs/STORAGE-BACKENDS.md) need a new home.
-
-If you still have an OrbitDB instance with the blocks in it, re-pin them somewhere else now: the data
-is only as alive as the peers that hold it.
-
-
-## What we want to accomplish
-
-### The Challenge of Distributed Data Persistence
-
-In local-first, peer-to-peer applications built on OrbitDB, data naturally replicates across participating peers through libp2p network connections. Under ideal conditions, this distributed architecture provides inherent redundancy—if one peer loses data, they can resynchronize from other active peers in the network. This peer-to-peer replication model represents the current state of OrbitDB technology.
-
-While relay nodes and pinning services (running Helia and OrbitDB instances) can provide additional decentralized persistence for database entries and IPFS-referenced content, the ecosystem still lacks comprehensive infrastructure for long-term archival of large-scale OrbitDB deployments.
-
-### Architectural Considerations for Local-First Applications
-
-OrbitDB's data model differs fundamentally from traditional centralized databases. In local-first architectures, users typically host their own data locally and selectively replicate with specific peers based on collaboration requirements—not with the entire network. This selective replication is essential for scalability and user experience.
-
-However, as OrbitDB instances grow (consider a blog database accumulating years of posts), replication times increase proportionally. At scale, databases require archival strategies and potential sharding to maintain performant synchronization and optimal user experience.
-
-### Use Cases
-
-This bridge addresses several critical scenarios:
-
-**1. Long-term Archival**
-Archive large OrbitDB instances to Storacha/Filecoin storage, enabling efficient cold storage for historical data while maintaining fast replication of active datasets.
-
-**2. Disaster Recovery**
-Provides recovery options when all active peers lose data simultaneously. Users can restore databases from Storacha using either their original identity or a new identity, ensuring business continuity beyond the peer-to-peer network's availability.
-
-**3. Network Resilience**
-Historically, various network environments (corporate networks, ISPs, regional restrictions) have blocked critical protocols including WebRTC and WebSocket/WebTransport. While libp2p's multi-transport architecture provides numerous fallback options, having an additional restoration pathway through IPFS/Storacha offers defense-in-depth for network-hostile environments. Users can restore databases directly from IPFS and maintain incremental backups after each database mutation.
-
-**4. Access Control & Delegation**
-The bridge supports UCAN (User Controlled Authorization Networks) authentication with planned delegation capabilities between OrbitDB instances. This enables fine-grained, time-bound access control for Storacha backup spaces, allowing users to securely share access with collaborators or recovery agents.
-
-### Architecture Notes
-
-Currently, Storacha backup and restore operations utilize Storacha's gateway infrastructure to interface with Filecoin's decentralized storage network. This hybrid approach balances accessibility with decentralization during the current phase of the Filecoin ecosystem's evolution.
-
-That description is now historical: the gateway infrastructure it relies on was decommissioned in 2026 (see [Status: Storacha sunset](#status-storacha-sunset-may-2026)). The hybrid shape of the design still holds — a backend that stores bytes, an IPFS network that serves them — but the backend slot is open. [docs/STORAGE-BACKENDS.md](docs/STORAGE-BACKENDS.md) compares the candidates.
-
-## What This Does
-
-Backup and restore between **OrbitDB databases** and **Storacha/Filecoin** with full hash and identity preservation. Works in both Node.js and browser environments.
-
-The project includes **Svelte components** for browser-based demos and integration (see [SVELTE-COMPONENTS.md](SVELTE-COMPONENTS.md) for detailed documentation).
-
-**Features:**
-
-- backup/restore between OrbitDB and Storacha in browsers and NodeJS via Storacha key and proof credential
-  - full backup per space
-  - timestamped backups (multiple backups per space - restore last backup by default)
-- Storacha Svelte components for integration into Svelte projects
-- UCAN authentication 
-- Backup/restore functionality with hash and identity preservation
-- OrbitDB CAR file storage [OrbitDB CustomStorage](https://github.com/orbitdb/orbitdb/blob/main/docs/STORAGE.md)
-
-## Roadmap
-
-> Being re-based on a backend interface instead of a single vendor — the plan is
-> [issue 54](https://github.com/NiKrause/orbitdb-storage-bridge/issues/54), not here. The WebAuthn/varsig items below survive
-> unchanged; the Storacha-named ones become backend-agnostic.
-
-- [ ] Live parallel persistence: hand an open database a backend-backed OrbitDB `ComposedStorage`, so every block is written to a backend **as it is created** — during sync and after each update — rather than only when a backup runs.
-  - [ ] Today a block becomes durable somewhere else only if bitswap reaches a peer that keeps it, gossipsub reaches a subscriber, or somebody runs a backup. All three depend on another party being reachable at that moment; a write-through storage does not.
-  - [ ] The seam already exists: [`lib/car-storage.js`](lib/car-storage.js) implements OrbitDB's storage shape (`put`/`get`/`del`/`iterator`/`merge`) against a CAR file, and `ComposedStorage` is designed to fan a write out to more than one place. This points the same seam at a network backend instead.
-  - [ ] Aleph Cloud is the first candidate, because it is the only evaluated backend a browser can write to with no key at all — `POST /api/v0/add` answers unauthenticated with an open CORS header. See [docs/STORAGE-BACKENDS.md](docs/STORAGE-BACKENDS.md); persistence there needs the wallet-signed STORE message, not the ingest alone.
-  - [ ] Open question worth settling first: what a write-through storage does when the backend is unreachable. Blocking a local write on a network round trip would make the database only as available as the backend, which is the opposite of the point — so it queues, and the queue is the design.
-
-- [ ] v0.4.4 (Feb 2026): Latest-backup pointer (single CID) to avoid listing via the Storacha SDK and restore from the IPFS network for initial OrbitDB syncs.
-  - [ ] After each backup, write a small pointer record (JSON) that stores the latest metadata CID, CAR CID, and last heads (block CID).
-  - [ ] Store that pointer in a user-controlled place (local storage, QR/share link, WebAuthN largetBlog extension or file download).
-- [ ] v0.5.0 (Feb 2026): OrbitDB CustomStorage (StorachaStorage) ([issue 23](https://github.com/NiKrause/orbitdb-storage-bridge/issues/23)).
-- [ ] v0.6.0 (Mar 2026): WebAuthN + varsig signing/verification (Ed25519 and P-256) for OrbitDB oplog. https://github.com/ChainAgnostic/varsig/blob/main/README.md
-- [ ] v0.6.1 (Mar 2026): WebAuthN + SimpleEncryption example that uses WebAuthN+PRF key material for encrypted backups and restore.
-- [ ] v0.7.0 (Apr 2026): WebAuthN + OrbitDB AccessController (store a UCAN instead of only a DID for admin/write access).
-  - [ ] Alice (authenticated via UCAN or Storacha credentials) can delegate/revoke access for Bob with custom/default capabilities ([issue 16](https://github.com/NiKrause/orbitdb-storage-bridge/issues/16)). See [WebAuthN Upload Wall](https://github.com/NiKrause/ucan-upload-wall/tree/browser-only/web) and the [live demo](https://bafybeibdcnp7pr26okzr6kbygcounsz3klyg3vydxwwovmz2ljyzfmprre.ipfs.w3s.link/).
-- [ ] v0.7.1 (May 2026): Storacha Backup & Restore Svelte widget with WebAuthN-varsig UCAN signing/verification (Ed25519/P-256).
-- [ ] v0.7.2 (May 2026): Storacha Backup & Restore React widget with WebAuthN-varsig UCAN signing/verification (Ed25519/P-256).
-- [ ] v0.7.3 (May 2026): Storacha Backup & Restore React widget with WebAuthN-varsig UCAN delegation (Ed25519/P-256).
-- [ ] v0.7.4 (May 2026): UI enhancement for the Storacha Backup & Restore widget (timestamped backup restore and management).
-- [ ] v0.8.0 (Jun 2026): Upgrade to UCAN 1.0 support.
-- [ ] v0.9.0 (Jul 2026): Social backup between devices with DKG (decentralized key generation).
-- [ ] v0.10.0 (Aug 2027): WebAuthN + Roaming Credentials: Have a browser and a mobile with one Yubikey creating one and the same DID and replicating the same OrbitDB
-
-Read more on Medium: [Bridging OrbitDB with Storacha: Decentralized Database Backups](https://medium.com/@akashjana663/bridging-orbitdb-with-storacha-decentralized-database-backups-44c7bee5c395)
-
-## Installation
-
-Install the package via npm:
+> Published as `orbitdb-storacha-bridge` up to 0.6.0 and as `@le-space/orbitdb-storage-bridge`
+> from 0.7.0, because Storacha is no longer the only backend it bridges to. The rename changed no
+> API — only the dependency and the import specifiers. Names that refer to Storacha itself stay as
+> they are: `OrbitDBStorachaBridge`, `backends/storacha`, and the `libp2p:orbitdb-storacha:*` debug
+> namespace.
+
+## What it does
+
+An OrbitDB database is a set of content-addressed blocks and a log whose heads tie them together.
+This package moves those blocks somewhere else and brings them back with their CIDs and the
+writer's identity intact — so a restored database is the same database, and its original author can
+still write to it.
+
+Where "somewhere else" is, is a choice:
+
+| Backend | Needs | |
+| --- | --- | --- |
+| `aleph` | nothing | the only backend a browser can write to with no key; ingest only |
+| `aleph-pin` | a wallet | the signed STORE message that makes Aleph keep it |
+| `pinata` | a scoped JWT | pin-by-CID as well as CAR upload |
+| `lighthouse` | an API key | pay once, stored in perpetuity |
+| `memory` | nothing | in-process, for tests and demos |
+| `mirror` | — | wraps others: one backup, several services |
+| `encryption` | — | wraps one: the service holds ciphertext |
+
+Reading back needs no account, because a CID is a name anyone can resolve: `peer-fetch` asks the
+peers that hold the blocks over libp2p, `gateway-fetch` asks an HTTP gateway. Two more entry
+points do not involve a storage service at all — `courier-sync` replicates over any byte courier
+(a LoRa mesh, a QR relay, a file), and `dehydrate`/`hydrate` put a database where a second device
+can find it from a passkey alone.
+
+## Install
 
 ```bash
 npm install @le-space/orbitdb-storage-bridge
 ```
 
-## Environment Setup
-
-`STORACHA_KEY` and `STORACHA_PROOF` in `.env` are still what the code reads, and existing credentials still parse — but there is no longer a service to present them to, and no way to mint new ones: the console and the quickstart docs are gone (see [Status: Storacha sunset](#status-storacha-sunset-may-2026)). The test suite's in-memory modes run without credentials; see `test/README.md`.
-
-## Demo
+## Quick start
 
-### NodeJS Demo Scripts (full backup with Manifest, Identity and AccessController and entries blocks)
-
-They back up to Aleph by default, which takes an upload without an account, so
-they run straight after cloning. `STORAGE=pinata` and `STORAGE=lighthouse`
-switch to the paid services and read their keys from the environment — see
-[`examples/storage.js`](examples/storage.js).
+The demos back up to Aleph, which accepts an upload without an account, so they run straight after
+cloning:
 
 ```sh
-node examples/backup-demo.js                          # prints the backup's CID
-BACKUP_CID=<cid> node examples/restore-demo.js        # restores it on a fresh node
+node examples/backup-demo.js                     # prints the backup's CID
+BACKUP_CID=<cid> node examples/restore-demo.js   # restores it on a node that has never seen it
 ```
 
-- `node` [`examples/demo.js`](examples/demo.js) - Complete backup/restore cycle, including what stops the second node from writing
-- `node` [`examples/backup-demo.js`](examples/backup-demo.js) - Backup only; prints the CID the restore needs
-- `node` [`examples/restore-demo.js`](examples/restore-demo.js) - Restore from that CID onto a node that has never seen the database
-- `node` [`examples/demo-different-identity.js`](examples/demo-different-identity.js) - Different identities with access control enforcement
-- `node` [`examples/demo-shared-identities.js`](examples/demo-shared-identities.js) - Shared identity backup/restore scenarios
-
-An Aleph upload is not kept: staying stored takes a wallet-signed STORE message
-this library does not send, so restore what you back up while the demo is still
-running.
-
-The scripts written against Storacha's space and UCAN model are in
-[`examples/storacha/`](examples/storacha/README.md). None of them runs end to
-end since the uploads stopped, and the README there says what replaced each.
-
-### In a browser: a database back on a device that has nothing
+`STORAGE=pinata` and `STORAGE=lighthouse` switch backends and read their keys from the environment
+— see [`examples/storage.js`](examples/storage.js). An Aleph upload is not retained without the
+signed STORE message, so restore what you back up while the demo is still running.
 
-[**funkpost's recovery page**](https://nikrause.github.io/funkpost/recovery/) runs the whole
-[recovery procedure](docs/RECOVERY-ON-A-SECOND-DEVICE.md) in a phone's browser, with a
-security key and nothing else ([source](https://github.com/NiKrause/funkpost/tree/main/examples/recovery)):
+Further scripts: identity and access-control scenarios in [`examples/`](examples/), and the
+Storacha-era scripts in [`examples/storacha/`](examples/storacha/README.md), none of which runs end
+to end any more.
 
-1. **the key gives the identity** — the DID, and a signing key derived from its PRF output,
-   the same on every device;
-2. **a list** is made and written to;
-3. **`dehydrate`** backs it up to Aleph as a CAR, and publishes an IPNS pointer under a name
-   the key derives;
-4. **on another device** — or the same one, wiped — the same key finds the pointer, **`hydrate`**
-   brings the list back, and the list takes new entries, because the writer is the same.
+## Recovery on a second device
 
-On 21 September 2026 it ran that way on two phones: a Galaxy Fold 5 backed up and was reset,
-and a Galaxy A57 with the same key brought the list back and wrote to it. The page says at
-every step which service it contacts; the technical details sit behind one button.
+[The recovery page](https://nikrause.github.io/orbitdb-storage-bridge/recovery/) runs the whole
+procedure in a phone's browser with a security key and nothing else: the passkey gives the
+identity, `dehydrate` backs the database up and publishes an IPNS pointer under a name derived from
+the key, and on another device the same key finds that pointer — so `hydrate` needs no CID, no
+address and no file. The restored database takes new entries, because the writer is the same.
 
-### Svelte Components
+On 21 September 2026 it ran that way on two phones: a Galaxy Fold 5 backed up and was reset, and a
+Galaxy A57 with the same key brought the database back and wrote to it. Source:
+[`examples/svelte/recovery/`](examples/svelte/recovery).
 
-For browser-based integration, this project includes Svelte components for authentication, backup/restore, P2P replication, and WebAuthn biometric authentication. See [**SVELTE-COMPONENTS.md**](SVELTE-COMPONENTS.md) for complete documentation of all available components and demonstrations.
+## Documentation
 
-That browser side is older than this repository's copy of it, and the UCAN delegation work now sits on a branch. [docs/STORACHA-UI-HISTORY.md](docs/STORACHA-UI-HISTORY.md) records what existed, what it could do, and where each piece is today.
-
-## How It Works
-
-1. **Extract Blocks** - Separates OrbitDB database into individual components (log entries, manifest, identities, access controls)
-2. **Upload to Storacha** - Each block is uploaded separately to IPFS/Filecoin via Storacha
-3. **Block Discovery** - Lists all files in Storacha space using Storacha SDK APIs
-4. **CID Bridging** - Converts between Storacha CIDs (`bafkre*`) and OrbitDB CIDs (`zdpu*`)
-5. **Reconstruct Database** - Reassembles blocks and opens database with original identity
-
-## Restore Mechanism
-
-The restore process uses a **ipfs-p2p-first approach with ipfs-http-gateway fallback** for downloading backups for restore. File listing and metadata discovery are currently performed via the Storacha SDK (using Storacha gateway API). We are working on an **IPNS-based mechanism** to find the latest heads blocks and OrbitDB address directly from the IPFS network via IPNS, eliminating the need to list all files via the centralized Storacha gateway API.
-
-## Logging
-
-The library uses **@libp2p/logger** for consistent logging across the libp2p ecosystem. Control logging with the `DEBUG` environment variable:
-
-**Node.js:**
-```bash
-# Enable all logs from this library (the namespace kept its old name)
-DEBUG=libp2p:orbitdb-storacha:* node your-script.js
-
-# Enable specific components
-DEBUG=libp2p:orbitdb-storacha:bridge node your-script.js
-
-# Enable all libp2p logs (includes this library + libp2p internals)
-DEBUG=libp2p:* node your-script.js
-```
-
-**Browser:**
-```javascript
-// In browser console or before loading the application
-localStorage.setItem('debug', 'libp2p:orbitdb-storacha:*')
-// Then refresh the page
-```
-
-The logger supports printf-style formatting:
-- `%s` - string
-- `%d` - number
-- `%o` - object
-- `%p` - peer ID
-- `%b` - base58btc encoded data
-- `%t` - base32 encoded data
-
-## Testing
-
-See `test/README.md` for detailed test documentation, modes (in-memory vs production),
-and how to run each suite.
+| | |
+| --- | --- |
+| [docs/DESIGN.md](docs/DESIGN.md) | the problem this solves, and how a backup and a restore actually work |
+| [docs/STORAGE-BACKENDS.md](docs/STORAGE-BACKENDS.md) | every backend evaluated — prices, limits, and what has been verified against a live account |
+| [docs/RECOVERY-ON-A-SECOND-DEVICE.md](docs/RECOVERY-ON-A-SECOND-DEVICE.md) | the passkey recovery procedure, step by step |
+| [docs/CAR-BACKUP.md](docs/CAR-BACKUP.md) | CAR-based timestamped backups |
+| [SVELTE-COMPONENTS.md](SVELTE-COMPONENTS.md) | the browser components — older than this repository's copy of them; [docs/STORACHA-UI-HISTORY.md](docs/STORACHA-UI-HISTORY.md) records what each piece could do and where it went |
+| [docs/LOGGING.md](docs/LOGGING.md) | debug namespaces, in Node and in a browser |
+| [ROADMAP.md](ROADMAP.md) | what is planned |
+| [docs/STORACHA-SUNSET.md](docs/STORACHA-SUNSET.md) | why Storacha stopped being the default, with dates and evidence |
+| [test/README.md](test/README.md) | the suites, and which of them need credentials |
+
+`examples/browser/storage-probe/` is a static page that asks these services from a real browser
+with no build step: whether they answer a page at all, whether their CORS survives a refusal as
+well as a success, and whether anyone on IPFS holds a given CID at an address a browser can dial.
 
 ## Contributing
 
-1. Fork the repository
-2. Create a feature branch
-3. Add tests for new functionality
-4. Submit a pull request
+Fork, branch, add tests for new behaviour, open a pull request.
 
 ## License
 
-MIT License
+MIT.
+
+LoRa® is a trademark of Semtech Corporation and Meshtastic® a registered trademark of Meshtastic
+LLC; OrbitDB, IPFS, Filecoin, Storacha, Pinata, Lighthouse and Aleph belong to their respective
+owners and appear here only to say what this package interoperates with. This project is
+affiliated with none of them and contains no code from the Meshtastic libraries, which are
+GPL-3.0-only — see [docs/TRADEMARKS.md](docs/TRADEMARKS.md) for why that line matters and which
+side of it this package is on.

package/lib/courier-sync.js

@@ -17,15 +17,84 @@
  *     all three. Returns an unsubscribe function.
  *
  * Wire messages (dag-cbor encoded, one per courier payload):
- *   { v, tag, p, t: "announce", heads: [hash] }
+ *   { v, tag, p, t: "announce", heads: [hash], r? }   r: please reconcile
  *   { v, tag, p, t: "want", cids: [hash], have: [hash] }
  *   { v, tag, p, t: "blocks", heads: [hash], blocks: [{ hash, bytes }] }
+ *   { v, tag, p, t: "op", id: bytes8, o: { op, key, value } }   one change
  *   { v, tag, p, t: "hello" }   is anybody keeping this database out there?
  *   { v, tag, p, t: "here" }    the answer
  * `tag` is a short hash of the database address, so couriers can be shared
  * between databases without cross-talk while the address itself stays off
  * the air (the mesh reads everything).
  *
+ * Two ways to carry a change, and the cheap one is not replication.
+ *
+ * `announce`/`want`/`blocks` move the log itself: signed entries, their
+ * ancestry, the identity that vouches for them. That is real OrbitDB
+ * replication — every peer ends up with the same log and the same hashes — and
+ * it is what a cold join needs, because a peer that has never seen the database
+ * has to be given the manifest and the whole history.
+ *
+ * It is also expensive out of all proportion to a todo. The operation inside
+ * that entry — `{op: "PUT", key, value}`, which OrbitDB hands to every
+ * `update` listener already — is the whole of what anyone meant to send.
+ *
+ * Measured over funkpost's Meshtastic courier, framing, ARQ and duty-cycle
+ * pacing included, for one further change to a list both sides already hold:
+ *
+ *   | plane        | bytes on the air |
+ *   | ------------ | ---------------- |
+ *   | `delta`      | 1774 B           |
+ *   | `operations` |   86 B           |
+ *
+ * Twenty times. On a carrier moving half a kilobyte a minute that is three and
+ * a half minutes against ten seconds, for one ticked box.
+ *
+ * So `op` ships the operation and lets the far side perform it locally, as its
+ * own write, with its own identity. What that buys and what it costs:
+ *
+ *   - The two logs **diverge**: each device holds its own entry for the same
+ *     change, with a different hash. For a keyvalue database the materialised
+ *     view still agrees, and once an IP path returns OrbitDB's own sync merges
+ *     both logs and the divergence becomes history rather than disagreement.
+ *   - Two devices editing the **same key** while both are offline can show
+ *     different values until that merge, because each applies the other's
+ *     operation after its own. A CRDT would not have this; an oplog with
+ *     last-write-wins does. For a shared todo list it is an acceptable trade
+ *     and it must be a deliberate one.
+ *   - **The receiver must be allowed to write.** This is the requirement that
+ *     surprises: on the delta plane the far side only *stores* an entry someone
+ *     else signed, so a database whose access controller names one writer
+ *     replicates fine. Here the far side performs the change as its own write,
+ *     so an access controller that excludes it refuses the operation outright —
+ *     *"Key … is not allowed to write to the log"*. `write: ["*"]` makes it
+ *     work, which is what funkpost's mesh-todo already does.
+ *
+ *     That is also where a pairing between two devices belongs, rather than in
+ *     a new protocol of its own: two devices that have met put each other's
+ *     OrbitDB identity in the write set, and "anyone holding the channel key"
+ *     becomes "these two". The access controller is the authorisation
+ *     mechanism this plane needs, and it already exists.
+ *   - Nothing is signed end to end. The receiver vouches for the change with
+ *     its own identity, so trust rests entirely on the carrier — which on a
+ *     Meshtastic channel means everyone holding the channel key. The sender id
+ *     in `p` is not authentication and must never be read as such.
+ *   - Reconciliation is asked for, not inferred: `announce()` sets `r` on the
+ *     wire and the far side answers with a delta. An announce without it is
+ *     noted and left alone, because divergent logs are this plane's ordinary
+ *     state rather than a gap.
+ *   - A lost `op` is simply lost; there is no ancestry to notice the hole.
+ *     Note that `send`'s own comment — that a dropped message is re-derivable
+ *     because a peer still wanting it asks again — holds for `announce`, `want`
+ *     and `blocks` and **not** for `op`: an operation the outbox sheds under
+ *     `maxOutbox` takes its change with it.
+ *     `announce()` is the repair: it runs the delta plane, which is complete by
+ *     construction. Cheap path for the ordinary change, expensive path when
+ *     something has to be made right.
+ *
+ * `liveUpdates: "operations"` opts in. The default stays `"delta"`, so nothing
+ * changes for a consumer that does not ask.
+ *
  * `p` is a four-byte sender id, and it is what makes *presence* possible: a
  * carrier can tell you a radio is in range, which is not the question. The
  * question is whether another program is keeping the same database, and only
@@ -52,6 +121,13 @@ export const COURIER_SYNC_VERSION = 1;
 
 const TAG_LENGTH = 8;
 
+// Eight bytes naming a change, derived from the entry hash so that every
+// retransmission of the same change carries the same id and applies once.
+const CHANGE_ID_LENGTH = 8;
+// How many change ids to remember. A duplicate arriving after this many others
+// is applied twice: one extra entry in the log, not a wrong value.
+const SEEN_OPS = 512;
+
 // Four bytes of sender id: enough that two peers in one conversation collide
 // with probability ~1 in 4 billion, small enough to ride on every message.
 const PEER_ID_LENGTH = 4;
@@ -185,6 +261,19 @@ function isOplogEntry(value) {
  * Reading the whole ancestry locally to avoid transmitting it is a good trade
  * on any carrier: the reads are a blockstore away, the bytes are airtime.
  */
+/**
+ * Eight bytes naming one change, the same on every retransmission of it.
+ *
+ * Derived from the entry hash rather than from the operation, so that two
+ * genuinely separate writes of the same value — ticking a box off and on and
+ * off again — stay separate changes, while a duplicate delivery of one write
+ * is recognised and applied once.
+ */
+async function changeId(entryHash) {
+  const digest = await sha256.digest(new TextEncoder().encode(entryHash));
+  return digest.digest.slice(0, CHANGE_ID_LENGTH);
+}
+
 async function reachableFrom(db, roots) {
   const held = new Set();
   const queue = [...roots];
@@ -192,6 +281,18 @@ async function reachableFrom(db, roots) {
     const hash = queue.shift();
     if (held.has(hash)) continue;
     held.add(hash);
+    // The index first, and only then the blocks. `IPFSBlockStorage.get` on a
+    // miss waits out a network timeout — measured at 20 s against a Helia node
+    // with no peers, which is every phone in the field — and the peer's heads
+    // are precisely where misses live: their newest entry is the one we have
+    // not got. `has` reads the log's index and answers at once.
+    //
+    // The rule is already written down twenty lines below, in the closure
+    // check, and this walk broke it: 0.14.0 built correct deltas and took
+    // twenty seconds to do it, which stalled the exchange past every timeout
+    // around it (funkpost#170). The suite could not see it, because its nodes
+    // are built offline and a miss there fails instantly.
+    if (!(await db.log.has(hash))) continue; // theirs, not ours: stop here
     const bytes = await db.log.storage.get(hash).catch(() => null);
     if (!bytes) continue; // not ours to follow; their ancestry ends here for us
     let value;
@@ -466,6 +567,11 @@ async function applyDeltaToStores({ blockstore, log, events, delta }) {
  * @param {string} [params.address] Database address, required when `db` is not given
  * @param {Object} params.courier The byte courier (see module docs)
  * @param {Object} [params.dbOptions] Extra options for the lazy `orbitdb.open`
+ * @param {"delta"|"operations"} [params.liveUpdates="delta"] How a local write
+ *   travels. `"delta"` announces and lets the peer ask, which moves signed
+ *   entries and keeps both logs identical. `"operations"` ships the operation
+ *   itself — thirteen times smaller, at the cost of divergent logs and no
+ *   end-to-end signature. See the module docs before choosing it.
  * @param {boolean} [params.announceOnLocalUpdate=true] Announce as soon as a
  *   local write lands. Default keeps the eager behaviour. Set false where the
  *   courier is expensive — a duty-cycled radio, say — and the application would
@@ -500,6 +606,7 @@ export async function createCourierSync({
   dbOptions = {},
   rejoinIntervalMs = 15000,
   announceOnLocalUpdate = true,
+  liveUpdates = "delta",
   peerId = randomPeerId(),
   peerTimeoutMs = PEER_TIMEOUT_MS,
   sendTimeoutMs = SEND_TIMEOUT_MS,
@@ -515,6 +622,9 @@ export async function createCourierSync({
   if (!(peerId instanceof Uint8Array) || peerId.length !== PEER_ID_LENGTH) {
     throw new Error(`peerId must be ${PEER_ID_LENGTH} bytes`);
   }
+  if (liveUpdates !== "delta" && liveUpdates !== "operations") {
+    throw new Error('liveUpdates must be "delta" or "operations"');
+  }
   const databaseAddress = address || (db && db.address);
   if (!databaseAddress) {
     throw new Error("Either an open db or a database address is required");
@@ -530,6 +640,9 @@ export async function createCourierSync({
   const peers = new Map(); // sender id (hex) -> when we last heard it
   let lastHeardAt = null; // any traffic for this database, identified or not
   const listeners = { synced: [], applied: [], message: [], error: [] };
+  // Change ids already applied, oldest first. Insertion-ordered, so the oldest
+  // key is the first one Map iteration yields.
+  const seenOps = new Map();
   let database = db;
   // Opened on first contact but not handed out: the bootstrap is not in it yet.
   // The protocol works on it all the same, so repair stays incremental.
@@ -714,7 +827,13 @@ export async function createCourierSync({
   const ourHeadHashes = async (target = local()) =>
     target ? (await target.log.heads()).map((entry) => entry.hash) : [];
 
-  const announce = async () => {
+  /**
+   * @param {boolean} [reconcile] Ask the peer to close the gap, not merely to
+   *   note where we stand. Only the operation plane distinguishes the two:
+   *   there, divergent logs are the ordinary state rather than a gap, so a
+   *   reconciliation has to be asked for or it would run after every change.
+   */
+  const announce = async (reconcile = false) => {
     if (!local()) {
       // Nothing local yet — not even the manifest. An announce of empty heads
       // cannot get one from a peer whose log is also empty, so first contact
@@ -723,7 +842,9 @@ export async function createCourierSync({
       await send({ t: "want", cids: [], have: [] });
       return;
     }
-    await send({ t: "announce", heads: await ourHeadHashes() });
+    const message = { t: "announce", heads: await ourHeadHashes() };
+    if (reconcile) message.r = true;
+    await send(message);
   };
 
   /**
@@ -762,10 +883,12 @@ export async function createCourierSync({
     // blocks is served — going quiet must not mean going deaf.
     if (!announceOnLocalUpdate) return;
     if (!database || offUpdate) return;
-    const onUpdate = () => {
+    const onUpdate = (entry) => {
       if (applying) return; // courier-applied entries already end in an announce
       queue = queue
-        .then(() => announceSoon())
+        .then(() =>
+          liveUpdates === "operations" ? shipOperation(entry) : announceSoon(),
+        )
         .catch((error) => emit("error", error));
     };
     database.events.on("update", onUpdate);
@@ -784,6 +907,16 @@ export async function createCourierSync({
         );
       return;
     }
+    // On the operation plane both sides write their own entry for the same
+    // change, so the peer's heads being strangers to us is the ordinary state
+    // and not a gap to close. Without this, the acknowledgement that ends a
+    // blocks delivery reads as a reconcile request, and on logs that are
+    // permanently divergent each answer earns another: measured over the memory
+    // courier at nine messages for one reconcile against six with it. Both
+    // terminate — this is chatter, not a loop — and it does not change what an
+    // operation costs, which is one message either way.
+    if (liveUpdates === "operations" && !message.r) return;
+
     const ours = await ourHeadHashes();
     const theirSet = new Set(theirHeads);
     const theyLack = ours.filter((hash) => !theirSet.has(hash));
@@ -875,8 +1008,114 @@ export async function createCourierSync({
       emit("synced", { joined: result.joined, entries: result.entries });
     }
     // Tells the peer where we now stand — their diff turns empty and the
-    // exchange goes quiet; doubles as an end-to-end acknowledgement.
-    await announce();
+    // exchange goes quiet; doubles as an end-to-end acknowledgement. Not a
+    // reconcile request: two peers answering each other's would never stop.
+    await announce(false);
+  };
+
+  /**
+   * Remember a change id. Answers whether it was new.
+   *
+   * Our own changes are remembered as they go out, so a mesh repeating us —
+   * or a peer echoing the operation onward — cannot make us perform our own
+   * write a second time.
+   */
+  const rememberOp = (id) => {
+    const key = hex(id);
+    if (seenOps.has(key)) return false;
+    seenOps.set(key, Date.now());
+    while (seenOps.size > SEEN_OPS) seenOps.delete(seenOps.keys().next().value);
+    return true;
+  };
+
+  /**
+   * Perform someone else's change as our own write.
+   *
+   * Only the operations a keyvalue, documents or events database emits. A type
+   * whose operations are named differently is not carried this way, and saying
+   * so out loud is better than writing something that silently does nothing.
+   */
+  const performOperation = async (target, op) => {
+    const run = () => {
+      if (op.op === "PUT") return target.put(op.key, op.value);
+      if (op.op === "DEL") return target.del(op.key);
+      if (op.op === "ADD") return target.add(op.value);
+      throw new Error(`cannot perform operation ${op.op} over the courier`);
+    };
+    try {
+      return await run();
+    } catch (error) {
+      // The refusal every consumer of this plane meets first, and OrbitDB's own
+      // message does not say why it is happening here rather than on the delta
+      // plane. Name the cause where it is read.
+      if (/not allowed to write/i.test(error?.message || "")) {
+        throw new Error(
+          "the operation plane performs the change as a local write, so this " +
+            "device must be in the database's write set — open it with an " +
+            'access controller that admits it (mesh-todo uses write: ["*"]), ' +
+            'or stay on liveUpdates: "delta"',
+          { cause: error },
+        );
+      }
+      throw error;
+    }
+  };
+
+  /** One local change, on its way as an operation rather than as a log entry. */
+  const shipOperation = async (entry) => {
+    const op = entry && entry.payload;
+    if (!op || typeof op.op !== "string" || !entry.hash) return;
+    const id = await changeId(entry.hash);
+    rememberOp(id); // ours: never perform it on ourselves
+    post({ t: "op", id, o: op });
+  };
+
+  const handleOperation = async (message) => {
+    const id = message.id;
+    const op = message.o;
+    if (!(id instanceof Uint8Array) || id.length !== CHANGE_ID_LENGTH) return;
+    if (!op || typeof op.op !== "string") return;
+
+    const target = local();
+    // An operation carries no manifest, so it cannot bootstrap a database. Ask
+    // for the whole thing instead and let the delta plane answer.
+    if (!target) {
+      post({ t: "want", cids: [], have: [] }, { to: senderOf(message) });
+      return;
+    }
+
+    const report = (applied) =>
+      emit("applied", {
+        complete: true,
+        heads: 1,
+        missing: 0,
+        joined: applied ? 1 : 0,
+        held: applied ? 0 : 1,
+        absent: 0,
+        malformed: 0,
+        refused: 0,
+        via: "operation",
+      });
+
+    // Duplicates are ordinary on a broadcast carrier with retransmission, and
+    // performing one twice would put a second entry in the log for a change
+    // that already happened.
+    if (!rememberOp(id)) return report(false);
+
+    // The write we are about to make fires the database's own "update", which
+    // is what an application listens to — so the far side's list moves without
+    // anything else being wired. The guard is what stops that same event from
+    // sending the change straight back out.
+    applying = true;
+    try {
+      await performOperation(target, op);
+    } finally {
+      applying = false;
+    }
+    // No "synced": that event carries the entries that were joined, and nothing
+    // was joined here — the database performed a write of its own. "applied"
+    // is where a courier delivery reports what it came to.
+    report(true);
   };
 
   const handlePayload = (bytes) => {
@@ -905,6 +1144,7 @@ export async function createCourierSync({
         if (message.t === "announce") return handleAnnounce(message);
         if (message.t === "want") return handleWant(message);
         if (message.t === "blocks") return handleBlocks(message);
+        if (message.t === "op") return handleOperation(message);
       })
       .catch((error) => emit("error", error));
   };
@@ -923,7 +1163,9 @@ export async function createCourierSync({
       started = true;
       unsubscribe = courier.onPayload(handlePayload);
       watchLocalUpdates();
-      await announce();
+      // A start reconciles: whatever happened while this program was not
+      // running is exactly what the delta plane is for. Once per session.
+      await announce(true);
       // A joiner that has not bootstrapped keeps re-asking on its own until
       // the database opens — so a bootstrap the lossy channel dropped heals
       // without the user pressing "join" again. Cleared the moment the
@@ -941,7 +1183,7 @@ export async function createCourierSync({
       }
     },
     /** Re-announce — recovery poke after suspected loss. */
-    announce: () => announce(),
+    announce: () => announce(true),
     /** This instance's sender id, as it appears on the wire. */
     peerId: hex(peerId),
     /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@le-space/orbitdb-storage-bridge",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "Back up, restore and replicate OrbitDB databases through pluggable storage backends, with hash and identity preservation",
   "main": "lib/orbitdb-storacha-bridge.js",
   "svelte": "dist/components/",
@@ -46,6 +46,7 @@
     "clear-space": "node examples/storacha/clear-space.js",
     "car-demo": "node examples/storacha/car-backup-demo.js",
     "ucan-demo": "node examples/storacha/ucan-demo.js",
+    "check:links": "node scripts/check-links.mjs",
     "lint": "eslint --config eslint.config.js '{lib,examples,test}/**/*.js' --ignore-pattern 'examples/svelte/*/build/**' --ignore-pattern 'examples/svelte/*/dist/**' --ignore-pattern 'examples/svelte/**/.svelte-kit/**' --ignore-pattern 'node_modules/**'",
     "format": "prettier --write lib/ examples/ test/ --ignore-path '.prettierignore'",
     "clean": "rm -rf ./cid-bridge-test* ./demo-test*",