@optimystic/db-p2p 0.24.2 → 0.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/{readme.md → README.md} +688 -668
- package/dist/src/cluster/block-transfer-service.d.ts +133 -10
- package/dist/src/cluster/block-transfer-service.d.ts.map +1 -1
- package/dist/src/cluster/block-transfer-service.js +155 -9
- package/dist/src/cluster/block-transfer-service.js.map +1 -1
- package/dist/src/cluster/block-transfer.d.ts +67 -7
- package/dist/src/cluster/block-transfer.d.ts.map +1 -1
- package/dist/src/cluster/block-transfer.js +127 -24
- package/dist/src/cluster/block-transfer.js.map +1 -1
- package/dist/src/cluster/certified-claims.d.ts +170 -0
- package/dist/src/cluster/certified-claims.d.ts.map +1 -0
- package/dist/src/cluster/certified-claims.js +200 -0
- package/dist/src/cluster/certified-claims.js.map +1 -0
- package/dist/src/cluster/client-signature-verifier.d.ts +23 -0
- package/dist/src/cluster/client-signature-verifier.d.ts.map +1 -0
- package/dist/src/cluster/client-signature-verifier.js +33 -0
- package/dist/src/cluster/client-signature-verifier.js.map +1 -0
- package/dist/src/cluster/cluster-policy.d.ts +9 -3
- package/dist/src/cluster/cluster-policy.d.ts.map +1 -1
- package/dist/src/cluster/cluster-policy.js +111 -27
- package/dist/src/cluster/cluster-policy.js.map +1 -1
- package/dist/src/cluster/cluster-repo.d.ts +140 -11
- package/dist/src/cluster/cluster-repo.d.ts.map +1 -1
- package/dist/src/cluster/cluster-repo.js +485 -29
- package/dist/src/cluster/cluster-repo.js.map +1 -1
- package/dist/src/cluster/commit-proof.d.ts +134 -0
- package/dist/src/cluster/commit-proof.d.ts.map +1 -0
- package/dist/src/cluster/commit-proof.js +263 -0
- package/dist/src/cluster/commit-proof.js.map +1 -0
- package/dist/src/cluster/quorum-restore.d.ts +95 -8
- package/dist/src/cluster/quorum-restore.d.ts.map +1 -1
- package/dist/src/cluster/quorum-restore.js +113 -14
- package/dist/src/cluster/quorum-restore.js.map +1 -1
- package/dist/src/cluster/rebalance-monitor.d.ts +84 -0
- package/dist/src/cluster/rebalance-monitor.d.ts.map +1 -1
- package/dist/src/cluster/rebalance-monitor.js +207 -5
- package/dist/src/cluster/rebalance-monitor.js.map +1 -1
- package/dist/src/cluster/reconcile-block.d.ts +27 -7
- package/dist/src/cluster/reconcile-block.d.ts.map +1 -1
- package/dist/src/cluster/reconcile-block.js +179 -38
- package/dist/src/cluster/reconcile-block.js.map +1 -1
- package/dist/src/cluster/service.d.ts +13 -4
- package/dist/src/cluster/service.d.ts.map +1 -1
- package/dist/src/cluster/service.js +17 -18
- package/dist/src/cluster/service.js.map +1 -1
- package/dist/src/cluster/spread-on-churn.d.ts +8 -2
- package/dist/src/cluster/spread-on-churn.d.ts.map +1 -1
- package/dist/src/cluster/spread-on-churn.js +7 -7
- package/dist/src/cluster/spread-on-churn.js.map +1 -1
- package/dist/src/cohort-topic/host.d.ts.map +1 -1
- package/dist/src/cohort-topic/host.js +8 -7
- package/dist/src/cohort-topic/host.js.map +1 -1
- package/dist/src/cohort-topic/stream-util.d.ts +7 -6
- package/dist/src/cohort-topic/stream-util.d.ts.map +1 -1
- package/dist/src/cohort-topic/stream-util.js +14 -41
- package/dist/src/cohort-topic/stream-util.js.map +1 -1
- package/dist/src/dispute/cascade.d.ts +7 -8
- package/dist/src/dispute/cascade.d.ts.map +1 -1
- package/dist/src/dispute/cascade.js +24 -7
- package/dist/src/dispute/cascade.js.map +1 -1
- package/dist/src/dispute/dispute-service.d.ts +5 -0
- package/dist/src/dispute/dispute-service.d.ts.map +1 -1
- package/dist/src/dispute/dispute-service.js +13 -3
- package/dist/src/dispute/dispute-service.js.map +1 -1
- package/dist/src/dispute/invalidation.d.ts +33 -15
- package/dist/src/dispute/invalidation.d.ts.map +1 -1
- package/dist/src/dispute/invalidation.js +127 -34
- package/dist/src/dispute/invalidation.js.map +1 -1
- package/dist/src/dispute/service.d.ts.map +1 -1
- package/dist/src/dispute/service.js +2 -1
- package/dist/src/dispute/service.js.map +1 -1
- package/dist/src/index.d.ts +7 -0
- package/dist/src/index.d.ts.map +1 -1
- package/dist/src/index.js +7 -0
- package/dist/src/index.js.map +1 -1
- package/dist/src/libp2p-key-network.d.ts +14 -10
- package/dist/src/libp2p-key-network.d.ts.map +1 -1
- package/dist/src/libp2p-key-network.js +28 -58
- package/dist/src/libp2p-key-network.js.map +1 -1
- package/dist/src/libp2p-node-base.d.ts +38 -7
- package/dist/src/libp2p-node-base.d.ts.map +1 -1
- package/dist/src/libp2p-node-base.js +148 -63
- package/dist/src/libp2p-node-base.js.map +1 -1
- package/dist/src/network/open-protocol-stream.d.ts +56 -0
- package/dist/src/network/open-protocol-stream.d.ts.map +1 -0
- package/dist/src/network/open-protocol-stream.js +60 -0
- package/dist/src/network/open-protocol-stream.js.map +1 -0
- package/dist/src/network/register-protocol-handler.d.ts +66 -0
- package/dist/src/network/register-protocol-handler.d.ts.map +1 -0
- package/dist/src/network/register-protocol-handler.js +51 -0
- package/dist/src/network/register-protocol-handler.js.map +1 -0
- package/dist/src/peer-address-book.d.ts +60 -2
- package/dist/src/peer-address-book.d.ts.map +1 -1
- package/dist/src/peer-address-book.js +89 -1
- package/dist/src/peer-address-book.js.map +1 -1
- package/dist/src/pend-validation.d.ts +65 -0
- package/dist/src/pend-validation.d.ts.map +1 -0
- package/dist/src/pend-validation.js +69 -0
- package/dist/src/pend-validation.js.map +1 -0
- package/dist/src/reactivity/notify-transport.d.ts.map +1 -1
- package/dist/src/reactivity/notify-transport.js +2 -1
- package/dist/src/reactivity/notify-transport.js.map +1 -1
- package/dist/src/reactivity/push-state-gossip.d.ts.map +1 -1
- package/dist/src/reactivity/push-state-gossip.js +2 -1
- package/dist/src/reactivity/push-state-gossip.js.map +1 -1
- package/dist/src/repo/cluster-coordinator.d.ts +32 -6
- package/dist/src/repo/cluster-coordinator.d.ts.map +1 -1
- package/dist/src/repo/cluster-coordinator.js +44 -10
- package/dist/src/repo/cluster-coordinator.js.map +1 -1
- package/dist/src/repo/coordinator-repo.d.ts +181 -25
- package/dist/src/repo/coordinator-repo.d.ts.map +1 -1
- package/dist/src/repo/coordinator-repo.js +650 -62
- package/dist/src/repo/coordinator-repo.js.map +1 -1
- package/dist/src/repo/served-repo-proxy.d.ts +30 -0
- package/dist/src/repo/served-repo-proxy.d.ts.map +1 -0
- package/dist/src/repo/served-repo-proxy.js +50 -0
- package/dist/src/repo/served-repo-proxy.js.map +1 -0
- package/dist/src/repo/service.d.ts +16 -1
- package/dist/src/repo/service.d.ts.map +1 -1
- package/dist/src/repo/service.js +18 -16
- package/dist/src/repo/service.js.map +1 -1
- package/dist/src/reputation/types.d.ts +3 -1
- package/dist/src/reputation/types.d.ts.map +1 -1
- package/dist/src/reputation/types.js +3 -1
- package/dist/src/reputation/types.js.map +1 -1
- package/dist/src/rn.d.ts +24 -2
- package/dist/src/rn.d.ts.map +1 -1
- package/dist/src/rn.js +33 -4
- package/dist/src/rn.js.map +1 -1
- package/dist/src/storage/block-archive.d.ts +153 -0
- package/dist/src/storage/block-archive.d.ts.map +1 -0
- package/dist/src/storage/block-archive.js +194 -0
- package/dist/src/storage/block-archive.js.map +1 -0
- package/dist/src/storage/block-latch.d.ts +92 -0
- package/dist/src/storage/block-latch.d.ts.map +1 -0
- package/dist/src/storage/block-latch.js +132 -0
- package/dist/src/storage/block-latch.js.map +1 -0
- package/dist/src/storage/block-storage.d.ts +169 -12
- package/dist/src/storage/block-storage.d.ts.map +1 -1
- package/dist/src/storage/block-storage.js +415 -152
- package/dist/src/storage/block-storage.js.map +1 -1
- package/dist/src/storage/cached-raw-storage.d.ts +5 -1
- package/dist/src/storage/cached-raw-storage.d.ts.map +1 -1
- package/dist/src/storage/cached-raw-storage.js +16 -1
- package/dist/src/storage/cached-raw-storage.js.map +1 -1
- package/dist/src/storage/cached-store-driver.d.ts +14 -4
- package/dist/src/storage/cached-store-driver.d.ts.map +1 -1
- package/dist/src/storage/cached-store-driver.js +39 -5
- package/dist/src/storage/cached-store-driver.js.map +1 -1
- package/dist/src/storage/i-block-storage.d.ts +70 -19
- package/dist/src/storage/i-block-storage.d.ts.map +1 -1
- package/dist/src/storage/i-block-storage.js +21 -1
- package/dist/src/storage/i-block-storage.js.map +1 -1
- package/dist/src/storage/i-raw-storage.d.ts +38 -0
- package/dist/src/storage/i-raw-storage.d.ts.map +1 -1
- package/dist/src/storage/kv-raw-storage.d.ts +14 -8
- package/dist/src/storage/kv-raw-storage.d.ts.map +1 -1
- package/dist/src/storage/kv-raw-storage.js +29 -9
- package/dist/src/storage/kv-raw-storage.js.map +1 -1
- package/dist/src/storage/memory-store-driver.d.ts +4 -1
- package/dist/src/storage/memory-store-driver.d.ts.map +1 -1
- package/dist/src/storage/memory-store-driver.js +22 -2
- package/dist/src/storage/memory-store-driver.js.map +1 -1
- package/dist/src/storage/raw-store-codec.js +2 -2
- package/dist/src/storage/raw-store-driver.d.ts +44 -3
- package/dist/src/storage/raw-store-driver.d.ts.map +1 -1
- package/dist/src/storage/shared-cache-pool.d.ts +60 -3
- package/dist/src/storage/shared-cache-pool.d.ts.map +1 -1
- package/dist/src/storage/shared-cache-pool.js +68 -4
- package/dist/src/storage/shared-cache-pool.js.map +1 -1
- package/dist/src/storage/storage-repo.d.ts +204 -26
- package/dist/src/storage/storage-repo.d.ts.map +1 -1
- package/dist/src/storage/storage-repo.js +422 -119
- package/dist/src/storage/storage-repo.js.map +1 -1
- package/dist/src/storage/store-identity.d.ts +21 -0
- package/dist/src/storage/store-identity.d.ts.map +1 -0
- package/dist/src/storage/store-identity.js +33 -0
- package/dist/src/storage/store-identity.js.map +1 -0
- package/dist/src/storage/struct.d.ts +15 -0
- package/dist/src/storage/struct.d.ts.map +1 -1
- package/dist/src/storage/with-read-cache.d.ts +92 -0
- package/dist/src/storage/with-read-cache.d.ts.map +1 -0
- package/dist/src/storage/with-read-cache.js +132 -0
- package/dist/src/storage/with-read-cache.js.map +1 -0
- package/dist/src/sync/service.d.ts +13 -3
- package/dist/src/sync/service.d.ts.map +1 -1
- package/dist/src/sync/service.js +7 -30
- package/dist/src/sync/service.js.map +1 -1
- package/dist/src/testing/mesh-harness.d.ts +75 -2
- package/dist/src/testing/mesh-harness.d.ts.map +1 -1
- package/dist/src/testing/mesh-harness.js +193 -81
- package/dist/src/testing/mesh-harness.js.map +1 -1
- package/dist/src/testing/raw-storage-conformance.d.ts +8 -0
- package/dist/src/testing/raw-storage-conformance.d.ts.map +1 -1
- package/dist/src/testing/raw-storage-conformance.js +97 -2
- package/dist/src/testing/raw-storage-conformance.js.map +1 -1
- package/package.json +5 -6
- package/src/cluster/block-transfer-service.ts +528 -312
- package/src/cluster/block-transfer.ts +547 -424
- package/src/cluster/certified-claims.ts +319 -0
- package/src/cluster/client-signature-verifier.ts +34 -0
- package/src/cluster/cluster-policy.ts +294 -203
- package/src/cluster/cluster-repo.ts +2501 -2034
- package/src/cluster/commit-proof.ts +344 -0
- package/src/cluster/quorum-restore.ts +348 -223
- package/src/cluster/rebalance-monitor.ts +539 -254
- package/src/cluster/reconcile-block.ts +404 -203
- package/src/cluster/service.ts +23 -20
- package/src/cluster/spread-on-churn.ts +352 -346
- package/src/cohort-topic/host.ts +8 -7
- package/src/cohort-topic/stream-util.ts +14 -43
- package/src/dispute/cascade.ts +31 -15
- package/src/dispute/dispute-service.ts +706 -696
- package/src/dispute/invalidation.ts +162 -52
- package/src/dispute/service.ts +126 -125
- package/src/index.ts +7 -0
- package/src/libp2p-key-network.ts +1204 -1235
- package/src/libp2p-node-base.ts +185 -70
- package/src/network/open-protocol-stream.ts +87 -0
- package/src/network/register-protocol-handler.ts +85 -0
- package/src/peer-address-book.ts +96 -2
- package/src/pend-validation.ts +89 -0
- package/src/reactivity/notify-transport.ts +2 -1
- package/src/reactivity/push-state-gossip.ts +2 -1
- package/src/repo/cluster-coordinator.ts +65 -11
- package/src/repo/coordinator-repo.ts +729 -71
- package/src/repo/served-repo-proxy.ts +55 -0
- package/src/repo/service.ts +25 -16
- package/src/reputation/types.ts +3 -1
- package/src/rn.ts +34 -4
- package/src/storage/block-archive.ts +242 -0
- package/src/storage/block-latch.ts +144 -0
- package/src/storage/block-storage.ts +787 -498
- package/src/storage/cached-raw-storage.ts +21 -1
- package/src/storage/cached-store-driver.ts +898 -859
- package/src/storage/i-block-storage.ts +172 -116
- package/src/storage/i-raw-storage.ts +40 -0
- package/src/storage/kv-raw-storage.ts +34 -9
- package/src/storage/memory-store-driver.ts +26 -2
- package/src/storage/raw-store-codec.ts +42 -42
- package/src/storage/raw-store-driver.ts +45 -3
- package/src/storage/shared-cache-pool.ts +516 -452
- package/src/storage/storage-repo.ts +1421 -1014
- package/src/storage/store-identity.ts +41 -0
- package/src/storage/struct.ts +52 -34
- package/src/storage/with-read-cache.ts +182 -0
- package/src/sync/service.ts +178 -198
- package/src/testing/mesh-harness.ts +524 -344
- package/src/testing/raw-storage-conformance.ts +114 -2
package/{readme.md → README.md}
RENAMED
|
@@ -1,668 +1,688 @@
|
|
|
1
|
-
# Optimystic DB-P2P
|
|
2
|
-
|
|
3
|
-
A distributed peer-to-peer database system that provides concrete implementations of the Optimystic database abstractions using pluggable storage backends and libp2p networking. This package transforms the interfaces and abstractions from `@optimystic/db-core` into a fully operational distributed database system.
|
|
4
|
-
|
|
5
|
-
> **Storage is pluggable.** `db-p2p` owns the distributed layer (repo/cluster/coordination) and the storage *interfaces* (`IRawStorage`, `IBlockStorage`, `IKVStore`) plus an in-memory backend. Concrete persistent backends live in separate packages — filesystem persistence is [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs); see also `-storage-ns` (NativeScript/SQLite), `-storage-rn` (React Native/LevelDB), and `-storage-web` (IndexedDB).
|
|
6
|
-
|
|
7
|
-
## Overview
|
|
8
|
-
|
|
9
|
-
The `@optimystic/db-core` package provides abstractions and interfaces for database operations—immutable blocks, versioned transactions, repository interfaces, and collection data structures. However, it doesn't provide concrete implementations for storage or distribution. `db-p2p` provides concrete implementations of the core abstractions using:
|
|
10
|
-
|
|
11
|
-
- **Versioned block storage** (`StorageRepo` / `BlockStorage`) layered over a pluggable `IRawStorage` backend — an in-memory backend ships here; filesystem persistence lives in `@optimystic/db-p2p-storage-fs`
|
|
12
|
-
- **libp2p networking** for decentralized peer-to-peer communication
|
|
13
|
-
- **Distributed consensus** using 2-phase commit protocols for consistency
|
|
14
|
-
- **Fault-tolerant coordination** with automatic recovery and data restoration
|
|
15
|
-
|
|
16
|
-
The result is a fully operational distributed database system that implements the interfaces defined in the core abstractions to provide a complete distributed database system.
|
|
17
|
-
|
|
18
|
-
## Architecture Overview
|
|
19
|
-
|
|
20
|
-
The system provides three distinct communication interfaces, each serving different roles in the distributed architecture:
|
|
21
|
-
|
|
22
|
-
```mermaid
|
|
23
|
-
graph TD
|
|
24
|
-
subgraph "External Client Process"
|
|
25
|
-
A[Application Code]
|
|
26
|
-
B[RepoClient]
|
|
27
|
-
A --> B
|
|
28
|
-
end
|
|
29
|
-
|
|
30
|
-
subgraph "Coordinator Node Process<br/>(Orchestrates Transactions)"
|
|
31
|
-
C[RepoService]
|
|
32
|
-
D[CoordinatorRepo]
|
|
33
|
-
E[ClusterClient]
|
|
34
|
-
F[StorageRepo]
|
|
35
|
-
subgraph "Local Storage"
|
|
36
|
-
F1[BlockStorage]
|
|
37
|
-
F2[IRawStorage backend]
|
|
38
|
-
F3[Backend store]
|
|
39
|
-
end
|
|
40
|
-
C --> D
|
|
41
|
-
D --> E
|
|
42
|
-
D --> F
|
|
43
|
-
F --> F1
|
|
44
|
-
F1 --> F2
|
|
45
|
-
F2 --> F3
|
|
46
|
-
end
|
|
47
|
-
|
|
48
|
-
subgraph "Cluster Peer 1 Process<br/>(Participates in Consensus)"
|
|
49
|
-
G1[ClusterService]
|
|
50
|
-
H1[ClusterMember]
|
|
51
|
-
I1[StorageRepo]
|
|
52
|
-
subgraph "Local Storage 1"
|
|
53
|
-
I1A[BlockStorage]
|
|
54
|
-
I1B[IRawStorage backend]
|
|
55
|
-
I1C[Backend store]
|
|
56
|
-
end
|
|
57
|
-
G1 --> H1
|
|
58
|
-
H1 --> I1
|
|
59
|
-
I1 --> I1A
|
|
60
|
-
I1A --> I1B
|
|
61
|
-
I1B --> I1C
|
|
62
|
-
end
|
|
63
|
-
|
|
64
|
-
subgraph "Cluster Peer 2 Process<br/>(Participates in Consensus)"
|
|
65
|
-
G2[ClusterService]
|
|
66
|
-
H2[ClusterMember]
|
|
67
|
-
I2[StorageRepo]
|
|
68
|
-
subgraph "Local Storage 2"
|
|
69
|
-
I2A[BlockStorage]
|
|
70
|
-
I2B[IRawStorage backend]
|
|
71
|
-
I2C[Backend store]
|
|
72
|
-
end
|
|
73
|
-
G2 --> H2
|
|
74
|
-
H2 --> I2
|
|
75
|
-
I2 --> I2A
|
|
76
|
-
I2A --> I2B
|
|
77
|
-
I2B --> I2C
|
|
78
|
-
end
|
|
79
|
-
|
|
80
|
-
subgraph "Cluster Peer N Process<br/>(Participates in Consensus)"
|
|
81
|
-
G3[ClusterService]
|
|
82
|
-
H3[ClusterMember]
|
|
83
|
-
I3[StorageRepo]
|
|
84
|
-
subgraph "Local Storage N"
|
|
85
|
-
I3A[BlockStorage]
|
|
86
|
-
I3B[IRawStorage backend]
|
|
87
|
-
I3C[Backend store]
|
|
88
|
-
end
|
|
89
|
-
G3 --> H3
|
|
90
|
-
H3 --> I3
|
|
91
|
-
I3 --> I3A
|
|
92
|
-
I3A --> I3B
|
|
93
|
-
I3B --> I3C
|
|
94
|
-
end
|
|
95
|
-
|
|
96
|
-
%% Network communication
|
|
97
|
-
B -.->|"/optimystic/<network>/repo/1.0.0"| C
|
|
98
|
-
E -.->|"/optimystic/<network>/cluster/1.0.0"| G1
|
|
99
|
-
E -.->|"/optimystic/<network>/cluster/1.0.0"| G2
|
|
100
|
-
E -.->|"/optimystic/<network>/cluster/1.0.0"| G3
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
### Repo Layer: External Client Interface
|
|
104
|
-
|
|
105
|
-
The repo layer provides the interface for external clients to communicate with the distributed database:
|
|
106
|
-
|
|
107
|
-
- **RepoClient**: Allows external clients to connect to any coordinator node
|
|
108
|
-
- **RepoService**: Handles incoming requests from external clients
|
|
109
|
-
- **CoordinatorRepo**: Orchestrates distributed transactions by coordinating with cluster peers
|
|
110
|
-
- **Network transparency**: Same `IRepo` interface whether local or distributed
|
|
111
|
-
|
|
112
|
-
### Cluster Layer: Peer-to-Peer Coordination Interface
|
|
113
|
-
|
|
114
|
-
The cluster layer provides the interface for coordinators to communicate with other peers in the cluster:
|
|
115
|
-
|
|
116
|
-
- **ClusterClient**: Allows coordinators to send consensus requests to cluster peers
|
|
117
|
-
- **ClusterService**: Handles incoming consensus requests from coordinators
|
|
118
|
-
- **ClusterMember**: Implements 2-phase commit protocol for distributed consensus
|
|
119
|
-
- **Byzantine fault tolerance**: Ensures consistency even with peer failures
|
|
120
|
-
|
|
121
|
-
### Storage Layer: Persistent Data Implementation
|
|
122
|
-
|
|
123
|
-
The storage layer implements the core database abstractions over a pluggable raw-storage backend:
|
|
124
|
-
|
|
125
|
-
- **StorageRepo**: Implements the `IRepo` interface on top of `BlockStorage` — backend-agnostic
|
|
126
|
-
- **BlockStorage**: Provides versioned block storage with conflict resolution, reading/writing through an injected `IRawStorage`
|
|
127
|
-
- **`IRawStorage`**: The raw persistence interface. An in-memory backend ships in this package; the filesystem backend (`FileRawStorage`) lives in `@optimystic/db-p2p-storage-fs`
|
|
128
|
-
- **Data restoration**: Pluggable restoration for missing data from network peers
|
|
129
|
-
|
|
130
|
-
## Detailed Component Architecture
|
|
131
|
-
|
|
132
|
-
### Repo Layer Components
|
|
133
|
-
|
|
134
|
-
#### `RepoClient`
|
|
135
|
-
Allows external clients to connect to coordinator nodes:
|
|
136
|
-
|
|
137
|
-
```typescript
|
|
138
|
-
class RepoClient extends ProtocolClient implements IRepo {
|
|
139
|
-
static create(peerId: PeerId, peerNetwork: IPeerNetwork): RepoClient
|
|
140
|
-
// Implements same IRepo interface as local storage
|
|
141
|
-
}
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
**Key Features:**
|
|
145
|
-
- Network-transparent database operations for external clients
|
|
146
|
-
- Automatic protocol handling and message serialization
|
|
147
|
-
- Error handling for network failures and timeouts
|
|
148
|
-
- Connection pooling and optimization
|
|
149
|
-
|
|
150
|
-
#### `RepoService`
|
|
151
|
-
Handles incoming repository protocol messages from external clients:
|
|
152
|
-
|
|
153
|
-
```typescript
|
|
154
|
-
class RepoService implements Startable {
|
|
155
|
-
// Protocol: /optimystic/<network>/repo/1.0.0 (see docs/repo.md § Protocol id conventions)
|
|
156
|
-
// Transport: Length-prefixed JSON over libp2p streams
|
|
157
|
-
async start(): Promise<void>
|
|
158
|
-
async stop(): Promise<void>
|
|
159
|
-
}
|
|
160
|
-
```
|
|
161
|
-
|
|
162
|
-
**Key Features:**
|
|
163
|
-
- Registers and handles the repository protocol
|
|
164
|
-
- Routes operations from external clients to coordinator implementation
|
|
165
|
-
- Manages concurrent connections and stream processing
|
|
166
|
-
- Provides comprehensive error handling and logging
|
|
167
|
-
|
|
168
|
-
#### `CoordinatorRepo`
|
|
169
|
-
Orchestrates distributed transactions by coordinating with cluster peers:
|
|
170
|
-
|
|
171
|
-
```typescript
|
|
172
|
-
class CoordinatorRepo implements IRepo {
|
|
173
|
-
constructor(
|
|
174
|
-
keyNetwork: IKeyNetwork,
|
|
175
|
-
createClusterClient: (peerId: PeerId) => ClusterClient,
|
|
176
|
-
storageRepo: IRepo
|
|
177
|
-
)
|
|
178
|
-
}
|
|
179
|
-
```
|
|
180
|
-
|
|
181
|
-
**Key Features:**
|
|
182
|
-
- Manages distributed transactions using cluster consensus
|
|
183
|
-
- Integrates with key network for peer discovery
|
|
184
|
-
- Uses cluster layer to coordinate with other peers
|
|
185
|
-
- Maintains local storage directly while coordinating remote operations
|
|
186
|
-
|
|
187
|
-
### Cluster Layer Components
|
|
188
|
-
|
|
189
|
-
#### `ClusterClient`
|
|
190
|
-
Allows coordinators to send consensus requests to cluster peers:
|
|
191
|
-
|
|
192
|
-
```typescript
|
|
193
|
-
class ClusterClient extends ProtocolClient implements ICluster {
|
|
194
|
-
async update(record: ClusterRecord): Promise<ClusterRecord>
|
|
195
|
-
}
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
**Key Features:**
|
|
199
|
-
- Network communication for consensus operations
|
|
200
|
-
- Handles cluster record updates and responses
|
|
201
|
-
- Manages timeouts and retries for consensus requests
|
|
202
|
-
|
|
203
|
-
#### `ClusterService`
|
|
204
|
-
Handles incoming consensus requests from coordinators:
|
|
205
|
-
|
|
206
|
-
```typescript
|
|
207
|
-
class ClusterService implements Startable {
|
|
208
|
-
// Protocol: /optimystic/<network>/cluster/1.0.0 (built from protocolPrefix)
|
|
209
|
-
// Handles ClusterRecord updates for consensus
|
|
210
|
-
}
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
**Key Features:**
|
|
214
|
-
- Registers and handles the cluster consensus protocol
|
|
215
|
-
- Routes consensus requests to local cluster member
|
|
216
|
-
- Manages concurrent consensus operations
|
|
217
|
-
|
|
218
|
-
#### `ClusterMember`
|
|
219
|
-
Implements 2-phase commit protocol for distributed consensus:
|
|
220
|
-
|
|
221
|
-
```typescript
|
|
222
|
-
class ClusterMember implements ICluster {
|
|
223
|
-
async update(record: ClusterRecord): Promise<ClusterRecord>
|
|
224
|
-
// Handles: Promise collection → Majority consensus → Commit execution
|
|
225
|
-
}
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
**Key Features:**
|
|
229
|
-
- Complete 2-phase commit protocol implementation
|
|
230
|
-
- Conflict detection and resolution
|
|
231
|
-
- Cryptographic signature verification
|
|
232
|
-
- Automatic timeout and cleanup management
|
|
233
|
-
- Uses storage layer to execute operations when consensus is reached
|
|
234
|
-
|
|
235
|
-
### Storage Layer Components
|
|
236
|
-
|
|
237
|
-
#### `StorageRepo`
|
|
238
|
-
Implements the `IRepo` interface over `BlockStorage` — independent of any particular storage backend:
|
|
239
|
-
|
|
240
|
-
```typescript
|
|
241
|
-
class StorageRepo implements IRepo {
|
|
242
|
-
async get(blockGets: BlockGets): Promise<GetBlockResults>
|
|
243
|
-
async pend(request: PendRequest): Promise<PendResult>
|
|
244
|
-
async commit(request: CommitRequest): Promise<CommitResult>
|
|
245
|
-
async cancel(actionRef: ActionBlocks): Promise<void>
|
|
246
|
-
}
|
|
247
|
-
```
|
|
248
|
-
|
|
249
|
-
**Key Features:**
|
|
250
|
-
- Concrete implementation of core database abstractions
|
|
251
|
-
- Orchestrates transactions across multiple blocks
|
|
252
|
-
- Handles revision conflicts and missing transaction detection
|
|
253
|
-
- Provides atomic commit operations
|
|
254
|
-
|
|
255
|
-
#### `BlockStorage`
|
|
256
|
-
Provides versioned block storage with conflict resolution, reading and writing through an injected `IRawStorage` backend:
|
|
257
|
-
|
|
258
|
-
```typescript
|
|
259
|
-
class BlockStorage implements IBlockStorage {
|
|
260
|
-
constructor(blockId: BlockId, storage: IRawStorage, /* … */)
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
async
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
- **
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
const
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
//
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
```
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
}
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
-
|
|
506
|
-
|
|
507
|
-
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
```
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
```
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
-
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
-
|
|
660
|
-
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
1
|
+
# Optimystic DB-P2P
|
|
2
|
+
|
|
3
|
+
A distributed peer-to-peer database system that provides concrete implementations of the Optimystic database abstractions using pluggable storage backends and libp2p networking. This package transforms the interfaces and abstractions from `@optimystic/db-core` into a fully operational distributed database system.
|
|
4
|
+
|
|
5
|
+
> **Storage is pluggable.** `db-p2p` owns the distributed layer (repo/cluster/coordination) and the storage *interfaces* (`IRawStorage`, `IBlockStorage`, `IKVStore`) plus an in-memory backend. Concrete persistent backends live in separate packages — filesystem persistence is [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs); see also `-storage-ns` (NativeScript/SQLite), `-storage-rn` (React Native/LevelDB), and `-storage-web` (IndexedDB).
|
|
6
|
+
|
|
7
|
+
## Overview
|
|
8
|
+
|
|
9
|
+
The `@optimystic/db-core` package provides abstractions and interfaces for database operations—immutable blocks, versioned transactions, repository interfaces, and collection data structures. However, it doesn't provide concrete implementations for storage or distribution. `db-p2p` provides concrete implementations of the core abstractions using:
|
|
10
|
+
|
|
11
|
+
- **Versioned block storage** (`StorageRepo` / `BlockStorage`) layered over a pluggable `IRawStorage` backend — an in-memory backend ships here; filesystem persistence lives in `@optimystic/db-p2p-storage-fs`
|
|
12
|
+
- **libp2p networking** for decentralized peer-to-peer communication
|
|
13
|
+
- **Distributed consensus** using 2-phase commit protocols for consistency
|
|
14
|
+
- **Fault-tolerant coordination** with automatic recovery and data restoration
|
|
15
|
+
|
|
16
|
+
The result is a fully operational distributed database system that implements the interfaces defined in the core abstractions to provide a complete distributed database system.
|
|
17
|
+
|
|
18
|
+
## Architecture Overview
|
|
19
|
+
|
|
20
|
+
The system provides three distinct communication interfaces, each serving different roles in the distributed architecture:
|
|
21
|
+
|
|
22
|
+
```mermaid
|
|
23
|
+
graph TD
|
|
24
|
+
subgraph "External Client Process"
|
|
25
|
+
A[Application Code]
|
|
26
|
+
B[RepoClient]
|
|
27
|
+
A --> B
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
subgraph "Coordinator Node Process<br/>(Orchestrates Transactions)"
|
|
31
|
+
C[RepoService]
|
|
32
|
+
D[CoordinatorRepo]
|
|
33
|
+
E[ClusterClient]
|
|
34
|
+
F[StorageRepo]
|
|
35
|
+
subgraph "Local Storage"
|
|
36
|
+
F1[BlockStorage]
|
|
37
|
+
F2[IRawStorage backend]
|
|
38
|
+
F3[Backend store]
|
|
39
|
+
end
|
|
40
|
+
C --> D
|
|
41
|
+
D --> E
|
|
42
|
+
D --> F
|
|
43
|
+
F --> F1
|
|
44
|
+
F1 --> F2
|
|
45
|
+
F2 --> F3
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
subgraph "Cluster Peer 1 Process<br/>(Participates in Consensus)"
|
|
49
|
+
G1[ClusterService]
|
|
50
|
+
H1[ClusterMember]
|
|
51
|
+
I1[StorageRepo]
|
|
52
|
+
subgraph "Local Storage 1"
|
|
53
|
+
I1A[BlockStorage]
|
|
54
|
+
I1B[IRawStorage backend]
|
|
55
|
+
I1C[Backend store]
|
|
56
|
+
end
|
|
57
|
+
G1 --> H1
|
|
58
|
+
H1 --> I1
|
|
59
|
+
I1 --> I1A
|
|
60
|
+
I1A --> I1B
|
|
61
|
+
I1B --> I1C
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
subgraph "Cluster Peer 2 Process<br/>(Participates in Consensus)"
|
|
65
|
+
G2[ClusterService]
|
|
66
|
+
H2[ClusterMember]
|
|
67
|
+
I2[StorageRepo]
|
|
68
|
+
subgraph "Local Storage 2"
|
|
69
|
+
I2A[BlockStorage]
|
|
70
|
+
I2B[IRawStorage backend]
|
|
71
|
+
I2C[Backend store]
|
|
72
|
+
end
|
|
73
|
+
G2 --> H2
|
|
74
|
+
H2 --> I2
|
|
75
|
+
I2 --> I2A
|
|
76
|
+
I2A --> I2B
|
|
77
|
+
I2B --> I2C
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
subgraph "Cluster Peer N Process<br/>(Participates in Consensus)"
|
|
81
|
+
G3[ClusterService]
|
|
82
|
+
H3[ClusterMember]
|
|
83
|
+
I3[StorageRepo]
|
|
84
|
+
subgraph "Local Storage N"
|
|
85
|
+
I3A[BlockStorage]
|
|
86
|
+
I3B[IRawStorage backend]
|
|
87
|
+
I3C[Backend store]
|
|
88
|
+
end
|
|
89
|
+
G3 --> H3
|
|
90
|
+
H3 --> I3
|
|
91
|
+
I3 --> I3A
|
|
92
|
+
I3A --> I3B
|
|
93
|
+
I3B --> I3C
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
%% Network communication
|
|
97
|
+
B -.->|"/optimystic/<network>/repo/1.0.0"| C
|
|
98
|
+
E -.->|"/optimystic/<network>/cluster/1.0.0"| G1
|
|
99
|
+
E -.->|"/optimystic/<network>/cluster/1.0.0"| G2
|
|
100
|
+
E -.->|"/optimystic/<network>/cluster/1.0.0"| G3
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
### Repo Layer: External Client Interface
|
|
104
|
+
|
|
105
|
+
The repo layer provides the interface for external clients to communicate with the distributed database:
|
|
106
|
+
|
|
107
|
+
- **RepoClient**: Allows external clients to connect to any coordinator node
|
|
108
|
+
- **RepoService**: Handles incoming requests from external clients
|
|
109
|
+
- **CoordinatorRepo**: Orchestrates distributed transactions by coordinating with cluster peers
|
|
110
|
+
- **Network transparency**: Same `IRepo` interface whether local or distributed
|
|
111
|
+
|
|
112
|
+
### Cluster Layer: Peer-to-Peer Coordination Interface
|
|
113
|
+
|
|
114
|
+
The cluster layer provides the interface for coordinators to communicate with other peers in the cluster:
|
|
115
|
+
|
|
116
|
+
- **ClusterClient**: Allows coordinators to send consensus requests to cluster peers
|
|
117
|
+
- **ClusterService**: Handles incoming consensus requests from coordinators
|
|
118
|
+
- **ClusterMember**: Implements 2-phase commit protocol for distributed consensus
|
|
119
|
+
- **Byzantine fault tolerance**: Ensures consistency even with peer failures
|
|
120
|
+
|
|
121
|
+
### Storage Layer: Persistent Data Implementation
|
|
122
|
+
|
|
123
|
+
The storage layer implements the core database abstractions over a pluggable raw-storage backend:
|
|
124
|
+
|
|
125
|
+
- **StorageRepo**: Implements the `IRepo` interface on top of `BlockStorage` — backend-agnostic
|
|
126
|
+
- **BlockStorage**: Provides versioned block storage with conflict resolution, reading/writing through an injected `IRawStorage`
|
|
127
|
+
- **`IRawStorage`**: The raw persistence interface. An in-memory backend ships in this package; the filesystem backend (`FileRawStorage`) lives in `@optimystic/db-p2p-storage-fs`
|
|
128
|
+
- **Data restoration**: Pluggable restoration for missing data from network peers
|
|
129
|
+
|
|
130
|
+
## Detailed Component Architecture
|
|
131
|
+
|
|
132
|
+
### Repo Layer Components
|
|
133
|
+
|
|
134
|
+
#### `RepoClient`
|
|
135
|
+
Allows external clients to connect to coordinator nodes:
|
|
136
|
+
|
|
137
|
+
```typescript
|
|
138
|
+
class RepoClient extends ProtocolClient implements IRepo {
|
|
139
|
+
static create(peerId: PeerId, peerNetwork: IPeerNetwork): RepoClient
|
|
140
|
+
// Implements same IRepo interface as local storage
|
|
141
|
+
}
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
**Key Features:**
|
|
145
|
+
- Network-transparent database operations for external clients
|
|
146
|
+
- Automatic protocol handling and message serialization
|
|
147
|
+
- Error handling for network failures and timeouts
|
|
148
|
+
- Connection pooling and optimization
|
|
149
|
+
|
|
150
|
+
#### `RepoService`
|
|
151
|
+
Handles incoming repository protocol messages from external clients:
|
|
152
|
+
|
|
153
|
+
```typescript
|
|
154
|
+
class RepoService implements Startable {
|
|
155
|
+
// Protocol: /optimystic/<network>/repo/1.0.0 (see docs/repo.md § Protocol id conventions)
|
|
156
|
+
// Transport: Length-prefixed JSON over libp2p streams
|
|
157
|
+
async start(): Promise<void>
|
|
158
|
+
async stop(): Promise<void>
|
|
159
|
+
}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
**Key Features:**
|
|
163
|
+
- Registers and handles the repository protocol
|
|
164
|
+
- Routes operations from external clients to coordinator implementation
|
|
165
|
+
- Manages concurrent connections and stream processing
|
|
166
|
+
- Provides comprehensive error handling and logging
|
|
167
|
+
|
|
168
|
+
#### `CoordinatorRepo`
|
|
169
|
+
Orchestrates distributed transactions by coordinating with cluster peers:
|
|
170
|
+
|
|
171
|
+
```typescript
|
|
172
|
+
class CoordinatorRepo implements IRepo {
|
|
173
|
+
constructor(
|
|
174
|
+
keyNetwork: IKeyNetwork,
|
|
175
|
+
createClusterClient: (peerId: PeerId) => ClusterClient,
|
|
176
|
+
storageRepo: IRepo
|
|
177
|
+
)
|
|
178
|
+
}
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
**Key Features:**
|
|
182
|
+
- Manages distributed transactions using cluster consensus
|
|
183
|
+
- Integrates with key network for peer discovery
|
|
184
|
+
- Uses cluster layer to coordinate with other peers
|
|
185
|
+
- Maintains local storage directly while coordinating remote operations
|
|
186
|
+
|
|
187
|
+
### Cluster Layer Components
|
|
188
|
+
|
|
189
|
+
#### `ClusterClient`
|
|
190
|
+
Allows coordinators to send consensus requests to cluster peers:
|
|
191
|
+
|
|
192
|
+
```typescript
|
|
193
|
+
class ClusterClient extends ProtocolClient implements ICluster {
|
|
194
|
+
async update(record: ClusterRecord): Promise<ClusterRecord>
|
|
195
|
+
}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
**Key Features:**
|
|
199
|
+
- Network communication for consensus operations
|
|
200
|
+
- Handles cluster record updates and responses
|
|
201
|
+
- Manages timeouts and retries for consensus requests
|
|
202
|
+
|
|
203
|
+
#### `ClusterService`
|
|
204
|
+
Handles incoming consensus requests from coordinators:
|
|
205
|
+
|
|
206
|
+
```typescript
|
|
207
|
+
class ClusterService implements Startable {
|
|
208
|
+
// Protocol: /optimystic/<network>/cluster/1.0.0 (built from protocolPrefix)
|
|
209
|
+
// Handles ClusterRecord updates for consensus
|
|
210
|
+
}
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
**Key Features:**
|
|
214
|
+
- Registers and handles the cluster consensus protocol
|
|
215
|
+
- Routes consensus requests to local cluster member
|
|
216
|
+
- Manages concurrent consensus operations
|
|
217
|
+
|
|
218
|
+
#### `ClusterMember`
|
|
219
|
+
Implements 2-phase commit protocol for distributed consensus:
|
|
220
|
+
|
|
221
|
+
```typescript
|
|
222
|
+
class ClusterMember implements ICluster {
|
|
223
|
+
async update(record: ClusterRecord): Promise<ClusterRecord>
|
|
224
|
+
// Handles: Promise collection → Majority consensus → Commit execution
|
|
225
|
+
}
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
**Key Features:**
|
|
229
|
+
- Complete 2-phase commit protocol implementation
|
|
230
|
+
- Conflict detection and resolution
|
|
231
|
+
- Cryptographic signature verification
|
|
232
|
+
- Automatic timeout and cleanup management
|
|
233
|
+
- Uses storage layer to execute operations when consensus is reached
|
|
234
|
+
|
|
235
|
+
### Storage Layer Components
|
|
236
|
+
|
|
237
|
+
#### `StorageRepo`
|
|
238
|
+
Implements the `IRepo` interface over `BlockStorage` — independent of any particular storage backend:
|
|
239
|
+
|
|
240
|
+
```typescript
|
|
241
|
+
class StorageRepo implements IRepo {
|
|
242
|
+
async get(blockGets: BlockGets): Promise<GetBlockResults>
|
|
243
|
+
async pend(request: PendRequest): Promise<PendResult>
|
|
244
|
+
async commit(request: CommitRequest): Promise<CommitResult>
|
|
245
|
+
async cancel(actionRef: ActionBlocks): Promise<void>
|
|
246
|
+
}
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
**Key Features:**
|
|
250
|
+
- Concrete implementation of core database abstractions
|
|
251
|
+
- Orchestrates transactions across multiple blocks
|
|
252
|
+
- Handles revision conflicts and missing transaction detection
|
|
253
|
+
- Provides atomic commit operations under one write latch per block, acquired in sorted block-id order so two commits over overlapping batches cannot deadlock
|
|
254
|
+
|
|
255
|
+
#### `BlockStorage`
|
|
256
|
+
Provides versioned block storage with conflict resolution, reading and writing through an injected `IRawStorage` backend:
|
|
257
|
+
|
|
258
|
+
```typescript
|
|
259
|
+
class BlockStorage implements IBlockStorage {
|
|
260
|
+
constructor(blockId: BlockId, storage: IRawStorage, restoreCallback?: RestoreCallback, /* … */)
|
|
261
|
+
|
|
262
|
+
// Reads take no latch.
|
|
263
|
+
async getBlock(rev?: number): Promise<{ block: IBlock, actionRev: ActionRev } | undefined>
|
|
264
|
+
|
|
265
|
+
// Every write takes a BlockWriteLatch token, minted only by acquiring the
|
|
266
|
+
// block's write latch — so an unlatched write does not compile.
|
|
267
|
+
async savePendingTransaction(actionId: ActionId, transform: Transform, latch: BlockWriteLatch): Promise<void>
|
|
268
|
+
async promotePendingTransaction(actionId: ActionId, latch: BlockWriteLatch): Promise<void>
|
|
269
|
+
async restoreRevision(rev: number, latch: BlockWriteLatch): Promise<void>
|
|
270
|
+
}
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
**Key Features:**
|
|
274
|
+
- Maintains complete revision history for each block
|
|
275
|
+
- Reconstructs blocks by applying transforms to base versions
|
|
276
|
+
- `getBlock` is **local-only** — it never fetches from a peer. A revision outside the
|
|
277
|
+
block's recorded coverage raises `RevisionNotCoveredError`; `StorageRepo.get` is the
|
|
278
|
+
one caller that heals that, by calling `restoreRevision` (which does consult the
|
|
279
|
+
restoration callback) under the block's write latch and re-reading. The commit path
|
|
280
|
+
never restores in line: it holds every batch block's latch and must do no network I/O
|
|
281
|
+
there, so an unreachable base is refused with `MissingBaseRevisionError` and healed
|
|
282
|
+
out-of-band by cohort reconcile plus a retry.
|
|
283
|
+
- Concurrency is one write latch per block (`Block.write:<blockId>`). The caller acquires
|
|
284
|
+
it and passes the resulting token into each write; `BlockStorage` rejects a token minted
|
|
285
|
+
for a different block, or one whose latch has already been released.
|
|
286
|
+
|
|
287
|
+
#### `IRawStorage` backends
|
|
288
|
+
`BlockStorage` persists through the `IRawStorage` interface, so the backing store is pluggable:
|
|
289
|
+
|
|
290
|
+
- **In-memory** (`MemoryRawStorage`) — ships in this package; used for tests and ephemeral nodes.
|
|
291
|
+
- **Filesystem** (`FileRawStorage`) — lives in the separate [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs) package; JSON files with atomic writes, organized per block:
|
|
292
|
+
|
|
293
|
+
```typescript
|
|
294
|
+
// Filesystem backend layout (@optimystic/db-p2p-storage-fs)
|
|
295
|
+
{basePath}/
|
|
296
|
+
├── {blockId}/
|
|
297
|
+
│ ├── meta.json # Block metadata and revision ranges
|
|
298
|
+
│ ├── revs/{rev}.json # Revision → ActionId mappings
|
|
299
|
+
│ ├── pend/{actionId}.json # Pending actions
|
|
300
|
+
│ ├── actions/{actionId}.json # Committed actions
|
|
301
|
+
│ ├── proofs/{rev}.json # Retained BlockCommitProof per revision
|
|
302
|
+
│ └── blocks/{actionId}.json # Materialized blocks
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
- **Other backends** — SQLite (`-storage-ns`), LevelDB (`-storage-rn`), IndexedDB (`-storage-web`), each in its own package.
|
|
306
|
+
|
|
307
|
+
## Integration Patterns
|
|
308
|
+
|
|
309
|
+
### Relationship to `@optimystic/db-core`
|
|
310
|
+
|
|
311
|
+
The `db-p2p` package provides concrete implementations of the core database abstractions:
|
|
312
|
+
|
|
313
|
+
```typescript
|
|
314
|
+
// Core provides interfaces and abstractions
|
|
315
|
+
import { IRepo, IBlock, Transform, IBlockStorage } from '@optimystic/db-core';
|
|
316
|
+
|
|
317
|
+
// P2P provides concrete implementations
|
|
318
|
+
class StorageRepo implements IRepo { /* backend-agnostic versioned repo */ }
|
|
319
|
+
class BlockStorage implements IBlockStorage { /* versioned block storage over IRawStorage */ }
|
|
320
|
+
class RepoClient implements IRepo { /* network-transparent repo */ }
|
|
321
|
+
class CoordinatorRepo implements IRepo { /* distributed consensus repo */ }
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
**Implementation Points:**
|
|
325
|
+
- **Storage Layer**: Implements the core storage interfaces over a pluggable `IRawStorage` backend (in-memory here; filesystem in `@optimystic/db-p2p-storage-fs`)
|
|
326
|
+
- **Network Layer**: Implements peer-to-peer communication using libp2p
|
|
327
|
+
- **Consensus Layer**: Adds distributed consensus while maintaining core interfaces
|
|
328
|
+
- **API Compatibility**: Same interfaces as core abstractions for seamless integration
|
|
329
|
+
|
|
330
|
+
### Libp2p Integration
|
|
331
|
+
|
|
332
|
+
The package integrates deeply with libp2p for networking:
|
|
333
|
+
|
|
334
|
+
```typescript
|
|
335
|
+
// Node creation with integrated services
|
|
336
|
+
const node = await createLibp2pNode({
|
|
337
|
+
services: {
|
|
338
|
+
repo: repoService({ protocolPrefix: '/optimystic/<network>' }),
|
|
339
|
+
cluster: clusterService({ protocolPrefix: '/optimystic/<network>' })
|
|
340
|
+
}
|
|
341
|
+
});
|
|
342
|
+
|
|
343
|
+
// Key network integration — use the node's OWN key network, do not construct a second one.
|
|
344
|
+
// `createLibp2pNode` returns an `OptimysticNode`, which declares this handle (and
|
|
345
|
+
// `coordinatedRepo`, `storageRepo`, `blockChangeNotifier`, `reputation`, `peerPrivateKey`).
|
|
346
|
+
// The attached instance carries the node's resolved cluster size and its network-namespaced
|
|
347
|
+
// protocol prefix; a separately constructed one selects a different-width cohort and has the
|
|
348
|
+
// "does this peer serve my network?" filter switched off.
|
|
349
|
+
const keyNetwork = node.keyNetwork;
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
**Integration Features:**
|
|
353
|
+
- **Protocol Management**: Registers custom protocols for repo and cluster operations
|
|
354
|
+
- **Stream Handling**: Uses libp2p streams for reliable message transport
|
|
355
|
+
- **Peer Discovery**: Integrates with libp2p DHT for peer discovery
|
|
356
|
+
- **Security**: Leverages libp2p's cryptographic peer identity
|
|
357
|
+
|
|
358
|
+
**Inbound stream authorization (optional).** The `repo`, `cluster`, `sync` and `block-transfer`
|
|
359
|
+
protocols are open to any peer that can connect. An embedder whose database is private supplies one
|
|
360
|
+
node-level, fail-closed predicate, consulted once per inbound stream before any decoding or execution:
|
|
361
|
+
|
|
362
|
+
```typescript
|
|
363
|
+
const node = await createLibp2pNode({
|
|
364
|
+
networkName: 'my-net',
|
|
365
|
+
bootstrapNodes: [],
|
|
366
|
+
// remotePeerId is the dialing peer's PeerId.toString(); only a literal `true` allows.
|
|
367
|
+
// false / throw / rejection / timeout all deny and abort the stream.
|
|
368
|
+
authorizeInboundStream: (remotePeerId, protocol) => memberSet.has(remotePeerId)
|
|
369
|
+
});
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
Omitting it is the default and leaves behavior unchanged. See
|
|
373
|
+
[internals](../../docs/internals.md) § Inbound Stream Authorization for the full contract.
|
|
374
|
+
|
|
375
|
+
**Announcing a different address than you bind (optional).** A node behind a NAT, port forward,
|
|
376
|
+
reverse proxy or DNS front binds one address but is reachable at another. `announceAddrs` advertises
|
|
377
|
+
the reachable address *instead of* whatever the transports report; `appendAnnounceAddrs` advertises
|
|
378
|
+
it *in addition to* them:
|
|
379
|
+
|
|
380
|
+
```typescript
|
|
381
|
+
const node = await createLibp2pNode({
|
|
382
|
+
networkName: 'my-net',
|
|
383
|
+
bootstrapNodes: ['...'],
|
|
384
|
+
listenAddrs: ['/ip4/0.0.0.0/tcp/4001'], // what the process binds
|
|
385
|
+
announceAddrs: ['/dns4/mynode.example.com/tcp/4001'] // what peers are told
|
|
386
|
+
});
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
`announceAddrs` replaces the advertised set **entirely** — observed and relayed addresses are
|
|
390
|
+
dropped from it, and `appendAnnounceAddrs` is ignored while it is non-empty. Use
|
|
391
|
+
`appendAnnounceAddrs` alone when the automatically discovered addresses are still worth advertising.
|
|
392
|
+
Both are straight passthroughs to libp2p's `addresses.announce` / `addresses.appendAnnounce`; an
|
|
393
|
+
empty array means "unset".
|
|
394
|
+
|
|
395
|
+
## Usage Examples
|
|
396
|
+
|
|
397
|
+
### Setting Up a Coordinator Node
|
|
398
|
+
|
|
399
|
+
`createLibp2pNode` assembles the whole stack and **starts** the node: it builds the storage layer,
|
|
400
|
+
the consensus coordinator and the key network from one resolved config, and hands them back on the
|
|
401
|
+
returned `OptimysticNode`. Read them off the node — do not construct your own, or the two copies
|
|
402
|
+
disagree about the same key's cohort.
|
|
403
|
+
|
|
404
|
+
```typescript
|
|
405
|
+
import { createLibp2pNode } from '@optimystic/db-p2p';
|
|
406
|
+
import { FileRawStorage } from '@optimystic/db-p2p-storage-fs';
|
|
407
|
+
|
|
408
|
+
// Create (and start) a libp2p node. The repo and cluster protocol services are registered by the
|
|
409
|
+
// factory itself — there is no `services` option to pass them in.
|
|
410
|
+
const node = await createLibp2pNode({
|
|
411
|
+
networkName: 'my-net',
|
|
412
|
+
bootstrapNodes: ['...'],
|
|
413
|
+
// In-memory when omitted; a raw-storage backend makes it durable
|
|
414
|
+
storage: new FileRawStorage('/var/lib/optimystic'),
|
|
415
|
+
});
|
|
416
|
+
|
|
417
|
+
// The handles the node already owns (see `OptimysticNodeAttachments`)
|
|
418
|
+
const storageRepo = node.storageRepo; // Local block storage
|
|
419
|
+
const coordinatedRepo = node.coordinatedRepo; // Cluster-consensus writes and reads
|
|
420
|
+
const keyNetwork = node.keyNetwork; // Peer/coordinator discovery
|
|
421
|
+
|
|
422
|
+
// ... use them, then:
|
|
423
|
+
await node.stop();
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
`createLibp2pNode` is all-or-nothing: the node is started early and then wired, so if any wiring step
|
|
427
|
+
fails the factory stops the node before rejecting and surfaces the original error. A rejection
|
|
428
|
+
therefore never leaves a running node behind — which matters because the caller gets no handle to
|
|
429
|
+
stop one. Retrying on the same port is safe.
|
|
430
|
+
|
|
431
|
+
Constructing a `StorageRepo`, `CoordinatorRepo` or `Libp2pKeyPeerNetwork` yourself is only for a
|
|
432
|
+
host assembling a stack *without* `createLibp2pNode`. A standalone `CoordinatorRepo` or
|
|
433
|
+
`Libp2pKeyPeerNetwork` must be told its cluster size; pass `DEFAULT_CLUSTER_SIZE` (exported from
|
|
434
|
+
this package) to match what an unconfigured node resolves to.
|
|
435
|
+
|
|
436
|
+
### External Client Operations
|
|
437
|
+
|
|
438
|
+
```typescript
|
|
439
|
+
// External clients connect to coordinator nodes
|
|
440
|
+
const client = RepoClient.create(coordinatorPeerId, peerNetwork);
|
|
441
|
+
|
|
442
|
+
// Operations work the same as core database interfaces
|
|
443
|
+
const blocks = await client.get({
|
|
444
|
+
blockIds: ['block1', 'block2'],
|
|
445
|
+
context: { rev: 10 }
|
|
446
|
+
});
|
|
447
|
+
|
|
448
|
+
// But automatically coordinate across the cluster
|
|
449
|
+
const pendResult = await client.pend({
|
|
450
|
+
actionId: 'tx1',
|
|
451
|
+
transforms: { block1: [/* operations */] },
|
|
452
|
+
rev: 11
|
|
453
|
+
});
|
|
454
|
+
|
|
455
|
+
const commitResult = await client.commit({
|
|
456
|
+
actionId: 'tx1',
|
|
457
|
+
blockIds: ['block1'],
|
|
458
|
+
rev: 11
|
|
459
|
+
});
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
### How Coordination Works
|
|
463
|
+
|
|
464
|
+
```mermaid
|
|
465
|
+
sequenceDiagram
|
|
466
|
+
participant Client as External Client
|
|
467
|
+
participant RC as RepoClient
|
|
468
|
+
participant CR as CoordinatorRepo
|
|
469
|
+
participant CC as ClusterClient
|
|
470
|
+
participant CS as ClusterService
|
|
471
|
+
participant CM as ClusterMember
|
|
472
|
+
participant SR as StorageRepo
|
|
473
|
+
|
|
474
|
+
Client->>RC: pend(transaction)
|
|
475
|
+
RC->>CR: pend(transaction)
|
|
476
|
+
CR->>CC: consensus request
|
|
477
|
+
CC->>CS: cluster protocol
|
|
478
|
+
CS->>CM: update(ClusterRecord)
|
|
479
|
+
CM->>CM: 2-phase commit
|
|
480
|
+
CM->>SR: execute operation
|
|
481
|
+
SR-->>CM: result
|
|
482
|
+
CM-->>CS: consensus result
|
|
483
|
+
CS-->>CC: response
|
|
484
|
+
CC-->>CR: consensus complete
|
|
485
|
+
CR-->>RC: transaction result
|
|
486
|
+
RC-->>Client: pend result
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
## Documentation
|
|
490
|
+
|
|
491
|
+
For detailed information about specific components:
|
|
492
|
+
|
|
493
|
+
- **[Storage System](./docs/storage.md)**: Versioned block storage and persistence
|
|
494
|
+
- **[Repo Interface](./docs/repo.md)**: Distributed database operations
|
|
495
|
+
- **[Cluster Consensus](./docs/cluster.md)**: 2-phase commit and distributed consensus
|
|
496
|
+
|
|
497
|
+
## Block Restoration & Arachnode Integration
|
|
498
|
+
|
|
499
|
+
The db-p2p package includes **dynamic Arachnode ring discovery** for automatic block restoration across storage tiers.
|
|
500
|
+
|
|
501
|
+
### How Block Restoration Works
|
|
502
|
+
|
|
503
|
+
When a node is missing a block or revision, the restoration system:
|
|
504
|
+
|
|
505
|
+
1. **Determines storage rings** - Nodes self-select ring depth based on capacity
|
|
506
|
+
2. **Discovers peers via FRET** - Ring membership propagates via existing neighbor exchange
|
|
507
|
+
3. **Queries rings intelligently** - Tries transaction ring first, then inner storage rings
|
|
508
|
+
4. **Filters by partition** - Only queries peers responsible for the block's keyspace
|
|
509
|
+
|
|
510
|
+
### Ring Selection
|
|
511
|
+
|
|
512
|
+
Nodes calculate their appropriate ring depth using:
|
|
513
|
+
|
|
514
|
+
```
|
|
515
|
+
ringDepth = ceil(-log2(available_capacity / estimated_demand))
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
**Examples:**
|
|
519
|
+
- 100% coverage → Ring 0 (full keyspace)
|
|
520
|
+
- 50% coverage → Ring 1 (2 partitions)
|
|
521
|
+
- 1% coverage → Ring 7 (128 partitions)
|
|
522
|
+
|
|
523
|
+
### Architecture
|
|
524
|
+
|
|
525
|
+
```
|
|
526
|
+
FRET (Pure DHT)
|
|
527
|
+
- Generic metadata transport
|
|
528
|
+
- Peer discovery & routing
|
|
529
|
+
|
|
530
|
+
ArachnodeFretAdapter (Plugin Layer)
|
|
531
|
+
- Arachnode-specific semantics
|
|
532
|
+
- Ring discovery methods
|
|
533
|
+
|
|
534
|
+
Restoration Components
|
|
535
|
+
- RingSelector: Capacity-based ring selection
|
|
536
|
+
- RestorationCoordinator: Multi-ring queries
|
|
537
|
+
- StorageMonitor: Capacity tracking
|
|
538
|
+
- SyncService/Client: Block request protocol
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
### Configuration
|
|
542
|
+
|
|
543
|
+
```typescript
|
|
544
|
+
const node = await createLibp2pNode({
|
|
545
|
+
port: 9000,
|
|
546
|
+
networkName: 'mynet',
|
|
547
|
+
bootstrapNodes: ['...'],
|
|
548
|
+
clusterSize: 10, // Cluster size for peer discovery
|
|
549
|
+
arachnode: {
|
|
550
|
+
enableRingZulu: true // Default: enabled
|
|
551
|
+
}
|
|
552
|
+
});
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
#### Solo / bootstrap / mobile nodes (no remote peers)
|
|
556
|
+
|
|
557
|
+
A node can operate fully on its own — no bootstrap peers, no listen addresses, no
|
|
558
|
+
connectivity to other peers. This is the common shape for a mobile app on first
|
|
559
|
+
launch or for a local-only sanity test.
|
|
560
|
+
|
|
561
|
+
```typescript
|
|
562
|
+
const soloNode = await createLibp2pNode({
|
|
563
|
+
networkName: 'mynet',
|
|
564
|
+
bootstrapNodes: [], // no bootstrap
|
|
565
|
+
listenAddrs: [], // no incoming connections
|
|
566
|
+
transports: [webSockets()], // transports still required
|
|
567
|
+
clusterSize: 1,
|
|
568
|
+
});
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
In solo mode:
|
|
572
|
+
|
|
573
|
+
- The node is its own coordinator. `CoordinatorRepo` short-circuits to local
|
|
574
|
+
storage when the cluster size is ≤ 1, so `pend`, `commit` and `cancel` do not
|
|
575
|
+
require cluster consensus and do not attempt to dial self. `pend` and `commit`
|
|
576
|
+
decide once (from the coordinating block); `cancel` decides per block id,
|
|
577
|
+
because a multi-block cancel can span cohorts of different sizes.
|
|
578
|
+
- Block restoration is skipped when the only discovered peer is self.
|
|
579
|
+
`RestorationCoordinator` will return `undefined` immediately rather than
|
|
580
|
+
attempt to dial self (which would hang on a node with no listen addrs).
|
|
581
|
+
- If an operation fails for some other reason, `findCoordinator` surfaces a
|
|
582
|
+
distinguishable `FindCoordinatorError` with `code ===
|
|
583
|
+
FIND_COORDINATOR_ERROR_CODES.SELF_COORDINATION_EXHAUSTED` rather than the
|
|
584
|
+
generic "all candidates excluded". The aggregate error thrown by
|
|
585
|
+
`NetworkTransactor` preserves the ORIGINAL first-attempt cause so callers can
|
|
586
|
+
diagnose the real problem.
|
|
587
|
+
|
|
588
|
+
Once remote peers become reachable, the node transitions out of solo mode
|
|
589
|
+
automatically as FRET discovers them. Persisted network state (HWM, FRET
|
|
590
|
+
table) carries across restarts when an `options.persistence` is supplied.
|
|
591
|
+
|
|
592
|
+
#### Custom transports (including React Native)
|
|
593
|
+
|
|
594
|
+
By default, `createLibp2pNode()` uses TCP + circuit-relay transport.
|
|
595
|
+
|
|
596
|
+
To use non-default transports, pass `transports` (and typically `listenAddrs`):
|
|
597
|
+
|
|
598
|
+
```typescript
|
|
599
|
+
import { webSockets } from '@libp2p/websockets';
|
|
600
|
+
import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
|
|
601
|
+
import { createLibp2pNode } from '@optimystic/db-p2p';
|
|
602
|
+
|
|
603
|
+
const node = await createLibp2pNode({
|
|
604
|
+
networkName: 'mynet',
|
|
605
|
+
bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
|
|
606
|
+
transports: [webSockets(), circuitRelayTransport()],
|
|
607
|
+
listenAddrs: [], // client-only; no incoming connections
|
|
608
|
+
});
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
#### React Native
|
|
612
|
+
|
|
613
|
+
For React Native, import from `@optimystic/db-p2p/rn` instead of the root entrypoint.
|
|
614
|
+
The `/rn` entrypoint does **not** import `@libp2p/tcp`, so Metro/Hermes won't try to
|
|
615
|
+
bundle Node-only native modules. It requires `options.transports` explicitly:
|
|
616
|
+
|
|
617
|
+
```typescript
|
|
618
|
+
import { webSockets } from '@libp2p/websockets';
|
|
619
|
+
import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
|
|
620
|
+
import { createLibp2pNode } from '@optimystic/db-p2p/rn';
|
|
621
|
+
|
|
622
|
+
const node = await createLibp2pNode({
|
|
623
|
+
networkName: 'mynet',
|
|
624
|
+
bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
|
|
625
|
+
transports: [webSockets(), circuitRelayTransport()],
|
|
626
|
+
});
|
|
627
|
+
```
|
|
628
|
+
|
|
629
|
+
Apart from that transport wiring, `/rn` exports exactly the same API as the root entrypoint,
|
|
630
|
+
so anything importable from `@optimystic/db-p2p` is importable from `@optimystic/db-p2p/rn`.
|
|
631
|
+
`test/entry-parity.spec.ts` fails if the two ever diverge.
|
|
632
|
+
|
|
633
|
+
RN environments (Hermes) need polyfills for globals that libp2p and its
|
|
634
|
+
dependencies expect. Install these early (e.g. in your app's entry file
|
|
635
|
+
before any other imports).
|
|
636
|
+
|
|
637
|
+
**Global polyfills** (patch `globalThis` before any library code):
|
|
638
|
+
|
|
639
|
+
| API | Required by | Notes |
|
|
640
|
+
|-----|-------------|-------|
|
|
641
|
+
| `crypto.getRandomValues()` | @noble/hashes, @libp2p/crypto | RN 0.76+ New Architecture provides natively; fallback via e.g. `react-native-get-random-values` |
|
|
642
|
+
| `crypto.subtle.digest()` | multiformats/hashes/sha2-browser | Async SHA-256/SHA-512 — implement via @noble/hashes |
|
|
643
|
+
| `structuredClone()` | @optimystic/db-core | JSON round-trip is sufficient |
|
|
644
|
+
| `Promise.withResolvers()` | @libp2p/utils, @chainsafe/libp2p-yamux, it-queue | ES2024 — simple shim |
|
|
645
|
+
| `AbortSignal.prototype.throwIfAborted()` | libp2p, @libp2p/circuit-relay-v2, it-pushable | DOM spec addition — simple shim |
|
|
646
|
+
| Timer `.ref()` / `.unref()` | @optimystic/db-p2p, undici | Wrap Hermes numeric timer IDs in objects with no-op `.ref()`/`.unref()` methods; patch `clearTimeout`/`clearInterval` to unwrap |
|
|
647
|
+
| `Event`, `CustomEvent`, `EventTarget` | libp2p, @libp2p/interface | Custom shim or npm `event-target-polyfill` |
|
|
648
|
+
| `Intl.PluralRules` | moat-maker | English-only ordinal/cardinal shim is sufficient |
|
|
649
|
+
|
|
650
|
+
**Node.js built-in module shims** (via Metro `extraNodeModules` or bundler aliases):
|
|
651
|
+
|
|
652
|
+
| Module | Required by | Recommended shim |
|
|
653
|
+
|--------|-------------|------------------|
|
|
654
|
+
| `os` / `node:os` | @libp2p/utils | Custom shim: `networkInterfaces()` → `{}`, `platform()` → `Platform.OS` |
|
|
655
|
+
| `crypto` / `node:crypto` | multiformats/hashes/sha2 | Custom shim: `createHash()` via @noble/hashes |
|
|
656
|
+
| `stream` / `node:stream` | libp2p | `readable-stream` (npm) |
|
|
657
|
+
| `buffer` / `node:buffer` | libp2p, multiformats | `buffer` (npm) |
|
|
658
|
+
|
|
659
|
+
**Built-in (no polyfill needed):**
|
|
660
|
+
- `TextEncoder` — built-in to Hermes
|
|
661
|
+
- `TextDecoder` — built-in to Expo SDK 52+ (UTF-8 only)
|
|
662
|
+
- `BigInt` — built-in to Hermes since RN 0.70
|
|
663
|
+
|
|
664
|
+
See the [Sereus reference-app-rn](https://github.com/gotchoices/sereus/tree/master/packages/reference-app-rn/polyfills) for working polyfill implementations.
|
|
665
|
+
|
|
666
|
+
### Ring Transitions
|
|
667
|
+
|
|
668
|
+
Nodes automatically transition between rings based on capacity thresholds:
|
|
669
|
+
- **Move OUT** (to more granular ring) when > 85% capacity used
|
|
670
|
+
- **Move IN** (to broader ring) when < 40% capacity used
|
|
671
|
+
|
|
672
|
+
## Related Packages
|
|
673
|
+
|
|
674
|
+
- **[@optimystic/db-core](../db-core)**: Core database interfaces and local operations
|
|
675
|
+
- **[@optimystic/db-p2p-storage-fs](../db-p2p-storage-fs)**: Filesystem `IRawStorage` backend (Node)
|
|
676
|
+
- **[@optimystic/db-p2p-storage-ns](../db-p2p-storage-ns)**: NativeScript SQLite storage backend
|
|
677
|
+
- **[@optimystic/db-p2p-storage-rn](../db-p2p-storage-rn)**: React Native LevelDB storage backend
|
|
678
|
+
- **[@optimystic/db-p2p-storage-web](../db-p2p-storage-web)**: Browser IndexedDB storage backend
|
|
679
|
+
- **[@optimystic/db-quereus](../db-quereus)**: Query engine and data access patterns
|
|
680
|
+
- **[p2p-fret](../fret)**: DHT implementation for peer discovery
|
|
681
|
+
|
|
682
|
+
### Internal Architecture
|
|
683
|
+
|
|
684
|
+
For developers and AI agents working on internals, see [Internals Guide](../../docs/internals.md) for:
|
|
685
|
+
- Data flow diagrams (read/write/commit paths)
|
|
686
|
+
- Mutation contracts (which functions mutate vs clone)
|
|
687
|
+
- Key invariants and common pitfalls
|
|
688
|
+
- Type glossary and debugging tips
|