@optimystic/db-p2p 0.24.1 → 0.25.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (257) hide show
  1. package/{readme.md → README.md} +688 -668
  2. package/dist/src/cluster/block-transfer-service.d.ts +133 -10
  3. package/dist/src/cluster/block-transfer-service.d.ts.map +1 -1
  4. package/dist/src/cluster/block-transfer-service.js +155 -9
  5. package/dist/src/cluster/block-transfer-service.js.map +1 -1
  6. package/dist/src/cluster/block-transfer.d.ts +67 -7
  7. package/dist/src/cluster/block-transfer.d.ts.map +1 -1
  8. package/dist/src/cluster/block-transfer.js +127 -24
  9. package/dist/src/cluster/block-transfer.js.map +1 -1
  10. package/dist/src/cluster/certified-claims.d.ts +170 -0
  11. package/dist/src/cluster/certified-claims.d.ts.map +1 -0
  12. package/dist/src/cluster/certified-claims.js +200 -0
  13. package/dist/src/cluster/certified-claims.js.map +1 -0
  14. package/dist/src/cluster/client-signature-verifier.d.ts +23 -0
  15. package/dist/src/cluster/client-signature-verifier.d.ts.map +1 -0
  16. package/dist/src/cluster/client-signature-verifier.js +33 -0
  17. package/dist/src/cluster/client-signature-verifier.js.map +1 -0
  18. package/dist/src/cluster/cluster-policy.d.ts +9 -3
  19. package/dist/src/cluster/cluster-policy.d.ts.map +1 -1
  20. package/dist/src/cluster/cluster-policy.js +111 -27
  21. package/dist/src/cluster/cluster-policy.js.map +1 -1
  22. package/dist/src/cluster/cluster-repo.d.ts +140 -11
  23. package/dist/src/cluster/cluster-repo.d.ts.map +1 -1
  24. package/dist/src/cluster/cluster-repo.js +485 -29
  25. package/dist/src/cluster/cluster-repo.js.map +1 -1
  26. package/dist/src/cluster/commit-proof.d.ts +134 -0
  27. package/dist/src/cluster/commit-proof.d.ts.map +1 -0
  28. package/dist/src/cluster/commit-proof.js +263 -0
  29. package/dist/src/cluster/commit-proof.js.map +1 -0
  30. package/dist/src/cluster/quorum-restore.d.ts +95 -8
  31. package/dist/src/cluster/quorum-restore.d.ts.map +1 -1
  32. package/dist/src/cluster/quorum-restore.js +113 -14
  33. package/dist/src/cluster/quorum-restore.js.map +1 -1
  34. package/dist/src/cluster/rebalance-monitor.d.ts +84 -0
  35. package/dist/src/cluster/rebalance-monitor.d.ts.map +1 -1
  36. package/dist/src/cluster/rebalance-monitor.js +207 -5
  37. package/dist/src/cluster/rebalance-monitor.js.map +1 -1
  38. package/dist/src/cluster/reconcile-block.d.ts +27 -7
  39. package/dist/src/cluster/reconcile-block.d.ts.map +1 -1
  40. package/dist/src/cluster/reconcile-block.js +179 -38
  41. package/dist/src/cluster/reconcile-block.js.map +1 -1
  42. package/dist/src/cluster/service.d.ts +21 -4
  43. package/dist/src/cluster/service.d.ts.map +1 -1
  44. package/dist/src/cluster/service.js +28 -17
  45. package/dist/src/cluster/service.js.map +1 -1
  46. package/dist/src/cluster/spread-on-churn.d.ts +8 -2
  47. package/dist/src/cluster/spread-on-churn.d.ts.map +1 -1
  48. package/dist/src/cluster/spread-on-churn.js +7 -7
  49. package/dist/src/cluster/spread-on-churn.js.map +1 -1
  50. package/dist/src/cohort-topic/host.d.ts.map +1 -1
  51. package/dist/src/cohort-topic/host.js +8 -7
  52. package/dist/src/cohort-topic/host.js.map +1 -1
  53. package/dist/src/cohort-topic/stream-util.d.ts +7 -6
  54. package/dist/src/cohort-topic/stream-util.d.ts.map +1 -1
  55. package/dist/src/cohort-topic/stream-util.js +14 -41
  56. package/dist/src/cohort-topic/stream-util.js.map +1 -1
  57. package/dist/src/dispute/cascade.d.ts +7 -8
  58. package/dist/src/dispute/cascade.d.ts.map +1 -1
  59. package/dist/src/dispute/cascade.js +24 -7
  60. package/dist/src/dispute/cascade.js.map +1 -1
  61. package/dist/src/dispute/dispute-service.d.ts +5 -0
  62. package/dist/src/dispute/dispute-service.d.ts.map +1 -1
  63. package/dist/src/dispute/dispute-service.js +13 -3
  64. package/dist/src/dispute/dispute-service.js.map +1 -1
  65. package/dist/src/dispute/invalidation.d.ts +33 -15
  66. package/dist/src/dispute/invalidation.d.ts.map +1 -1
  67. package/dist/src/dispute/invalidation.js +127 -34
  68. package/dist/src/dispute/invalidation.js.map +1 -1
  69. package/dist/src/dispute/service.d.ts.map +1 -1
  70. package/dist/src/dispute/service.js +2 -1
  71. package/dist/src/dispute/service.js.map +1 -1
  72. package/dist/src/index.d.ts +7 -0
  73. package/dist/src/index.d.ts.map +1 -1
  74. package/dist/src/index.js +7 -0
  75. package/dist/src/index.js.map +1 -1
  76. package/dist/src/libp2p-key-network.d.ts +78 -6
  77. package/dist/src/libp2p-key-network.d.ts.map +1 -1
  78. package/dist/src/libp2p-key-network.js +136 -57
  79. package/dist/src/libp2p-key-network.js.map +1 -1
  80. package/dist/src/libp2p-node-base.d.ts +38 -7
  81. package/dist/src/libp2p-node-base.d.ts.map +1 -1
  82. package/dist/src/libp2p-node-base.js +150 -62
  83. package/dist/src/libp2p-node-base.js.map +1 -1
  84. package/dist/src/logger.d.ts +2 -2
  85. package/dist/src/logger.js +2 -2
  86. package/dist/src/network/open-protocol-stream.d.ts +56 -0
  87. package/dist/src/network/open-protocol-stream.d.ts.map +1 -0
  88. package/dist/src/network/open-protocol-stream.js +60 -0
  89. package/dist/src/network/open-protocol-stream.js.map +1 -0
  90. package/dist/src/network/register-protocol-handler.d.ts +66 -0
  91. package/dist/src/network/register-protocol-handler.d.ts.map +1 -0
  92. package/dist/src/network/register-protocol-handler.js +51 -0
  93. package/dist/src/network/register-protocol-handler.js.map +1 -0
  94. package/dist/src/peer-address-book.d.ts +128 -1
  95. package/dist/src/peer-address-book.d.ts.map +1 -1
  96. package/dist/src/peer-address-book.js +198 -15
  97. package/dist/src/peer-address-book.js.map +1 -1
  98. package/dist/src/pend-validation.d.ts +65 -0
  99. package/dist/src/pend-validation.d.ts.map +1 -0
  100. package/dist/src/pend-validation.js +69 -0
  101. package/dist/src/pend-validation.js.map +1 -0
  102. package/dist/src/reactivity/notify-transport.d.ts.map +1 -1
  103. package/dist/src/reactivity/notify-transport.js +2 -1
  104. package/dist/src/reactivity/notify-transport.js.map +1 -1
  105. package/dist/src/reactivity/push-state-gossip.d.ts.map +1 -1
  106. package/dist/src/reactivity/push-state-gossip.js +2 -1
  107. package/dist/src/reactivity/push-state-gossip.js.map +1 -1
  108. package/dist/src/repo/cluster-coordinator.d.ts +32 -6
  109. package/dist/src/repo/cluster-coordinator.d.ts.map +1 -1
  110. package/dist/src/repo/cluster-coordinator.js +44 -10
  111. package/dist/src/repo/cluster-coordinator.js.map +1 -1
  112. package/dist/src/repo/coordinator-repo.d.ts +181 -25
  113. package/dist/src/repo/coordinator-repo.d.ts.map +1 -1
  114. package/dist/src/repo/coordinator-repo.js +650 -62
  115. package/dist/src/repo/coordinator-repo.js.map +1 -1
  116. package/dist/src/repo/served-repo-proxy.d.ts +30 -0
  117. package/dist/src/repo/served-repo-proxy.d.ts.map +1 -0
  118. package/dist/src/repo/served-repo-proxy.js +50 -0
  119. package/dist/src/repo/served-repo-proxy.js.map +1 -0
  120. package/dist/src/repo/service.d.ts +22 -1
  121. package/dist/src/repo/service.d.ts.map +1 -1
  122. package/dist/src/repo/service.js +25 -13
  123. package/dist/src/repo/service.js.map +1 -1
  124. package/dist/src/reputation/types.d.ts +3 -1
  125. package/dist/src/reputation/types.d.ts.map +1 -1
  126. package/dist/src/reputation/types.js +3 -1
  127. package/dist/src/reputation/types.js.map +1 -1
  128. package/dist/src/rn.d.ts +24 -2
  129. package/dist/src/rn.d.ts.map +1 -1
  130. package/dist/src/rn.js +33 -4
  131. package/dist/src/rn.js.map +1 -1
  132. package/dist/src/routing/libp2p-known-peers.d.ts.map +1 -1
  133. package/dist/src/routing/libp2p-known-peers.js +5 -0
  134. package/dist/src/routing/libp2p-known-peers.js.map +1 -1
  135. package/dist/src/storage/block-archive.d.ts +153 -0
  136. package/dist/src/storage/block-archive.d.ts.map +1 -0
  137. package/dist/src/storage/block-archive.js +194 -0
  138. package/dist/src/storage/block-archive.js.map +1 -0
  139. package/dist/src/storage/block-latch.d.ts +92 -0
  140. package/dist/src/storage/block-latch.d.ts.map +1 -0
  141. package/dist/src/storage/block-latch.js +132 -0
  142. package/dist/src/storage/block-latch.js.map +1 -0
  143. package/dist/src/storage/block-storage.d.ts +169 -12
  144. package/dist/src/storage/block-storage.d.ts.map +1 -1
  145. package/dist/src/storage/block-storage.js +415 -152
  146. package/dist/src/storage/block-storage.js.map +1 -1
  147. package/dist/src/storage/cached-raw-storage.d.ts +5 -1
  148. package/dist/src/storage/cached-raw-storage.d.ts.map +1 -1
  149. package/dist/src/storage/cached-raw-storage.js +16 -1
  150. package/dist/src/storage/cached-raw-storage.js.map +1 -1
  151. package/dist/src/storage/cached-store-driver.d.ts +14 -4
  152. package/dist/src/storage/cached-store-driver.d.ts.map +1 -1
  153. package/dist/src/storage/cached-store-driver.js +39 -5
  154. package/dist/src/storage/cached-store-driver.js.map +1 -1
  155. package/dist/src/storage/i-block-storage.d.ts +70 -19
  156. package/dist/src/storage/i-block-storage.d.ts.map +1 -1
  157. package/dist/src/storage/i-block-storage.js +21 -1
  158. package/dist/src/storage/i-block-storage.js.map +1 -1
  159. package/dist/src/storage/i-raw-storage.d.ts +38 -0
  160. package/dist/src/storage/i-raw-storage.d.ts.map +1 -1
  161. package/dist/src/storage/kv-raw-storage.d.ts +14 -8
  162. package/dist/src/storage/kv-raw-storage.d.ts.map +1 -1
  163. package/dist/src/storage/kv-raw-storage.js +29 -9
  164. package/dist/src/storage/kv-raw-storage.js.map +1 -1
  165. package/dist/src/storage/memory-store-driver.d.ts +4 -1
  166. package/dist/src/storage/memory-store-driver.d.ts.map +1 -1
  167. package/dist/src/storage/memory-store-driver.js +22 -2
  168. package/dist/src/storage/memory-store-driver.js.map +1 -1
  169. package/dist/src/storage/raw-store-codec.js +2 -2
  170. package/dist/src/storage/raw-store-driver.d.ts +44 -3
  171. package/dist/src/storage/raw-store-driver.d.ts.map +1 -1
  172. package/dist/src/storage/shared-cache-pool.d.ts +60 -3
  173. package/dist/src/storage/shared-cache-pool.d.ts.map +1 -1
  174. package/dist/src/storage/shared-cache-pool.js +68 -4
  175. package/dist/src/storage/shared-cache-pool.js.map +1 -1
  176. package/dist/src/storage/storage-repo.d.ts +204 -26
  177. package/dist/src/storage/storage-repo.d.ts.map +1 -1
  178. package/dist/src/storage/storage-repo.js +422 -119
  179. package/dist/src/storage/storage-repo.js.map +1 -1
  180. package/dist/src/storage/store-identity.d.ts +21 -0
  181. package/dist/src/storage/store-identity.d.ts.map +1 -0
  182. package/dist/src/storage/store-identity.js +33 -0
  183. package/dist/src/storage/store-identity.js.map +1 -0
  184. package/dist/src/storage/struct.d.ts +15 -0
  185. package/dist/src/storage/struct.d.ts.map +1 -1
  186. package/dist/src/storage/with-read-cache.d.ts +92 -0
  187. package/dist/src/storage/with-read-cache.d.ts.map +1 -0
  188. package/dist/src/storage/with-read-cache.js +132 -0
  189. package/dist/src/storage/with-read-cache.js.map +1 -0
  190. package/dist/src/sync/service.d.ts +13 -3
  191. package/dist/src/sync/service.d.ts.map +1 -1
  192. package/dist/src/sync/service.js +7 -30
  193. package/dist/src/sync/service.js.map +1 -1
  194. package/dist/src/testing/mesh-harness.d.ts +75 -2
  195. package/dist/src/testing/mesh-harness.d.ts.map +1 -1
  196. package/dist/src/testing/mesh-harness.js +193 -81
  197. package/dist/src/testing/mesh-harness.js.map +1 -1
  198. package/dist/src/testing/raw-storage-conformance.d.ts +8 -0
  199. package/dist/src/testing/raw-storage-conformance.d.ts.map +1 -1
  200. package/dist/src/testing/raw-storage-conformance.js +97 -2
  201. package/dist/src/testing/raw-storage-conformance.js.map +1 -1
  202. package/package.json +5 -6
  203. package/src/cluster/block-transfer-service.ts +528 -312
  204. package/src/cluster/block-transfer.ts +547 -424
  205. package/src/cluster/certified-claims.ts +319 -0
  206. package/src/cluster/client-signature-verifier.ts +34 -0
  207. package/src/cluster/cluster-policy.ts +294 -203
  208. package/src/cluster/cluster-repo.ts +2501 -2034
  209. package/src/cluster/commit-proof.ts +344 -0
  210. package/src/cluster/quorum-restore.ts +348 -223
  211. package/src/cluster/rebalance-monitor.ts +539 -254
  212. package/src/cluster/reconcile-block.ts +404 -203
  213. package/src/cluster/service.ts +308 -293
  214. package/src/cluster/spread-on-churn.ts +352 -346
  215. package/src/cohort-topic/host.ts +8 -7
  216. package/src/cohort-topic/stream-util.ts +14 -43
  217. package/src/dispute/cascade.ts +31 -15
  218. package/src/dispute/dispute-service.ts +706 -696
  219. package/src/dispute/invalidation.ts +162 -52
  220. package/src/dispute/service.ts +126 -125
  221. package/src/index.ts +7 -0
  222. package/src/libp2p-key-network.ts +142 -58
  223. package/src/libp2p-node-base.ts +1793 -1675
  224. package/src/logger.ts +27 -27
  225. package/src/network/open-protocol-stream.ts +87 -0
  226. package/src/network/register-protocol-handler.ts +85 -0
  227. package/src/peer-address-book.ts +360 -149
  228. package/src/pend-validation.ts +89 -0
  229. package/src/reactivity/notify-transport.ts +145 -144
  230. package/src/reactivity/push-state-gossip.ts +292 -291
  231. package/src/repo/cluster-coordinator.ts +65 -11
  232. package/src/repo/coordinator-repo.ts +729 -71
  233. package/src/repo/served-repo-proxy.ts +55 -0
  234. package/src/repo/service.ts +332 -313
  235. package/src/reputation/types.ts +3 -1
  236. package/src/rn.ts +34 -4
  237. package/src/routing/libp2p-known-peers.ts +31 -26
  238. package/src/storage/block-archive.ts +242 -0
  239. package/src/storage/block-latch.ts +144 -0
  240. package/src/storage/block-storage.ts +787 -498
  241. package/src/storage/cached-raw-storage.ts +21 -1
  242. package/src/storage/cached-store-driver.ts +898 -859
  243. package/src/storage/i-block-storage.ts +172 -116
  244. package/src/storage/i-raw-storage.ts +40 -0
  245. package/src/storage/kv-raw-storage.ts +34 -9
  246. package/src/storage/memory-store-driver.ts +26 -2
  247. package/src/storage/raw-store-codec.ts +42 -42
  248. package/src/storage/raw-store-driver.ts +45 -3
  249. package/src/storage/shared-cache-pool.ts +516 -452
  250. package/src/storage/storage-repo.ts +1421 -1014
  251. package/src/storage/store-identity.ts +41 -0
  252. package/src/storage/struct.ts +52 -34
  253. package/src/storage/with-read-cache.ts +182 -0
  254. package/src/sync/service.ts +178 -198
  255. package/src/testing/cohort-topic-mesh-harness.ts +673 -673
  256. package/src/testing/mesh-harness.ts +524 -344
  257. package/src/testing/raw-storage-conformance.ts +114 -2
@@ -1,668 +1,688 @@
1
- # Optimystic DB-P2P
2
-
3
- A distributed peer-to-peer database system that provides concrete implementations of the Optimystic database abstractions using pluggable storage backends and libp2p networking. This package transforms the interfaces and abstractions from `@optimystic/db-core` into a fully operational distributed database system.
4
-
5
- > **Storage is pluggable.** `db-p2p` owns the distributed layer (repo/cluster/coordination) and the storage *interfaces* (`IRawStorage`, `IBlockStorage`, `IKVStore`) plus an in-memory backend. Concrete persistent backends live in separate packages — filesystem persistence is [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs); see also `-storage-ns` (NativeScript/SQLite), `-storage-rn` (React Native/LevelDB), and `-storage-web` (IndexedDB).
6
-
7
- ## Overview
8
-
9
- The `@optimystic/db-core` package provides abstractions and interfaces for database operations—immutable blocks, versioned transactions, repository interfaces, and collection data structures. However, it doesn't provide concrete implementations for storage or distribution. `db-p2p` provides concrete implementations of the core abstractions using:
10
-
11
- - **Versioned block storage** (`StorageRepo` / `BlockStorage`) layered over a pluggable `IRawStorage` backend — an in-memory backend ships here; filesystem persistence lives in `@optimystic/db-p2p-storage-fs`
12
- - **libp2p networking** for decentralized peer-to-peer communication
13
- - **Distributed consensus** using 2-phase commit protocols for consistency
14
- - **Fault-tolerant coordination** with automatic recovery and data restoration
15
-
16
- The result is a fully operational distributed database system that implements the interfaces defined in the core abstractions to provide a complete distributed database system.
17
-
18
- ## Architecture Overview
19
-
20
- The system provides three distinct communication interfaces, each serving different roles in the distributed architecture:
21
-
22
- ```mermaid
23
- graph TD
24
- subgraph "External Client Process"
25
- A[Application Code]
26
- B[RepoClient]
27
- A --> B
28
- end
29
-
30
- subgraph "Coordinator Node Process<br/>(Orchestrates Transactions)"
31
- C[RepoService]
32
- D[CoordinatorRepo]
33
- E[ClusterClient]
34
- F[StorageRepo]
35
- subgraph "Local Storage"
36
- F1[BlockStorage]
37
- F2[IRawStorage backend]
38
- F3[Backend store]
39
- end
40
- C --> D
41
- D --> E
42
- D --> F
43
- F --> F1
44
- F1 --> F2
45
- F2 --> F3
46
- end
47
-
48
- subgraph "Cluster Peer 1 Process<br/>(Participates in Consensus)"
49
- G1[ClusterService]
50
- H1[ClusterMember]
51
- I1[StorageRepo]
52
- subgraph "Local Storage 1"
53
- I1A[BlockStorage]
54
- I1B[IRawStorage backend]
55
- I1C[Backend store]
56
- end
57
- G1 --> H1
58
- H1 --> I1
59
- I1 --> I1A
60
- I1A --> I1B
61
- I1B --> I1C
62
- end
63
-
64
- subgraph "Cluster Peer 2 Process<br/>(Participates in Consensus)"
65
- G2[ClusterService]
66
- H2[ClusterMember]
67
- I2[StorageRepo]
68
- subgraph "Local Storage 2"
69
- I2A[BlockStorage]
70
- I2B[IRawStorage backend]
71
- I2C[Backend store]
72
- end
73
- G2 --> H2
74
- H2 --> I2
75
- I2 --> I2A
76
- I2A --> I2B
77
- I2B --> I2C
78
- end
79
-
80
- subgraph "Cluster Peer N Process<br/>(Participates in Consensus)"
81
- G3[ClusterService]
82
- H3[ClusterMember]
83
- I3[StorageRepo]
84
- subgraph "Local Storage N"
85
- I3A[BlockStorage]
86
- I3B[IRawStorage backend]
87
- I3C[Backend store]
88
- end
89
- G3 --> H3
90
- H3 --> I3
91
- I3 --> I3A
92
- I3A --> I3B
93
- I3B --> I3C
94
- end
95
-
96
- %% Network communication
97
- B -.->|"/optimystic/&lt;network&gt;/repo/1.0.0"| C
98
- E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G1
99
- E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G2
100
- E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G3
101
- ```
102
-
103
- ### Repo Layer: External Client Interface
104
-
105
- The repo layer provides the interface for external clients to communicate with the distributed database:
106
-
107
- - **RepoClient**: Allows external clients to connect to any coordinator node
108
- - **RepoService**: Handles incoming requests from external clients
109
- - **CoordinatorRepo**: Orchestrates distributed transactions by coordinating with cluster peers
110
- - **Network transparency**: Same `IRepo` interface whether local or distributed
111
-
112
- ### Cluster Layer: Peer-to-Peer Coordination Interface
113
-
114
- The cluster layer provides the interface for coordinators to communicate with other peers in the cluster:
115
-
116
- - **ClusterClient**: Allows coordinators to send consensus requests to cluster peers
117
- - **ClusterService**: Handles incoming consensus requests from coordinators
118
- - **ClusterMember**: Implements 2-phase commit protocol for distributed consensus
119
- - **Byzantine fault tolerance**: Ensures consistency even with peer failures
120
-
121
- ### Storage Layer: Persistent Data Implementation
122
-
123
- The storage layer implements the core database abstractions over a pluggable raw-storage backend:
124
-
125
- - **StorageRepo**: Implements the `IRepo` interface on top of `BlockStorage` — backend-agnostic
126
- - **BlockStorage**: Provides versioned block storage with conflict resolution, reading/writing through an injected `IRawStorage`
127
- - **`IRawStorage`**: The raw persistence interface. An in-memory backend ships in this package; the filesystem backend (`FileRawStorage`) lives in `@optimystic/db-p2p-storage-fs`
128
- - **Data restoration**: Pluggable restoration for missing data from network peers
129
-
130
- ## Detailed Component Architecture
131
-
132
- ### Repo Layer Components
133
-
134
- #### `RepoClient`
135
- Allows external clients to connect to coordinator nodes:
136
-
137
- ```typescript
138
- class RepoClient extends ProtocolClient implements IRepo {
139
- static create(peerId: PeerId, peerNetwork: IPeerNetwork): RepoClient
140
- // Implements same IRepo interface as local storage
141
- }
142
- ```
143
-
144
- **Key Features:**
145
- - Network-transparent database operations for external clients
146
- - Automatic protocol handling and message serialization
147
- - Error handling for network failures and timeouts
148
- - Connection pooling and optimization
149
-
150
- #### `RepoService`
151
- Handles incoming repository protocol messages from external clients:
152
-
153
- ```typescript
154
- class RepoService implements Startable {
155
- // Protocol: /optimystic/<network>/repo/1.0.0 (see docs/repo.md § Protocol id conventions)
156
- // Transport: Length-prefixed JSON over libp2p streams
157
- async start(): Promise<void>
158
- async stop(): Promise<void>
159
- }
160
- ```
161
-
162
- **Key Features:**
163
- - Registers and handles the repository protocol
164
- - Routes operations from external clients to coordinator implementation
165
- - Manages concurrent connections and stream processing
166
- - Provides comprehensive error handling and logging
167
-
168
- #### `CoordinatorRepo`
169
- Orchestrates distributed transactions by coordinating with cluster peers:
170
-
171
- ```typescript
172
- class CoordinatorRepo implements IRepo {
173
- constructor(
174
- keyNetwork: IKeyNetwork,
175
- createClusterClient: (peerId: PeerId) => ClusterClient,
176
- storageRepo: IRepo
177
- )
178
- }
179
- ```
180
-
181
- **Key Features:**
182
- - Manages distributed transactions using cluster consensus
183
- - Integrates with key network for peer discovery
184
- - Uses cluster layer to coordinate with other peers
185
- - Maintains local storage directly while coordinating remote operations
186
-
187
- ### Cluster Layer Components
188
-
189
- #### `ClusterClient`
190
- Allows coordinators to send consensus requests to cluster peers:
191
-
192
- ```typescript
193
- class ClusterClient extends ProtocolClient implements ICluster {
194
- async update(record: ClusterRecord): Promise<ClusterRecord>
195
- }
196
- ```
197
-
198
- **Key Features:**
199
- - Network communication for consensus operations
200
- - Handles cluster record updates and responses
201
- - Manages timeouts and retries for consensus requests
202
-
203
- #### `ClusterService`
204
- Handles incoming consensus requests from coordinators:
205
-
206
- ```typescript
207
- class ClusterService implements Startable {
208
- // Protocol: /optimystic/<network>/cluster/1.0.0 (built from protocolPrefix)
209
- // Handles ClusterRecord updates for consensus
210
- }
211
- ```
212
-
213
- **Key Features:**
214
- - Registers and handles the cluster consensus protocol
215
- - Routes consensus requests to local cluster member
216
- - Manages concurrent consensus operations
217
-
218
- #### `ClusterMember`
219
- Implements 2-phase commit protocol for distributed consensus:
220
-
221
- ```typescript
222
- class ClusterMember implements ICluster {
223
- async update(record: ClusterRecord): Promise<ClusterRecord>
224
- // Handles: Promise collection → Majority consensus → Commit execution
225
- }
226
- ```
227
-
228
- **Key Features:**
229
- - Complete 2-phase commit protocol implementation
230
- - Conflict detection and resolution
231
- - Cryptographic signature verification
232
- - Automatic timeout and cleanup management
233
- - Uses storage layer to execute operations when consensus is reached
234
-
235
- ### Storage Layer Components
236
-
237
- #### `StorageRepo`
238
- Implements the `IRepo` interface over `BlockStorage` — independent of any particular storage backend:
239
-
240
- ```typescript
241
- class StorageRepo implements IRepo {
242
- async get(blockGets: BlockGets): Promise<GetBlockResults>
243
- async pend(request: PendRequest): Promise<PendResult>
244
- async commit(request: CommitRequest): Promise<CommitResult>
245
- async cancel(actionRef: ActionBlocks): Promise<void>
246
- }
247
- ```
248
-
249
- **Key Features:**
250
- - Concrete implementation of core database abstractions
251
- - Orchestrates transactions across multiple blocks
252
- - Handles revision conflicts and missing transaction detection
253
- - Provides atomic commit operations with proper locking
254
-
255
- #### `BlockStorage`
256
- Provides versioned block storage with conflict resolution, reading and writing through an injected `IRawStorage` backend:
257
-
258
- ```typescript
259
- class BlockStorage implements IBlockStorage {
260
- constructor(blockId: BlockId, storage: IRawStorage, /* … */)
261
- async getBlock(rev?: number): Promise<{ block: IBlock, actionRev: ActionRev }>
262
- async savePendingAction(actionId: ActionId, transform: Transform): Promise<void>
263
- async promotePendingAction(actionId: ActionId): Promise<void>
264
- async ensureRevision(rev: number): Promise<void>
265
- }
266
- ```
267
-
268
- **Key Features:**
269
- - Maintains complete revision history for each block
270
- - Reconstructs blocks by applying transforms to base versions
271
- - Integrates with restoration callbacks for missing data
272
- - Uses latches for thread-safe concurrent access
273
-
274
- #### `IRawStorage` backends
275
- `BlockStorage` persists through the `IRawStorage` interface, so the backing store is pluggable:
276
-
277
- - **In-memory** (`MemoryRawStorage`) ships in this package; used for tests and ephemeral nodes.
278
- - **Filesystem** (`FileRawStorage`) lives in the separate [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs) package; JSON files with atomic writes, organized per block:
279
-
280
- ```typescript
281
- // Filesystem backend layout (@optimystic/db-p2p-storage-fs)
282
- {basePath}/
283
- ├── {blockId}/
284
- │ ├── meta.json # Block metadata and revision ranges
285
- │ ├── revs/{rev}.json # Revision ActionId mappings
286
- │ ├── pend/{actionId}.json # Pending actions
287
- │ ├── actions/{actionId}.json # Committed actions
288
- │ └── blocks/{actionId}.json # Materialized blocks
289
- ```
290
-
291
- - **Other backends** — SQLite (`-storage-ns`), LevelDB (`-storage-rn`), IndexedDB (`-storage-web`), each in its own package.
292
-
293
- ## Integration Patterns
294
-
295
- ### Relationship to `@optimystic/db-core`
296
-
297
- The `db-p2p` package provides concrete implementations of the core database abstractions:
298
-
299
- ```typescript
300
- // Core provides interfaces and abstractions
301
- import { IRepo, IBlock, Transform, IBlockStorage } from '@optimystic/db-core';
302
-
303
- // P2P provides concrete implementations
304
- class StorageRepo implements IRepo { /* backend-agnostic versioned repo */ }
305
- class BlockStorage implements IBlockStorage { /* versioned block storage over IRawStorage */ }
306
- class RepoClient implements IRepo { /* network-transparent repo */ }
307
- class CoordinatorRepo implements IRepo { /* distributed consensus repo */ }
308
- ```
309
-
310
- **Implementation Points:**
311
- - **Storage Layer**: Implements the core storage interfaces over a pluggable `IRawStorage` backend (in-memory here; filesystem in `@optimystic/db-p2p-storage-fs`)
312
- - **Network Layer**: Implements peer-to-peer communication using libp2p
313
- - **Consensus Layer**: Adds distributed consensus while maintaining core interfaces
314
- - **API Compatibility**: Same interfaces as core abstractions for seamless integration
315
-
316
- ### Libp2p Integration
317
-
318
- The package integrates deeply with libp2p for networking:
319
-
320
- ```typescript
321
- // Node creation with integrated services
322
- const node = await createLibp2pNode({
323
- services: {
324
- repo: repoService({ protocolPrefix: '/optimystic/<network>' }),
325
- cluster: clusterService({ protocolPrefix: '/optimystic/<network>' })
326
- }
327
- });
328
-
329
- // Key network integration — use the node's OWN key network, do not construct a second one.
330
- // `createLibp2pNode` returns an `OptimysticNode`, which declares this handle (and
331
- // `coordinatedRepo`, `storageRepo`, `blockChangeNotifier`, `reputation`, `peerPrivateKey`).
332
- // The attached instance carries the node's resolved cluster size and its network-namespaced
333
- // protocol prefix; a separately constructed one selects a different-width cohort and has the
334
- // "does this peer serve my network?" filter switched off.
335
- const keyNetwork = node.keyNetwork;
336
- ```
337
-
338
- **Integration Features:**
339
- - **Protocol Management**: Registers custom protocols for repo and cluster operations
340
- - **Stream Handling**: Uses libp2p streams for reliable message transport
341
- - **Peer Discovery**: Integrates with libp2p DHT for peer discovery
342
- - **Security**: Leverages libp2p's cryptographic peer identity
343
-
344
- **Inbound stream authorization (optional).** The `repo`, `cluster`, `sync` and `block-transfer`
345
- protocols are open to any peer that can connect. An embedder whose database is private supplies one
346
- node-level, fail-closed predicate, consulted once per inbound stream before any decoding or execution:
347
-
348
- ```typescript
349
- const node = await createLibp2pNode({
350
- networkName: 'my-net',
351
- bootstrapNodes: [],
352
- // remotePeerId is the dialing peer's PeerId.toString(); only a literal `true` allows.
353
- // false / throw / rejection / timeout all deny and abort the stream.
354
- authorizeInboundStream: (remotePeerId, protocol) => memberSet.has(remotePeerId)
355
- });
356
- ```
357
-
358
- Omitting it is the default and leaves behavior unchanged. See
359
- [internals](../../docs/internals.md) § Inbound Stream Authorization for the full contract.
360
-
361
- **Announcing a different address than you bind (optional).** A node behind a NAT, port forward,
362
- reverse proxy or DNS front binds one address but is reachable at another. `announceAddrs` advertises
363
- the reachable address *instead of* whatever the transports report; `appendAnnounceAddrs` advertises
364
- it *in addition to* them:
365
-
366
- ```typescript
367
- const node = await createLibp2pNode({
368
- networkName: 'my-net',
369
- bootstrapNodes: ['...'],
370
- listenAddrs: ['/ip4/0.0.0.0/tcp/4001'], // what the process binds
371
- announceAddrs: ['/dns4/mynode.example.com/tcp/4001'] // what peers are told
372
- });
373
- ```
374
-
375
- `announceAddrs` replaces the advertised set **entirely** observed and relayed addresses are
376
- dropped from it, and `appendAnnounceAddrs` is ignored while it is non-empty. Use
377
- `appendAnnounceAddrs` alone when the automatically discovered addresses are still worth advertising.
378
- Both are straight passthroughs to libp2p's `addresses.announce` / `addresses.appendAnnounce`; an
379
- empty array means "unset".
380
-
381
- ## Usage Examples
382
-
383
- ### Setting Up a Coordinator Node
384
-
385
- `createLibp2pNode` assembles the whole stack and **starts** the node: it builds the storage layer,
386
- the consensus coordinator and the key network from one resolved config, and hands them back on the
387
- returned `OptimysticNode`. Read them off the node — do not construct your own, or the two copies
388
- disagree about the same key's cohort.
389
-
390
- ```typescript
391
- import { createLibp2pNode } from '@optimystic/db-p2p';
392
- import { FileRawStorage } from '@optimystic/db-p2p-storage-fs';
393
-
394
- // Create (and start) a libp2p node. The repo and cluster protocol services are registered by the
395
- // factory itself — there is no `services` option to pass them in.
396
- const node = await createLibp2pNode({
397
- networkName: 'my-net',
398
- bootstrapNodes: ['...'],
399
- // In-memory when omitted; a raw-storage backend makes it durable
400
- storage: new FileRawStorage('/var/lib/optimystic'),
401
- });
402
-
403
- // The handles the node already owns (see `OptimysticNodeAttachments`)
404
- const storageRepo = node.storageRepo; // Local block storage
405
- const coordinatedRepo = node.coordinatedRepo; // Cluster-consensus writes and reads
406
- const keyNetwork = node.keyNetwork; // Peer/coordinator discovery
407
-
408
- // ... use them, then:
409
- await node.stop();
410
- ```
411
-
412
- `createLibp2pNode` is all-or-nothing: the node is started early and then wired, so if any wiring step
413
- fails the factory stops the node before rejecting and surfaces the original error. A rejection
414
- therefore never leaves a running node behind — which matters because the caller gets no handle to
415
- stop one. Retrying on the same port is safe.
416
-
417
- Constructing a `StorageRepo`, `CoordinatorRepo` or `Libp2pKeyPeerNetwork` yourself is only for a
418
- host assembling a stack *without* `createLibp2pNode`. A standalone `CoordinatorRepo` or
419
- `Libp2pKeyPeerNetwork` must be told its cluster size; pass `DEFAULT_CLUSTER_SIZE` (exported from
420
- this package) to match what an unconfigured node resolves to.
421
-
422
- ### External Client Operations
423
-
424
- ```typescript
425
- // External clients connect to coordinator nodes
426
- const client = RepoClient.create(coordinatorPeerId, peerNetwork);
427
-
428
- // Operations work the same as core database interfaces
429
- const blocks = await client.get({
430
- blockIds: ['block1', 'block2'],
431
- context: { rev: 10 }
432
- });
433
-
434
- // But automatically coordinate across the cluster
435
- const pendResult = await client.pend({
436
- actionId: 'tx1',
437
- transforms: { block1: [/* operations */] },
438
- rev: 11
439
- });
440
-
441
- const commitResult = await client.commit({
442
- actionId: 'tx1',
443
- blockIds: ['block1'],
444
- rev: 11
445
- });
446
- ```
447
-
448
- ### How Coordination Works
449
-
450
- ```mermaid
451
- sequenceDiagram
452
- participant Client as External Client
453
- participant RC as RepoClient
454
- participant CR as CoordinatorRepo
455
- participant CC as ClusterClient
456
- participant CS as ClusterService
457
- participant CM as ClusterMember
458
- participant SR as StorageRepo
459
-
460
- Client->>RC: pend(transaction)
461
- RC->>CR: pend(transaction)
462
- CR->>CC: consensus request
463
- CC->>CS: cluster protocol
464
- CS->>CM: update(ClusterRecord)
465
- CM->>CM: 2-phase commit
466
- CM->>SR: execute operation
467
- SR-->>CM: result
468
- CM-->>CS: consensus result
469
- CS-->>CC: response
470
- CC-->>CR: consensus complete
471
- CR-->>RC: transaction result
472
- RC-->>Client: pend result
473
- ```
474
-
475
- ## Documentation
476
-
477
- For detailed information about specific components:
478
-
479
- - **[Storage System](./docs/storage.md)**: Versioned block storage and persistence
480
- - **[Repo Interface](./docs/repo.md)**: Distributed database operations
481
- - **[Cluster Consensus](./docs/cluster.md)**: 2-phase commit and distributed consensus
482
-
483
- ## Block Restoration & Arachnode Integration
484
-
485
- The db-p2p package includes **dynamic Arachnode ring discovery** for automatic block restoration across storage tiers.
486
-
487
- ### How Block Restoration Works
488
-
489
- When a node is missing a block or revision, the restoration system:
490
-
491
- 1. **Determines storage rings** - Nodes self-select ring depth based on capacity
492
- 2. **Discovers peers via FRET** - Ring membership propagates via existing neighbor exchange
493
- 3. **Queries rings intelligently** - Tries transaction ring first, then inner storage rings
494
- 4. **Filters by partition** - Only queries peers responsible for the block's keyspace
495
-
496
- ### Ring Selection
497
-
498
- Nodes calculate their appropriate ring depth using:
499
-
500
- ```
501
- ringDepth = ceil(-log2(available_capacity / estimated_demand))
502
- ```
503
-
504
- **Examples:**
505
- - 100% coverage Ring 0 (full keyspace)
506
- - 50% coverage Ring 1 (2 partitions)
507
- - 1% coverage Ring 7 (128 partitions)
508
-
509
- ### Architecture
510
-
511
- ```
512
- FRET (Pure DHT)
513
- - Generic metadata transport
514
- - Peer discovery & routing
515
-
516
- ArachnodeFretAdapter (Plugin Layer)
517
- - Arachnode-specific semantics
518
- - Ring discovery methods
519
-
520
- Restoration Components
521
- - RingSelector: Capacity-based ring selection
522
- - RestorationCoordinator: Multi-ring queries
523
- - StorageMonitor: Capacity tracking
524
- - SyncService/Client: Block request protocol
525
- ```
526
-
527
- ### Configuration
528
-
529
- ```typescript
530
- const node = await createLibp2pNode({
531
- port: 9000,
532
- networkName: 'mynet',
533
- bootstrapNodes: ['...'],
534
- clusterSize: 10, // Cluster size for peer discovery
535
- arachnode: {
536
- enableRingZulu: true // Default: enabled
537
- }
538
- });
539
- ```
540
-
541
- #### Solo / bootstrap / mobile nodes (no remote peers)
542
-
543
- A node can operate fully on its own — no bootstrap peers, no listen addresses, no
544
- connectivity to other peers. This is the common shape for a mobile app on first
545
- launch or for a local-only sanity test.
546
-
547
- ```typescript
548
- const soloNode = await createLibp2pNode({
549
- networkName: 'mynet',
550
- bootstrapNodes: [], // no bootstrap
551
- listenAddrs: [], // no incoming connections
552
- transports: [webSockets()], // transports still required
553
- clusterSize: 1,
554
- });
555
- ```
556
-
557
- In solo mode:
558
-
559
- - The node is its own coordinator. `CoordinatorRepo` short-circuits to local
560
- storage when the cluster size is ≤ 1, so `pend`/`commit` do not require
561
- cluster consensus and do not attempt to dial self.
562
- - Block restoration is skipped when the only discovered peer is self.
563
- `RestorationCoordinator` will return `undefined` immediately rather than
564
- attempt to dial self (which would hang on a node with no listen addrs).
565
- - If an operation fails for some other reason, `findCoordinator` surfaces a
566
- distinguishable `FindCoordinatorError` with `code ===
567
- FIND_COORDINATOR_ERROR_CODES.SELF_COORDINATION_EXHAUSTED` rather than the
568
- generic "all candidates excluded". The aggregate error thrown by
569
- `NetworkTransactor` preserves the ORIGINAL first-attempt cause so callers can
570
- diagnose the real problem.
571
-
572
- Once remote peers become reachable, the node transitions out of solo mode
573
- automatically as FRET discovers them. Persisted network state (HWM, FRET
574
- table) carries across restarts when an `options.persistence` is supplied.
575
-
576
- #### Custom transports (including React Native)
577
-
578
- By default, `createLibp2pNode()` uses TCP + circuit-relay transport.
579
-
580
- To use non-default transports, pass `transports` (and typically `listenAddrs`):
581
-
582
- ```typescript
583
- import { webSockets } from '@libp2p/websockets';
584
- import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
585
- import { createLibp2pNode } from '@optimystic/db-p2p';
586
-
587
- const node = await createLibp2pNode({
588
- networkName: 'mynet',
589
- bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
590
- transports: [webSockets(), circuitRelayTransport()],
591
- listenAddrs: [], // client-only; no incoming connections
592
- });
593
- ```
594
-
595
- #### React Native
596
-
597
- For React Native, import from `@optimystic/db-p2p/rn` instead of the root entrypoint.
598
- The `/rn` entrypoint does **not** import `@libp2p/tcp`, so Metro/Hermes won't try to
599
- bundle Node-only native modules. It requires `options.transports` explicitly:
600
-
601
- ```typescript
602
- import { webSockets } from '@libp2p/websockets';
603
- import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
604
- import { createLibp2pNode } from '@optimystic/db-p2p/rn';
605
-
606
- const node = await createLibp2pNode({
607
- networkName: 'mynet',
608
- bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
609
- transports: [webSockets(), circuitRelayTransport()],
610
- });
611
- ```
612
-
613
- RN environments (Hermes) need polyfills for globals that libp2p and its
614
- dependencies expect. Install these early (e.g. in your app's entry file
615
- before any other imports).
616
-
617
- **Global polyfills** (patch `globalThis` before any library code):
618
-
619
- | API | Required by | Notes |
620
- |-----|-------------|-------|
621
- | `crypto.getRandomValues()` | @noble/hashes, @libp2p/crypto | RN 0.76+ New Architecture provides natively; fallback via e.g. `react-native-get-random-values` |
622
- | `crypto.subtle.digest()` | multiformats/hashes/sha2-browser | Async SHA-256/SHA-512 — implement via @noble/hashes |
623
- | `structuredClone()` | @optimystic/db-core | JSON round-trip is sufficient |
624
- | `Promise.withResolvers()` | @libp2p/utils, @chainsafe/libp2p-yamux, it-queue | ES2024 — simple shim |
625
- | `AbortSignal.prototype.throwIfAborted()` | libp2p, @libp2p/circuit-relay-v2, it-pushable | DOM spec addition — simple shim |
626
- | Timer `.ref()` / `.unref()` | @optimystic/db-p2p, undici | Wrap Hermes numeric timer IDs in objects with no-op `.ref()`/`.unref()` methods; patch `clearTimeout`/`clearInterval` to unwrap |
627
- | `Event`, `CustomEvent`, `EventTarget` | libp2p, @libp2p/interface | Custom shim or npm `event-target-polyfill` |
628
- | `Intl.PluralRules` | moat-maker | English-only ordinal/cardinal shim is sufficient |
629
-
630
- **Node.js built-in module shims** (via Metro `extraNodeModules` or bundler aliases):
631
-
632
- | Module | Required by | Recommended shim |
633
- |--------|-------------|------------------|
634
- | `os` / `node:os` | @libp2p/utils | Custom shim: `networkInterfaces()` `{}`, `platform()` `Platform.OS` |
635
- | `crypto` / `node:crypto` | multiformats/hashes/sha2 | Custom shim: `createHash()` via @noble/hashes |
636
- | `stream` / `node:stream` | libp2p | `readable-stream` (npm) |
637
- | `buffer` / `node:buffer` | libp2p, multiformats | `buffer` (npm) |
638
-
639
- **Built-in (no polyfill needed):**
640
- - `TextEncoder` — built-in to Hermes
641
- - `TextDecoder` built-in to Expo SDK 52+ (UTF-8 only)
642
- - `BigInt` built-in to Hermes since RN 0.70
643
-
644
- See the [Sereus reference-app-rn](https://github.com/gotchoices/sereus/tree/master/packages/reference-app-rn/polyfills) for working polyfill implementations.
645
-
646
- ### Ring Transitions
647
-
648
- Nodes automatically transition between rings based on capacity thresholds:
649
- - **Move OUT** (to more granular ring) when > 85% capacity used
650
- - **Move IN** (to broader ring) when < 40% capacity used
651
-
652
- ## Related Packages
653
-
654
- - **[@optimystic/db-core](../db-core)**: Core database interfaces and local operations
655
- - **[@optimystic/db-p2p-storage-fs](../db-p2p-storage-fs)**: Filesystem `IRawStorage` backend (Node)
656
- - **[@optimystic/db-p2p-storage-ns](../db-p2p-storage-ns)**: NativeScript SQLite storage backend
657
- - **[@optimystic/db-p2p-storage-rn](../db-p2p-storage-rn)**: React Native LevelDB storage backend
658
- - **[@optimystic/db-p2p-storage-web](../db-p2p-storage-web)**: Browser IndexedDB storage backend
659
- - **[@optimystic/db-quereus](../db-quereus)**: Query engine and data access patterns
660
- - **[p2p-fret](../fret)**: DHT implementation for peer discovery
661
-
662
- ### Internal Architecture
663
-
664
- For developers and AI agents working on internals, see [Internals Guide](../../docs/internals.md) for:
665
- - Data flow diagrams (read/write/commit paths)
666
- - Mutation contracts (which functions mutate vs clone)
667
- - Key invariants and common pitfalls
668
- - Type glossary and debugging tips
1
+ # Optimystic DB-P2P
2
+
3
+ A distributed peer-to-peer database system that provides concrete implementations of the Optimystic database abstractions using pluggable storage backends and libp2p networking. This package transforms the interfaces and abstractions from `@optimystic/db-core` into a fully operational distributed database system.
4
+
5
+ > **Storage is pluggable.** `db-p2p` owns the distributed layer (repo/cluster/coordination) and the storage *interfaces* (`IRawStorage`, `IBlockStorage`, `IKVStore`) plus an in-memory backend. Concrete persistent backends live in separate packages — filesystem persistence is [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs); see also `-storage-ns` (NativeScript/SQLite), `-storage-rn` (React Native/LevelDB), and `-storage-web` (IndexedDB).
6
+
7
+ ## Overview
8
+
9
+ The `@optimystic/db-core` package provides abstractions and interfaces for database operations—immutable blocks, versioned transactions, repository interfaces, and collection data structures. However, it doesn't provide concrete implementations for storage or distribution. `db-p2p` provides concrete implementations of the core abstractions using:
10
+
11
+ - **Versioned block storage** (`StorageRepo` / `BlockStorage`) layered over a pluggable `IRawStorage` backend — an in-memory backend ships here; filesystem persistence lives in `@optimystic/db-p2p-storage-fs`
12
+ - **libp2p networking** for decentralized peer-to-peer communication
13
+ - **Distributed consensus** using 2-phase commit protocols for consistency
14
+ - **Fault-tolerant coordination** with automatic recovery and data restoration
15
+
16
+ The result is a fully operational distributed database system that implements the interfaces defined in the core abstractions to provide a complete distributed database system.
17
+
18
+ ## Architecture Overview
19
+
20
+ The system provides three distinct communication interfaces, each serving different roles in the distributed architecture:
21
+
22
+ ```mermaid
23
+ graph TD
24
+ subgraph "External Client Process"
25
+ A[Application Code]
26
+ B[RepoClient]
27
+ A --> B
28
+ end
29
+
30
+ subgraph "Coordinator Node Process<br/>(Orchestrates Transactions)"
31
+ C[RepoService]
32
+ D[CoordinatorRepo]
33
+ E[ClusterClient]
34
+ F[StorageRepo]
35
+ subgraph "Local Storage"
36
+ F1[BlockStorage]
37
+ F2[IRawStorage backend]
38
+ F3[Backend store]
39
+ end
40
+ C --> D
41
+ D --> E
42
+ D --> F
43
+ F --> F1
44
+ F1 --> F2
45
+ F2 --> F3
46
+ end
47
+
48
+ subgraph "Cluster Peer 1 Process<br/>(Participates in Consensus)"
49
+ G1[ClusterService]
50
+ H1[ClusterMember]
51
+ I1[StorageRepo]
52
+ subgraph "Local Storage 1"
53
+ I1A[BlockStorage]
54
+ I1B[IRawStorage backend]
55
+ I1C[Backend store]
56
+ end
57
+ G1 --> H1
58
+ H1 --> I1
59
+ I1 --> I1A
60
+ I1A --> I1B
61
+ I1B --> I1C
62
+ end
63
+
64
+ subgraph "Cluster Peer 2 Process<br/>(Participates in Consensus)"
65
+ G2[ClusterService]
66
+ H2[ClusterMember]
67
+ I2[StorageRepo]
68
+ subgraph "Local Storage 2"
69
+ I2A[BlockStorage]
70
+ I2B[IRawStorage backend]
71
+ I2C[Backend store]
72
+ end
73
+ G2 --> H2
74
+ H2 --> I2
75
+ I2 --> I2A
76
+ I2A --> I2B
77
+ I2B --> I2C
78
+ end
79
+
80
+ subgraph "Cluster Peer N Process<br/>(Participates in Consensus)"
81
+ G3[ClusterService]
82
+ H3[ClusterMember]
83
+ I3[StorageRepo]
84
+ subgraph "Local Storage N"
85
+ I3A[BlockStorage]
86
+ I3B[IRawStorage backend]
87
+ I3C[Backend store]
88
+ end
89
+ G3 --> H3
90
+ H3 --> I3
91
+ I3 --> I3A
92
+ I3A --> I3B
93
+ I3B --> I3C
94
+ end
95
+
96
+ %% Network communication
97
+ B -.->|"/optimystic/&lt;network&gt;/repo/1.0.0"| C
98
+ E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G1
99
+ E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G2
100
+ E -.->|"/optimystic/&lt;network&gt;/cluster/1.0.0"| G3
101
+ ```
102
+
103
+ ### Repo Layer: External Client Interface
104
+
105
+ The repo layer provides the interface for external clients to communicate with the distributed database:
106
+
107
+ - **RepoClient**: Allows external clients to connect to any coordinator node
108
+ - **RepoService**: Handles incoming requests from external clients
109
+ - **CoordinatorRepo**: Orchestrates distributed transactions by coordinating with cluster peers
110
+ - **Network transparency**: Same `IRepo` interface whether local or distributed
111
+
112
+ ### Cluster Layer: Peer-to-Peer Coordination Interface
113
+
114
+ The cluster layer provides the interface for coordinators to communicate with other peers in the cluster:
115
+
116
+ - **ClusterClient**: Allows coordinators to send consensus requests to cluster peers
117
+ - **ClusterService**: Handles incoming consensus requests from coordinators
118
+ - **ClusterMember**: Implements 2-phase commit protocol for distributed consensus
119
+ - **Byzantine fault tolerance**: Ensures consistency even with peer failures
120
+
121
+ ### Storage Layer: Persistent Data Implementation
122
+
123
+ The storage layer implements the core database abstractions over a pluggable raw-storage backend:
124
+
125
+ - **StorageRepo**: Implements the `IRepo` interface on top of `BlockStorage` — backend-agnostic
126
+ - **BlockStorage**: Provides versioned block storage with conflict resolution, reading/writing through an injected `IRawStorage`
127
+ - **`IRawStorage`**: The raw persistence interface. An in-memory backend ships in this package; the filesystem backend (`FileRawStorage`) lives in `@optimystic/db-p2p-storage-fs`
128
+ - **Data restoration**: Pluggable restoration for missing data from network peers
129
+
130
+ ## Detailed Component Architecture
131
+
132
+ ### Repo Layer Components
133
+
134
+ #### `RepoClient`
135
+ Allows external clients to connect to coordinator nodes:
136
+
137
+ ```typescript
138
+ class RepoClient extends ProtocolClient implements IRepo {
139
+ static create(peerId: PeerId, peerNetwork: IPeerNetwork): RepoClient
140
+ // Implements same IRepo interface as local storage
141
+ }
142
+ ```
143
+
144
+ **Key Features:**
145
+ - Network-transparent database operations for external clients
146
+ - Automatic protocol handling and message serialization
147
+ - Error handling for network failures and timeouts
148
+ - Connection pooling and optimization
149
+
150
+ #### `RepoService`
151
+ Handles incoming repository protocol messages from external clients:
152
+
153
+ ```typescript
154
+ class RepoService implements Startable {
155
+ // Protocol: /optimystic/<network>/repo/1.0.0 (see docs/repo.md § Protocol id conventions)
156
+ // Transport: Length-prefixed JSON over libp2p streams
157
+ async start(): Promise<void>
158
+ async stop(): Promise<void>
159
+ }
160
+ ```
161
+
162
+ **Key Features:**
163
+ - Registers and handles the repository protocol
164
+ - Routes operations from external clients to coordinator implementation
165
+ - Manages concurrent connections and stream processing
166
+ - Provides comprehensive error handling and logging
167
+
168
+ #### `CoordinatorRepo`
169
+ Orchestrates distributed transactions by coordinating with cluster peers:
170
+
171
+ ```typescript
172
+ class CoordinatorRepo implements IRepo {
173
+ constructor(
174
+ keyNetwork: IKeyNetwork,
175
+ createClusterClient: (peerId: PeerId) => ClusterClient,
176
+ storageRepo: IRepo
177
+ )
178
+ }
179
+ ```
180
+
181
+ **Key Features:**
182
+ - Manages distributed transactions using cluster consensus
183
+ - Integrates with key network for peer discovery
184
+ - Uses cluster layer to coordinate with other peers
185
+ - Maintains local storage directly while coordinating remote operations
186
+
187
+ ### Cluster Layer Components
188
+
189
+ #### `ClusterClient`
190
+ Allows coordinators to send consensus requests to cluster peers:
191
+
192
+ ```typescript
193
+ class ClusterClient extends ProtocolClient implements ICluster {
194
+ async update(record: ClusterRecord): Promise<ClusterRecord>
195
+ }
196
+ ```
197
+
198
+ **Key Features:**
199
+ - Network communication for consensus operations
200
+ - Handles cluster record updates and responses
201
+ - Manages timeouts and retries for consensus requests
202
+
203
+ #### `ClusterService`
204
+ Handles incoming consensus requests from coordinators:
205
+
206
+ ```typescript
207
+ class ClusterService implements Startable {
208
+ // Protocol: /optimystic/<network>/cluster/1.0.0 (built from protocolPrefix)
209
+ // Handles ClusterRecord updates for consensus
210
+ }
211
+ ```
212
+
213
+ **Key Features:**
214
+ - Registers and handles the cluster consensus protocol
215
+ - Routes consensus requests to local cluster member
216
+ - Manages concurrent consensus operations
217
+
218
+ #### `ClusterMember`
219
+ Implements 2-phase commit protocol for distributed consensus:
220
+
221
+ ```typescript
222
+ class ClusterMember implements ICluster {
223
+ async update(record: ClusterRecord): Promise<ClusterRecord>
224
+ // Handles: Promise collection → Majority consensus → Commit execution
225
+ }
226
+ ```
227
+
228
+ **Key Features:**
229
+ - Complete 2-phase commit protocol implementation
230
+ - Conflict detection and resolution
231
+ - Cryptographic signature verification
232
+ - Automatic timeout and cleanup management
233
+ - Uses storage layer to execute operations when consensus is reached
234
+
235
+ ### Storage Layer Components
236
+
237
+ #### `StorageRepo`
238
+ Implements the `IRepo` interface over `BlockStorage` — independent of any particular storage backend:
239
+
240
+ ```typescript
241
+ class StorageRepo implements IRepo {
242
+ async get(blockGets: BlockGets): Promise<GetBlockResults>
243
+ async pend(request: PendRequest): Promise<PendResult>
244
+ async commit(request: CommitRequest): Promise<CommitResult>
245
+ async cancel(actionRef: ActionBlocks): Promise<void>
246
+ }
247
+ ```
248
+
249
+ **Key Features:**
250
+ - Concrete implementation of core database abstractions
251
+ - Orchestrates transactions across multiple blocks
252
+ - Handles revision conflicts and missing transaction detection
253
+ - Provides atomic commit operations under one write latch per block, acquired in sorted block-id order so two commits over overlapping batches cannot deadlock
254
+
255
+ #### `BlockStorage`
256
+ Provides versioned block storage with conflict resolution, reading and writing through an injected `IRawStorage` backend:
257
+
258
+ ```typescript
259
+ class BlockStorage implements IBlockStorage {
260
+ constructor(blockId: BlockId, storage: IRawStorage, restoreCallback?: RestoreCallback, /* … */)
261
+
262
+ // Reads take no latch.
263
+ async getBlock(rev?: number): Promise<{ block: IBlock, actionRev: ActionRev } | undefined>
264
+
265
+ // Every write takes a BlockWriteLatch token, minted only by acquiring the
266
+ // block's write latch — so an unlatched write does not compile.
267
+ async savePendingTransaction(actionId: ActionId, transform: Transform, latch: BlockWriteLatch): Promise<void>
268
+ async promotePendingTransaction(actionId: ActionId, latch: BlockWriteLatch): Promise<void>
269
+ async restoreRevision(rev: number, latch: BlockWriteLatch): Promise<void>
270
+ }
271
+ ```
272
+
273
+ **Key Features:**
274
+ - Maintains complete revision history for each block
275
+ - Reconstructs blocks by applying transforms to base versions
276
+ - `getBlock` is **local-only** — it never fetches from a peer. A revision outside the
277
+ block's recorded coverage raises `RevisionNotCoveredError`; `StorageRepo.get` is the
278
+ one caller that heals that, by calling `restoreRevision` (which does consult the
279
+ restoration callback) under the block's write latch and re-reading. The commit path
280
+ never restores in line: it holds every batch block's latch and must do no network I/O
281
+ there, so an unreachable base is refused with `MissingBaseRevisionError` and healed
282
+ out-of-band by cohort reconcile plus a retry.
283
+ - Concurrency is one write latch per block (`Block.write:<blockId>`). The caller acquires
284
+ it and passes the resulting token into each write; `BlockStorage` rejects a token minted
285
+ for a different block, or one whose latch has already been released.
286
+
287
+ #### `IRawStorage` backends
288
+ `BlockStorage` persists through the `IRawStorage` interface, so the backing store is pluggable:
289
+
290
+ - **In-memory** (`MemoryRawStorage`) — ships in this package; used for tests and ephemeral nodes.
291
+ - **Filesystem** (`FileRawStorage`) lives in the separate [`@optimystic/db-p2p-storage-fs`](../db-p2p-storage-fs) package; JSON files with atomic writes, organized per block:
292
+
293
+ ```typescript
294
+ // Filesystem backend layout (@optimystic/db-p2p-storage-fs)
295
+ {basePath}/
296
+ ├── {blockId}/
297
+ │ ├── meta.json # Block metadata and revision ranges
298
+ │ ├── revs/{rev}.json # Revision → ActionId mappings
299
+ │ ├── pend/{actionId}.json # Pending actions
300
+ │ ├── actions/{actionId}.json # Committed actions
301
+ │ ├── proofs/{rev}.json # Retained BlockCommitProof per revision
302
+ │ └── blocks/{actionId}.json # Materialized blocks
303
+ ```
304
+
305
+ - **Other backends** SQLite (`-storage-ns`), LevelDB (`-storage-rn`), IndexedDB (`-storage-web`), each in its own package.
306
+
307
+ ## Integration Patterns
308
+
309
+ ### Relationship to `@optimystic/db-core`
310
+
311
+ The `db-p2p` package provides concrete implementations of the core database abstractions:
312
+
313
+ ```typescript
314
+ // Core provides interfaces and abstractions
315
+ import { IRepo, IBlock, Transform, IBlockStorage } from '@optimystic/db-core';
316
+
317
+ // P2P provides concrete implementations
318
+ class StorageRepo implements IRepo { /* backend-agnostic versioned repo */ }
319
+ class BlockStorage implements IBlockStorage { /* versioned block storage over IRawStorage */ }
320
+ class RepoClient implements IRepo { /* network-transparent repo */ }
321
+ class CoordinatorRepo implements IRepo { /* distributed consensus repo */ }
322
+ ```
323
+
324
+ **Implementation Points:**
325
+ - **Storage Layer**: Implements the core storage interfaces over a pluggable `IRawStorage` backend (in-memory here; filesystem in `@optimystic/db-p2p-storage-fs`)
326
+ - **Network Layer**: Implements peer-to-peer communication using libp2p
327
+ - **Consensus Layer**: Adds distributed consensus while maintaining core interfaces
328
+ - **API Compatibility**: Same interfaces as core abstractions for seamless integration
329
+
330
+ ### Libp2p Integration
331
+
332
+ The package integrates deeply with libp2p for networking:
333
+
334
+ ```typescript
335
+ // Node creation with integrated services
336
+ const node = await createLibp2pNode({
337
+ services: {
338
+ repo: repoService({ protocolPrefix: '/optimystic/<network>' }),
339
+ cluster: clusterService({ protocolPrefix: '/optimystic/<network>' })
340
+ }
341
+ });
342
+
343
+ // Key network integration — use the node's OWN key network, do not construct a second one.
344
+ // `createLibp2pNode` returns an `OptimysticNode`, which declares this handle (and
345
+ // `coordinatedRepo`, `storageRepo`, `blockChangeNotifier`, `reputation`, `peerPrivateKey`).
346
+ // The attached instance carries the node's resolved cluster size and its network-namespaced
347
+ // protocol prefix; a separately constructed one selects a different-width cohort and has the
348
+ // "does this peer serve my network?" filter switched off.
349
+ const keyNetwork = node.keyNetwork;
350
+ ```
351
+
352
+ **Integration Features:**
353
+ - **Protocol Management**: Registers custom protocols for repo and cluster operations
354
+ - **Stream Handling**: Uses libp2p streams for reliable message transport
355
+ - **Peer Discovery**: Integrates with libp2p DHT for peer discovery
356
+ - **Security**: Leverages libp2p's cryptographic peer identity
357
+
358
+ **Inbound stream authorization (optional).** The `repo`, `cluster`, `sync` and `block-transfer`
359
+ protocols are open to any peer that can connect. An embedder whose database is private supplies one
360
+ node-level, fail-closed predicate, consulted once per inbound stream before any decoding or execution:
361
+
362
+ ```typescript
363
+ const node = await createLibp2pNode({
364
+ networkName: 'my-net',
365
+ bootstrapNodes: [],
366
+ // remotePeerId is the dialing peer's PeerId.toString(); only a literal `true` allows.
367
+ // false / throw / rejection / timeout all deny and abort the stream.
368
+ authorizeInboundStream: (remotePeerId, protocol) => memberSet.has(remotePeerId)
369
+ });
370
+ ```
371
+
372
+ Omitting it is the default and leaves behavior unchanged. See
373
+ [internals](../../docs/internals.md) § Inbound Stream Authorization for the full contract.
374
+
375
+ **Announcing a different address than you bind (optional).** A node behind a NAT, port forward,
376
+ reverse proxy or DNS front binds one address but is reachable at another. `announceAddrs` advertises
377
+ the reachable address *instead of* whatever the transports report; `appendAnnounceAddrs` advertises
378
+ it *in addition to* them:
379
+
380
+ ```typescript
381
+ const node = await createLibp2pNode({
382
+ networkName: 'my-net',
383
+ bootstrapNodes: ['...'],
384
+ listenAddrs: ['/ip4/0.0.0.0/tcp/4001'], // what the process binds
385
+ announceAddrs: ['/dns4/mynode.example.com/tcp/4001'] // what peers are told
386
+ });
387
+ ```
388
+
389
+ `announceAddrs` replaces the advertised set **entirely** — observed and relayed addresses are
390
+ dropped from it, and `appendAnnounceAddrs` is ignored while it is non-empty. Use
391
+ `appendAnnounceAddrs` alone when the automatically discovered addresses are still worth advertising.
392
+ Both are straight passthroughs to libp2p's `addresses.announce` / `addresses.appendAnnounce`; an
393
+ empty array means "unset".
394
+
395
+ ## Usage Examples
396
+
397
+ ### Setting Up a Coordinator Node
398
+
399
+ `createLibp2pNode` assembles the whole stack and **starts** the node: it builds the storage layer,
400
+ the consensus coordinator and the key network from one resolved config, and hands them back on the
401
+ returned `OptimysticNode`. Read them off the node — do not construct your own, or the two copies
402
+ disagree about the same key's cohort.
403
+
404
+ ```typescript
405
+ import { createLibp2pNode } from '@optimystic/db-p2p';
406
+ import { FileRawStorage } from '@optimystic/db-p2p-storage-fs';
407
+
408
+ // Create (and start) a libp2p node. The repo and cluster protocol services are registered by the
409
+ // factory itself — there is no `services` option to pass them in.
410
+ const node = await createLibp2pNode({
411
+ networkName: 'my-net',
412
+ bootstrapNodes: ['...'],
413
+ // In-memory when omitted; a raw-storage backend makes it durable
414
+ storage: new FileRawStorage('/var/lib/optimystic'),
415
+ });
416
+
417
+ // The handles the node already owns (see `OptimysticNodeAttachments`)
418
+ const storageRepo = node.storageRepo; // Local block storage
419
+ const coordinatedRepo = node.coordinatedRepo; // Cluster-consensus writes and reads
420
+ const keyNetwork = node.keyNetwork; // Peer/coordinator discovery
421
+
422
+ // ... use them, then:
423
+ await node.stop();
424
+ ```
425
+
426
+ `createLibp2pNode` is all-or-nothing: the node is started early and then wired, so if any wiring step
427
+ fails the factory stops the node before rejecting and surfaces the original error. A rejection
428
+ therefore never leaves a running node behind — which matters because the caller gets no handle to
429
+ stop one. Retrying on the same port is safe.
430
+
431
+ Constructing a `StorageRepo`, `CoordinatorRepo` or `Libp2pKeyPeerNetwork` yourself is only for a
432
+ host assembling a stack *without* `createLibp2pNode`. A standalone `CoordinatorRepo` or
433
+ `Libp2pKeyPeerNetwork` must be told its cluster size; pass `DEFAULT_CLUSTER_SIZE` (exported from
434
+ this package) to match what an unconfigured node resolves to.
435
+
436
+ ### External Client Operations
437
+
438
+ ```typescript
439
+ // External clients connect to coordinator nodes
440
+ const client = RepoClient.create(coordinatorPeerId, peerNetwork);
441
+
442
+ // Operations work the same as core database interfaces
443
+ const blocks = await client.get({
444
+ blockIds: ['block1', 'block2'],
445
+ context: { rev: 10 }
446
+ });
447
+
448
+ // But automatically coordinate across the cluster
449
+ const pendResult = await client.pend({
450
+ actionId: 'tx1',
451
+ transforms: { block1: [/* operations */] },
452
+ rev: 11
453
+ });
454
+
455
+ const commitResult = await client.commit({
456
+ actionId: 'tx1',
457
+ blockIds: ['block1'],
458
+ rev: 11
459
+ });
460
+ ```
461
+
462
+ ### How Coordination Works
463
+
464
+ ```mermaid
465
+ sequenceDiagram
466
+ participant Client as External Client
467
+ participant RC as RepoClient
468
+ participant CR as CoordinatorRepo
469
+ participant CC as ClusterClient
470
+ participant CS as ClusterService
471
+ participant CM as ClusterMember
472
+ participant SR as StorageRepo
473
+
474
+ Client->>RC: pend(transaction)
475
+ RC->>CR: pend(transaction)
476
+ CR->>CC: consensus request
477
+ CC->>CS: cluster protocol
478
+ CS->>CM: update(ClusterRecord)
479
+ CM->>CM: 2-phase commit
480
+ CM->>SR: execute operation
481
+ SR-->>CM: result
482
+ CM-->>CS: consensus result
483
+ CS-->>CC: response
484
+ CC-->>CR: consensus complete
485
+ CR-->>RC: transaction result
486
+ RC-->>Client: pend result
487
+ ```
488
+
489
+ ## Documentation
490
+
491
+ For detailed information about specific components:
492
+
493
+ - **[Storage System](./docs/storage.md)**: Versioned block storage and persistence
494
+ - **[Repo Interface](./docs/repo.md)**: Distributed database operations
495
+ - **[Cluster Consensus](./docs/cluster.md)**: 2-phase commit and distributed consensus
496
+
497
+ ## Block Restoration & Arachnode Integration
498
+
499
+ The db-p2p package includes **dynamic Arachnode ring discovery** for automatic block restoration across storage tiers.
500
+
501
+ ### How Block Restoration Works
502
+
503
+ When a node is missing a block or revision, the restoration system:
504
+
505
+ 1. **Determines storage rings** - Nodes self-select ring depth based on capacity
506
+ 2. **Discovers peers via FRET** - Ring membership propagates via existing neighbor exchange
507
+ 3. **Queries rings intelligently** - Tries transaction ring first, then inner storage rings
508
+ 4. **Filters by partition** - Only queries peers responsible for the block's keyspace
509
+
510
+ ### Ring Selection
511
+
512
+ Nodes calculate their appropriate ring depth using:
513
+
514
+ ```
515
+ ringDepth = ceil(-log2(available_capacity / estimated_demand))
516
+ ```
517
+
518
+ **Examples:**
519
+ - 100% coverage → Ring 0 (full keyspace)
520
+ - 50% coverage → Ring 1 (2 partitions)
521
+ - 1% coverage Ring 7 (128 partitions)
522
+
523
+ ### Architecture
524
+
525
+ ```
526
+ FRET (Pure DHT)
527
+ - Generic metadata transport
528
+ - Peer discovery & routing
529
+
530
+ ArachnodeFretAdapter (Plugin Layer)
531
+ - Arachnode-specific semantics
532
+ - Ring discovery methods
533
+
534
+ Restoration Components
535
+ - RingSelector: Capacity-based ring selection
536
+ - RestorationCoordinator: Multi-ring queries
537
+ - StorageMonitor: Capacity tracking
538
+ - SyncService/Client: Block request protocol
539
+ ```
540
+
541
+ ### Configuration
542
+
543
+ ```typescript
544
+ const node = await createLibp2pNode({
545
+ port: 9000,
546
+ networkName: 'mynet',
547
+ bootstrapNodes: ['...'],
548
+ clusterSize: 10, // Cluster size for peer discovery
549
+ arachnode: {
550
+ enableRingZulu: true // Default: enabled
551
+ }
552
+ });
553
+ ```
554
+
555
+ #### Solo / bootstrap / mobile nodes (no remote peers)
556
+
557
+ A node can operate fully on its own — no bootstrap peers, no listen addresses, no
558
+ connectivity to other peers. This is the common shape for a mobile app on first
559
+ launch or for a local-only sanity test.
560
+
561
+ ```typescript
562
+ const soloNode = await createLibp2pNode({
563
+ networkName: 'mynet',
564
+ bootstrapNodes: [], // no bootstrap
565
+ listenAddrs: [], // no incoming connections
566
+ transports: [webSockets()], // transports still required
567
+ clusterSize: 1,
568
+ });
569
+ ```
570
+
571
+ In solo mode:
572
+
573
+ - The node is its own coordinator. `CoordinatorRepo` short-circuits to local
574
+ storage when the cluster size is ≤ 1, so `pend`, `commit` and `cancel` do not
575
+ require cluster consensus and do not attempt to dial self. `pend` and `commit`
576
+ decide once (from the coordinating block); `cancel` decides per block id,
577
+ because a multi-block cancel can span cohorts of different sizes.
578
+ - Block restoration is skipped when the only discovered peer is self.
579
+ `RestorationCoordinator` will return `undefined` immediately rather than
580
+ attempt to dial self (which would hang on a node with no listen addrs).
581
+ - If an operation fails for some other reason, `findCoordinator` surfaces a
582
+ distinguishable `FindCoordinatorError` with `code ===
583
+ FIND_COORDINATOR_ERROR_CODES.SELF_COORDINATION_EXHAUSTED` rather than the
584
+ generic "all candidates excluded". The aggregate error thrown by
585
+ `NetworkTransactor` preserves the ORIGINAL first-attempt cause so callers can
586
+ diagnose the real problem.
587
+
588
+ Once remote peers become reachable, the node transitions out of solo mode
589
+ automatically as FRET discovers them. Persisted network state (HWM, FRET
590
+ table) carries across restarts when an `options.persistence` is supplied.
591
+
592
+ #### Custom transports (including React Native)
593
+
594
+ By default, `createLibp2pNode()` uses TCP + circuit-relay transport.
595
+
596
+ To use non-default transports, pass `transports` (and typically `listenAddrs`):
597
+
598
+ ```typescript
599
+ import { webSockets } from '@libp2p/websockets';
600
+ import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
601
+ import { createLibp2pNode } from '@optimystic/db-p2p';
602
+
603
+ const node = await createLibp2pNode({
604
+ networkName: 'mynet',
605
+ bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
606
+ transports: [webSockets(), circuitRelayTransport()],
607
+ listenAddrs: [], // client-only; no incoming connections
608
+ });
609
+ ```
610
+
611
+ #### React Native
612
+
613
+ For React Native, import from `@optimystic/db-p2p/rn` instead of the root entrypoint.
614
+ The `/rn` entrypoint does **not** import `@libp2p/tcp`, so Metro/Hermes won't try to
615
+ bundle Node-only native modules. It requires `options.transports` explicitly:
616
+
617
+ ```typescript
618
+ import { webSockets } from '@libp2p/websockets';
619
+ import { circuitRelayTransport } from '@libp2p/circuit-relay-v2';
620
+ import { createLibp2pNode } from '@optimystic/db-p2p/rn';
621
+
622
+ const node = await createLibp2pNode({
623
+ networkName: 'mynet',
624
+ bootstrapNodes: ['/dns4/relay.example.com/tcp/443/wss/p2p/12D3...'],
625
+ transports: [webSockets(), circuitRelayTransport()],
626
+ });
627
+ ```
628
+
629
+ Apart from that transport wiring, `/rn` exports exactly the same API as the root entrypoint,
630
+ so anything importable from `@optimystic/db-p2p` is importable from `@optimystic/db-p2p/rn`.
631
+ `test/entry-parity.spec.ts` fails if the two ever diverge.
632
+
633
+ RN environments (Hermes) need polyfills for globals that libp2p and its
634
+ dependencies expect. Install these early (e.g. in your app's entry file
635
+ before any other imports).
636
+
637
+ **Global polyfills** (patch `globalThis` before any library code):
638
+
639
+ | API | Required by | Notes |
640
+ |-----|-------------|-------|
641
+ | `crypto.getRandomValues()` | @noble/hashes, @libp2p/crypto | RN 0.76+ New Architecture provides natively; fallback via e.g. `react-native-get-random-values` |
642
+ | `crypto.subtle.digest()` | multiformats/hashes/sha2-browser | Async SHA-256/SHA-512 implement via @noble/hashes |
643
+ | `structuredClone()` | @optimystic/db-core | JSON round-trip is sufficient |
644
+ | `Promise.withResolvers()` | @libp2p/utils, @chainsafe/libp2p-yamux, it-queue | ES2024 simple shim |
645
+ | `AbortSignal.prototype.throwIfAborted()` | libp2p, @libp2p/circuit-relay-v2, it-pushable | DOM spec addition — simple shim |
646
+ | Timer `.ref()` / `.unref()` | @optimystic/db-p2p, undici | Wrap Hermes numeric timer IDs in objects with no-op `.ref()`/`.unref()` methods; patch `clearTimeout`/`clearInterval` to unwrap |
647
+ | `Event`, `CustomEvent`, `EventTarget` | libp2p, @libp2p/interface | Custom shim or npm `event-target-polyfill` |
648
+ | `Intl.PluralRules` | moat-maker | English-only ordinal/cardinal shim is sufficient |
649
+
650
+ **Node.js built-in module shims** (via Metro `extraNodeModules` or bundler aliases):
651
+
652
+ | Module | Required by | Recommended shim |
653
+ |--------|-------------|------------------|
654
+ | `os` / `node:os` | @libp2p/utils | Custom shim: `networkInterfaces()` `{}`, `platform()` `Platform.OS` |
655
+ | `crypto` / `node:crypto` | multiformats/hashes/sha2 | Custom shim: `createHash()` via @noble/hashes |
656
+ | `stream` / `node:stream` | libp2p | `readable-stream` (npm) |
657
+ | `buffer` / `node:buffer` | libp2p, multiformats | `buffer` (npm) |
658
+
659
+ **Built-in (no polyfill needed):**
660
+ - `TextEncoder` built-in to Hermes
661
+ - `TextDecoder` — built-in to Expo SDK 52+ (UTF-8 only)
662
+ - `BigInt` — built-in to Hermes since RN 0.70
663
+
664
+ See the [Sereus reference-app-rn](https://github.com/gotchoices/sereus/tree/master/packages/reference-app-rn/polyfills) for working polyfill implementations.
665
+
666
+ ### Ring Transitions
667
+
668
+ Nodes automatically transition between rings based on capacity thresholds:
669
+ - **Move OUT** (to more granular ring) when > 85% capacity used
670
+ - **Move IN** (to broader ring) when < 40% capacity used
671
+
672
+ ## Related Packages
673
+
674
+ - **[@optimystic/db-core](../db-core)**: Core database interfaces and local operations
675
+ - **[@optimystic/db-p2p-storage-fs](../db-p2p-storage-fs)**: Filesystem `IRawStorage` backend (Node)
676
+ - **[@optimystic/db-p2p-storage-ns](../db-p2p-storage-ns)**: NativeScript SQLite storage backend
677
+ - **[@optimystic/db-p2p-storage-rn](../db-p2p-storage-rn)**: React Native LevelDB storage backend
678
+ - **[@optimystic/db-p2p-storage-web](../db-p2p-storage-web)**: Browser IndexedDB storage backend
679
+ - **[@optimystic/db-quereus](../db-quereus)**: Query engine and data access patterns
680
+ - **[p2p-fret](../fret)**: DHT implementation for peer discovery
681
+
682
+ ### Internal Architecture
683
+
684
+ For developers and AI agents working on internals, see [Internals Guide](../../docs/internals.md) for:
685
+ - Data flow diagrams (read/write/commit paths)
686
+ - Mutation contracts (which functions mutate vs clone)
687
+ - Key invariants and common pitfalls
688
+ - Type glossary and debugging tips