@rljson/bs 0.0.19 → 0.0.20

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,9 +1,1302 @@
1
1
  <!--
2
2
  @license
3
- Copyright (c) 2025 Rljson
3
+ Copyright (c) 2026 Rljson
4
4
 
5
5
  Use of this source code is governed by terms that can be
6
6
  found in the LICENSE file in the root of this package.
7
7
  -->
8
8
 
9
- # Architecture
9
+ # @rljson/bs - Architecture Documentation
10
+
11
+ Deep dive into the architectural design, implementation patterns, and internal workings of the rljson blob storage system.
12
+
13
+ ## Table of Contents
14
+
15
+ 1. [Overview](#overview)
16
+ 2. [Core Design Principles](#core-design-principles)
17
+ 3. [Component Architecture](#component-architecture)
18
+ 4. [Content-Addressable Storage](#content-addressable-storage)
19
+ 5. [Network Architecture](#network-architecture)
20
+ 6. [PULL vs PUSH Architecture](#pull-vs-push-architecture)
21
+ 7. [Multi-Tier Storage](#multi-tier-storage)
22
+ 8. [Socket Abstraction Layer](#socket-abstraction-layer)
23
+ 9. [Implementation Details](#implementation-details)
24
+ 10. [Design Patterns](#design-patterns)
25
+ 11. [Performance Optimization](#performance-optimization)
26
+ 12. [Testing Strategy](#testing-strategy)
27
+
28
+ ## Overview
29
+
30
+ `@rljson/bs` implements a content-addressable blob storage system with a layered architecture designed for flexibility, performance, and type safety. The system follows a modular design where each component has a single responsibility and can be composed with others to create complex storage hierarchies.
31
+
32
+ ### System Goals
33
+
34
+ 1. **Content Addressability**: Every blob is identified by its SHA256 hash
35
+ 2. **Automatic Deduplication**: Identical content stored only once
36
+ 3. **Composability**: Components can be combined to create multi-tier architectures
37
+ 4. **Network Transparency**: Local and remote storage use the same interface
38
+ 5. **Type Safety**: Full TypeScript support with comprehensive types
39
+ 6. **Testability**: 100% test coverage with comprehensive test utilities
40
+
41
+ ## Core Design Principles
42
+
43
+ ### 1. Interface-First Design
44
+
45
+ All storage implementations conform to the `Bs` interface, ensuring uniform behavior:
46
+
47
+ ```typescript
48
+ export interface Bs {
49
+ setBlob(content: Buffer | string | ReadableStream): Promise<BlobProperties>;
50
+ getBlob(blobId: string, options?: DownloadBlobOptions): Promise<{ content: Buffer; properties: BlobProperties }>;
51
+ getBlobStream(blobId: string): Promise<ReadableStream>;
52
+ deleteBlob(blobId: string): Promise<void>;
53
+ blobExists(blobId: string): Promise<boolean>;
54
+ getBlobProperties(blobId: string): Promise<BlobProperties>;
55
+ listBlobs(options?: ListBlobsOptions): Promise<ListBlobsResult>;
56
+ generateSignedUrl(blobId: string, expiresIn: number, permissions?: 'read' | 'delete'): Promise<string>;
57
+ }
58
+ ```
59
+
60
+ **Benefits:**
61
+ - Implementations can be swapped without changing client code
62
+ - Easy to add new storage backends
63
+ - Consistent behavior across all implementations
64
+ - Simplified testing with mocks
65
+
66
+ ### 2. Content-Addressable Storage
67
+
68
+ Blobs are identified by SHA256 hash of their content, not by user-defined names.
69
+
70
+ **Advantages:**
71
+ - **Automatic Deduplication**: Same content = same ID
72
+ - **Data Integrity**: Hash serves as checksum
73
+ - **Location Independence**: Content can be verified anywhere
74
+ - **Efficient Caching**: Content can be cached safely by hash
75
+
76
+ **Trade-offs:**
77
+ - Cannot choose blob IDs
78
+ - Metadata must be stored separately
79
+ - Changing content changes ID
80
+
81
+ ### 3. Flat Storage Pool
82
+
83
+ No hierarchy, containers, or folders at the Bs level:
84
+
85
+ ```
86
+ Traditional: Content-Addressable:
87
+ ├── container1/ All blobs in flat pool:
88
+ │ ├── file1.txt ├── dffd6021bb2bd5b0...
89
+ │ └── file2.txt ├── a591a6d40bf420...<br/>└── container2/ └── e3b0c44298fc1c1...
90
+ └── file3.txt
91
+ ```
92
+
93
+ **Benefits:**
94
+ - Simpler implementation
95
+ - Natural deduplication
96
+ - No path traversal issues
97
+ - Efficient lookups
98
+
99
+ **Metadata Storage:**
100
+ Organizational metadata (folders, tags, filenames) is stored separately, typically in `@rljson/io` (table storage).
101
+
102
+ ### 4. Composition Over Inheritance
103
+
104
+ Complex behaviors are achieved by composing simple components:
105
+
106
+ ```typescript
107
+ // Composition: BsMulti wraps multiple Bs instances
108
+ const bs = new BsMulti([
109
+ { bs: localCache, priority: 0 }, // Simple BsMem
110
+ { bs: remotePeer, priority: 1 }, // Simple BsPeer
111
+ ]);
112
+
113
+ // Not inheritance: class BsCachedRemote extends Bs
114
+ ```
115
+
116
+ ## Component Architecture
117
+
118
+ ### Layered Architecture Diagram
119
+
120
+ ```
121
+ ┌─────────────────────────────────────────────────────────┐
122
+ │ Application Layer │
123
+ │ (Your code using @rljson/bs) │
124
+ └──────────────────┬──────────────────────────────────────┘
125
+
126
+ ┌──────────────────▼──────────────────────────────────────┐
127
+ │ Composition Layer │
128
+ │ │
129
+ │ ┌─────────┐ Combines multiple storage backends │
130
+ │ │BsMulti │ - Priority-based reads │
131
+ │ │ │ - Parallel writes │
132
+ │ └────┬────┘ - Hot-swapping/caching │
133
+ │ │ │
134
+ └───────┼─────────────────────────────────────────────────┘
135
+
136
+ ├──────┬──────┬──────┬──────┐
137
+ │ │ │ │ │
138
+ ┌───────▼──┐ ┌─▼────┐ ┌────▼┐ ┌────▼─────┐
139
+ │ BsMem │ │BsPeer│ │... │ │ Custom │ <- Storage Layer
140
+ │ (local) │ │(net) │ │ │ │ Impl │
141
+ └──────────┘ └──┬───┘ └─────┘ └──────────┘
142
+
143
+ ┌───────▼────────┐
144
+ │ Socket Layer │ <- Transport Layer
145
+ │ (abstraction) │
146
+ └───────┬────────┘
147
+
148
+ ┌───────▼────────┐
149
+ │ BsServer │ <- Server Layer
150
+ │ BsPeerBridge │
151
+ └────────────────┘
152
+ ```
153
+
154
+ ### Component Responsibilities
155
+
156
+ #### 1. **Bs Interface** (`bs.ts`)
157
+
158
+ The core interface that all implementations must follow.
159
+
160
+ **Responsibilities:**
161
+ - Define the contract for blob storage operations
162
+ - Specify type definitions for parameters and return values
163
+ - Document expected behavior
164
+
165
+ **Key Types:**
166
+ - `BlobProperties`: Metadata about a blob
167
+ - `DownloadBlobOptions`: Options for retrieving blobs
168
+ - `ListBlobsOptions`: Options for listing blobs
169
+ - `ListBlobsResult`: Result structure for list operations
170
+
171
+ #### 2. **BsMem** (`bs-mem.ts`)
172
+
173
+ In-memory implementation of the Bs interface.
174
+
175
+ **Responsibilities:**
176
+ - Store blobs in a Map (memory)
177
+ - Calculate SHA256 hashes
178
+ - Handle deduplication automatically
179
+ - Provide fast, ephemeral storage
180
+
181
+ **Internal Structure:**
182
+ ```typescript
183
+ class BsMem implements Bs {
184
+ private readonly blobs = new Map<string, StoredBlob>();
185
+
186
+ // StoredBlob structure:
187
+ interface StoredBlob {
188
+ content: Buffer;
189
+ properties: BlobProperties;
190
+ }
191
+ }
192
+ ```
193
+
194
+ **Key Operations:**
195
+ - `setBlob`: Hash content → Check if exists → Store if new
196
+ - `getBlob`: Lookup by blobId → Return content + properties
197
+ - `listBlobs`: Convert Map entries → Sort → Filter → Paginate
198
+
199
+ #### 3. **BsPeer** (`bs-peer.ts`)
200
+
201
+ Client-side implementation that accesses remote storage via socket.
202
+
203
+ **Responsibilities:**
204
+ - Translate Bs method calls to socket messages
205
+ - Handle async socket communication
206
+ - Manage connection state
207
+ - Convert between local types and wire format
208
+
209
+ **Architecture Pattern:**
210
+ ```
211
+ BsPeer (Client) BsServer (Remote)
212
+ │ │
213
+ │ socket.emit('setBlob', ...) │
214
+ ├──────────────────────────────────>│
215
+ │ │ storage.setBlob(...)
216
+ │ callback(null, result) │
217
+ │<──────────────────────────────────┤
218
+ │ │
219
+ ```
220
+
221
+ **Connection Lifecycle:**
222
+ 1. `init()`: Connect socket and wait for 'connect' event
223
+ 2. Operations: Send requests via socket, receive callbacks
224
+ 3. `close()`: Disconnect socket and wait for 'disconnect' event
225
+
226
+ **Error Handling:**
227
+ - Uses error-first callbacks: `(error, result)`
228
+ - Network errors propagated as promise rejections
229
+ - Connection state tracked via `isOpen` property
230
+
231
+ #### 4. **BsServer** (`bs-server.ts`)
232
+
233
+ Server-side component that exposes a Bs instance over sockets.
234
+
235
+ **Responsibilities:**
236
+ - Accept socket connections from multiple clients
237
+ - Route socket messages to underlying Bs methods
238
+ - Translate results back to socket responses
239
+ - Handle multiple concurrent clients
240
+
241
+ **Transport Layer Generation:**
242
+ ```typescript
243
+ private _generateTransportLayer(bs: Bs) {
244
+ return {
245
+ setBlob: (content) => bs.setBlob(content),
246
+ getBlob: (blobId, options) => bs.getBlob(blobId, options),
247
+ // ... map all Bs methods
248
+ };
249
+ }
250
+ ```
251
+
252
+ **Multi-Client Support:**
253
+ ```typescript
254
+ // Each client has its own socket
255
+ await server.addSocket(clientSocket1);
256
+ await server.addSocket(clientSocket2);
257
+
258
+ // Clients share the same underlying Bs instance
259
+ // Content-addressable nature ensures consistency
260
+ ```
261
+
262
+ #### 5. **BsPeerBridge** (`bs-peer-bridge.ts`)
263
+
264
+ **CRITICAL: Read-Only PULL Architecture**
265
+
266
+ Exposes local storage for remote access (server pulls from client).
267
+
268
+ **Responsibilities:**
269
+ - Listen for socket events from server
270
+ - Translate events to Bs method calls on local storage
271
+ - **Only expose READ operations** (PULL pattern)
272
+ - Use error-first callback pattern
273
+
274
+ **Architectural Pattern:**
275
+ ```
276
+ Client Server
277
+ ┌─────────────────┐ ┌──────────────────┐
278
+ │ Local Storage │ │ Server Bs │
279
+ │ (BsMem) │ │ │
280
+ └────────┬────────┘ └────────┬─────────┘
281
+ │ │
282
+ │ │
283
+ ┌────▼─────────┐ ┌────▼────────┐
284
+ │BsPeerBridge │◄───Socket (READ)───│ BsPeer │
285
+ │ (Exposes │ │ (Requests) │
286
+ │ READ only) │ │ │
287
+ └──────────────┘ └─────────────┘
288
+
289
+ Client exposes local storage for SERVER to PULL from (read-only)
290
+ Server CANNOT push writes to client
291
+ ```
292
+
293
+ **Read-Only Methods Exposed:**
294
+ - `getBlob` - Read blob content
295
+ - `getBlobStream` - Stream blob content
296
+ - `blobExists` - Check existence
297
+ - `getBlobProperties` - Get metadata
298
+ - `listBlobs` - List available blobs
299
+
300
+ **NOT Exposed (Write Operations):**
301
+ - `setBlob` - Would allow server to write to client
302
+ - `deleteBlob` - Would allow server to delete from client
303
+ - `generateSignedUrl` - Management operation
304
+
305
+ **Implementation:**
306
+ ```typescript
307
+ private _registerBsMethods(): void {
308
+ const bsMethods = [
309
+ 'getBlob', // ✅ READ
310
+ 'getBlobStream', // ✅ READ
311
+ 'blobExists', // ✅ READ
312
+ 'getBlobProperties', // ✅ READ
313
+ 'listBlobs', // ✅ READ
314
+ // NOT: 'setBlob', 'deleteBlob', 'generateSignedUrl'
315
+ ];
316
+
317
+ for (const methodName of bsMethods) {
318
+ this.registerEvent(methodName);
319
+ }
320
+ }
321
+ ```
322
+
323
+ **Callback Pattern (Error-First):**
324
+ ```typescript
325
+ const handler = (...args: any[]) => {
326
+ const callback = args[args.length - 1];
327
+ const methodArgs = args.slice(0, -1);
328
+
329
+ bsMethod.apply(this._bs, methodArgs)
330
+ .then((result) => {
331
+ callback(null, result); // ✅ Error-first: (error, result)
332
+ })
333
+ .catch((error) => {
334
+ callback(error, null); // ✅ Error-first: (error, result)
335
+ });
336
+ };
337
+ ```
338
+
339
+ **Why Read-Only?**
340
+
341
+ 1. **Security**: Prevents server from pushing malicious data to clients
342
+ 2. **Architectural Clarity**: Clear unidirectional data flow (PULL)
343
+ 3. **Consistency with IoPeerBridge**: Matches the pattern from `@rljson/io`
344
+ 4. **Client Control**: Client owns its local storage, server can only read
345
+
346
+ #### 6. **BsMulti** (`bs-multi.ts`)
347
+
348
+ Multi-tier storage that composes multiple Bs instances.
349
+
350
+ **Responsibilities:**
351
+ - Manage multiple storage backends with priorities
352
+ - Route reads to highest priority readable
353
+ - Write to all writable stores in parallel
354
+ - Implement hot-swapping (cache population)
355
+ - Merge results from multiple stores
356
+
357
+ **Priority System:**
358
+ ```typescript
359
+ type BsMultiBs = {
360
+ bs: Bs;
361
+ id?: string;
362
+ priority: number; // Lower number = higher priority
363
+ read: boolean; // Can read from this store
364
+ write: boolean; // Can write to this store
365
+ };
366
+ ```
367
+
368
+ **Read Strategy:**
369
+ 1. Sort stores by priority (ascending)
370
+ 2. Try each readable store in order
371
+ 3. First successful read wins
372
+ 4. Hot-swap: Write to all writable stores for caching
373
+ 5. Return result
374
+
375
+ **Write Strategy:**
376
+ 1. Collect all writable stores
377
+ 2. Write to all in parallel using `Promise.all`
378
+ 3. All writes must succeed (or error)
379
+ 4. Content-addressable ensures consistency
380
+
381
+ **Hot-Swapping:**
382
+ ```typescript
383
+ async getBlob(blobId: string): Promise<{ content: Buffer; properties: BlobProperties }> {
384
+ // Try stores in priority order
385
+ for (const readable of this.readables) {
386
+ try {
387
+ result = await readable.bs.getBlob(blobId);
388
+ readFrom = readable.id;
389
+ break;
390
+ } catch (e) {
391
+ continue;
392
+ }
393
+ }
394
+
395
+ // Hot-swap: Cache in all writables (except source)
396
+ const hotSwapWrites = this.writables
397
+ .filter((writable) => writable.id !== readFrom)
398
+ .map(({ bs }) => bs.setBlob(result.content).catch(() => {}));
399
+
400
+ await Promise.all(hotSwapWrites);
401
+ return result;
402
+ }
403
+ ```
404
+
405
+ **List Strategy (Deduplication):**
406
+ ```typescript
407
+ async listBlobs(): Promise<ListBlobsResult> {
408
+ // Query all readable stores in parallel
409
+ const allResults = await Promise.all(
410
+ this.readables.map(({ bs }) => bs.listBlobs())
411
+ );
412
+
413
+ // Deduplicate by blobId (Set)
414
+ const uniqueBlobs = new Map<string, BlobProperties>();
415
+ for (const result of allResults) {
416
+ for (const blob of result.blobs) {
417
+ uniqueBlobs.set(blob.blobId, blob);
418
+ }
419
+ }
420
+
421
+ return { blobs: Array.from(uniqueBlobs.values()) };
422
+ }
423
+ ```
424
+
425
+ ## Content-Addressable Storage
426
+
427
+ ### SHA256 Hashing
428
+
429
+ Content addressing uses SHA256 to generate a unique identifier for each blob.
430
+
431
+ **Hash Calculation:**
432
+ ```typescript
433
+ import { createHash } from 'crypto';
434
+
435
+ function calculateBlobId(content: Buffer): string {
436
+ return createHash('sha256').update(content).digest('hex');
437
+ }
438
+ ```
439
+
440
+ **Properties:**
441
+ - **Deterministic**: Same content always produces same hash
442
+ - **Collision-Resistant**: Practically impossible to find two different contents with same hash
443
+ - **Fixed Length**: Always 64 hexadecimal characters (256 bits)
444
+ - **One-Way**: Cannot derive content from hash
445
+
446
+ ### Deduplication Mechanics
447
+
448
+ **Automatic Deduplication:**
449
+ ```typescript
450
+ async setBlob(content: Buffer | string | ReadableStream): Promise<BlobProperties> {
451
+ const buffer = await this.toBuffer(content);
452
+ const blobId = hshBuffer(buffer);
453
+
454
+ // Check if blob already exists
455
+ const existing = this.blobs.get(blobId);
456
+ if (existing) {
457
+ return existing.properties; // Return existing, don't store again
458
+ }
459
+
460
+ // Store new blob
461
+ const properties: BlobProperties = {
462
+ blobId,
463
+ size: buffer.length,
464
+ createdAt: new Date(),
465
+ };
466
+
467
+ this.blobs.set(blobId, { content: buffer, properties });
468
+ return properties;
469
+ }
470
+ ```
471
+
472
+ **Benefits in Multi-Tier:**
473
+ ```typescript
474
+ // Same content stored in multiple tiers = same blobId
475
+ // BsMulti writes to all writables, but no duplicate storage cost
476
+ // because content-addressable storage deduplicates automatically
477
+
478
+ await bsMulti.setBlob('same content'); // Writes to tier1, tier2
479
+ await bsMulti.setBlob('same content'); // No-op (already exists)
480
+ ```
481
+
482
+ ### Content Verification
483
+
484
+ Any party can verify content integrity:
485
+
486
+ ```typescript
487
+ async verifyBlob(blobId: string, content: Buffer): boolean {
488
+ const computedId = calculateBlobId(content);
489
+ return computedId === blobId;
490
+ }
491
+ ```
492
+
493
+ ## Network Architecture
494
+
495
+ ### Socket Abstraction
496
+
497
+ The `Socket` interface abstracts network transport:
498
+
499
+ ```typescript
500
+ export interface Socket {
501
+ connected: boolean;
502
+ disconnected: boolean;
503
+ connect(): void;
504
+ disconnect(): void;
505
+ on(eventName: string | symbol, listener: (...args: any[]) => void): this;
506
+ emit(eventName: string | symbol, ...args: any[]): boolean | this;
507
+ off(eventName: string | symbol, listener: (...args: any[]) => void): this;
508
+ removeAllListeners(eventName?: string | symbol): this;
509
+ }
510
+ ```
511
+
512
+ **Implementations:**
513
+ - `SocketMock`: In-process mock for testing (synchronous)
514
+ - `PeerSocketMock`: Direct Bs invocation (testing without network)
515
+ - `DirectionalSocketMock`: Bidirectional socket pair (testing)
516
+ - **Production**: Use Socket.IO, WebSocket, or custom implementation
517
+
518
+ ### Protocol
519
+
520
+ **Message Format (Socket Events):**
521
+ ```
522
+ Event: <methodName>
523
+ Args: [...methodArgs, callback]
524
+
525
+ Example:
526
+ socket.emit('getBlob', blobId, options, (error, result) => {
527
+ if (error) console.error(error);
528
+ else console.log(result);
529
+ });
530
+ ```
531
+
532
+ **Error-First Callbacks:**
533
+ ```typescript
534
+ callback(error: Error | null, result?: T);
535
+
536
+ // Success
537
+ callback(null, { content: buffer, properties: props });
538
+
539
+ // Error
540
+ callback(new Error('Blob not found'), null);
541
+ ```
542
+
543
+ ### Client-Server Flow
544
+
545
+ **Complete Request/Response Cycle:**
546
+ ```
547
+ 1. Client: BsPeer.getBlob(blobId)
548
+ ├─> Create promise
549
+ └─> socket.emit('getBlob', blobId, callback)
550
+
551
+ 2. Network: Transport socket event
552
+
553
+ 3. Server: BsServer receives 'getBlob' event
554
+ ├─> Extract args and callback
555
+ ├─> Call underlying bs.getBlob(blobId)
556
+ ├─> Wait for result
557
+ └─> Invoke callback(null, result)
558
+
559
+ 4. Network: Transport callback response
560
+
561
+ 5. Client: BsPeer callback invoked
562
+ ├─> Resolve or reject promise
563
+ └─> Return to caller
564
+ ```
565
+
566
+ ## PULL vs PUSH Architecture
567
+
568
+ ### PULL Architecture (Recommended)
569
+
570
+ **Pattern: Server pulls data from client when needed**
571
+
572
+ ```
573
+ ┌─────────┐ ┌────────┐
574
+ │ Client │ │ Server │
575
+ │ │ │ │
576
+ │ Local │◄────READ ONLY──────│ Peer │
577
+ │ Store │ (BsPeerBridge) │ │
578
+ │ │ │ │
579
+ └─────────┘ └────────┘
580
+
581
+ Server requests: "Give me blob X"
582
+ Client responds: "Here is blob X"
583
+ Server CANNOT push data to client
584
+ ```
585
+
586
+ **Implementation:**
587
+ ```typescript
588
+ // Client exposes local storage via BsPeerBridge (read-only)
589
+ const bridge = new BsPeerBridge(localStorage, socket);
590
+ bridge.start(); // Only read operations exposed
591
+
592
+ // Server pulls from client
593
+ const peer = new BsPeer(socket);
594
+ const { content } = await peer.getBlob(blobId); // ✅ Can read
595
+ // await peer.setBlob('data'); // ❌ Would fail if bridge is read-only on client
596
+ ```
597
+
598
+ **Benefits:**
599
+ - ✅ Client controls its data
600
+ - ✅ Server cannot inject malicious content
601
+ - ✅ Clear unidirectional data flow
602
+ - ✅ Matches real-world HTTP patterns (server pulls via GET)
603
+
604
+ ### PUSH Architecture (Anti-Pattern for BsPeerBridge)
605
+
606
+ **Pattern: Server pushes data to client (NOT RECOMMENDED)**
607
+
608
+ ```
609
+ ┌─────────┐ ┌────────┐
610
+ │ Client │ │ Server │
611
+ │ │ │ │
612
+ │ Local │◄────READ/WRITE─────│ Peer │
613
+ │ Store │ (Full Access) │ │
614
+ │ │ │ │
615
+ └─────────┘ └────────┘
616
+
617
+ Server command: "Store this blob"
618
+ Client stores: "Blob stored"
619
+ Server can write to client ❌
620
+ ```
621
+
622
+ **Why NOT to expose writes in BsPeerBridge:**
623
+ - ❌ Security risk: Server can push malicious data
624
+ - ❌ Unclear data ownership
625
+ - ❌ Violates principle of least privilege
626
+ - ❌ Client loses control over its storage
627
+
628
+ **Correct Alternative:**
629
+ If client wants to send data to server, use `BsPeer` for client-to-server writes:
630
+
631
+ ```typescript
632
+ // Client pushes to server using BsPeer
633
+ const serverPeer = new BsPeer(socketToServer);
634
+ await serverPeer.setBlob('data'); // Client-initiated write to server
635
+ ```
636
+
637
+ ### Comparison with IoPeerBridge
638
+
639
+ `BsPeerBridge` follows the same read-only pattern as `IoPeerBridge` from `@rljson/io`:
640
+
641
+ **IoPeerBridge (Table Storage):**
642
+ ```typescript
643
+ // Only exposes reads:
644
+ - readRows // ✅ READ
645
+ - tableExists // ✅ READ
646
+ - rawTableCfgs // ✅ READ
647
+ // NOT: insertRow, updateRow, deleteRow
648
+ ```
649
+
650
+ **BsPeerBridge (Blob Storage):**
651
+ ```typescript
652
+ // Only exposes reads:
653
+ - getBlob // ✅ READ
654
+ - getBlobStream // ✅ READ
655
+ - blobExists // ✅ READ
656
+ - getBlobProperties // ✅ READ
657
+ - listBlobs // ✅ READ
658
+ // NOT: setBlob, deleteBlob, generateSignedUrl
659
+ ```
660
+
661
+ **Consistency:** Both use PULL architecture for client-server interactions.
662
+
663
+ ## Multi-Tier Storage
664
+
665
+ ### Hierarchical Storage Management (HSM)
666
+
667
+ BsMulti implements a form of HSM with automatic tier management:
668
+
669
+ ```
670
+ Priority 0 (Highest)
671
+ ├─ Fast local cache (BsMem)
672
+ │ └─ Small, fast, volatile
673
+
674
+ Priority 1
675
+ ├─ Network storage (BsPeer)
676
+ │ └─ Larger, slower, persistent
677
+
678
+ Priority 2 (Lowest)
679
+ ├─ Backup/Archive (Read-only)
680
+ └─ Largest, slowest, persistent
681
+ ```
682
+
683
+ ### Read Path with Hot-Swapping
684
+
685
+ ```
686
+ 1. Request: getBlob(blobId)
687
+
688
+ 2. Check Priority 0 (cache)
689
+ ├─ HIT: Return immediately ✅
690
+ └─ MISS: Continue to Priority 1
691
+
692
+ 3. Check Priority 1 (network)
693
+ ├─ HIT:
694
+ │ ├─ Return content
695
+ │ └─ Hot-swap to Priority 0 (cache for next time)
696
+ └─ MISS: Continue to Priority 2
697
+
698
+ 4. Check Priority 2 (backup)
699
+ ├─ HIT:
700
+ │ ├─ Return content
701
+ │ └─ Hot-swap to Priority 0 and 1
702
+ └─ MISS: Throw "Blob not found"
703
+ ```
704
+
705
+ **Code:**
706
+ ```typescript
707
+ async getBlob(blobId: string): Promise<{ content: Buffer; properties: BlobProperties }> {
708
+ for (const readable of this.readables) {
709
+ try {
710
+ result = await readable.bs.getBlob(blobId);
711
+ readFrom = readable.id;
712
+ break;
713
+ } catch (e) {
714
+ continue; // Try next priority
715
+ }
716
+ }
717
+
718
+ if (!result) {
719
+ throw new Error(`Blob not found: ${blobId}`);
720
+ }
721
+
722
+ // Hot-swap to all writables (except source)
723
+ const hotSwapWrites = this.writables
724
+ .filter((writable) => writable.id !== readFrom)
725
+ .map(({ bs }) => bs.setBlob(result.content).catch(() => {}));
726
+
727
+ await Promise.all(hotSwapWrites);
728
+ return result;
729
+ }
730
+ ```
731
+
732
+ ### Write Path (Parallel Replication)
733
+
734
+ ```
735
+ 1. Request: setBlob(content)
736
+
737
+ 2. Collect all writable stores
738
+ ├─ Priority 0: Local cache (write=true)
739
+ ├─ Priority 1: Network storage (write=true)
740
+ └─ Priority 2: Backup (write=false) ← Skip
741
+
742
+ 3. Write to all writables in parallel
743
+ ├─ Promise.all([
744
+ │ cache.setBlob(content),
745
+ │ network.setBlob(content),
746
+ │ ])
747
+
748
+ 4. All must succeed or error propagates
749
+ ├─ Content-addressable ensures same blobId everywhere
750
+ └─ Deduplication prevents storage waste
751
+ ```
752
+
753
+ ### List Operations (Deduplication)
754
+
755
+ ```
756
+ 1. Request: listBlobs()
757
+
758
+ 2. Query all readable stores in parallel
759
+ ├─ Promise.all([
760
+ │ cache.listBlobs(),
761
+ │ network.listBlobs(),
762
+ │ backup.listBlobs(),
763
+ │ ])
764
+
765
+ 3. Deduplicate results by blobId
766
+ ├─ Use Map<blobId, BlobProperties>
767
+ └─ Content-addressable means same blobId = same content
768
+
769
+ 4. Return merged list
770
+ ```
771
+
772
+ ## Socket Abstraction Layer
773
+
774
+ ### Design Philosophy
775
+
776
+ The Socket interface abstracts the transport layer, allowing:
777
+
778
+ 1. **Different transports**: WebSocket, Socket.IO, custom protocols
779
+ 2. **Testing**: In-process mocks without network
780
+ 3. **Flexibility**: Swap implementations without changing Bs code
781
+
782
+ ### Mock Implementations
783
+
784
+ #### SocketMock (Basic)
785
+
786
+ Simulates a socket with EventEmitter-like behavior:
787
+
788
+ ```typescript
789
+ class SocketMock implements Socket {
790
+ private _listeners: Map<string | symbol, Array<(...args: any[]) => void>>;
791
+
792
+ emit(eventName: string | symbol, ...args: any[]): boolean {
793
+ const listeners = this._listeners.get(eventName) || [];
794
+ for (const listener of listeners) {
795
+ listener(...args);
796
+ }
797
+ return listeners.length > 0;
798
+ }
799
+ }
800
+ ```
801
+
802
+ **Use:** General-purpose testing, client-server simulation.
803
+
804
+ #### PeerSocketMock (Direct)
805
+
806
+ Directly invokes Bs methods without socket events:
807
+
808
+ ```typescript
809
+ class PeerSocketMock implements Socket {
810
+ constructor(private _bs: Bs) {}
811
+
812
+ emit(eventName: string, ...args: any[]): this {
813
+ const callback = args[args.length - 1];
814
+ const methodArgs = args.slice(0, -1);
815
+
816
+ // Direct method invocation
817
+ this._bs[eventName](...methodArgs)
818
+ .then(result => callback(null, result))
819
+ .catch(error => callback(error, null));
820
+
821
+ return this;
822
+ }
823
+ }
824
+ ```
825
+
826
+ **Use:** Testing BsPeer without BsServer, fastest mock.
827
+
828
+ #### DirectionalSocketMock (Bidirectional)
829
+
830
+ Creates a pair of connected sockets:
831
+
832
+ ```typescript
833
+ const socket1 = new DirectionalSocketMock();
834
+ const socket2 = new DirectionalSocketMock();
835
+
836
+ socket1.setPeer(socket2);
837
+ socket2.setPeer(socket1);
838
+
839
+ // Events on socket1 trigger listeners on socket2
840
+ socket1.emit('event', 'data'); // socket2 receives 'event'
841
+ ```
842
+
843
+ **Use:** Testing full client-server interaction, network simulation.
844
+
845
+ ### Production Integration
846
+
847
+ **Socket.IO Example:**
848
+ ```typescript
849
+ import { io, Socket as SocketIOSocket } from 'socket.io-client';
850
+
851
+ // Wrap Socket.IO in Socket interface
852
+ class SocketIOAdapter implements Socket {
853
+ constructor(private _socket: SocketIOSocket) {}
854
+
855
+ get connected() { return this._socket.connected; }
856
+ get disconnected() { return this._socket.disconnected; }
857
+ connect() { this._socket.connect(); }
858
+ disconnect() { this._socket.disconnect(); }
859
+ on(event, listener) { this._socket.on(event, listener); return this; }
860
+ emit(event, ...args) { this._socket.emit(event, ...args); return this; }
861
+ off(event, listener) { this._socket.off(event, listener); return this; }
862
+ removeAllListeners(event) { this._socket.removeAllListeners(event); return this; }
863
+ }
864
+
865
+ // Use with BsPeer
866
+ const socket = new SocketIOAdapter(io('http://server'));
867
+ const peer = new BsPeer(socket);
868
+ await peer.init();
869
+ ```
870
+
871
+ ## Implementation Details
872
+
873
+ ### Error Handling
874
+
875
+ **Error Propagation:**
876
+ ```typescript
877
+ // BsMem: Synchronous errors
878
+ if (!this.blobs.has(blobId)) {
879
+ throw new Error(`Blob not found: ${blobId}`);
880
+ }
881
+
882
+ // BsPeer: Async errors from callbacks
883
+ socket.emit('getBlob', blobId, (error, result) => {
884
+ if (error) reject(error);
885
+ else resolve(result);
886
+ });
887
+
888
+ // BsMulti: Fallback on errors
889
+ for (const readable of this.readables) {
890
+ try {
891
+ return await readable.bs.getBlob(blobId);
892
+ } catch (e) {
893
+ // Try next priority
894
+ continue;
895
+ }
896
+ }
897
+ throw new Error('Blob not found in all stores');
898
+ ```
899
+
900
+ **Error Types:**
901
+ - `Blob not found`: blobId doesn't exist
902
+ - `No readable/writable Bs available`: BsMulti configuration error
903
+ - `Method not found`: BsPeerBridge received unknown method
904
+ - Network errors: Socket connection failures
905
+
906
+ ### Stream Handling
907
+
908
+ **Buffer to Stream Conversion:**
909
+ ```typescript
910
+ async getBlobStream(blobId: string): Promise<ReadableStream> {
911
+ const stored = this.blobs.get(blobId);
912
+ if (!stored) {
913
+ throw new Error(`Blob not found: ${blobId}`);
914
+ }
915
+
916
+ // Node.js Readable to Web ReadableStream
917
+ const nodeStream = Readable.from(stored.content);
918
+ return Readable.toWeb(nodeStream) as ReadableStream;
919
+ }
920
+ ```
921
+
922
+ **Stream to Buffer Conversion:**
923
+ ```typescript
924
+ private async toBuffer(content: ReadableStream): Promise<Buffer> {
925
+ const reader = content.getReader();
926
+ const chunks: Uint8Array[] = [];
927
+
928
+ while (true) {
929
+ const { done, value } = await reader.read();
930
+ if (done) break;
931
+ chunks.push(value);
932
+ }
933
+
934
+ return Buffer.concat(chunks);
935
+ }
936
+ ```
937
+
938
+ ### Pagination
939
+
940
+ **List Blobs with Continuation:**
941
+ ```typescript
942
+ async listBlobs(options?: ListBlobsOptions): Promise<ListBlobsResult> {
943
+ let blobs = Array.from(this.blobs.values()).map(s => s.properties);
944
+
945
+ // Filter by prefix
946
+ if (options?.prefix) {
947
+ blobs = blobs.filter(b => b.blobId.startsWith(options.prefix!));
948
+ }
949
+
950
+ // Sort for consistent ordering
951
+ blobs.sort((a, b) => a.blobId.localeCompare(b.blobId));
952
+
953
+ // Pagination
954
+ let startIndex = 0;
955
+ if (options?.continuationToken) {
956
+ startIndex = parseInt(options.continuationToken, 10);
957
+ }
958
+
959
+ const maxResults = options?.maxResults || blobs.length;
960
+ const endIndex = startIndex + maxResults;
961
+ const pageBlobs = blobs.slice(startIndex, endIndex);
962
+
963
+ const result: ListBlobsResult = { blobs: pageBlobs };
964
+
965
+ if (endIndex < blobs.length) {
966
+ result.continuationToken = endIndex.toString();
967
+ }
968
+
969
+ return result;
970
+ }
971
+ ```
972
+
973
+ ## Design Patterns
974
+
975
+ ### 1. Strategy Pattern (Storage Backend)
976
+
977
+ Different implementations of Bs interface:
978
+
979
+ ```typescript
980
+ interface Bs {
981
+ setBlob(...): Promise<BlobProperties>;
982
+ getBlob(...): Promise<{ content: Buffer; properties: BlobProperties }>;
983
+ // ...
984
+ }
985
+
986
+ // Strategies:
987
+ class BsMem implements Bs { /* In-memory */ }
988
+ class BsPeer implements Bs { /* Network */ }
989
+ class BsMulti implements Bs { /* Composite */ }
990
+ ```
991
+
992
+ **Benefit:** Swap storage strategies without changing client code.
993
+
994
+ ### 2. Composite Pattern (BsMulti)
995
+
996
+ BsMulti composes multiple Bs instances:
997
+
998
+ ```typescript
999
+ class BsMulti implements Bs {
1000
+ constructor(private _stores: Array<BsMultiBs>) {}
1001
+
1002
+ async getBlob(blobId: string) {
1003
+ for (const store of this._stores) {
1004
+ try {
1005
+ return await store.bs.getBlob(blobId); // Delegate
1006
+ } catch (e) {
1007
+ continue;
1008
+ }
1009
+ }
1010
+ throw new Error('Blob not found');
1011
+ }
1012
+ }
1013
+ ```
1014
+
1015
+ **Benefit:** Treat single and composite storage uniformly.
1016
+
1017
+ ### 3. Proxy Pattern (BsPeer, BsServer)
1018
+
1019
+ BsPeer proxies requests to remote Bs:
1020
+
1021
+ ```typescript
1022
+ class BsPeer implements Bs {
1023
+ async getBlob(blobId: string) {
1024
+ // Proxy to remote via socket
1025
+ return new Promise((resolve, reject) => {
1026
+ this._socket.emit('getBlob', blobId, (error, result) => {
1027
+ if (error) reject(error);
1028
+ else resolve(result);
1029
+ });
1030
+ });
1031
+ }
1032
+ }
1033
+ ```
1034
+
1035
+ **Benefit:** Network transparency - remote storage looks like local.
1036
+
1037
+ ### 4. Bridge Pattern (BsPeerBridge)
1038
+
1039
+ BsPeerBridge bridges socket events to Bs method calls:
1040
+
1041
+ ```typescript
1042
+ class BsPeerBridge {
1043
+ constructor(private _bs: Bs, private _socket: Socket) {}
1044
+
1045
+ start() {
1046
+ this._socket.on('getBlob', (blobId, callback) => {
1047
+ this._bs.getBlob(blobId)
1048
+ .then(result => callback(null, result))
1049
+ .catch(error => callback(error, null));
1050
+ });
1051
+ }
1052
+ }
1053
+ ```
1054
+
1055
+ **Benefit:** Decouple socket abstraction from Bs implementation.
1056
+
1057
+ ### 5. Adapter Pattern (Socket Implementations)
1058
+
1059
+ Different socket libraries adapted to Socket interface:
1060
+
1061
+ ```typescript
1062
+ // SocketMock adapts Map to Socket
1063
+ class SocketMock implements Socket { /* ... */ }
1064
+
1065
+ // SocketIOAdapter adapts Socket.IO to Socket
1066
+ class SocketIOAdapter implements Socket {
1067
+ constructor(private _socket: SocketIOSocket) {}
1068
+ // Adapt methods...
1069
+ }
1070
+ ```
1071
+
1072
+ **Benefit:** Use any socket library with Bs components.
1073
+
1074
+ ## Performance Optimization
1075
+
1076
+ ### 1. Deduplication
1077
+
1078
+ **Space Efficiency:**
1079
+ - Same content stored only once
1080
+ - 10 copies of same file = 1 storage cost
1081
+ - Particularly effective for:
1082
+ - Version control (many unchanged files)
1083
+ - Backup systems (incremental with dedup)
1084
+ - Build artifacts (shared dependencies)
1085
+
1086
+ **Example:**
1087
+ ```typescript
1088
+ // Traditional storage (10 copies = 10x space)
1089
+ await bs.put('file1', content);
1090
+ await bs.put('file2', content); // Same content, different name
1091
+ // ... 8 more copies
1092
+
1093
+ // Content-addressable (10 copies = 1x space)
1094
+ const { blobId: id1 } = await bs.setBlob(content);
1095
+ const { blobId: id2 } = await bs.setBlob(content); // Same blobId
1096
+ console.log(id1 === id2); // true - only stored once
1097
+ ```
1098
+
1099
+ ### 2. Hot-Swapping (Cache Population)
1100
+
1101
+ **Automatic Caching:**
1102
+ ```typescript
1103
+ // First read from remote (slow)
1104
+ const result = await bsMulti.getBlob(blobId); // Fetches from network
1105
+
1106
+ // Hot-swapped to cache automatically
1107
+
1108
+ // Second read from cache (fast)
1109
+ const result2 = await bsMulti.getBlob(blobId); // Instant from cache
1110
+ ```
1111
+
1112
+ **Benefits:**
1113
+ - Transparent caching
1114
+ - No manual cache management
1115
+ - Reduces network requests
1116
+ - Improves read latency
1117
+
1118
+ ### 3. Parallel Operations
1119
+
1120
+ **Parallel Writes:**
1121
+ ```typescript
1122
+ // BsMulti writes to all writables in parallel
1123
+ await bsMulti.setBlob(content);
1124
+
1125
+ // Instead of serial:
1126
+ // await tier1.setBlob(content);
1127
+ // await tier2.setBlob(content);
1128
+ // await tier3.setBlob(content);
1129
+
1130
+ // Uses Promise.all for parallelism
1131
+ ```
1132
+
1133
+ **Parallel Reads (List):**
1134
+ ```typescript
1135
+ // Query all stores simultaneously
1136
+ const results = await Promise.all(
1137
+ stores.map(store => store.bs.listBlobs())
1138
+ );
1139
+ // Merge results
1140
+ ```
1141
+
1142
+ ### 4. Stream Support
1143
+
1144
+ **Large Blob Handling:**
1145
+ ```typescript
1146
+ // Avoid loading entire blob into memory
1147
+ const stream = await bs.getBlobStream(blobId);
1148
+ const reader = stream.getReader();
1149
+
1150
+ while (true) {
1151
+ const { done, value } = await reader.read();
1152
+ if (done) break;
1153
+ // Process chunk without buffering entire blob
1154
+ processChunk(value);
1155
+ }
1156
+ ```
1157
+
1158
+ **Benefits:**
1159
+ - Constant memory usage
1160
+ - Faster time to first byte
1161
+ - Suitable for large files (GB+)
1162
+
1163
+ ### 5. Range Requests
1164
+
1165
+ **Partial Content Retrieval:**
1166
+ ```typescript
1167
+ // Download only first 1MB of 100MB blob
1168
+ const { content } = await bs.getBlob(blobId, {
1169
+ range: { start: 0, end: 1024 * 1024 - 1 }
1170
+ });
1171
+
1172
+ // Use for:
1173
+ // - Previews (first few KB)
1174
+ // - Resume downloads
1175
+ // - Random access to large files
1176
+ ```
1177
+
1178
+ ## Testing Strategy
1179
+
1180
+ ### Test Pyramid
1181
+
1182
+ ```
1183
+ ┌─────────────┐
1184
+ │ Integration │ Full client-server tests
1185
+ │ Tests │ (bs-integration.spec.ts)
1186
+ └─────────────┘
1187
+ ┌───────────────┐
1188
+ │ Component │ Per-component tests
1189
+ │ Tests │ (bs-mem.spec.ts, etc.)
1190
+ └───────────────┘
1191
+ ┌─────────────────┐
1192
+ │ Unit Tests │ Internal methods, helpers
1193
+ └─────────────────┘
1194
+ ```
1195
+
1196
+ ### Testing Implementations
1197
+
1198
+ **1. BsMem Testing:**
1199
+ ```typescript
1200
+ it('should deduplicate identical content', async () => {
1201
+ const bs = new BsMem();
1202
+ const { blobId: id1 } = await bs.setBlob('same');
1203
+ const { blobId: id2 } = await bs.setBlob('same');
1204
+ expect(id1).toBe(id2);
1205
+ });
1206
+ ```
1207
+
1208
+ **2. BsPeer + BsServer Testing:**
1209
+ ```typescript
1210
+ it('should handle client-server communication', async () => {
1211
+ const storage = new BsMem();
1212
+ const server = new BsServer(storage);
1213
+
1214
+ const socket = new SocketMock();
1215
+ await server.addSocket(socket);
1216
+
1217
+ const client = new BsPeer(socket);
1218
+ await client.init();
1219
+
1220
+ const { blobId } = await client.setBlob('data');
1221
+ const { content } = await client.getBlob(blobId);
1222
+ expect(content.toString()).toBe('data');
1223
+ });
1224
+ ```
1225
+
1226
+ **3. BsMulti Testing:**
1227
+ ```typescript
1228
+ it('should hot-swap from remote to cache', async () => {
1229
+ const cache = new BsMem();
1230
+ const remote = new BsMem();
1231
+
1232
+ // Store in remote only
1233
+ const { blobId } = await remote.setBlob('data');
1234
+
1235
+ const multi = new BsMulti([
1236
+ { bs: cache, priority: 0, read: true, write: true },
1237
+ { bs: remote, priority: 1, read: true, write: false },
1238
+ ]);
1239
+ await multi.init();
1240
+
1241
+ // First read from remote, hot-swaps to cache
1242
+ await multi.getBlob(blobId);
1243
+
1244
+ // Verify cached
1245
+ expect(await cache.blobExists(blobId)).toBe(true);
1246
+ });
1247
+ ```
1248
+
1249
+ **4. BsPeerBridge Testing:**
1250
+ ```typescript
1251
+ it('should only expose read operations', async () => {
1252
+ const localStorage = new BsMem();
1253
+ const socket = new SocketMock();
1254
+ const bridge = new BsPeerBridge(localStorage, socket);
1255
+ bridge.start();
1256
+
1257
+ // Verify read operations are registered
1258
+ const readOps = ['getBlob', 'blobExists', 'getBlobProperties', 'listBlobs', 'getBlobStream'];
1259
+ for (const op of readOps) {
1260
+ const listeners = socket._listeners.get(op);
1261
+ expect(listeners.length).toBeGreaterThan(0);
1262
+ }
1263
+
1264
+ // Verify write operations are NOT registered
1265
+ const writeOps = ['setBlob', 'deleteBlob', 'generateSignedUrl'];
1266
+ for (const op of writeOps) {
1267
+ const listeners = socket._listeners.get(op);
1268
+ expect(listeners).toBeUndefined();
1269
+ }
1270
+ });
1271
+ ```
1272
+
1273
+ ### Conformance Tests
1274
+
1275
+ **Golden Tests:**
1276
+ - Store expected behavior in `test/goldens/`
1277
+ - Compare actual output against golden files
1278
+ - Ensures consistent behavior across versions
1279
+
1280
+ **Coverage:**
1281
+ - Target: 100% code coverage
1282
+ - Tools: Vitest + v8 coverage
1283
+ - All branches, statements, functions tested
1284
+
1285
+ ## Conclusion
1286
+
1287
+ The `@rljson/bs` architecture provides a flexible, composable, and type-safe blob storage system with:
1288
+
1289
+ 1. **Content-addressable storage** for automatic deduplication
1290
+ 2. **Modular components** that can be composed into complex hierarchies
1291
+ 3. **Network transparency** via socket abstraction
1292
+ 4. **PULL architecture** for secure client-server interactions
1293
+ 5. **Multi-tier storage** with automatic caching and hot-swapping
1294
+ 6. **Full test coverage** ensuring reliability
1295
+
1296
+ The system follows established design patterns (Strategy, Composite, Proxy, Bridge, Adapter) and provides a solid foundation for building distributed storage solutions.
1297
+
1298
+ For further details, see:
1299
+ - [README.public.md](README.public.md) - Usage documentation
1300
+ - [README.contributors.md](README.contributors.md) - Development guide
1301
+ - Source code in `src/` directory
1302
+ - Tests in `test/` directory