@rljson/bs 0.0.19 → 0.0.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.architecture.md +1295 -2
- package/README.public.md +324 -169
- package/dist/README.architecture.md +1295 -2
- package/dist/README.public.md +324 -169
- package/dist/bs-peer-bridge.d.ts +2 -0
- package/dist/bs.js +163 -14
- package/dist/directional-socket-mock.d.ts +10 -0
- package/dist/index.d.ts +1 -0
- package/package.json +17 -17
|
@@ -1,9 +1,1302 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
@license
|
|
3
|
-
Copyright (c)
|
|
3
|
+
Copyright (c) 2026 Rljson
|
|
4
4
|
|
|
5
5
|
Use of this source code is governed by terms that can be
|
|
6
6
|
found in the LICENSE file in the root of this package.
|
|
7
7
|
-->
|
|
8
8
|
|
|
9
|
-
# Architecture
|
|
9
|
+
# @rljson/bs - Architecture Documentation
|
|
10
|
+
|
|
11
|
+
Deep dive into the architectural design, implementation patterns, and internal workings of the rljson blob storage system.
|
|
12
|
+
|
|
13
|
+
## Table of Contents
|
|
14
|
+
|
|
15
|
+
1. [Overview](#overview)
|
|
16
|
+
2. [Core Design Principles](#core-design-principles)
|
|
17
|
+
3. [Component Architecture](#component-architecture)
|
|
18
|
+
4. [Content-Addressable Storage](#content-addressable-storage)
|
|
19
|
+
5. [Network Architecture](#network-architecture)
|
|
20
|
+
6. [PULL vs PUSH Architecture](#pull-vs-push-architecture)
|
|
21
|
+
7. [Multi-Tier Storage](#multi-tier-storage)
|
|
22
|
+
8. [Socket Abstraction Layer](#socket-abstraction-layer)
|
|
23
|
+
9. [Implementation Details](#implementation-details)
|
|
24
|
+
10. [Design Patterns](#design-patterns)
|
|
25
|
+
11. [Performance Optimization](#performance-optimization)
|
|
26
|
+
12. [Testing Strategy](#testing-strategy)
|
|
27
|
+
|
|
28
|
+
## Overview
|
|
29
|
+
|
|
30
|
+
`@rljson/bs` implements a content-addressable blob storage system with a layered architecture designed for flexibility, performance, and type safety. The system follows a modular design where each component has a single responsibility and can be composed with others to create complex storage hierarchies.
|
|
31
|
+
|
|
32
|
+
### System Goals
|
|
33
|
+
|
|
34
|
+
1. **Content Addressability**: Every blob is identified by its SHA256 hash
|
|
35
|
+
2. **Automatic Deduplication**: Identical content stored only once
|
|
36
|
+
3. **Composability**: Components can be combined to create multi-tier architectures
|
|
37
|
+
4. **Network Transparency**: Local and remote storage use the same interface
|
|
38
|
+
5. **Type Safety**: Full TypeScript support with comprehensive types
|
|
39
|
+
6. **Testability**: 100% test coverage with comprehensive test utilities
|
|
40
|
+
|
|
41
|
+
## Core Design Principles
|
|
42
|
+
|
|
43
|
+
### 1. Interface-First Design
|
|
44
|
+
|
|
45
|
+
All storage implementations conform to the `Bs` interface, ensuring uniform behavior:
|
|
46
|
+
|
|
47
|
+
```typescript
|
|
48
|
+
export interface Bs {
|
|
49
|
+
setBlob(content: Buffer | string | ReadableStream): Promise<BlobProperties>;
|
|
50
|
+
getBlob(blobId: string, options?: DownloadBlobOptions): Promise<{ content: Buffer; properties: BlobProperties }>;
|
|
51
|
+
getBlobStream(blobId: string): Promise<ReadableStream>;
|
|
52
|
+
deleteBlob(blobId: string): Promise<void>;
|
|
53
|
+
blobExists(blobId: string): Promise<boolean>;
|
|
54
|
+
getBlobProperties(blobId: string): Promise<BlobProperties>;
|
|
55
|
+
listBlobs(options?: ListBlobsOptions): Promise<ListBlobsResult>;
|
|
56
|
+
generateSignedUrl(blobId: string, expiresIn: number, permissions?: 'read' | 'delete'): Promise<string>;
|
|
57
|
+
}
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Benefits:**
|
|
61
|
+
- Implementations can be swapped without changing client code
|
|
62
|
+
- Easy to add new storage backends
|
|
63
|
+
- Consistent behavior across all implementations
|
|
64
|
+
- Simplified testing with mocks
|
|
65
|
+
|
|
66
|
+
### 2. Content-Addressable Storage
|
|
67
|
+
|
|
68
|
+
Blobs are identified by SHA256 hash of their content, not by user-defined names.
|
|
69
|
+
|
|
70
|
+
**Advantages:**
|
|
71
|
+
- **Automatic Deduplication**: Same content = same ID
|
|
72
|
+
- **Data Integrity**: Hash serves as checksum
|
|
73
|
+
- **Location Independence**: Content can be verified anywhere
|
|
74
|
+
- **Efficient Caching**: Content can be cached safely by hash
|
|
75
|
+
|
|
76
|
+
**Trade-offs:**
|
|
77
|
+
- Cannot choose blob IDs
|
|
78
|
+
- Metadata must be stored separately
|
|
79
|
+
- Changing content changes ID
|
|
80
|
+
|
|
81
|
+
### 3. Flat Storage Pool
|
|
82
|
+
|
|
83
|
+
No hierarchy, containers, or folders at the Bs level:
|
|
84
|
+
|
|
85
|
+
```
|
|
86
|
+
Traditional: Content-Addressable:
|
|
87
|
+
├── container1/ All blobs in flat pool:
|
|
88
|
+
│ ├── file1.txt ├── dffd6021bb2bd5b0...
|
|
89
|
+
│ └── file2.txt ├── a591a6d40bf420...<br/>└── container2/ └── e3b0c44298fc1c1...
|
|
90
|
+
└── file3.txt
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
**Benefits:**
|
|
94
|
+
- Simpler implementation
|
|
95
|
+
- Natural deduplication
|
|
96
|
+
- No path traversal issues
|
|
97
|
+
- Efficient lookups
|
|
98
|
+
|
|
99
|
+
**Metadata Storage:**
|
|
100
|
+
Organizational metadata (folders, tags, filenames) is stored separately, typically in `@rljson/io` (table storage).
|
|
101
|
+
|
|
102
|
+
### 4. Composition Over Inheritance
|
|
103
|
+
|
|
104
|
+
Complex behaviors are achieved by composing simple components:
|
|
105
|
+
|
|
106
|
+
```typescript
|
|
107
|
+
// Composition: BsMulti wraps multiple Bs instances
|
|
108
|
+
const bs = new BsMulti([
|
|
109
|
+
{ bs: localCache, priority: 0 }, // Simple BsMem
|
|
110
|
+
{ bs: remotePeer, priority: 1 }, // Simple BsPeer
|
|
111
|
+
]);
|
|
112
|
+
|
|
113
|
+
// Not inheritance: class BsCachedRemote extends Bs
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## Component Architecture
|
|
117
|
+
|
|
118
|
+
### Layered Architecture Diagram
|
|
119
|
+
|
|
120
|
+
```
|
|
121
|
+
┌─────────────────────────────────────────────────────────┐
|
|
122
|
+
│ Application Layer │
|
|
123
|
+
│ (Your code using @rljson/bs) │
|
|
124
|
+
└──────────────────┬──────────────────────────────────────┘
|
|
125
|
+
│
|
|
126
|
+
┌──────────────────▼──────────────────────────────────────┐
|
|
127
|
+
│ Composition Layer │
|
|
128
|
+
│ │
|
|
129
|
+
│ ┌─────────┐ Combines multiple storage backends │
|
|
130
|
+
│ │BsMulti │ - Priority-based reads │
|
|
131
|
+
│ │ │ - Parallel writes │
|
|
132
|
+
│ └────┬────┘ - Hot-swapping/caching │
|
|
133
|
+
│ │ │
|
|
134
|
+
└───────┼─────────────────────────────────────────────────┘
|
|
135
|
+
│
|
|
136
|
+
├──────┬──────┬──────┬──────┐
|
|
137
|
+
│ │ │ │ │
|
|
138
|
+
┌───────▼──┐ ┌─▼────┐ ┌────▼┐ ┌────▼─────┐
|
|
139
|
+
│ BsMem │ │BsPeer│ │... │ │ Custom │ <- Storage Layer
|
|
140
|
+
│ (local) │ │(net) │ │ │ │ Impl │
|
|
141
|
+
└──────────┘ └──┬───┘ └─────┘ └──────────┘
|
|
142
|
+
│
|
|
143
|
+
┌───────▼────────┐
|
|
144
|
+
│ Socket Layer │ <- Transport Layer
|
|
145
|
+
│ (abstraction) │
|
|
146
|
+
└───────┬────────┘
|
|
147
|
+
│
|
|
148
|
+
┌───────▼────────┐
|
|
149
|
+
│ BsServer │ <- Server Layer
|
|
150
|
+
│ BsPeerBridge │
|
|
151
|
+
└────────────────┘
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
### Component Responsibilities
|
|
155
|
+
|
|
156
|
+
#### 1. **Bs Interface** (`bs.ts`)
|
|
157
|
+
|
|
158
|
+
The core interface that all implementations must follow.
|
|
159
|
+
|
|
160
|
+
**Responsibilities:**
|
|
161
|
+
- Define the contract for blob storage operations
|
|
162
|
+
- Specify type definitions for parameters and return values
|
|
163
|
+
- Document expected behavior
|
|
164
|
+
|
|
165
|
+
**Key Types:**
|
|
166
|
+
- `BlobProperties`: Metadata about a blob
|
|
167
|
+
- `DownloadBlobOptions`: Options for retrieving blobs
|
|
168
|
+
- `ListBlobsOptions`: Options for listing blobs
|
|
169
|
+
- `ListBlobsResult`: Result structure for list operations
|
|
170
|
+
|
|
171
|
+
#### 2. **BsMem** (`bs-mem.ts`)
|
|
172
|
+
|
|
173
|
+
In-memory implementation of the Bs interface.
|
|
174
|
+
|
|
175
|
+
**Responsibilities:**
|
|
176
|
+
- Store blobs in a Map (memory)
|
|
177
|
+
- Calculate SHA256 hashes
|
|
178
|
+
- Handle deduplication automatically
|
|
179
|
+
- Provide fast, ephemeral storage
|
|
180
|
+
|
|
181
|
+
**Internal Structure:**
|
|
182
|
+
```typescript
|
|
183
|
+
class BsMem implements Bs {
|
|
184
|
+
private readonly blobs = new Map<string, StoredBlob>();
|
|
185
|
+
|
|
186
|
+
// StoredBlob structure:
|
|
187
|
+
interface StoredBlob {
|
|
188
|
+
content: Buffer;
|
|
189
|
+
properties: BlobProperties;
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
**Key Operations:**
|
|
195
|
+
- `setBlob`: Hash content → Check if exists → Store if new
|
|
196
|
+
- `getBlob`: Lookup by blobId → Return content + properties
|
|
197
|
+
- `listBlobs`: Convert Map entries → Sort → Filter → Paginate
|
|
198
|
+
|
|
199
|
+
#### 3. **BsPeer** (`bs-peer.ts`)
|
|
200
|
+
|
|
201
|
+
Client-side implementation that accesses remote storage via socket.
|
|
202
|
+
|
|
203
|
+
**Responsibilities:**
|
|
204
|
+
- Translate Bs method calls to socket messages
|
|
205
|
+
- Handle async socket communication
|
|
206
|
+
- Manage connection state
|
|
207
|
+
- Convert between local types and wire format
|
|
208
|
+
|
|
209
|
+
**Architecture Pattern:**
|
|
210
|
+
```
|
|
211
|
+
BsPeer (Client) BsServer (Remote)
|
|
212
|
+
│ │
|
|
213
|
+
│ socket.emit('setBlob', ...) │
|
|
214
|
+
├──────────────────────────────────>│
|
|
215
|
+
│ │ storage.setBlob(...)
|
|
216
|
+
│ callback(null, result) │
|
|
217
|
+
│<──────────────────────────────────┤
|
|
218
|
+
│ │
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
**Connection Lifecycle:**
|
|
222
|
+
1. `init()`: Connect socket and wait for 'connect' event
|
|
223
|
+
2. Operations: Send requests via socket, receive callbacks
|
|
224
|
+
3. `close()`: Disconnect socket and wait for 'disconnect' event
|
|
225
|
+
|
|
226
|
+
**Error Handling:**
|
|
227
|
+
- Uses error-first callbacks: `(error, result)`
|
|
228
|
+
- Network errors propagated as promise rejections
|
|
229
|
+
- Connection state tracked via `isOpen` property
|
|
230
|
+
|
|
231
|
+
#### 4. **BsServer** (`bs-server.ts`)
|
|
232
|
+
|
|
233
|
+
Server-side component that exposes a Bs instance over sockets.
|
|
234
|
+
|
|
235
|
+
**Responsibilities:**
|
|
236
|
+
- Accept socket connections from multiple clients
|
|
237
|
+
- Route socket messages to underlying Bs methods
|
|
238
|
+
- Translate results back to socket responses
|
|
239
|
+
- Handle multiple concurrent clients
|
|
240
|
+
|
|
241
|
+
**Transport Layer Generation:**
|
|
242
|
+
```typescript
|
|
243
|
+
private _generateTransportLayer(bs: Bs) {
|
|
244
|
+
return {
|
|
245
|
+
setBlob: (content) => bs.setBlob(content),
|
|
246
|
+
getBlob: (blobId, options) => bs.getBlob(blobId, options),
|
|
247
|
+
// ... map all Bs methods
|
|
248
|
+
};
|
|
249
|
+
}
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
**Multi-Client Support:**
|
|
253
|
+
```typescript
|
|
254
|
+
// Each client has its own socket
|
|
255
|
+
await server.addSocket(clientSocket1);
|
|
256
|
+
await server.addSocket(clientSocket2);
|
|
257
|
+
|
|
258
|
+
// Clients share the same underlying Bs instance
|
|
259
|
+
// Content-addressable nature ensures consistency
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
#### 5. **BsPeerBridge** (`bs-peer-bridge.ts`)
|
|
263
|
+
|
|
264
|
+
**CRITICAL: Read-Only PULL Architecture**
|
|
265
|
+
|
|
266
|
+
Exposes local storage for remote access (server pulls from client).
|
|
267
|
+
|
|
268
|
+
**Responsibilities:**
|
|
269
|
+
- Listen for socket events from server
|
|
270
|
+
- Translate events to Bs method calls on local storage
|
|
271
|
+
- **Only expose READ operations** (PULL pattern)
|
|
272
|
+
- Use error-first callback pattern
|
|
273
|
+
|
|
274
|
+
**Architectural Pattern:**
|
|
275
|
+
```
|
|
276
|
+
Client Server
|
|
277
|
+
┌─────────────────┐ ┌──────────────────┐
|
|
278
|
+
│ Local Storage │ │ Server Bs │
|
|
279
|
+
│ (BsMem) │ │ │
|
|
280
|
+
└────────┬────────┘ └────────┬─────────┘
|
|
281
|
+
│ │
|
|
282
|
+
│ │
|
|
283
|
+
┌────▼─────────┐ ┌────▼────────┐
|
|
284
|
+
│BsPeerBridge │◄───Socket (READ)───│ BsPeer │
|
|
285
|
+
│ (Exposes │ │ (Requests) │
|
|
286
|
+
│ READ only) │ │ │
|
|
287
|
+
└──────────────┘ └─────────────┘
|
|
288
|
+
|
|
289
|
+
Client exposes local storage for SERVER to PULL from (read-only)
|
|
290
|
+
Server CANNOT push writes to client
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
**Read-Only Methods Exposed:**
|
|
294
|
+
- `getBlob` - Read blob content
|
|
295
|
+
- `getBlobStream` - Stream blob content
|
|
296
|
+
- `blobExists` - Check existence
|
|
297
|
+
- `getBlobProperties` - Get metadata
|
|
298
|
+
- `listBlobs` - List available blobs
|
|
299
|
+
|
|
300
|
+
**NOT Exposed (Write Operations):**
|
|
301
|
+
- `setBlob` - Would allow server to write to client
|
|
302
|
+
- `deleteBlob` - Would allow server to delete from client
|
|
303
|
+
- `generateSignedUrl` - Management operation
|
|
304
|
+
|
|
305
|
+
**Implementation:**
|
|
306
|
+
```typescript
|
|
307
|
+
private _registerBsMethods(): void {
|
|
308
|
+
const bsMethods = [
|
|
309
|
+
'getBlob', // ✅ READ
|
|
310
|
+
'getBlobStream', // ✅ READ
|
|
311
|
+
'blobExists', // ✅ READ
|
|
312
|
+
'getBlobProperties', // ✅ READ
|
|
313
|
+
'listBlobs', // ✅ READ
|
|
314
|
+
// NOT: 'setBlob', 'deleteBlob', 'generateSignedUrl'
|
|
315
|
+
];
|
|
316
|
+
|
|
317
|
+
for (const methodName of bsMethods) {
|
|
318
|
+
this.registerEvent(methodName);
|
|
319
|
+
}
|
|
320
|
+
}
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
**Callback Pattern (Error-First):**
|
|
324
|
+
```typescript
|
|
325
|
+
const handler = (...args: any[]) => {
|
|
326
|
+
const callback = args[args.length - 1];
|
|
327
|
+
const methodArgs = args.slice(0, -1);
|
|
328
|
+
|
|
329
|
+
bsMethod.apply(this._bs, methodArgs)
|
|
330
|
+
.then((result) => {
|
|
331
|
+
callback(null, result); // ✅ Error-first: (error, result)
|
|
332
|
+
})
|
|
333
|
+
.catch((error) => {
|
|
334
|
+
callback(error, null); // ✅ Error-first: (error, result)
|
|
335
|
+
});
|
|
336
|
+
};
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
**Why Read-Only?**
|
|
340
|
+
|
|
341
|
+
1. **Security**: Prevents server from pushing malicious data to clients
|
|
342
|
+
2. **Architectural Clarity**: Clear unidirectional data flow (PULL)
|
|
343
|
+
3. **Consistency with IoPeerBridge**: Matches the pattern from `@rljson/io`
|
|
344
|
+
4. **Client Control**: Client owns its local storage, server can only read
|
|
345
|
+
|
|
346
|
+
#### 6. **BsMulti** (`bs-multi.ts`)
|
|
347
|
+
|
|
348
|
+
Multi-tier storage that composes multiple Bs instances.
|
|
349
|
+
|
|
350
|
+
**Responsibilities:**
|
|
351
|
+
- Manage multiple storage backends with priorities
|
|
352
|
+
- Route reads to highest priority readable
|
|
353
|
+
- Write to all writable stores in parallel
|
|
354
|
+
- Implement hot-swapping (cache population)
|
|
355
|
+
- Merge results from multiple stores
|
|
356
|
+
|
|
357
|
+
**Priority System:**
|
|
358
|
+
```typescript
|
|
359
|
+
type BsMultiBs = {
|
|
360
|
+
bs: Bs;
|
|
361
|
+
id?: string;
|
|
362
|
+
priority: number; // Lower number = higher priority
|
|
363
|
+
read: boolean; // Can read from this store
|
|
364
|
+
write: boolean; // Can write to this store
|
|
365
|
+
};
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
**Read Strategy:**
|
|
369
|
+
1. Sort stores by priority (ascending)
|
|
370
|
+
2. Try each readable store in order
|
|
371
|
+
3. First successful read wins
|
|
372
|
+
4. Hot-swap: Write to all writable stores for caching
|
|
373
|
+
5. Return result
|
|
374
|
+
|
|
375
|
+
**Write Strategy:**
|
|
376
|
+
1. Collect all writable stores
|
|
377
|
+
2. Write to all in parallel using `Promise.all`
|
|
378
|
+
3. All writes must succeed (or error)
|
|
379
|
+
4. Content-addressable ensures consistency
|
|
380
|
+
|
|
381
|
+
**Hot-Swapping:**
|
|
382
|
+
```typescript
|
|
383
|
+
async getBlob(blobId: string): Promise<{ content: Buffer; properties: BlobProperties }> {
|
|
384
|
+
// Try stores in priority order
|
|
385
|
+
for (const readable of this.readables) {
|
|
386
|
+
try {
|
|
387
|
+
result = await readable.bs.getBlob(blobId);
|
|
388
|
+
readFrom = readable.id;
|
|
389
|
+
break;
|
|
390
|
+
} catch (e) {
|
|
391
|
+
continue;
|
|
392
|
+
}
|
|
393
|
+
}
|
|
394
|
+
|
|
395
|
+
// Hot-swap: Cache in all writables (except source)
|
|
396
|
+
const hotSwapWrites = this.writables
|
|
397
|
+
.filter((writable) => writable.id !== readFrom)
|
|
398
|
+
.map(({ bs }) => bs.setBlob(result.content).catch(() => {}));
|
|
399
|
+
|
|
400
|
+
await Promise.all(hotSwapWrites);
|
|
401
|
+
return result;
|
|
402
|
+
}
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
**List Strategy (Deduplication):**
|
|
406
|
+
```typescript
|
|
407
|
+
async listBlobs(): Promise<ListBlobsResult> {
|
|
408
|
+
// Query all readable stores in parallel
|
|
409
|
+
const allResults = await Promise.all(
|
|
410
|
+
this.readables.map(({ bs }) => bs.listBlobs())
|
|
411
|
+
);
|
|
412
|
+
|
|
413
|
+
// Deduplicate by blobId (Set)
|
|
414
|
+
const uniqueBlobs = new Map<string, BlobProperties>();
|
|
415
|
+
for (const result of allResults) {
|
|
416
|
+
for (const blob of result.blobs) {
|
|
417
|
+
uniqueBlobs.set(blob.blobId, blob);
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
|
|
421
|
+
return { blobs: Array.from(uniqueBlobs.values()) };
|
|
422
|
+
}
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
## Content-Addressable Storage
|
|
426
|
+
|
|
427
|
+
### SHA256 Hashing
|
|
428
|
+
|
|
429
|
+
Content addressing uses SHA256 to generate a unique identifier for each blob.
|
|
430
|
+
|
|
431
|
+
**Hash Calculation:**
|
|
432
|
+
```typescript
|
|
433
|
+
import { createHash } from 'crypto';
|
|
434
|
+
|
|
435
|
+
function calculateBlobId(content: Buffer): string {
|
|
436
|
+
return createHash('sha256').update(content).digest('hex');
|
|
437
|
+
}
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
**Properties:**
|
|
441
|
+
- **Deterministic**: Same content always produces same hash
|
|
442
|
+
- **Collision-Resistant**: Practically impossible to find two different contents with same hash
|
|
443
|
+
- **Fixed Length**: Always 64 hexadecimal characters (256 bits)
|
|
444
|
+
- **One-Way**: Cannot derive content from hash
|
|
445
|
+
|
|
446
|
+
### Deduplication Mechanics
|
|
447
|
+
|
|
448
|
+
**Automatic Deduplication:**
|
|
449
|
+
```typescript
|
|
450
|
+
async setBlob(content: Buffer | string | ReadableStream): Promise<BlobProperties> {
|
|
451
|
+
const buffer = await this.toBuffer(content);
|
|
452
|
+
const blobId = hshBuffer(buffer);
|
|
453
|
+
|
|
454
|
+
// Check if blob already exists
|
|
455
|
+
const existing = this.blobs.get(blobId);
|
|
456
|
+
if (existing) {
|
|
457
|
+
return existing.properties; // Return existing, don't store again
|
|
458
|
+
}
|
|
459
|
+
|
|
460
|
+
// Store new blob
|
|
461
|
+
const properties: BlobProperties = {
|
|
462
|
+
blobId,
|
|
463
|
+
size: buffer.length,
|
|
464
|
+
createdAt: new Date(),
|
|
465
|
+
};
|
|
466
|
+
|
|
467
|
+
this.blobs.set(blobId, { content: buffer, properties });
|
|
468
|
+
return properties;
|
|
469
|
+
}
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
**Benefits in Multi-Tier:**
|
|
473
|
+
```typescript
|
|
474
|
+
// Same content stored in multiple tiers = same blobId
|
|
475
|
+
// BsMulti writes to all writables, but no duplicate storage cost
|
|
476
|
+
// because content-addressable storage deduplicates automatically
|
|
477
|
+
|
|
478
|
+
await bsMulti.setBlob('same content'); // Writes to tier1, tier2
|
|
479
|
+
await bsMulti.setBlob('same content'); // No-op (already exists)
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
### Content Verification
|
|
483
|
+
|
|
484
|
+
Any party can verify content integrity:
|
|
485
|
+
|
|
486
|
+
```typescript
|
|
487
|
+
async verifyBlob(blobId: string, content: Buffer): boolean {
|
|
488
|
+
const computedId = calculateBlobId(content);
|
|
489
|
+
return computedId === blobId;
|
|
490
|
+
}
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
## Network Architecture
|
|
494
|
+
|
|
495
|
+
### Socket Abstraction
|
|
496
|
+
|
|
497
|
+
The `Socket` interface abstracts network transport:
|
|
498
|
+
|
|
499
|
+
```typescript
|
|
500
|
+
export interface Socket {
|
|
501
|
+
connected: boolean;
|
|
502
|
+
disconnected: boolean;
|
|
503
|
+
connect(): void;
|
|
504
|
+
disconnect(): void;
|
|
505
|
+
on(eventName: string | symbol, listener: (...args: any[]) => void): this;
|
|
506
|
+
emit(eventName: string | symbol, ...args: any[]): boolean | this;
|
|
507
|
+
off(eventName: string | symbol, listener: (...args: any[]) => void): this;
|
|
508
|
+
removeAllListeners(eventName?: string | symbol): this;
|
|
509
|
+
}
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
**Implementations:**
|
|
513
|
+
- `SocketMock`: In-process mock for testing (synchronous)
|
|
514
|
+
- `PeerSocketMock`: Direct Bs invocation (testing without network)
|
|
515
|
+
- `DirectionalSocketMock`: Bidirectional socket pair (testing)
|
|
516
|
+
- **Production**: Use Socket.IO, WebSocket, or custom implementation
|
|
517
|
+
|
|
518
|
+
### Protocol
|
|
519
|
+
|
|
520
|
+
**Message Format (Socket Events):**
|
|
521
|
+
```
|
|
522
|
+
Event: <methodName>
|
|
523
|
+
Args: [...methodArgs, callback]
|
|
524
|
+
|
|
525
|
+
Example:
|
|
526
|
+
socket.emit('getBlob', blobId, options, (error, result) => {
|
|
527
|
+
if (error) console.error(error);
|
|
528
|
+
else console.log(result);
|
|
529
|
+
});
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
**Error-First Callbacks:**
|
|
533
|
+
```typescript
|
|
534
|
+
callback(error: Error | null, result?: T);
|
|
535
|
+
|
|
536
|
+
// Success
|
|
537
|
+
callback(null, { content: buffer, properties: props });
|
|
538
|
+
|
|
539
|
+
// Error
|
|
540
|
+
callback(new Error('Blob not found'), null);
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
### Client-Server Flow
|
|
544
|
+
|
|
545
|
+
**Complete Request/Response Cycle:**
|
|
546
|
+
```
|
|
547
|
+
1. Client: BsPeer.getBlob(blobId)
|
|
548
|
+
├─> Create promise
|
|
549
|
+
└─> socket.emit('getBlob', blobId, callback)
|
|
550
|
+
|
|
551
|
+
2. Network: Transport socket event
|
|
552
|
+
|
|
553
|
+
3. Server: BsServer receives 'getBlob' event
|
|
554
|
+
├─> Extract args and callback
|
|
555
|
+
├─> Call underlying bs.getBlob(blobId)
|
|
556
|
+
├─> Wait for result
|
|
557
|
+
└─> Invoke callback(null, result)
|
|
558
|
+
|
|
559
|
+
4. Network: Transport callback response
|
|
560
|
+
|
|
561
|
+
5. Client: BsPeer callback invoked
|
|
562
|
+
├─> Resolve or reject promise
|
|
563
|
+
└─> Return to caller
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
## PULL vs PUSH Architecture
|
|
567
|
+
|
|
568
|
+
### PULL Architecture (Recommended)
|
|
569
|
+
|
|
570
|
+
**Pattern: Server pulls data from client when needed**
|
|
571
|
+
|
|
572
|
+
```
|
|
573
|
+
┌─────────┐ ┌────────┐
|
|
574
|
+
│ Client │ │ Server │
|
|
575
|
+
│ │ │ │
|
|
576
|
+
│ Local │◄────READ ONLY──────│ Peer │
|
|
577
|
+
│ Store │ (BsPeerBridge) │ │
|
|
578
|
+
│ │ │ │
|
|
579
|
+
└─────────┘ └────────┘
|
|
580
|
+
|
|
581
|
+
Server requests: "Give me blob X"
|
|
582
|
+
Client responds: "Here is blob X"
|
|
583
|
+
Server CANNOT push data to client
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
**Implementation:**
|
|
587
|
+
```typescript
|
|
588
|
+
// Client exposes local storage via BsPeerBridge (read-only)
|
|
589
|
+
const bridge = new BsPeerBridge(localStorage, socket);
|
|
590
|
+
bridge.start(); // Only read operations exposed
|
|
591
|
+
|
|
592
|
+
// Server pulls from client
|
|
593
|
+
const peer = new BsPeer(socket);
|
|
594
|
+
const { content } = await peer.getBlob(blobId); // ✅ Can read
|
|
595
|
+
// await peer.setBlob('data'); // ❌ Would fail if bridge is read-only on client
|
|
596
|
+
```
|
|
597
|
+
|
|
598
|
+
**Benefits:**
|
|
599
|
+
- ✅ Client controls its data
|
|
600
|
+
- ✅ Server cannot inject malicious content
|
|
601
|
+
- ✅ Clear unidirectional data flow
|
|
602
|
+
- ✅ Matches real-world HTTP patterns (server pulls via GET)
|
|
603
|
+
|
|
604
|
+
### PUSH Architecture (Anti-Pattern for BsPeerBridge)
|
|
605
|
+
|
|
606
|
+
**Pattern: Server pushes data to client (NOT RECOMMENDED)**
|
|
607
|
+
|
|
608
|
+
```
|
|
609
|
+
┌─────────┐ ┌────────┐
|
|
610
|
+
│ Client │ │ Server │
|
|
611
|
+
│ │ │ │
|
|
612
|
+
│ Local │◄────READ/WRITE─────│ Peer │
|
|
613
|
+
│ Store │ (Full Access) │ │
|
|
614
|
+
│ │ │ │
|
|
615
|
+
└─────────┘ └────────┘
|
|
616
|
+
|
|
617
|
+
Server command: "Store this blob"
|
|
618
|
+
Client stores: "Blob stored"
|
|
619
|
+
Server can write to client ❌
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
**Why NOT to expose writes in BsPeerBridge:**
|
|
623
|
+
- ❌ Security risk: Server can push malicious data
|
|
624
|
+
- ❌ Unclear data ownership
|
|
625
|
+
- ❌ Violates principle of least privilege
|
|
626
|
+
- ❌ Client loses control over its storage
|
|
627
|
+
|
|
628
|
+
**Correct Alternative:**
|
|
629
|
+
If client wants to send data to server, use `BsPeer` for client-to-server writes:
|
|
630
|
+
|
|
631
|
+
```typescript
|
|
632
|
+
// Client pushes to server using BsPeer
|
|
633
|
+
const serverPeer = new BsPeer(socketToServer);
|
|
634
|
+
await serverPeer.setBlob('data'); // Client-initiated write to server
|
|
635
|
+
```
|
|
636
|
+
|
|
637
|
+
### Comparison with IoPeerBridge
|
|
638
|
+
|
|
639
|
+
`BsPeerBridge` follows the same read-only pattern as `IoPeerBridge` from `@rljson/io`:
|
|
640
|
+
|
|
641
|
+
**IoPeerBridge (Table Storage):**
|
|
642
|
+
```typescript
|
|
643
|
+
// Only exposes reads:
|
|
644
|
+
- readRows // ✅ READ
|
|
645
|
+
- tableExists // ✅ READ
|
|
646
|
+
- rawTableCfgs // ✅ READ
|
|
647
|
+
// NOT: insertRow, updateRow, deleteRow
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
**BsPeerBridge (Blob Storage):**
|
|
651
|
+
```typescript
|
|
652
|
+
// Only exposes reads:
|
|
653
|
+
- getBlob // ✅ READ
|
|
654
|
+
- getBlobStream // ✅ READ
|
|
655
|
+
- blobExists // ✅ READ
|
|
656
|
+
- getBlobProperties // ✅ READ
|
|
657
|
+
- listBlobs // ✅ READ
|
|
658
|
+
// NOT: setBlob, deleteBlob, generateSignedUrl
|
|
659
|
+
```
|
|
660
|
+
|
|
661
|
+
**Consistency:** Both use PULL architecture for client-server interactions.
|
|
662
|
+
|
|
663
|
+
## Multi-Tier Storage
|
|
664
|
+
|
|
665
|
+
### Hierarchical Storage Management (HSM)
|
|
666
|
+
|
|
667
|
+
BsMulti implements a form of HSM with automatic tier management:
|
|
668
|
+
|
|
669
|
+
```
|
|
670
|
+
Priority 0 (Highest)
|
|
671
|
+
├─ Fast local cache (BsMem)
|
|
672
|
+
│ └─ Small, fast, volatile
|
|
673
|
+
|
|
674
|
+
Priority 1
|
|
675
|
+
├─ Network storage (BsPeer)
|
|
676
|
+
│ └─ Larger, slower, persistent
|
|
677
|
+
|
|
678
|
+
Priority 2 (Lowest)
|
|
679
|
+
├─ Backup/Archive (Read-only)
|
|
680
|
+
└─ Largest, slowest, persistent
|
|
681
|
+
```
|
|
682
|
+
|
|
683
|
+
### Read Path with Hot-Swapping
|
|
684
|
+
|
|
685
|
+
```
|
|
686
|
+
1. Request: getBlob(blobId)
|
|
687
|
+
|
|
688
|
+
2. Check Priority 0 (cache)
|
|
689
|
+
├─ HIT: Return immediately ✅
|
|
690
|
+
└─ MISS: Continue to Priority 1
|
|
691
|
+
|
|
692
|
+
3. Check Priority 1 (network)
|
|
693
|
+
├─ HIT:
|
|
694
|
+
│ ├─ Return content
|
|
695
|
+
│ └─ Hot-swap to Priority 0 (cache for next time)
|
|
696
|
+
└─ MISS: Continue to Priority 2
|
|
697
|
+
|
|
698
|
+
4. Check Priority 2 (backup)
|
|
699
|
+
├─ HIT:
|
|
700
|
+
│ ├─ Return content
|
|
701
|
+
│ └─ Hot-swap to Priority 0 and 1
|
|
702
|
+
└─ MISS: Throw "Blob not found"
|
|
703
|
+
```
|
|
704
|
+
|
|
705
|
+
**Code:**
|
|
706
|
+
```typescript
|
|
707
|
+
async getBlob(blobId: string): Promise<{ content: Buffer; properties: BlobProperties }> {
|
|
708
|
+
for (const readable of this.readables) {
|
|
709
|
+
try {
|
|
710
|
+
result = await readable.bs.getBlob(blobId);
|
|
711
|
+
readFrom = readable.id;
|
|
712
|
+
break;
|
|
713
|
+
} catch (e) {
|
|
714
|
+
continue; // Try next priority
|
|
715
|
+
}
|
|
716
|
+
}
|
|
717
|
+
|
|
718
|
+
if (!result) {
|
|
719
|
+
throw new Error(`Blob not found: ${blobId}`);
|
|
720
|
+
}
|
|
721
|
+
|
|
722
|
+
// Hot-swap to all writables (except source)
|
|
723
|
+
const hotSwapWrites = this.writables
|
|
724
|
+
.filter((writable) => writable.id !== readFrom)
|
|
725
|
+
.map(({ bs }) => bs.setBlob(result.content).catch(() => {}));
|
|
726
|
+
|
|
727
|
+
await Promise.all(hotSwapWrites);
|
|
728
|
+
return result;
|
|
729
|
+
}
|
|
730
|
+
```
|
|
731
|
+
|
|
732
|
+
### Write Path (Parallel Replication)
|
|
733
|
+
|
|
734
|
+
```
|
|
735
|
+
1. Request: setBlob(content)
|
|
736
|
+
|
|
737
|
+
2. Collect all writable stores
|
|
738
|
+
├─ Priority 0: Local cache (write=true)
|
|
739
|
+
├─ Priority 1: Network storage (write=true)
|
|
740
|
+
└─ Priority 2: Backup (write=false) ← Skip
|
|
741
|
+
|
|
742
|
+
3. Write to all writables in parallel
|
|
743
|
+
├─ Promise.all([
|
|
744
|
+
│ cache.setBlob(content),
|
|
745
|
+
│ network.setBlob(content),
|
|
746
|
+
│ ])
|
|
747
|
+
|
|
748
|
+
4. All must succeed or error propagates
|
|
749
|
+
├─ Content-addressable ensures same blobId everywhere
|
|
750
|
+
└─ Deduplication prevents storage waste
|
|
751
|
+
```
|
|
752
|
+
|
|
753
|
+
### List Operations (Deduplication)
|
|
754
|
+
|
|
755
|
+
```
|
|
756
|
+
1. Request: listBlobs()
|
|
757
|
+
|
|
758
|
+
2. Query all readable stores in parallel
|
|
759
|
+
├─ Promise.all([
|
|
760
|
+
│ cache.listBlobs(),
|
|
761
|
+
│ network.listBlobs(),
|
|
762
|
+
│ backup.listBlobs(),
|
|
763
|
+
│ ])
|
|
764
|
+
|
|
765
|
+
3. Deduplicate results by blobId
|
|
766
|
+
├─ Use Map<blobId, BlobProperties>
|
|
767
|
+
└─ Content-addressable means same blobId = same content
|
|
768
|
+
|
|
769
|
+
4. Return merged list
|
|
770
|
+
```
|
|
771
|
+
|
|
772
|
+
## Socket Abstraction Layer
|
|
773
|
+
|
|
774
|
+
### Design Philosophy
|
|
775
|
+
|
|
776
|
+
The Socket interface abstracts the transport layer, allowing:
|
|
777
|
+
|
|
778
|
+
1. **Different transports**: WebSocket, Socket.IO, custom protocols
|
|
779
|
+
2. **Testing**: In-process mocks without network
|
|
780
|
+
3. **Flexibility**: Swap implementations without changing Bs code
|
|
781
|
+
|
|
782
|
+
### Mock Implementations
|
|
783
|
+
|
|
784
|
+
#### SocketMock (Basic)
|
|
785
|
+
|
|
786
|
+
Simulates a socket with EventEmitter-like behavior:
|
|
787
|
+
|
|
788
|
+
```typescript
|
|
789
|
+
class SocketMock implements Socket {
|
|
790
|
+
private _listeners: Map<string | symbol, Array<(...args: any[]) => void>>;
|
|
791
|
+
|
|
792
|
+
emit(eventName: string | symbol, ...args: any[]): boolean {
|
|
793
|
+
const listeners = this._listeners.get(eventName) || [];
|
|
794
|
+
for (const listener of listeners) {
|
|
795
|
+
listener(...args);
|
|
796
|
+
}
|
|
797
|
+
return listeners.length > 0;
|
|
798
|
+
}
|
|
799
|
+
}
|
|
800
|
+
```
|
|
801
|
+
|
|
802
|
+
**Use:** General-purpose testing, client-server simulation.
|
|
803
|
+
|
|
804
|
+
#### PeerSocketMock (Direct)
|
|
805
|
+
|
|
806
|
+
Directly invokes Bs methods without socket events:
|
|
807
|
+
|
|
808
|
+
```typescript
|
|
809
|
+
class PeerSocketMock implements Socket {
|
|
810
|
+
constructor(private _bs: Bs) {}
|
|
811
|
+
|
|
812
|
+
emit(eventName: string, ...args: any[]): this {
|
|
813
|
+
const callback = args[args.length - 1];
|
|
814
|
+
const methodArgs = args.slice(0, -1);
|
|
815
|
+
|
|
816
|
+
// Direct method invocation
|
|
817
|
+
this._bs[eventName](...methodArgs)
|
|
818
|
+
.then(result => callback(null, result))
|
|
819
|
+
.catch(error => callback(error, null));
|
|
820
|
+
|
|
821
|
+
return this;
|
|
822
|
+
}
|
|
823
|
+
}
|
|
824
|
+
```
|
|
825
|
+
|
|
826
|
+
**Use:** Testing BsPeer without BsServer, fastest mock.
|
|
827
|
+
|
|
828
|
+
#### DirectionalSocketMock (Bidirectional)
|
|
829
|
+
|
|
830
|
+
Creates a pair of connected sockets:
|
|
831
|
+
|
|
832
|
+
```typescript
|
|
833
|
+
const socket1 = new DirectionalSocketMock();
|
|
834
|
+
const socket2 = new DirectionalSocketMock();
|
|
835
|
+
|
|
836
|
+
socket1.setPeer(socket2);
|
|
837
|
+
socket2.setPeer(socket1);
|
|
838
|
+
|
|
839
|
+
// Events on socket1 trigger listeners on socket2
|
|
840
|
+
socket1.emit('event', 'data'); // socket2 receives 'event'
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
**Use:** Testing full client-server interaction, network simulation.
|
|
844
|
+
|
|
845
|
+
### Production Integration
|
|
846
|
+
|
|
847
|
+
**Socket.IO Example:**
|
|
848
|
+
```typescript
|
|
849
|
+
import { io, Socket as SocketIOSocket } from 'socket.io-client';
|
|
850
|
+
|
|
851
|
+
// Wrap Socket.IO in Socket interface
|
|
852
|
+
class SocketIOAdapter implements Socket {
|
|
853
|
+
constructor(private _socket: SocketIOSocket) {}
|
|
854
|
+
|
|
855
|
+
get connected() { return this._socket.connected; }
|
|
856
|
+
get disconnected() { return this._socket.disconnected; }
|
|
857
|
+
connect() { this._socket.connect(); }
|
|
858
|
+
disconnect() { this._socket.disconnect(); }
|
|
859
|
+
on(event, listener) { this._socket.on(event, listener); return this; }
|
|
860
|
+
emit(event, ...args) { this._socket.emit(event, ...args); return this; }
|
|
861
|
+
off(event, listener) { this._socket.off(event, listener); return this; }
|
|
862
|
+
removeAllListeners(event) { this._socket.removeAllListeners(event); return this; }
|
|
863
|
+
}
|
|
864
|
+
|
|
865
|
+
// Use with BsPeer
|
|
866
|
+
const socket = new SocketIOAdapter(io('http://server'));
|
|
867
|
+
const peer = new BsPeer(socket);
|
|
868
|
+
await peer.init();
|
|
869
|
+
```
|
|
870
|
+
|
|
871
|
+
## Implementation Details
|
|
872
|
+
|
|
873
|
+
### Error Handling
|
|
874
|
+
|
|
875
|
+
**Error Propagation:**
|
|
876
|
+
```typescript
|
|
877
|
+
// BsMem: Synchronous errors
|
|
878
|
+
if (!this.blobs.has(blobId)) {
|
|
879
|
+
throw new Error(`Blob not found: ${blobId}`);
|
|
880
|
+
}
|
|
881
|
+
|
|
882
|
+
// BsPeer: Async errors from callbacks
|
|
883
|
+
socket.emit('getBlob', blobId, (error, result) => {
|
|
884
|
+
if (error) reject(error);
|
|
885
|
+
else resolve(result);
|
|
886
|
+
});
|
|
887
|
+
|
|
888
|
+
// BsMulti: Fallback on errors
|
|
889
|
+
for (const readable of this.readables) {
|
|
890
|
+
try {
|
|
891
|
+
return await readable.bs.getBlob(blobId);
|
|
892
|
+
} catch (e) {
|
|
893
|
+
// Try next priority
|
|
894
|
+
continue;
|
|
895
|
+
}
|
|
896
|
+
}
|
|
897
|
+
throw new Error('Blob not found in all stores');
|
|
898
|
+
```
|
|
899
|
+
|
|
900
|
+
**Error Types:**
|
|
901
|
+
- `Blob not found`: blobId doesn't exist
|
|
902
|
+
- `No readable/writable Bs available`: BsMulti configuration error
|
|
903
|
+
- `Method not found`: BsPeerBridge received unknown method
|
|
904
|
+
- Network errors: Socket connection failures
|
|
905
|
+
|
|
906
|
+
### Stream Handling
|
|
907
|
+
|
|
908
|
+
**Buffer to Stream Conversion:**
|
|
909
|
+
```typescript
|
|
910
|
+
async getBlobStream(blobId: string): Promise<ReadableStream> {
|
|
911
|
+
const stored = this.blobs.get(blobId);
|
|
912
|
+
if (!stored) {
|
|
913
|
+
throw new Error(`Blob not found: ${blobId}`);
|
|
914
|
+
}
|
|
915
|
+
|
|
916
|
+
// Node.js Readable to Web ReadableStream
|
|
917
|
+
const nodeStream = Readable.from(stored.content);
|
|
918
|
+
return Readable.toWeb(nodeStream) as ReadableStream;
|
|
919
|
+
}
|
|
920
|
+
```
|
|
921
|
+
|
|
922
|
+
**Stream to Buffer Conversion:**
|
|
923
|
+
```typescript
|
|
924
|
+
private async toBuffer(content: ReadableStream): Promise<Buffer> {
|
|
925
|
+
const reader = content.getReader();
|
|
926
|
+
const chunks: Uint8Array[] = [];
|
|
927
|
+
|
|
928
|
+
while (true) {
|
|
929
|
+
const { done, value } = await reader.read();
|
|
930
|
+
if (done) break;
|
|
931
|
+
chunks.push(value);
|
|
932
|
+
}
|
|
933
|
+
|
|
934
|
+
return Buffer.concat(chunks);
|
|
935
|
+
}
|
|
936
|
+
```
|
|
937
|
+
|
|
938
|
+
### Pagination
|
|
939
|
+
|
|
940
|
+
**List Blobs with Continuation:**
|
|
941
|
+
```typescript
|
|
942
|
+
async listBlobs(options?: ListBlobsOptions): Promise<ListBlobsResult> {
|
|
943
|
+
let blobs = Array.from(this.blobs.values()).map(s => s.properties);
|
|
944
|
+
|
|
945
|
+
// Filter by prefix
|
|
946
|
+
if (options?.prefix) {
|
|
947
|
+
blobs = blobs.filter(b => b.blobId.startsWith(options.prefix!));
|
|
948
|
+
}
|
|
949
|
+
|
|
950
|
+
// Sort for consistent ordering
|
|
951
|
+
blobs.sort((a, b) => a.blobId.localeCompare(b.blobId));
|
|
952
|
+
|
|
953
|
+
// Pagination
|
|
954
|
+
let startIndex = 0;
|
|
955
|
+
if (options?.continuationToken) {
|
|
956
|
+
startIndex = parseInt(options.continuationToken, 10);
|
|
957
|
+
}
|
|
958
|
+
|
|
959
|
+
const maxResults = options?.maxResults || blobs.length;
|
|
960
|
+
const endIndex = startIndex + maxResults;
|
|
961
|
+
const pageBlobs = blobs.slice(startIndex, endIndex);
|
|
962
|
+
|
|
963
|
+
const result: ListBlobsResult = { blobs: pageBlobs };
|
|
964
|
+
|
|
965
|
+
if (endIndex < blobs.length) {
|
|
966
|
+
result.continuationToken = endIndex.toString();
|
|
967
|
+
}
|
|
968
|
+
|
|
969
|
+
return result;
|
|
970
|
+
}
|
|
971
|
+
```
|
|
972
|
+
|
|
973
|
+
## Design Patterns
|
|
974
|
+
|
|
975
|
+
### 1. Strategy Pattern (Storage Backend)
|
|
976
|
+
|
|
977
|
+
Different implementations of Bs interface:
|
|
978
|
+
|
|
979
|
+
```typescript
|
|
980
|
+
interface Bs {
|
|
981
|
+
setBlob(...): Promise<BlobProperties>;
|
|
982
|
+
getBlob(...): Promise<{ content: Buffer; properties: BlobProperties }>;
|
|
983
|
+
// ...
|
|
984
|
+
}
|
|
985
|
+
|
|
986
|
+
// Strategies:
|
|
987
|
+
class BsMem implements Bs { /* In-memory */ }
|
|
988
|
+
class BsPeer implements Bs { /* Network */ }
|
|
989
|
+
class BsMulti implements Bs { /* Composite */ }
|
|
990
|
+
```
|
|
991
|
+
|
|
992
|
+
**Benefit:** Swap storage strategies without changing client code.
|
|
993
|
+
|
|
994
|
+
### 2. Composite Pattern (BsMulti)
|
|
995
|
+
|
|
996
|
+
BsMulti composes multiple Bs instances:
|
|
997
|
+
|
|
998
|
+
```typescript
|
|
999
|
+
class BsMulti implements Bs {
|
|
1000
|
+
constructor(private _stores: Array<BsMultiBs>) {}
|
|
1001
|
+
|
|
1002
|
+
async getBlob(blobId: string) {
|
|
1003
|
+
for (const store of this._stores) {
|
|
1004
|
+
try {
|
|
1005
|
+
return await store.bs.getBlob(blobId); // Delegate
|
|
1006
|
+
} catch (e) {
|
|
1007
|
+
continue;
|
|
1008
|
+
}
|
|
1009
|
+
}
|
|
1010
|
+
throw new Error('Blob not found');
|
|
1011
|
+
}
|
|
1012
|
+
}
|
|
1013
|
+
```
|
|
1014
|
+
|
|
1015
|
+
**Benefit:** Treat single and composite storage uniformly.
|
|
1016
|
+
|
|
1017
|
+
### 3. Proxy Pattern (BsPeer, BsServer)
|
|
1018
|
+
|
|
1019
|
+
BsPeer proxies requests to remote Bs:
|
|
1020
|
+
|
|
1021
|
+
```typescript
|
|
1022
|
+
class BsPeer implements Bs {
|
|
1023
|
+
async getBlob(blobId: string) {
|
|
1024
|
+
// Proxy to remote via socket
|
|
1025
|
+
return new Promise((resolve, reject) => {
|
|
1026
|
+
this._socket.emit('getBlob', blobId, (error, result) => {
|
|
1027
|
+
if (error) reject(error);
|
|
1028
|
+
else resolve(result);
|
|
1029
|
+
});
|
|
1030
|
+
});
|
|
1031
|
+
}
|
|
1032
|
+
}
|
|
1033
|
+
```
|
|
1034
|
+
|
|
1035
|
+
**Benefit:** Network transparency - remote storage looks like local.
|
|
1036
|
+
|
|
1037
|
+
### 4. Bridge Pattern (BsPeerBridge)
|
|
1038
|
+
|
|
1039
|
+
BsPeerBridge bridges socket events to Bs method calls:
|
|
1040
|
+
|
|
1041
|
+
```typescript
|
|
1042
|
+
class BsPeerBridge {
|
|
1043
|
+
constructor(private _bs: Bs, private _socket: Socket) {}
|
|
1044
|
+
|
|
1045
|
+
start() {
|
|
1046
|
+
this._socket.on('getBlob', (blobId, callback) => {
|
|
1047
|
+
this._bs.getBlob(blobId)
|
|
1048
|
+
.then(result => callback(null, result))
|
|
1049
|
+
.catch(error => callback(error, null));
|
|
1050
|
+
});
|
|
1051
|
+
}
|
|
1052
|
+
}
|
|
1053
|
+
```
|
|
1054
|
+
|
|
1055
|
+
**Benefit:** Decouple socket abstraction from Bs implementation.
|
|
1056
|
+
|
|
1057
|
+
### 5. Adapter Pattern (Socket Implementations)
|
|
1058
|
+
|
|
1059
|
+
Different socket libraries adapted to Socket interface:
|
|
1060
|
+
|
|
1061
|
+
```typescript
|
|
1062
|
+
// SocketMock adapts Map to Socket
|
|
1063
|
+
class SocketMock implements Socket { /* ... */ }
|
|
1064
|
+
|
|
1065
|
+
// SocketIOAdapter adapts Socket.IO to Socket
|
|
1066
|
+
class SocketIOAdapter implements Socket {
|
|
1067
|
+
constructor(private _socket: SocketIOSocket) {}
|
|
1068
|
+
// Adapt methods...
|
|
1069
|
+
}
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
**Benefit:** Use any socket library with Bs components.
|
|
1073
|
+
|
|
1074
|
+
## Performance Optimization
|
|
1075
|
+
|
|
1076
|
+
### 1. Deduplication
|
|
1077
|
+
|
|
1078
|
+
**Space Efficiency:**
|
|
1079
|
+
- Same content stored only once
|
|
1080
|
+
- 10 copies of same file = 1 storage cost
|
|
1081
|
+
- Particularly effective for:
|
|
1082
|
+
- Version control (many unchanged files)
|
|
1083
|
+
- Backup systems (incremental with dedup)
|
|
1084
|
+
- Build artifacts (shared dependencies)
|
|
1085
|
+
|
|
1086
|
+
**Example:**
|
|
1087
|
+
```typescript
|
|
1088
|
+
// Traditional storage (10 copies = 10x space)
|
|
1089
|
+
await bs.put('file1', content);
|
|
1090
|
+
await bs.put('file2', content); // Same content, different name
|
|
1091
|
+
// ... 8 more copies
|
|
1092
|
+
|
|
1093
|
+
// Content-addressable (10 copies = 1x space)
|
|
1094
|
+
const { blobId: id1 } = await bs.setBlob(content);
|
|
1095
|
+
const { blobId: id2 } = await bs.setBlob(content); // Same blobId
|
|
1096
|
+
console.log(id1 === id2); // true - only stored once
|
|
1097
|
+
```
|
|
1098
|
+
|
|
1099
|
+
### 2. Hot-Swapping (Cache Population)
|
|
1100
|
+
|
|
1101
|
+
**Automatic Caching:**
|
|
1102
|
+
```typescript
|
|
1103
|
+
// First read from remote (slow)
|
|
1104
|
+
const result = await bsMulti.getBlob(blobId); // Fetches from network
|
|
1105
|
+
|
|
1106
|
+
// Hot-swapped to cache automatically
|
|
1107
|
+
|
|
1108
|
+
// Second read from cache (fast)
|
|
1109
|
+
const result2 = await bsMulti.getBlob(blobId); // Instant from cache
|
|
1110
|
+
```
|
|
1111
|
+
|
|
1112
|
+
**Benefits:**
|
|
1113
|
+
- Transparent caching
|
|
1114
|
+
- No manual cache management
|
|
1115
|
+
- Reduces network requests
|
|
1116
|
+
- Improves read latency
|
|
1117
|
+
|
|
1118
|
+
### 3. Parallel Operations
|
|
1119
|
+
|
|
1120
|
+
**Parallel Writes:**
|
|
1121
|
+
```typescript
|
|
1122
|
+
// BsMulti writes to all writables in parallel
|
|
1123
|
+
await bsMulti.setBlob(content);
|
|
1124
|
+
|
|
1125
|
+
// Instead of serial:
|
|
1126
|
+
// await tier1.setBlob(content);
|
|
1127
|
+
// await tier2.setBlob(content);
|
|
1128
|
+
// await tier3.setBlob(content);
|
|
1129
|
+
|
|
1130
|
+
// Uses Promise.all for parallelism
|
|
1131
|
+
```
|
|
1132
|
+
|
|
1133
|
+
**Parallel Reads (List):**
|
|
1134
|
+
```typescript
|
|
1135
|
+
// Query all stores simultaneously
|
|
1136
|
+
const results = await Promise.all(
|
|
1137
|
+
stores.map(store => store.bs.listBlobs())
|
|
1138
|
+
);
|
|
1139
|
+
// Merge results
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
### 4. Stream Support
|
|
1143
|
+
|
|
1144
|
+
**Large Blob Handling:**
|
|
1145
|
+
```typescript
|
|
1146
|
+
// Avoid loading entire blob into memory
|
|
1147
|
+
const stream = await bs.getBlobStream(blobId);
|
|
1148
|
+
const reader = stream.getReader();
|
|
1149
|
+
|
|
1150
|
+
while (true) {
|
|
1151
|
+
const { done, value } = await reader.read();
|
|
1152
|
+
if (done) break;
|
|
1153
|
+
// Process chunk without buffering entire blob
|
|
1154
|
+
processChunk(value);
|
|
1155
|
+
}
|
|
1156
|
+
```
|
|
1157
|
+
|
|
1158
|
+
**Benefits:**
|
|
1159
|
+
- Constant memory usage
|
|
1160
|
+
- Faster time to first byte
|
|
1161
|
+
- Suitable for large files (GB+)
|
|
1162
|
+
|
|
1163
|
+
### 5. Range Requests
|
|
1164
|
+
|
|
1165
|
+
**Partial Content Retrieval:**
|
|
1166
|
+
```typescript
|
|
1167
|
+
// Download only first 1MB of 100MB blob
|
|
1168
|
+
const { content } = await bs.getBlob(blobId, {
|
|
1169
|
+
range: { start: 0, end: 1024 * 1024 - 1 }
|
|
1170
|
+
});
|
|
1171
|
+
|
|
1172
|
+
// Use for:
|
|
1173
|
+
// - Previews (first few KB)
|
|
1174
|
+
// - Resume downloads
|
|
1175
|
+
// - Random access to large files
|
|
1176
|
+
```
|
|
1177
|
+
|
|
1178
|
+
## Testing Strategy
|
|
1179
|
+
|
|
1180
|
+
### Test Pyramid
|
|
1181
|
+
|
|
1182
|
+
```
|
|
1183
|
+
┌─────────────┐
|
|
1184
|
+
│ Integration │ Full client-server tests
|
|
1185
|
+
│ Tests │ (bs-integration.spec.ts)
|
|
1186
|
+
└─────────────┘
|
|
1187
|
+
┌───────────────┐
|
|
1188
|
+
│ Component │ Per-component tests
|
|
1189
|
+
│ Tests │ (bs-mem.spec.ts, etc.)
|
|
1190
|
+
└───────────────┘
|
|
1191
|
+
┌─────────────────┐
|
|
1192
|
+
│ Unit Tests │ Internal methods, helpers
|
|
1193
|
+
└─────────────────┘
|
|
1194
|
+
```
|
|
1195
|
+
|
|
1196
|
+
### Testing Implementations
|
|
1197
|
+
|
|
1198
|
+
**1. BsMem Testing:**
|
|
1199
|
+
```typescript
|
|
1200
|
+
it('should deduplicate identical content', async () => {
|
|
1201
|
+
const bs = new BsMem();
|
|
1202
|
+
const { blobId: id1 } = await bs.setBlob('same');
|
|
1203
|
+
const { blobId: id2 } = await bs.setBlob('same');
|
|
1204
|
+
expect(id1).toBe(id2);
|
|
1205
|
+
});
|
|
1206
|
+
```
|
|
1207
|
+
|
|
1208
|
+
**2. BsPeer + BsServer Testing:**
|
|
1209
|
+
```typescript
|
|
1210
|
+
it('should handle client-server communication', async () => {
|
|
1211
|
+
const storage = new BsMem();
|
|
1212
|
+
const server = new BsServer(storage);
|
|
1213
|
+
|
|
1214
|
+
const socket = new SocketMock();
|
|
1215
|
+
await server.addSocket(socket);
|
|
1216
|
+
|
|
1217
|
+
const client = new BsPeer(socket);
|
|
1218
|
+
await client.init();
|
|
1219
|
+
|
|
1220
|
+
const { blobId } = await client.setBlob('data');
|
|
1221
|
+
const { content } = await client.getBlob(blobId);
|
|
1222
|
+
expect(content.toString()).toBe('data');
|
|
1223
|
+
});
|
|
1224
|
+
```
|
|
1225
|
+
|
|
1226
|
+
**3. BsMulti Testing:**
|
|
1227
|
+
```typescript
|
|
1228
|
+
it('should hot-swap from remote to cache', async () => {
|
|
1229
|
+
const cache = new BsMem();
|
|
1230
|
+
const remote = new BsMem();
|
|
1231
|
+
|
|
1232
|
+
// Store in remote only
|
|
1233
|
+
const { blobId } = await remote.setBlob('data');
|
|
1234
|
+
|
|
1235
|
+
const multi = new BsMulti([
|
|
1236
|
+
{ bs: cache, priority: 0, read: true, write: true },
|
|
1237
|
+
{ bs: remote, priority: 1, read: true, write: false },
|
|
1238
|
+
]);
|
|
1239
|
+
await multi.init();
|
|
1240
|
+
|
|
1241
|
+
// First read from remote, hot-swaps to cache
|
|
1242
|
+
await multi.getBlob(blobId);
|
|
1243
|
+
|
|
1244
|
+
// Verify cached
|
|
1245
|
+
expect(await cache.blobExists(blobId)).toBe(true);
|
|
1246
|
+
});
|
|
1247
|
+
```
|
|
1248
|
+
|
|
1249
|
+
**4. BsPeerBridge Testing:**
|
|
1250
|
+
```typescript
|
|
1251
|
+
it('should only expose read operations', async () => {
|
|
1252
|
+
const localStorage = new BsMem();
|
|
1253
|
+
const socket = new SocketMock();
|
|
1254
|
+
const bridge = new BsPeerBridge(localStorage, socket);
|
|
1255
|
+
bridge.start();
|
|
1256
|
+
|
|
1257
|
+
// Verify read operations are registered
|
|
1258
|
+
const readOps = ['getBlob', 'blobExists', 'getBlobProperties', 'listBlobs', 'getBlobStream'];
|
|
1259
|
+
for (const op of readOps) {
|
|
1260
|
+
const listeners = socket._listeners.get(op);
|
|
1261
|
+
expect(listeners.length).toBeGreaterThan(0);
|
|
1262
|
+
}
|
|
1263
|
+
|
|
1264
|
+
// Verify write operations are NOT registered
|
|
1265
|
+
const writeOps = ['setBlob', 'deleteBlob', 'generateSignedUrl'];
|
|
1266
|
+
for (const op of writeOps) {
|
|
1267
|
+
const listeners = socket._listeners.get(op);
|
|
1268
|
+
expect(listeners).toBeUndefined();
|
|
1269
|
+
}
|
|
1270
|
+
});
|
|
1271
|
+
```
|
|
1272
|
+
|
|
1273
|
+
### Conformance Tests
|
|
1274
|
+
|
|
1275
|
+
**Golden Tests:**
|
|
1276
|
+
- Store expected behavior in `test/goldens/`
|
|
1277
|
+
- Compare actual output against golden files
|
|
1278
|
+
- Ensures consistent behavior across versions
|
|
1279
|
+
|
|
1280
|
+
**Coverage:**
|
|
1281
|
+
- Target: 100% code coverage
|
|
1282
|
+
- Tools: Vitest + v8 coverage
|
|
1283
|
+
- All branches, statements, functions tested
|
|
1284
|
+
|
|
1285
|
+
## Conclusion
|
|
1286
|
+
|
|
1287
|
+
The `@rljson/bs` architecture provides a flexible, composable, and type-safe blob storage system with:
|
|
1288
|
+
|
|
1289
|
+
1. **Content-addressable storage** for automatic deduplication
|
|
1290
|
+
2. **Modular components** that can be composed into complex hierarchies
|
|
1291
|
+
3. **Network transparency** via socket abstraction
|
|
1292
|
+
4. **PULL architecture** for secure client-server interactions
|
|
1293
|
+
5. **Multi-tier storage** with automatic caching and hot-swapping
|
|
1294
|
+
6. **Full test coverage** ensuring reliability
|
|
1295
|
+
|
|
1296
|
+
The system follows established design patterns (Strategy, Composite, Proxy, Bridge, Adapter) and provides a solid foundation for building distributed storage solutions.
|
|
1297
|
+
|
|
1298
|
+
For further details, see:
|
|
1299
|
+
- [README.public.md](README.public.md) - Usage documentation
|
|
1300
|
+
- [README.contributors.md](README.contributors.md) - Development guide
|
|
1301
|
+
- Source code in `src/` directory
|
|
1302
|
+
- Tests in `test/` directory
|