@genobank/biofs 1.6.2 β†’ 1.6.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,557 @@
1
+ # 🧬 BioFS Node vs Client Architecture Analysis
2
+
3
+ **Date**: October 24, 2025
4
+ **Status**: Architectural Planning
5
+ **Decision Required**: Node/Client separation strategy
6
+
7
+ ---
8
+
9
+ ## 🎯 Vision: BioFS as a Distributed Data Network
10
+
11
+ ### Current State
12
+ **biofs-cli** = Client-only tool for individual researchers
13
+
14
+ ### Desired State
15
+ **BioFS Network** = Nodes (data providers) + Clients (data consumers)
16
+
17
+ ```
18
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
19
+ β”‚ AUGenomics Node │◄───── Dr. Claudia Client β”‚
20
+ β”‚ (biofs-node) β”‚ β”‚ (biofs-cli) β”‚
21
+ β”‚ β”‚ β”‚ β”‚
22
+ β”‚ β€’ Serves S3 bucket β”‚ β”‚ β€’ Downloads files β”‚
23
+ β”‚ β€’ Exposes API β”‚ β”‚ β€’ LLM analysis β”‚
24
+ β”‚ β€’ Multi-user β”‚ β”‚ β€’ Personal BioWalletβ”‚
25
+ β”‚ β€’ Institutional β”‚ β”‚ β€’ Single-user β”‚
26
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
27
+ β–² β”‚
28
+ β”‚ β”‚
29
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
30
+ β”‚
31
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
32
+ β”‚ Story Protocol β”‚
33
+ β”‚ (License NFTs) β”‚
34
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
35
+ ```
36
+
37
+ ---
38
+
39
+ ## πŸ“Š Current Architecture Analysis
40
+
41
+ ### What Exists in biofs-cli v1.6.2
42
+
43
+ #### βœ… Client Commands
44
+ ```bash
45
+ biofs login # Authenticate with BioWallet
46
+ biofs files # List user's files
47
+ biofs download # Download genomic data
48
+ biofs upload # Upload new files
49
+ biofs tokenize # Mint BioIP NFTs
50
+ biofs access grant # Grant PIL licenses
51
+ biofs mount # Mount granted files locally
52
+ biofs job create # Run BioOS pipelines
53
+ ```
54
+
55
+ #### βœ… Infrastructure Already Present
56
+ | Component | Status | Location | Purpose |
57
+ |-----------|--------|----------|---------|
58
+ | **Express Server** | βœ… Exists | `src/lib/auth/server.ts` | OAuth callback handling |
59
+ | **BioNFS Spec** | βœ… Documented | `BIONFS_PROTOCOL_SPECIFICATION.md` | Network protocol design |
60
+ | **Mount Command** | βœ… Implemented | `src/commands/mount.ts` | Local filesystem mounting |
61
+ | **Consent Flow** | βœ… Implemented | `src/lib/consent/` | GDPR compliance |
62
+ | **Story Protocol** | βœ… Integrated | `src/lib/bioip/licenses.ts` | License verification |
63
+ | **Chunked Upload** | βœ… Implemented | `src/lib/upload/chunked.ts` | Large file handling |
64
+
65
+ #### πŸ” Current Server Capabilities
66
+
67
+ **CallbackServer** (`src/lib/auth/server.ts`):
68
+ - Runs Express on configurable port
69
+ - Handles OAuth callbacks
70
+ - Auto-closes after auth
71
+ - **Limitation**: Temporary, not persistent
72
+
73
+ ---
74
+
75
+ ## πŸ—οΈ Proposed Architectures
76
+
77
+ ### **Option 1: Flag-Based (Single Package)** ⚑ FASTEST
78
+
79
+ Add `biofs serve` command to existing CLI:
80
+
81
+ ```bash
82
+ # Client mode (unchanged)
83
+ biofs files
84
+ biofs download genome.vcf
85
+
86
+ # NEW: Node mode
87
+ biofs serve --bucket s3://augenomics-genomics \
88
+ --wallet 0xAUGenomics... \
89
+ --port 8080 \
90
+ --daemon
91
+ ```
92
+
93
+ **Implementation**:
94
+ ```typescript
95
+ // src/commands/serve.ts (NEW FILE)
96
+ export async function serveCommand(options: ServeOptions) {
97
+ const nodeServer = new BioFSNodeServer({
98
+ bucket: options.bucket,
99
+ wallet: options.wallet,
100
+ port: options.port,
101
+ daemon: options.daemon
102
+ });
103
+
104
+ await nodeServer.start();
105
+ }
106
+ ```
107
+
108
+ **Pros**:
109
+ - βœ… Fastest to implement (1-2 days)
110
+ - βœ… Single npm package
111
+ - βœ… Reuse all existing code
112
+ - βœ… Natural CLI evolution
113
+
114
+ **Cons**:
115
+ - ❌ Mixes client/server concerns
116
+ - ❌ Larger binary size
117
+ - ❌ Confusing documentation
118
+
119
+ ---
120
+
121
+ ### **Option 2: Separate Packages** πŸ† RECOMMENDED
122
+
123
+ Create three packages:
124
+
125
+ ```
126
+ @genobank/biofs-core (Shared library)
127
+ @genobank/biofs (Client CLI - researchers)
128
+ @genobank/biofs-node (Server daemon - institutions)
129
+ ```
130
+
131
+ #### Package Responsibilities
132
+
133
+ **@genobank/biofs-core** (New - Shared SDK)
134
+ ```typescript
135
+ export * from './auth/credentials';
136
+ export * from './bioip/licenses';
137
+ export * from './consent/manager';
138
+ export * from './api/client';
139
+ export { BioCIDResolver } from './biofiles/biocid';
140
+ ```
141
+
142
+ **@genobank/biofs** (Current - Refactored)
143
+ ```bash
144
+ # Researcher-focused commands
145
+ biofs login --wallet 0xDrClaudia...
146
+ biofs files
147
+ biofs download biocid://...
148
+ biofs job create "Find cancer variants" genome.vcf
149
+ ```
150
+
151
+ **@genobank/biofs-node** (New - Server)
152
+ ```bash
153
+ # Institution-focused daemon
154
+ biofs-node start --config /etc/biofs/augenomics.yaml
155
+ biofs-node status
156
+ biofs-node stop
157
+ ```
158
+
159
+ **Node Configuration** (`/etc/biofs/augenomics.yaml`):
160
+ ```yaml
161
+ node:
162
+ name: "AUGenomics Genomics Node"
163
+ wallet: "0x..." # Institutional wallet
164
+
165
+ storage:
166
+ bucket: "s3://augenomics-genomics"
167
+ region: "us-east-1"
168
+
169
+ api:
170
+ port: 8080
171
+ host: "0.0.0.0"
172
+ cors_origins:
173
+ - "https://lab.augenomics.org"
174
+ rate_limit:
175
+ requests_per_minute: 100
176
+
177
+ access:
178
+ verify_story_licenses: true
179
+ require_gdpr_consent: true
180
+
181
+ discovery:
182
+ register_with_network: true
183
+ announce_datasets: true
184
+ ```
185
+
186
+ **Pros**:
187
+ - βœ… Clear separation of concerns
188
+ - βœ… Optimized binaries
189
+ - βœ… Better documentation
190
+ - βœ… Different security models
191
+ - βœ… Easier testing
192
+
193
+ **Cons**:
194
+ - ❌ More repos to maintain
195
+ - ❌ Longer development time (1-2 weeks)
196
+ - ❌ Migration effort
197
+
198
+ ---
199
+
200
+ ### **Option 3: Hybrid** πŸ”€ COMPROMISE
201
+
202
+ Keep CLI as-is, add minimal server command:
203
+
204
+ ```bash
205
+ # Client commands (unchanged)
206
+ biofs files
207
+
208
+ # Simple server mode
209
+ biofs serve --bucket s3://augenomics-genomics --port 8080
210
+
211
+ # Later: Full node
212
+ npm install -g @genobank/biofs-node
213
+ biofs-node start --config augenomics.yaml
214
+ ```
215
+
216
+ **Implementation Path**:
217
+ 1. **Phase 1** (Now): Add `biofs serve` to CLI
218
+ 2. **Phase 2** (Later): Extract to `biofs-node` when needed
219
+
220
+ ---
221
+
222
+ ## 🎭 Use Cases Comparison
223
+
224
+ ### Researcher (Dr. Claudia) - Client Mode
225
+
226
+ **Current Workflow**:
227
+ ```bash
228
+ # Authenticate
229
+ biofs login --wallet 0xDrClaudia...
230
+
231
+ # Discover granted files
232
+ biofs files --granted
233
+
234
+ # Download for analysis
235
+ biofs download biocid://0x.../vcf/patient-123.vcf
236
+
237
+ # Run LLM analysis
238
+ biofs job create "Find hereditary cancer variants" patient-123.vcf
239
+
240
+ # Results go to her BioWallet
241
+ ```
242
+
243
+ **Needs**:
244
+ - βœ… Personal authentication
245
+ - βœ… Access granted files
246
+ - βœ… Download capabilities
247
+ - βœ… LLM integration
248
+ - ❌ Does NOT need server mode
249
+
250
+ ---
251
+
252
+ ### Institution (AUGenomics) - Node Mode
253
+
254
+ **Desired Workflow**:
255
+ ```bash
256
+ # Setup node
257
+ biofs-node init --bucket s3://augenomics-genomics \
258
+ --wallet 0xAUGenomics...
259
+
260
+ # Start server
261
+ biofs-node start --config /etc/biofs/augenomics.yaml
262
+
263
+ # Node is now running:
264
+ # - Serves files from s3://augenomics-genomics
265
+ # - Exposes REST API on port 8080
266
+ # - Verifies Story Protocol licenses
267
+ # - Logs all access (GDPR audit)
268
+ ```
269
+
270
+ **API Endpoints for Data Scientists**:
271
+ ```bash
272
+ # Discovery
273
+ GET /api/v1/datasets
274
+ GET /api/v1/datasets/:id/metadata
275
+
276
+ # Access (requires license token)
277
+ GET /api/v1/files/:file_id/download
278
+ GET /api/v1/files/:file_id/stream
279
+
280
+ # License verification
281
+ POST /api/v1/verify-access
282
+ ```
283
+
284
+ **Needs**:
285
+ - βœ… Long-running daemon
286
+ - βœ… Multi-user support
287
+ - βœ… API for programmatic access
288
+ - βœ… Institutional bucket serving
289
+ - βœ… Monitoring/health checks
290
+ - ❌ Does NOT need client features
291
+
292
+ ---
293
+
294
+ ## πŸ” Security Differences
295
+
296
+ | Feature | Client (biofs) | Node (biofs-node) |
297
+ |---------|----------------|-------------------|
298
+ | **Auth Model** | Personal wallet signature | Institutional wallet + JWT |
299
+ | **Session** | Short-lived (30 days) | Long-lived service |
300
+ | **Users** | Single user (researcher) | Multi-tenant |
301
+ | **API** | Calls GenoBank backend | Exposes own API |
302
+ | **Data Source** | GenoBank S3 | Institutional S3 bucket |
303
+ | **Access Control** | Granted licenses | License verification |
304
+ | **Deployment** | User's laptop | Production server |
305
+ | **Monitoring** | None | Prometheus, logs |
306
+
307
+ ---
308
+
309
+ ## πŸš€ Recommended Implementation: **Option 2** (Separate Packages)
310
+
311
+ ### Why?
312
+
313
+ 1. **Clear Mental Model**: Researchers use CLI, institutions use Node
314
+ 2. **Security**: Different threat models require different implementations
315
+ 3. **Maintenance**: Easier to version and update independently
316
+ 4. **Documentation**: Each tool has focused docs
317
+ 5. **Future-Proof**: Enables node federation, discovery protocol
318
+
319
+ ### Implementation Plan (2 Weeks)
320
+
321
+ #### Week 1: Extract Core
322
+ ```bash
323
+ # Create core package
324
+ mkdir -p biofs-core/src
325
+ # Extract shared code:
326
+ - auth/credentials
327
+ - bioip/licenses
328
+ - consent/manager
329
+ - api/client
330
+ - biofiles/biocid
331
+
332
+ # Publish @genobank/biofs-core@1.0.0
333
+ ```
334
+
335
+ #### Week 2: Build Node
336
+ ```bash
337
+ # Create node package
338
+ mkdir -p biofs-node/src
339
+ # Implement:
340
+ - Daemon process (systemd)
341
+ - REST API server (Express/Fastify)
342
+ - Multi-user auth
343
+ - License verification middleware
344
+ - Health checks
345
+ - Logging
346
+
347
+ # Publish @genobank/biofs-node@1.0.0
348
+ ```
349
+
350
+ ---
351
+
352
+ ## πŸ“‹ Technical Specifications
353
+
354
+ ### BioFS Node API (REST)
355
+
356
+ **Base URL**: `https://node.augenomics.org/api/v1`
357
+
358
+ #### Authentication
359
+ ```http
360
+ POST /auth/verify
361
+ Content-Type: application/json
362
+
363
+ {
364
+ "wallet": "0x...",
365
+ "signature": "0x...",
366
+ "message": "Access request"
367
+ }
368
+
369
+ Response:
370
+ {
371
+ "token": "eyJ...", // JWT
372
+ "expires_at": "2025-11-01T00:00:00Z"
373
+ }
374
+ ```
375
+
376
+ #### Discovery
377
+ ```http
378
+ GET /datasets
379
+ Authorization: Bearer eyJ...
380
+
381
+ Response:
382
+ {
383
+ "datasets": [
384
+ {
385
+ "id": "uuid",
386
+ "name": "AUGenomics WES Collection",
387
+ "type": "vcf",
388
+ "files_count": 1500,
389
+ "total_size_gb": 450,
390
+ "license_type": "PIL_NON_COMMERCIAL_REMIX",
391
+ "requires_consent": true
392
+ }
393
+ ]
394
+ }
395
+ ```
396
+
397
+ #### File Access
398
+ ```http
399
+ GET /files/:file_id/download
400
+ Authorization: Bearer eyJ...
401
+ X-License-Token-ID: 40447
402
+
403
+ Response: [Binary stream or 403 Forbidden]
404
+ ```
405
+
406
+ #### License Verification
407
+ ```http
408
+ POST /verify-access
409
+ Authorization: Bearer eyJ...
410
+
411
+ {
412
+ "ip_id": "0x...",
413
+ "wallet": "0x...",
414
+ "action": "download"
415
+ }
416
+
417
+ Response:
418
+ {
419
+ "authorized": true,
420
+ "license_token_id": 40447,
421
+ "license_type": "GDPR_RESEARCH",
422
+ "expires_at": null # Or timestamp if temporary
423
+ }
424
+ ```
425
+
426
+ ---
427
+
428
+ ## 🎯 Decision Matrix
429
+
430
+ | Criteria | Flag-Based | Separate Packages | Hybrid |
431
+ |----------|------------|-------------------|--------|
432
+ | **Time to MVP** | ⭐⭐⭐⭐⭐ 2 days | ⭐⭐⭐ 2 weeks | ⭐⭐⭐⭐ 1 week |
433
+ | **Maintainability** | ⭐⭐ Confusing | ⭐⭐⭐⭐⭐ Clear | ⭐⭐⭐⭐ Good |
434
+ | **User Experience** | ⭐⭐⭐ Okay | ⭐⭐⭐⭐⭐ Excellent | ⭐⭐⭐⭐ Good |
435
+ | **Security** | ⭐⭐⭐ Mixed | ⭐⭐⭐⭐⭐ Separated | ⭐⭐⭐⭐ Good |
436
+ | **Documentation** | ⭐⭐ Complex | ⭐⭐⭐⭐⭐ Clear | ⭐⭐⭐⭐ Good |
437
+ | **Future-Proof** | ⭐⭐ Limited | ⭐⭐⭐⭐⭐ Scalable | ⭐⭐⭐⭐ Good |
438
+
439
+ **Winner**: πŸ† **Separate Packages** (Option 2)
440
+
441
+ ---
442
+
443
+ ## πŸ› οΈ Alternative: Quick Start with Flag
444
+
445
+ If we need something **NOW** (within 48 hours):
446
+
447
+ ### Minimal Implementation
448
+
449
+ ```typescript
450
+ // src/commands/serve.ts (NEW)
451
+ import express from 'express';
452
+ import { GenoBankAPIClient } from '../lib/api/client';
453
+
454
+ export async function serveCommand(options: ServeOptions) {
455
+ const app = express();
456
+
457
+ // Middleware
458
+ app.use(express.json());
459
+ app.use(cors(options.corsOrigins));
460
+
461
+ // API endpoints
462
+ app.get('/api/v1/files', async (req, res) => {
463
+ const { authorization } = req.headers;
464
+ // Verify license token
465
+ // Return file list from options.bucket
466
+ });
467
+
468
+ app.get('/api/v1/files/:id/download', async (req, res) => {
469
+ // Verify license
470
+ // Stream from S3
471
+ });
472
+
473
+ // Start server
474
+ app.listen(options.port, () => {
475
+ console.log(`πŸš€ BioFS Node running on port ${options.port}`);
476
+ });
477
+ }
478
+ ```
479
+
480
+ **Add to CLI**:
481
+ ```typescript
482
+ // src/index.ts
483
+ program
484
+ .command('serve')
485
+ .description('Start BioFS node server (institutional use)')
486
+ .option('--bucket <s3_uri>', 'S3 bucket to serve')
487
+ .option('--wallet <address>', 'Institutional wallet')
488
+ .option('--port <number>', 'API port', parseInt, 8080)
489
+ .option('--daemon', 'Run as background daemon')
490
+ .action(serveCommand);
491
+ ```
492
+
493
+ **Timeline**: ⚑ 1-2 days
494
+
495
+ ---
496
+
497
+ ## 🎬 Recommended Action Plan
498
+
499
+ ### Immediate (This Week)
500
+ 1. **Implement `biofs serve`** (Flag-based approach)
501
+ 2. **Test with AUGenomics bucket**
502
+ 3. **Document API endpoints**
503
+ 4. **Get user feedback**
504
+
505
+ ### Near-Term (Next Month)
506
+ 5. **Extract shared code** to `@genobank/biofs-core`
507
+ 6. **Create `@genobank/biofs-node`** as separate package
508
+ 7. **Migrate AUGenomics** to dedicated node
509
+ 8. **Deprecate `biofs serve`** in favor of `biofs-node`
510
+
511
+ ### Long-Term (Q1 2026)
512
+ 9. **Node discovery protocol**
513
+ 10. **Federation between nodes**
514
+ 11. **Distributed catalog**
515
+ 12. **Cross-node queries**
516
+
517
+ ---
518
+
519
+ ## πŸ“ž Questions for Decision
520
+
521
+ 1. **Timeline**: Do we need node functionality NOW (2 days) or can we do it right (2 weeks)?
522
+ 2. **Use Cases**: Is AUGenomics the only institution, or will there be many nodes?
523
+ 3. **API Requirements**: What specific endpoints do data scientists need?
524
+ 4. **Authentication**: Should nodes have their own auth, or proxy to GenoBank?
525
+ 5. **Deployment**: Docker, systemd, or Kubernetes?
526
+
527
+ ---
528
+
529
+ ## πŸ’‘ My Recommendation
530
+
531
+ **Start Fast, Refactor Later**:
532
+
533
+ 1. **This Week**: Add `biofs serve --bucket s3://... --port 8080`
534
+ - Quick win for AUGenomics
535
+ - Test the concept
536
+ - Gather requirements
537
+
538
+ 2. **Next Sprint**: Build `@genobank/biofs-node` properly
539
+ - Extract after learning from serve command
540
+ - Professional daemon with systemd
541
+ - Production-ready API
542
+
543
+ 3. **Future**: Node federation
544
+ - Discovery protocol
545
+ - Cross-node queries
546
+ - Decentralized catalog
547
+
548
+ **Rationale**: "Make it work, then make it right, then make it fast"
549
+
550
+ ---
551
+
552
+ **Decision needed**: Which approach should we take?
553
+
554
+ A. **Quick Flag** (`biofs serve`) - 2 days
555
+ B. **Separate Package** (`biofs-node`) - 2 weeks
556
+ C. **Hybrid** (Flag now, package later) - Best of both βœ…
557
+