@genobank/biofs 1.3.4 → 1.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/BIONFS_DEPLOYMENT_GUIDE.md +408 -0
  2. package/BIONFS_MOUNT_INTEGRATION_PLAN.md +415 -0
  3. package/BIONFS_PROTOCOL_SPECIFICATION.md +688 -0
  4. package/BIONFS_SEQUENTIAS_INTEGRATION.md +457 -0
  5. package/BIONFT_NFS_PROTOCOL_ARCHITECTURE.md +583 -0
  6. package/GDPR_BIONFS_CONSENT_ARCHITECTURE.md +798 -0
  7. package/GDPR_BIONFS_IMPLEMENTATION_STATUS.md +462 -0
  8. package/SEQUENTIAS_PROTOCOL_ARCHITECTURE.md +880 -0
  9. package/ULTRATHINK_BIONFS_COMPLETE.md +458 -0
  10. package/WEB3FUSE_LICENSE_TOKEN_ANALYSIS.md +494 -0
  11. package/WHERE_PERMISSIONS_LIVE.md +165 -0
  12. package/dist/commands/access/revoke-consent.d.ts +13 -0
  13. package/dist/commands/access/revoke-consent.d.ts.map +1 -0
  14. package/dist/commands/access/revoke-consent.js +171 -0
  15. package/dist/commands/access/revoke-consent.js.map +1 -0
  16. package/dist/commands/download-with-consent.d.ts +12 -0
  17. package/dist/commands/download-with-consent.d.ts.map +1 -0
  18. package/dist/commands/download-with-consent.js +126 -0
  19. package/dist/commands/download-with-consent.js.map +1 -0
  20. package/dist/commands/mount.d.ts +11 -0
  21. package/dist/commands/mount.d.ts.map +1 -0
  22. package/dist/commands/mount.js +179 -0
  23. package/dist/commands/mount.js.map +1 -0
  24. package/dist/index.js +46 -10
  25. package/dist/index.js.map +1 -1
  26. package/dist/lib/biofiles/resolver.d.ts.map +1 -1
  27. package/dist/lib/biofiles/resolver.js +6 -1
  28. package/dist/lib/biofiles/resolver.js.map +1 -1
  29. package/dist/lib/consent/consent-manager.d.ts +34 -0
  30. package/dist/lib/consent/consent-manager.d.ts.map +1 -0
  31. package/dist/lib/consent/consent-manager.js +96 -0
  32. package/dist/lib/consent/consent-manager.js.map +1 -0
  33. package/dist/lib/consent/consent-prompt.d.ts +27 -0
  34. package/dist/lib/consent/consent-prompt.d.ts.map +1 -0
  35. package/dist/lib/consent/consent-prompt.js +118 -0
  36. package/dist/lib/consent/consent-prompt.js.map +1 -0
  37. package/dist/types/biofiles.d.ts +3 -0
  38. package/dist/types/biofiles.d.ts.map +1 -1
  39. package/lib/bionfs-client.js +125 -0
  40. package/lib/consent-manager.js +88 -0
  41. package/lib/consent-prompt.js +118 -0
  42. package/package.json +2 -2
@@ -0,0 +1,415 @@
1
+ # BioNFS Mount Integration for biofs CLI
2
+
3
+ ## Current Status: Dr. Claudia's Granted Files
4
+
5
+ ### āœ… What's Working
6
+
7
+ Dr. Claudia (wallet `0xb3c3a584491b8ca4df45116a1e250098a0d6192d`) has:
8
+
9
+ **2 License Tokens for VCF File:**
10
+ ```json
11
+ {
12
+ "ip_id": "0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7",
13
+ "filename": "55052008714000.deepvariant.vcf",
14
+ "s3_path": "users/0x5f5a60EaEf242c0D51A21c703f520347b96Ed19a/bioip/3931d9ff-54df-43c5-bd9b-10a273a8d37e/55052008714000.deepvariant.vcf",
15
+ "license_token_id": 40205, // First license
16
+ "license_type": "non-commercial"
17
+ }
18
+ {
19
+ "license_token_id": 40249 // Second license
20
+ }
21
+ ```
22
+
23
+ **BioCID:**
24
+ ```
25
+ biocid://0x5f5a60EaEf242c0D51A21c703f520347b96Ed19a/bioip/0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7
26
+ ```
27
+
28
+ ---
29
+
30
+ ## šŸŽÆ Goal: Enable Dr. Claudia to Mount VCF via BioNFS
31
+
32
+ ### What We Need to Build
33
+
34
+ **1. `biofs mount` Command**
35
+ ```bash
36
+ # Mount all granted files to local directory
37
+ biofs mount ~/genomic-data
38
+
39
+ # Mount specific BioCID
40
+ biofs mount biocid://0x5f5a...d19a/bioip/0xCCe...59f7 ~/my-vcf
41
+
42
+ # Mount with read-only flag
43
+ biofs mount --read-only ~/genomic-data
44
+ ```
45
+
46
+ **2. BioNFS Client Integration**
47
+ - Authenticate with BioNFS server using Web3 signature
48
+ - Stream files using BioNFS protocol (HTTP/2 or gRPC)
49
+ - Create local FUSE mount point
50
+
51
+ **3. BioNFS Server Deployment**
52
+ - Deploy on main API server (genobank.app) - has internet for Story Protocol queries
53
+ - Or add BioNFS endpoints to existing API
54
+
55
+ ---
56
+
57
+ ## Architecture Options
58
+
59
+ ### Option 1: FUSE Mount (Full Linux Compatibility)
60
+
61
+ **Pros:**
62
+ - āœ… Works with any Linux app (IGV, bcftools, samtools)
63
+ - āœ… Transparent to applications
64
+ - āœ… Standard file system semantics
65
+
66
+ **Cons:**
67
+ - āŒ Requires FUSE library
68
+ - āŒ Complex to implement
69
+ - āŒ macOS/Windows require different approaches
70
+
71
+ **Implementation:**
72
+ ```javascript
73
+ // Using fusejs or node-fuse-bindings
74
+ const fuse = require('fusejs');
75
+
76
+ fuse.mount('./mount-point', {
77
+ readdir: async (path) => {
78
+ // List files from BioNFS API
79
+ const files = await bionfs.listFiles(session_id);
80
+ return files.map(f => f.filename);
81
+ },
82
+
83
+ getattr: async (path) => {
84
+ // Get file metadata
85
+ const file = await bionfs.getMetadata(session_id, path);
86
+ return {
87
+ size: file.size,
88
+ mode: 0o444, // Read-only
89
+ mtime: new Date(file.created_at)
90
+ };
91
+ },
92
+
93
+ open: async (path) => {
94
+ // Verify permissions when opening file
95
+ const permitted = await bionfs.checkPermission(session_id, path);
96
+ if (!permitted) throw new Error('Permission denied');
97
+ return 0; // File descriptor
98
+ },
99
+
100
+ read: async (path, fd, buffer, length, position) => {
101
+ // Stream chunk from BioNFS
102
+ const chunk = await bionfs.streamChunk(session_id, path, position, length);
103
+ chunk.copy(buffer);
104
+ return chunk.length;
105
+ }
106
+ });
107
+ ```
108
+
109
+ ---
110
+
111
+ ### Option 2: Smart Download with Caching (Simpler)
112
+
113
+ **Pros:**
114
+ - āœ… No FUSE required
115
+ - āœ… Works on all platforms
116
+ - āœ… Easy to implement
117
+ - āœ… Can cache frequently used files
118
+
119
+ **Cons:**
120
+ - āŒ Not a "real" mount (downloads to temp directory)
121
+ - āŒ Uses disk space
122
+
123
+ **Implementation:**
124
+ ```javascript
125
+ // biofs mount ~/genomic-data
126
+ // Actually downloads files to ~/genomic-data with symlinks
127
+
128
+ async function mountCommand(mountPoint) {
129
+ // 1. Create mount directory
130
+ fs.mkdirSync(mountPoint, { recursive: true });
131
+
132
+ // 2. Get all granted files
133
+ const files = await api.getGrantedBioIPs(userSignature);
134
+
135
+ // 3. For each file, create download job
136
+ for (const file of files) {
137
+ const localPath = path.join(mountPoint, file.filename);
138
+
139
+ // Download in background
140
+ await streamFile(file.biocid, localPath);
141
+
142
+ console.log(`āœ… Mounted: ${file.filename}`);
143
+ }
144
+
145
+ console.log(`\nšŸ“ Files available at: ${mountPoint}`);
146
+ }
147
+ ```
148
+
149
+ ---
150
+
151
+ ### Option 3: Hybrid Approach (Recommended)
152
+
153
+ **Combine both:**
154
+ - Use download/caching for initial access
155
+ - Add FUSE support later for advanced users
156
+
157
+ **Phase 1: Download-based mount (Week 1)**
158
+ ```bash
159
+ biofs mount ~/genomic-data
160
+ # Downloads all granted files to ~/genomic-data
161
+ # Creates .bionfs-cache directory for metadata
162
+ ```
163
+
164
+ **Phase 2: FUSE support (Week 2)**
165
+ ```bash
166
+ biofs mount --fuse ~/genomic-data
167
+ # Real mount point with on-demand streaming
168
+ # No upfront downloads
169
+ ```
170
+
171
+ ---
172
+
173
+ ## Implementation Plan
174
+
175
+ ### Step 1: Add BioNFS Client to biofs CLI
176
+
177
+ **File:** `/home/ubuntu/genobank-cli/lib/bionfs-client.js`
178
+
179
+ ```javascript
180
+ const axios = require('axios');
181
+ const crypto = require('crypto');
182
+
183
+ class BioNFSClient {
184
+ constructor(serverUrl = 'https://genobank.app/bionfs/v1') {
185
+ this.serverUrl = serverUrl;
186
+ this.sessionId = null;
187
+ }
188
+
189
+ /**
190
+ * Authenticate with BioNFS server using Web3 signature
191
+ */
192
+ async authenticate(wallet, signature) {
193
+ const response = await axios.post(`${this.serverUrl}/auth`, {
194
+ wallet: wallet,
195
+ signature: signature,
196
+ message: 'I want to access BioNFS'
197
+ });
198
+
199
+ this.sessionId = response.data.session_id;
200
+ this.permissions = response.data.permissions;
201
+
202
+ return this.sessionId;
203
+ }
204
+
205
+ /**
206
+ * Stream file by BioCID
207
+ */
208
+ async streamFile(biocid, outputPath) {
209
+ // Parse BioCID to extract IP asset ID
210
+ const ipAssetId = this.parseBioCID(biocid);
211
+
212
+ // Stream from BioNFS server
213
+ const response = await axios.get(
214
+ `${this.serverUrl}/stream/${ipAssetId}`,
215
+ {
216
+ headers: { 'X-Session-ID': this.sessionId },
217
+ responseType: 'stream'
218
+ }
219
+ );
220
+
221
+ // Save to disk
222
+ const writer = fs.createWriteStream(outputPath);
223
+ response.data.pipe(writer);
224
+
225
+ return new Promise((resolve, reject) => {
226
+ writer.on('finish', resolve);
227
+ writer.on('error', reject);
228
+ });
229
+ }
230
+
231
+ /**
232
+ * Parse BioCID format:
233
+ * biocid://OWNER/bioip/IP_ASSET_ID
234
+ */
235
+ parseBioCID(biocid) {
236
+ const match = biocid.match(/biocid:\/\/0x[a-fA-F0-9]{40}\/bioip\/(0x[a-fA-F0-9]{40})/);
237
+ if (!match) throw new Error('Invalid BioCID format');
238
+ return match[1]; // IP Asset ID
239
+ }
240
+ }
241
+
242
+ module.exports = BioNFSClient;
243
+ ```
244
+
245
+ ---
246
+
247
+ ### Step 2: Add `mount` Command to biofs CLI
248
+
249
+ **File:** `/home/ubuntu/genobank-cli/commands/mount.js`
250
+
251
+ ```javascript
252
+ const BioNFSClient = require('../lib/bionfs-client');
253
+ const fs = require('fs');
254
+ const path = require('path');
255
+
256
+ async function mountCommand(mountPoint, options) {
257
+ const spinner = ora('Mounting BioFiles...').start();
258
+
259
+ try {
260
+ // 1. Load authentication
261
+ const { wallet, signature } = loadAuth();
262
+
263
+ // 2. Create mount directory
264
+ fs.mkdirSync(mountPoint, { recursive: true });
265
+
266
+ // 3. Authenticate with BioNFS
267
+ const bionfs = new BioNFSClient();
268
+ await bionfs.authenticate(wallet, signature);
269
+ spinner.text = 'Authenticated with BioNFS server';
270
+
271
+ // 4. Get all granted files
272
+ const files = await api.getGrantedBioIPs(signature);
273
+ spinner.text = `Found ${files.length} granted files`;
274
+
275
+ // 5. Mount each file
276
+ for (let i = 0; i < files.length; i++) {
277
+ const file = files[i];
278
+ const localPath = path.join(mountPoint, file.filename);
279
+
280
+ spinner.text = `Mounting ${file.filename} (${i+1}/${files.length})`;
281
+
282
+ // Stream from BioNFS
283
+ await bionfs.streamFile(file.biocid, localPath);
284
+ }
285
+
286
+ spinner.succeed(`āœ… Mounted ${files.length} files to ${mountPoint}`);
287
+
288
+ // 6. Create .bionfs-cache for metadata
289
+ const cacheDir = path.join(mountPoint, '.bionfs-cache');
290
+ fs.mkdirSync(cacheDir, { recursive: true });
291
+ fs.writeFileSync(
292
+ path.join(cacheDir, 'manifest.json'),
293
+ JSON.stringify({ files, mounted_at: new Date() }, null, 2)
294
+ );
295
+
296
+ console.log(`\nšŸ“ Your files are ready at: ${mountPoint}`);
297
+ console.log(`\nšŸ’” Use with any tool:`);
298
+ console.log(` bcftools view ${mountPoint}/55052008714000.deepvariant.vcf`);
299
+ console.log(` IGV ${mountPoint}/55052008714000.deepvariant.vcf`);
300
+
301
+ } catch (error) {
302
+ spinner.fail('Mount failed');
303
+ console.error(error.message);
304
+ process.exit(1);
305
+ }
306
+ }
307
+
308
+ module.exports = mountCommand;
309
+ ```
310
+
311
+ ---
312
+
313
+ ### Step 3: Add BioNFS Endpoints to Main API
314
+
315
+ **Option A: Add to existing `/production_api/run/runweb.py`**
316
+
317
+ ```python
318
+ # Add BioNFS endpoints to main API
319
+ from libs.service.bionfs_service import BioNFSService
320
+
321
+ bionfs_service = BioNFSService()
322
+
323
+ @app.route('/bionfs/v1/auth', methods=['POST'])
324
+ def bionfs_authenticate():
325
+ """
326
+ Authenticate user with Web3 signature
327
+ Returns session_id for subsequent requests
328
+ """
329
+ wallet = request.json.get('wallet')
330
+ signature = request.json.get('signature')
331
+
332
+ # Verify signature
333
+ recovered = recover_wallet(signature)
334
+ if recovered.lower() != wallet.lower():
335
+ return jsonify({'error': 'Invalid signature'}), 401
336
+
337
+ # Get permissions from Story Protocol + MongoDB
338
+ session_id = bionfs_service.create_session(wallet, signature)
339
+
340
+ return jsonify({
341
+ 'session_id': session_id,
342
+ 'wallet': wallet,
343
+ 'expires_at': (datetime.utcnow() + timedelta(hours=24)).isoformat()
344
+ })
345
+
346
+
347
+ @app.route('/bionfs/v1/stream/<ip_asset_id>', methods=['GET'])
348
+ def bionfs_stream_file(ip_asset_id):
349
+ """
350
+ Stream file by IP Asset ID
351
+ Requires X-Session-ID header
352
+ """
353
+ session_id = request.headers.get('X-Session-ID')
354
+
355
+ # Verify session and permissions
356
+ permitted = bionfs_service.check_permission(session_id, ip_asset_id)
357
+ if not permitted:
358
+ return jsonify({'error': 'Permission denied'}), 403
359
+
360
+ # Get S3 path from IP metadata
361
+ s3_path = bionfs_service.get_s3_path(ip_asset_id)
362
+
363
+ # Stream from S3
364
+ return stream_s3_file(s3_path)
365
+ ```
366
+
367
+ ---
368
+
369
+ ## Quick Test for Dr. Claudia
370
+
371
+ ### Test 1: Verify Download Works
372
+
373
+ ```bash
374
+ # Try downloading the granted VCF
375
+ biofs download biocid://0x5f5a60EaEf242c0D51A21c703f520347b96Ed19a/bioip/0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7
376
+
377
+ # Should download 55052008714000.deepvariant.vcf
378
+ ```
379
+
380
+ ### Test 2: Once Mount is Implemented
381
+
382
+ ```bash
383
+ # Mount all granted files
384
+ biofs mount ~/dr-claudia-genomic-data
385
+
386
+ # Use with standard tools
387
+ bcftools view ~/dr-claudia-genomic-data/55052008714000.deepvariant.vcf | head -20
388
+ ```
389
+
390
+ ---
391
+
392
+ ## Next Steps
393
+
394
+ 1. **Verify current download works** - Test Dr. Claudia's permissions
395
+ 2. **Add BioNFS client library** - Implement authentication + streaming
396
+ 3. **Add mount command** - Phase 1 (download-based)
397
+ 4. **Deploy BioNFS endpoints** - Add to main API server
398
+ 5. **Test with Dr. Claudia** - End-to-end verification
399
+
400
+ ---
401
+
402
+ ## Timeline
403
+
404
+ - **Today**: Verify download works, add BioNFS client
405
+ - **Tomorrow**: Implement mount command, deploy endpoints
406
+ - **Day 3**: Test with Dr. Claudia's files
407
+
408
+ ---
409
+
410
+ ## Success Criteria
411
+
412
+ āœ… Dr. Claudia runs: `biofs mount ~/genomic-data`
413
+ āœ… File appears: `~/genomic-data/55052008714000.deepvariant.vcf`
414
+ āœ… Can analyze with: `bcftools stats ~/genomic-data/55052008714000.deepvariant.vcf`
415
+ āœ… Permission verified via Story Protocol license token #40205