@genobank/biofs 2.5.0 → 2.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -0
- package/LICENSE +37 -0
- package/README.md +2 -1
- package/bin/biofs.js +2 -1
- package/contracts/ClaraJobNFT.sol +2 -0
- package/dist/commands/link-clara.d.ts.map +1 -1
- package/dist/commands/link-clara.js +9 -3
- package/dist/commands/link-clara.js.map +1 -1
- package/dist/commands/tokenize-biosample.d.ts.map +1 -1
- package/dist/commands/tokenize-biosample.js +9 -3
- package/dist/commands/tokenize-biosample.js.map +1 -1
- package/dist/index.js +1 -1
- package/docs/AI_AGENT_MCP_MANUAL.md +2 -0
- package/docs/USER_MANUAL.md +2 -0
- package/docs/technical-details.html +555 -0
- package/lib/bionfs-client.js +2 -0
- package/lib/consent-manager.js +2 -0
- package/lib/consent-prompt.js +2 -0
- package/package.json +3 -3
- package/presentation/biofs-x402-kite-presentation.html +2 -0
- package/scripts/build_annotator_dictionary.py +1 -0
- package/scripts/deploy-clara-nft.js +2 -0
- package/test-complete-flow.sh +1 -1
- package/test.vcf +1 -0
- package/test_23andme.txt +1 -0
- package/ADMIN_REGISTER_LAB.md +0 -252
- package/ANNOTATOR_RECOMMENDATIONS_API.md +0 -490
- package/BIOFS_QUICK_START.md +0 -200
- package/BIOFS_TOKENIZE_COMPLETE.md +0 -541
- package/BIOFS_V1.2.7_PUBLISHED.md +0 -317
- package/BIOFS_V1.2.9_FIXES.md +0 -306
- package/BIOFS_V1.3.0_DIRECT_GRANT.md +0 -381
- package/BIOFS_V1.3.1_GRANTED_FILES.md +0 -328
- package/BIONFS_DEPLOYMENT_GUIDE.md +0 -408
- package/BIONFS_MOUNT_INTEGRATION_PLAN.md +0 -415
- package/BIONFS_PROTOCOL_SPECIFICATION.md +0 -688
- package/BIONFS_SEQUENTIAS_INTEGRATION.md +0 -457
- package/BIONFT_NFS_PROTOCOL_ARCHITECTURE.md +0 -583
- package/BIOOS_CLI_IMPLEMENTATION_COMPLETE.md +0 -422
- package/BIOOS_JOB_MANAGEMENT.md +0 -681
- package/BUGFIX_V2.1.3.md +0 -72
- package/CHANGELOG_V2.1.0.md +0 -386
- package/CHANGELOG_V2.2.0.md +0 -284
- package/ENHANCEMENT_DOCUMENTATION.md +0 -218
- package/GDPR_BIONFS_CONSENT_ARCHITECTURE.md +0 -798
- package/GDPR_BIONFS_IMPLEMENTATION_STATUS.md +0 -462
- package/INSTALL_V1.2.7_ON_MAC.md +0 -306
- package/LABNFT_BIOFS_INTEGRATION.md +0 -396
- package/OPENCRAVAT_ANNOTATORS_REFERENCE.md +0 -340
- package/OPENCRAVAT_ANNOTATOR_INTELLIGENCE_COMPLETE.md +0 -479
- package/QUICK_START_V2.1.0.md +0 -116
- package/RELEASE_SUMMARY_V1.2.7.md +0 -479
- package/RELEASE_SUMMARY_V1.3.0.md +0 -324
- package/SEQUENTIAS_PROTOCOL_ARCHITECTURE.md +0 -880
- package/SEQUENTIA_MIGRATION_GUIDE.md +0 -337
- package/SESSION_SUMMARY_DIRECT_GRANT.md +0 -321
- package/ULTRATHINK_BIONFS_COMPLETE.md +0 -458
- package/WEB3FUSE_LICENSE_TOKEN_ANALYSIS.md +0 -494
- package/WHERE_PERMISSIONS_LIVE.md +0 -165
- package/artifacts/@openzeppelin/contracts/access/Ownable.sol/Ownable.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/access/Ownable.sol/Ownable.json +0 -63
- package/artifacts/@openzeppelin/contracts/interfaces/IERC4906.sol/IERC4906.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/interfaces/IERC4906.sol/IERC4906.json +0 -328
- package/artifacts/@openzeppelin/contracts/token/ERC721/ERC721.sol/ERC721.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/token/ERC721/ERC721.sol/ERC721.json +0 -357
- package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721.sol/IERC721.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721.sol/IERC721.json +0 -296
- package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol/IERC721Receiver.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol/IERC721Receiver.json +0 -45
- package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol/ERC721URIStorage.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol/ERC721URIStorage.json +0 -373
- package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol/IERC721Metadata.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol/IERC721Metadata.json +0 -341
- package/artifacts/@openzeppelin/contracts/utils/Address.sol/Address.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/Address.sol/Address.json +0 -10
- package/artifacts/@openzeppelin/contracts/utils/Context.sol/Context.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/Context.sol/Context.json +0 -10
- package/artifacts/@openzeppelin/contracts/utils/Strings.sol/Strings.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/Strings.sol/Strings.json +0 -10
- package/artifacts/@openzeppelin/contracts/utils/introspection/ERC165.sol/ERC165.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/introspection/ERC165.sol/ERC165.json +0 -30
- package/artifacts/@openzeppelin/contracts/utils/introspection/IERC165.sol/IERC165.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/introspection/IERC165.sol/IERC165.json +0 -30
- package/artifacts/@openzeppelin/contracts/utils/math/Math.sol/Math.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/math/Math.sol/Math.json +0 -10
- package/artifacts/@openzeppelin/contracts/utils/math/SignedMath.sol/SignedMath.dbg.json +0 -4
- package/artifacts/@openzeppelin/contracts/utils/math/SignedMath.sol/SignedMath.json +0 -10
- package/artifacts/build-info/1adfbc50ae3e1821260f858a9c689a37.json +0 -1
- package/artifacts/contracts/ClaraJobNFT.sol/ClaraJobNFT.dbg.json +0 -4
- package/artifacts/contracts/ClaraJobNFT.sol/ClaraJobNFT.json +0 -667
- package/cache/solidity-files-cache.json +0 -628
- package/check_clara2.js +0 -70
- package/check_clara3.js +0 -65
- package/config.json +0 -21
- package/hardhat.config.js +0 -25
- package/jest.config.js +0 -27
- package/mint_and_link_all.js +0 -128
- package/opencravat_annotators_dictionary.json +0 -2356
- package/package.json.working.v1.9.2.20251105_023952 +0 -48
- package/test_v2_mint.js +0 -101
|
@@ -1,583 +0,0 @@
|
|
|
1
|
-
# 🧬 BioNFT-NFS Protocol Architecture
|
|
2
|
-
|
|
3
|
-
## 🎯 Vision: True Network File System for Blockchain-Gated Genomic Data
|
|
4
|
-
|
|
5
|
-
**Problem with Presigned URLs**:
|
|
6
|
-
- ❌ Temporary (expire in 1 hour)
|
|
7
|
-
- ❌ AWS-dependent (not our protocol)
|
|
8
|
-
- ❌ HTTP-based (inefficient for large files)
|
|
9
|
-
- ❌ No innovation (just using AWS's existing solution)
|
|
10
|
-
|
|
11
|
-
**BioNFT-NFS Solution**:
|
|
12
|
-
- ✅ Persistent blockchain-based authentication
|
|
13
|
-
- ✅ Our own protocol (not dependent on AWS)
|
|
14
|
-
- ✅ Optimized for genomic data streaming
|
|
15
|
-
- ✅ True innovation (first blockchain NFS)
|
|
16
|
-
|
|
17
|
-
---
|
|
18
|
-
|
|
19
|
-
## 🏗️ Architecture Overview
|
|
20
|
-
|
|
21
|
-
```
|
|
22
|
-
┌─────────────────────────────────────────────────────────────┐
|
|
23
|
-
│ BioNFT-NFS Client │
|
|
24
|
-
│ ┌──────────────┬──────────────┬──────────────────────┐ │
|
|
25
|
-
│ │ biofs mount │ biofs stream │ Application Direct │ │
|
|
26
|
-
│ │ (FUSE) │ (CLI) │ Access (API) │ │
|
|
27
|
-
│ └──────┬───────┴──────┬───────┴──────┬───────────────┘ │
|
|
28
|
-
│ │ │ │ │
|
|
29
|
-
│ └──────────────┴──────────────┘ │
|
|
30
|
-
│ │ │
|
|
31
|
-
│ ┌────────────▼────────────┐ │
|
|
32
|
-
│ │ BioNFS Client Library │ │
|
|
33
|
-
│ │ - Web3 Auth │ │
|
|
34
|
-
│ │ - License Verification │ │
|
|
35
|
-
│ │ - Chunk Streaming │ │
|
|
36
|
-
│ └────────────┬────────────┘ │
|
|
37
|
-
└────────────────────────┼────────────────────────────────────┘
|
|
38
|
-
│
|
|
39
|
-
BioNFS Protocol
|
|
40
|
-
(gRPC / HTTP/2)
|
|
41
|
-
│
|
|
42
|
-
┌────────────────────────▼────────────────────────────────────┐
|
|
43
|
-
│ BioNFT-NFS Server │
|
|
44
|
-
│ ┌──────────────────────────────────────────────────────┐ │
|
|
45
|
-
│ │ Authentication Layer (Story Protocol) │ │
|
|
46
|
-
│ │ - Verify Web3 signature │ │
|
|
47
|
-
│ │ - Check NFT ownership │ │
|
|
48
|
-
│ │ - Verify License Tokens │ │
|
|
49
|
-
│ └──────────────────┬───────────────────────────────────┘ │
|
|
50
|
-
│ │ │
|
|
51
|
-
│ ┌──────────────────▼───────────────────────────────────┐ │
|
|
52
|
-
│ │ File Access Layer │ │
|
|
53
|
-
│ │ - MongoDB metadata lookup │ │
|
|
54
|
-
│ │ - S3 path resolution │ │
|
|
55
|
-
│ │ - Permission caching │ │
|
|
56
|
-
│ └──────────────────┬───────────────────────────────────┘ │
|
|
57
|
-
│ │ │
|
|
58
|
-
│ ┌──────────────────▼───────────────────────────────────┐ │
|
|
59
|
-
│ │ Streaming Layer │ │
|
|
60
|
-
│ │ - Chunked file delivery │ │
|
|
61
|
-
│ │ - Range requests (seek support) │ │
|
|
62
|
-
│ │ - Compression (gzip/zstd) │ │
|
|
63
|
-
│ └──────────────────┬───────────────────────────────────┘ │
|
|
64
|
-
└────────────────────┬┼───────────────────────────────────────┘
|
|
65
|
-
││
|
|
66
|
-
┌──────────────┼┴──────────────┐
|
|
67
|
-
│ │ │
|
|
68
|
-
▼ ▼ ▼
|
|
69
|
-
MongoDB S3 Buckets Story Protocol
|
|
70
|
-
(Metadata) (Files) (Permissions)
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
---
|
|
74
|
-
|
|
75
|
-
## 📋 Protocol Design
|
|
76
|
-
|
|
77
|
-
### 1. **BioNFS Wire Protocol**
|
|
78
|
-
|
|
79
|
-
```protobuf
|
|
80
|
-
// BioNFS Protocol Buffers Definition
|
|
81
|
-
|
|
82
|
-
service BioNFS {
|
|
83
|
-
// Authentication
|
|
84
|
-
rpc Authenticate(AuthRequest) returns (AuthResponse);
|
|
85
|
-
|
|
86
|
-
// File operations
|
|
87
|
-
rpc Open(OpenRequest) returns (OpenResponse);
|
|
88
|
-
rpc Read(ReadRequest) returns (stream ReadResponse);
|
|
89
|
-
rpc Stat(StatRequest) returns (StatResponse);
|
|
90
|
-
rpc List(ListRequest) returns (ListResponse);
|
|
91
|
-
|
|
92
|
-
// License operations
|
|
93
|
-
rpc VerifyLicense(LicenseRequest) returns (LicenseResponse);
|
|
94
|
-
}
|
|
95
|
-
|
|
96
|
-
message AuthRequest {
|
|
97
|
-
string wallet_address = 1;
|
|
98
|
-
string signature = 2;
|
|
99
|
-
int64 timestamp = 3;
|
|
100
|
-
}
|
|
101
|
-
|
|
102
|
-
message AuthResponse {
|
|
103
|
-
string session_token = 1;
|
|
104
|
-
int64 expires_at = 2;
|
|
105
|
-
repeated string granted_ip_ids = 3;
|
|
106
|
-
}
|
|
107
|
-
|
|
108
|
-
message OpenRequest {
|
|
109
|
-
string session_token = 1;
|
|
110
|
-
string ip_id = 2; // IP Asset ID or BioCID
|
|
111
|
-
string mode = 3; // "read", "stream"
|
|
112
|
-
}
|
|
113
|
-
|
|
114
|
-
message OpenResponse {
|
|
115
|
-
string file_handle = 1;
|
|
116
|
-
int64 file_size = 2;
|
|
117
|
-
string content_type = 3;
|
|
118
|
-
FileMetadata metadata = 4;
|
|
119
|
-
}
|
|
120
|
-
|
|
121
|
-
message ReadRequest {
|
|
122
|
-
string file_handle = 1;
|
|
123
|
-
int64 offset = 2;
|
|
124
|
-
int64 length = 3;
|
|
125
|
-
}
|
|
126
|
-
|
|
127
|
-
message ReadResponse {
|
|
128
|
-
bytes data = 1;
|
|
129
|
-
int64 bytes_read = 2;
|
|
130
|
-
bool eof = 3;
|
|
131
|
-
}
|
|
132
|
-
|
|
133
|
-
message FileMetadata {
|
|
134
|
-
string filename = 1;
|
|
135
|
-
string ip_id = 2;
|
|
136
|
-
string owner = 3;
|
|
137
|
-
string license_type = 4;
|
|
138
|
-
int64 created_at = 5;
|
|
139
|
-
string file_type = 6;
|
|
140
|
-
map<string, string> attributes = 7;
|
|
141
|
-
}
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
### 2. **Authentication Flow**
|
|
145
|
-
|
|
146
|
-
```javascript
|
|
147
|
-
// Client authenticates with Web3 signature
|
|
148
|
-
const signature = await wallet.signMessage("BioNFS Access Request");
|
|
149
|
-
|
|
150
|
-
// Server verifies and creates session
|
|
151
|
-
const session = bionfs.authenticate({
|
|
152
|
-
wallet: "0xb3c3a584491b8ca4df45116a1e250098a0d6192d",
|
|
153
|
-
signature: signature,
|
|
154
|
-
timestamp: Date.now()
|
|
155
|
-
});
|
|
156
|
-
|
|
157
|
-
// Session token valid for 24 hours (not 1 hour like presigned URLs!)
|
|
158
|
-
console.log(session.token); // "bionfs_sess_abc123..."
|
|
159
|
-
console.log(session.granted_files); // List of accessible IP IDs
|
|
160
|
-
```
|
|
161
|
-
|
|
162
|
-
### 3. **File Access Flow**
|
|
163
|
-
|
|
164
|
-
```javascript
|
|
165
|
-
// Open file using IP Asset ID
|
|
166
|
-
const file = await bionfs.open({
|
|
167
|
-
session: session.token,
|
|
168
|
-
ip_id: "0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7"
|
|
169
|
-
});
|
|
170
|
-
|
|
171
|
-
// Stream file in chunks (efficient for large VCF files)
|
|
172
|
-
for await (const chunk of bionfs.read(file.handle)) {
|
|
173
|
-
process(chunk); // Process 1MB chunks at a time
|
|
174
|
-
}
|
|
175
|
-
|
|
176
|
-
// Or mount entire filesystem
|
|
177
|
-
await bionfs.mount("/mnt/genobank", {
|
|
178
|
-
session: session.token,
|
|
179
|
-
auto_discover: true // Mount all granted files
|
|
180
|
-
});
|
|
181
|
-
```
|
|
182
|
-
|
|
183
|
-
---
|
|
184
|
-
|
|
185
|
-
## 🛠️ Implementation Plan
|
|
186
|
-
|
|
187
|
-
### Phase 1: BioNFS Server (Python/FastAPI) - 3 days
|
|
188
|
-
|
|
189
|
-
```python
|
|
190
|
-
# /home/ubuntu/Genobank_APIs/bionfs_server/server.py
|
|
191
|
-
|
|
192
|
-
from fastapi import FastAPI, WebSocket, HTTPException
|
|
193
|
-
from fastapi.responses import StreamingResponse
|
|
194
|
-
import grpc
|
|
195
|
-
from concurrent import futures
|
|
196
|
-
|
|
197
|
-
app = FastAPI()
|
|
198
|
-
|
|
199
|
-
class BioNFSServer:
|
|
200
|
-
def __init__(self):
|
|
201
|
-
self.signature_service = SignatureService()
|
|
202
|
-
self.bioip_registry = BioIPRegistryDAO()
|
|
203
|
-
self.license_tokens = LicenseTokensDAO()
|
|
204
|
-
self.bucket_dao = BucketDAO()
|
|
205
|
-
self.sessions = {} # In-memory session cache
|
|
206
|
-
|
|
207
|
-
async def authenticate(self, wallet: str, signature: str):
|
|
208
|
-
"""Authenticate user and return session token"""
|
|
209
|
-
# Verify Web3 signature
|
|
210
|
-
message = "BioNFS Access Request"
|
|
211
|
-
recovered = self.signature_service.recover_from_signature(signature, message)
|
|
212
|
-
|
|
213
|
-
if recovered.lower() != wallet.lower():
|
|
214
|
-
raise HTTPException(401, "Invalid signature")
|
|
215
|
-
|
|
216
|
-
# Get all files user can access
|
|
217
|
-
owned = self.bioip_registry.find({"wallet_address": wallet})
|
|
218
|
-
licensed = self.license_tokens.find({
|
|
219
|
-
"receiver": wallet,
|
|
220
|
-
"status": "active"
|
|
221
|
-
})
|
|
222
|
-
|
|
223
|
-
# Create session (valid 24 hours)
|
|
224
|
-
session_token = self.create_session(wallet, owned, licensed)
|
|
225
|
-
|
|
226
|
-
return {
|
|
227
|
-
"session_token": session_token,
|
|
228
|
-
"expires_at": time.time() + 86400, # 24 hours
|
|
229
|
-
"granted_files": [f["ip_id"] for f in owned + licensed]
|
|
230
|
-
}
|
|
231
|
-
|
|
232
|
-
async def stream_file(self, session_token: str, ip_id: str, offset: int = 0):
|
|
233
|
-
"""Stream file in chunks"""
|
|
234
|
-
# Verify session
|
|
235
|
-
session = self.verify_session(session_token)
|
|
236
|
-
|
|
237
|
-
# Check permission
|
|
238
|
-
if ip_id not in session["granted_files"]:
|
|
239
|
-
raise HTTPException(403, "Access denied")
|
|
240
|
-
|
|
241
|
-
# Get file metadata
|
|
242
|
-
bioip = self.bioip_registry.fetch_one({"ip_id": ip_id})
|
|
243
|
-
s3_path = bioip["s3_path"]
|
|
244
|
-
|
|
245
|
-
# Stream from S3 without presigned URL!
|
|
246
|
-
async def chunk_generator():
|
|
247
|
-
s3_client = self.bucket_dao.get_client()
|
|
248
|
-
response = s3_client.get_object(
|
|
249
|
-
Bucket="test.vault.genoverse.io",
|
|
250
|
-
Key=s3_path,
|
|
251
|
-
Range=f"bytes={offset}-"
|
|
252
|
-
)
|
|
253
|
-
|
|
254
|
-
# Stream 1MB chunks
|
|
255
|
-
for chunk in response["Body"].iter_chunks(chunk_size=1024*1024):
|
|
256
|
-
yield chunk
|
|
257
|
-
|
|
258
|
-
return StreamingResponse(
|
|
259
|
-
chunk_generator(),
|
|
260
|
-
media_type="application/octet-stream",
|
|
261
|
-
headers={
|
|
262
|
-
"Content-Disposition": f"attachment; filename={bioip['filename']}",
|
|
263
|
-
"X-BioNFS-IP-ID": ip_id,
|
|
264
|
-
"X-BioNFS-Owner": bioip["wallet_address"]
|
|
265
|
-
}
|
|
266
|
-
)
|
|
267
|
-
|
|
268
|
-
# REST API endpoints
|
|
269
|
-
@app.post("/bionfs/v1/auth")
|
|
270
|
-
async def authenticate_endpoint(request: AuthRequest):
|
|
271
|
-
server = BioNFSServer()
|
|
272
|
-
return await server.authenticate(request.wallet, request.signature)
|
|
273
|
-
|
|
274
|
-
@app.get("/bionfs/v1/stream/{ip_id}")
|
|
275
|
-
async def stream_file_endpoint(ip_id: str, session: str, offset: int = 0):
|
|
276
|
-
server = BioNFSServer()
|
|
277
|
-
return await server.stream_file(session, ip_id, offset)
|
|
278
|
-
```
|
|
279
|
-
|
|
280
|
-
### Phase 2: BioNFS Client Library (TypeScript) - 2 days
|
|
281
|
-
|
|
282
|
-
```typescript
|
|
283
|
-
// /home/ubuntu/genobank-cli/src/lib/bionfs/client.ts
|
|
284
|
-
|
|
285
|
-
import axios from 'axios';
|
|
286
|
-
import { ethers } from 'ethers';
|
|
287
|
-
|
|
288
|
-
export class BioNFSClient {
|
|
289
|
-
private baseUrl = 'https://bionfs.genobank.app/v1';
|
|
290
|
-
private session?: BioNFSSession;
|
|
291
|
-
|
|
292
|
-
async authenticate(wallet: ethers.Wallet): Promise<BioNFSSession> {
|
|
293
|
-
// Sign authentication message
|
|
294
|
-
const message = "BioNFS Access Request";
|
|
295
|
-
const signature = await wallet.signMessage(message);
|
|
296
|
-
|
|
297
|
-
// Request session from server
|
|
298
|
-
const response = await axios.post(`${this.baseUrl}/auth`, {
|
|
299
|
-
wallet: wallet.address,
|
|
300
|
-
signature: signature,
|
|
301
|
-
timestamp: Date.now()
|
|
302
|
-
});
|
|
303
|
-
|
|
304
|
-
this.session = {
|
|
305
|
-
token: response.data.session_token,
|
|
306
|
-
expiresAt: response.data.expires_at,
|
|
307
|
-
grantedFiles: response.data.granted_files
|
|
308
|
-
};
|
|
309
|
-
|
|
310
|
-
return this.session;
|
|
311
|
-
}
|
|
312
|
-
|
|
313
|
-
async *streamFile(ipId: string): AsyncGenerator<Buffer> {
|
|
314
|
-
if (!this.session) throw new Error("Not authenticated");
|
|
315
|
-
|
|
316
|
-
// Server-side chunking (no presigned URL!)
|
|
317
|
-
const response = await axios.get(`${this.baseUrl}/stream/${ipId}`, {
|
|
318
|
-
params: { session: this.session.token },
|
|
319
|
-
responseType: 'stream'
|
|
320
|
-
});
|
|
321
|
-
|
|
322
|
-
// Yield chunks as they arrive
|
|
323
|
-
for await (const chunk of response.data) {
|
|
324
|
-
yield chunk;
|
|
325
|
-
}
|
|
326
|
-
}
|
|
327
|
-
|
|
328
|
-
async downloadFile(ipId: string, outputPath: string): Promise<void> {
|
|
329
|
-
const writer = fs.createWriteStream(outputPath);
|
|
330
|
-
|
|
331
|
-
for await (const chunk of this.streamFile(ipId)) {
|
|
332
|
-
writer.write(chunk);
|
|
333
|
-
}
|
|
334
|
-
|
|
335
|
-
writer.close();
|
|
336
|
-
}
|
|
337
|
-
}
|
|
338
|
-
```
|
|
339
|
-
|
|
340
|
-
### Phase 3: FUSE Integration - 2 days
|
|
341
|
-
|
|
342
|
-
```c
|
|
343
|
-
// /home/ubuntu/web3fuse/src/bionfs_fuse.c
|
|
344
|
-
|
|
345
|
-
#include <fuse.h>
|
|
346
|
-
#include <curl/curl.h>
|
|
347
|
-
#include "bionfs_client.h"
|
|
348
|
-
|
|
349
|
-
static struct bionfs_session *session;
|
|
350
|
-
|
|
351
|
-
static int bionfs_getattr(const char *path, struct stat *stbuf) {
|
|
352
|
-
// Parse path to get IP ID
|
|
353
|
-
// Example: /0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7/file.vcf
|
|
354
|
-
|
|
355
|
-
bionfs_file_info_t *file = bionfs_stat(session, ip_id);
|
|
356
|
-
|
|
357
|
-
stbuf->st_mode = S_IFREG | 0444; // Read-only file
|
|
358
|
-
stbuf->st_size = file->size;
|
|
359
|
-
stbuf->st_mtime = file->created_at;
|
|
360
|
-
|
|
361
|
-
return 0;
|
|
362
|
-
}
|
|
363
|
-
|
|
364
|
-
static int bionfs_open(const char *path, struct fuse_file_info *fi) {
|
|
365
|
-
// Verify session is valid
|
|
366
|
-
if (!bionfs_verify_session(session)) {
|
|
367
|
-
return -EACCES;
|
|
368
|
-
}
|
|
369
|
-
|
|
370
|
-
// Open file handle
|
|
371
|
-
bionfs_file_t *file = bionfs_open_file(session, ip_id);
|
|
372
|
-
fi->fh = (uint64_t)file;
|
|
373
|
-
|
|
374
|
-
return 0;
|
|
375
|
-
}
|
|
376
|
-
|
|
377
|
-
static int bionfs_read(const char *path, char *buf, size_t size, off_t offset,
|
|
378
|
-
struct fuse_file_info *fi) {
|
|
379
|
-
bionfs_file_t *file = (bionfs_file_t *)fi->fh;
|
|
380
|
-
|
|
381
|
-
// Stream from server
|
|
382
|
-
return bionfs_read_chunk(session, file, buf, size, offset);
|
|
383
|
-
}
|
|
384
|
-
|
|
385
|
-
static struct fuse_operations bionfs_oper = {
|
|
386
|
-
.getattr = bionfs_getattr,
|
|
387
|
-
.open = bionfs_open,
|
|
388
|
-
.read = bionfs_read,
|
|
389
|
-
};
|
|
390
|
-
|
|
391
|
-
int main(int argc, char *argv[]) {
|
|
392
|
-
// Initialize BioNFS session
|
|
393
|
-
session = bionfs_init(wallet_address, signature);
|
|
394
|
-
|
|
395
|
-
// Mount filesystem
|
|
396
|
-
return fuse_main(argc, argv, &bionfs_oper, NULL);
|
|
397
|
-
}
|
|
398
|
-
```
|
|
399
|
-
|
|
400
|
-
---
|
|
401
|
-
|
|
402
|
-
## 🚀 Usage Examples
|
|
403
|
-
|
|
404
|
-
### Example 1: Mount All Granted Files
|
|
405
|
-
|
|
406
|
-
```bash
|
|
407
|
-
# Authenticate once
|
|
408
|
-
biofs login
|
|
409
|
-
|
|
410
|
-
# Mount all granted files to local directory
|
|
411
|
-
biofs mount ~/genomics
|
|
412
|
-
|
|
413
|
-
# Files appear as if they're local!
|
|
414
|
-
ls ~/genomics/
|
|
415
|
-
# 0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7/
|
|
416
|
-
# 55052008714000.deepvariant.vcf
|
|
417
|
-
# 0x89224559242246F93479Fc44B0d8a1AFF5950faB/
|
|
418
|
-
# analysis_results.sqlite
|
|
419
|
-
|
|
420
|
-
# Open in any tool (no download needed!)
|
|
421
|
-
bcftools view ~/genomics/0xCCe1.../55052008714000.deepvariant.vcf
|
|
422
|
-
```
|
|
423
|
-
|
|
424
|
-
### Example 2: Stream Large VCF File
|
|
425
|
-
|
|
426
|
-
```bash
|
|
427
|
-
# Stream file directly (no full download)
|
|
428
|
-
biofs stream 0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7 | grep "CHROM"
|
|
429
|
-
|
|
430
|
-
# Works with any Unix tool
|
|
431
|
-
biofs stream 0xCCe1... | head -n 1000 | less
|
|
432
|
-
```
|
|
433
|
-
|
|
434
|
-
### Example 3: Application Integration
|
|
435
|
-
|
|
436
|
-
```python
|
|
437
|
-
# Python app using BioNFS directly
|
|
438
|
-
from genobank_bionfs import BioNFSClient
|
|
439
|
-
|
|
440
|
-
client = BioNFSClient()
|
|
441
|
-
await client.authenticate(wallet_address, signature)
|
|
442
|
-
|
|
443
|
-
# Stream file without downloading
|
|
444
|
-
async for variant in client.stream_vcf("0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7"):
|
|
445
|
-
if variant.is_pathogenic():
|
|
446
|
-
print(f"Found pathogenic variant: {variant}")
|
|
447
|
-
```
|
|
448
|
-
|
|
449
|
-
---
|
|
450
|
-
|
|
451
|
-
## 🎯 Advantages Over Presigned URLs
|
|
452
|
-
|
|
453
|
-
| Feature | Presigned URLs | BioNFT-NFS |
|
|
454
|
-
|---------|---------------|------------|
|
|
455
|
-
| **Expiration** | 1 hour | 24 hours (session) |
|
|
456
|
-
| **Protocol** | AWS HTTP | Our custom protocol |
|
|
457
|
-
| **Streaming** | Download full file | Chunk-by-chunk streaming |
|
|
458
|
-
| **Mounting** | Not possible | FUSE mount support |
|
|
459
|
-
| **Innovation** | AWS solution | GenoBank innovation 🚀 |
|
|
460
|
-
| **Blockchain Auth** | Separate | Native integration |
|
|
461
|
-
| **License Verification** | Per-request | Session-based caching |
|
|
462
|
-
| **Seek Support** | No | Yes (range requests) |
|
|
463
|
-
|
|
464
|
-
---
|
|
465
|
-
|
|
466
|
-
## 📊 Performance Benefits
|
|
467
|
-
|
|
468
|
-
### Scenario: 150GB WGS VCF File
|
|
469
|
-
|
|
470
|
-
**Presigned URL Approach**:
|
|
471
|
-
```
|
|
472
|
-
1. Request presigned URL (1 second)
|
|
473
|
-
2. Download entire 150GB file (30+ minutes on fast connection)
|
|
474
|
-
3. Process file locally
|
|
475
|
-
Total time: 30+ minutes before analysis starts
|
|
476
|
-
```
|
|
477
|
-
|
|
478
|
-
**BioNFS Approach**:
|
|
479
|
-
```
|
|
480
|
-
1. Authenticate session (1 second)
|
|
481
|
-
2. Stream first 1MB chunk (immediate)
|
|
482
|
-
3. Start processing while streaming rest
|
|
483
|
-
Total time: Analysis starts in <2 seconds!
|
|
484
|
-
```
|
|
485
|
-
|
|
486
|
-
### Scenario: Checking First 1000 Variants
|
|
487
|
-
|
|
488
|
-
**Presigned URL**:
|
|
489
|
-
```bash
|
|
490
|
-
# Download entire 150GB file first
|
|
491
|
-
curl -o file.vcf "https://s3.amazonaws.com/...?signature=..."
|
|
492
|
-
head -n 1000 file.vcf
|
|
493
|
-
# 30 minutes wasted
|
|
494
|
-
```
|
|
495
|
-
|
|
496
|
-
**BioNFS**:
|
|
497
|
-
```bash
|
|
498
|
-
# Stream only what you need
|
|
499
|
-
biofs stream 0xCCe1... | head -n 1000
|
|
500
|
-
# 2 seconds total
|
|
501
|
-
```
|
|
502
|
-
|
|
503
|
-
---
|
|
504
|
-
|
|
505
|
-
## 🔒 Security Model
|
|
506
|
-
|
|
507
|
-
### Session-Based Authentication (Better than Presigned URLs)
|
|
508
|
-
|
|
509
|
-
```
|
|
510
|
-
Traditional Presigned URL:
|
|
511
|
-
URL contains all credentials → Anyone with URL has access
|
|
512
|
-
|
|
513
|
-
BioNFS Session:
|
|
514
|
-
Session token tied to specific wallet address
|
|
515
|
-
Server verifies blockchain permissions on every request
|
|
516
|
-
Token cannot be transferred (wallet-bound)
|
|
517
|
-
```
|
|
518
|
-
|
|
519
|
-
### Audit Trail
|
|
520
|
-
|
|
521
|
-
```python
|
|
522
|
-
# Every file access logged
|
|
523
|
-
{
|
|
524
|
-
"wallet": "0xb3c3a584491b8ca4df45116a1e250098a0d6192d",
|
|
525
|
-
"ip_id": "0xCCe14315eE3D6a41596EeB4a2839eE50A8ec59f7",
|
|
526
|
-
"access_type": "stream",
|
|
527
|
-
"bytes_read": 1048576,
|
|
528
|
-
"timestamp": "2025-10-07T04:30:00Z",
|
|
529
|
-
"license_token_id": 40205
|
|
530
|
-
}
|
|
531
|
-
```
|
|
532
|
-
|
|
533
|
-
---
|
|
534
|
-
|
|
535
|
-
## 🛣️ Roadmap
|
|
536
|
-
|
|
537
|
-
### Week 1: Core Protocol
|
|
538
|
-
- ✅ Protocol specification (gRPC/protobuf)
|
|
539
|
-
- ✅ Server implementation (FastAPI)
|
|
540
|
-
- ✅ Basic authentication
|
|
541
|
-
|
|
542
|
-
### Week 2: Client Library
|
|
543
|
-
- ✅ TypeScript client
|
|
544
|
-
- ✅ Streaming support
|
|
545
|
-
- ✅ Session management
|
|
546
|
-
|
|
547
|
-
### Week 3: FUSE Integration
|
|
548
|
-
- ✅ Update web3fuse
|
|
549
|
-
- ✅ License token verification
|
|
550
|
-
- ✅ Mount command
|
|
551
|
-
|
|
552
|
-
### Week 4: Production
|
|
553
|
-
- ✅ Performance testing
|
|
554
|
-
- ✅ Security audit
|
|
555
|
-
- ✅ Documentation
|
|
556
|
-
- ✅ Deploy to bionfs.genobank.app
|
|
557
|
-
|
|
558
|
-
---
|
|
559
|
-
|
|
560
|
-
## 💡 Why This Is Innovative
|
|
561
|
-
|
|
562
|
-
1. **First Blockchain NFS**: No one has built a network file system with native blockchain authentication
|
|
563
|
-
2. **Genomics-Optimized**: Designed specifically for large genomic files (not generic cloud storage)
|
|
564
|
-
3. **License-Aware**: Story Protocol PIL terms integrated at protocol level
|
|
565
|
-
4. **GDPR-Compliant**: Audit trail + right to erasure (can revoke sessions)
|
|
566
|
-
5. **Open Protocol**: Can be adopted by other genomics platforms
|
|
567
|
-
|
|
568
|
-
---
|
|
569
|
-
|
|
570
|
-
## 🎉 Result
|
|
571
|
-
|
|
572
|
-
Instead of temporary AWS presigned URLs, we have:
|
|
573
|
-
|
|
574
|
-
```bash
|
|
575
|
-
# The GenoBank Way 🧬
|
|
576
|
-
biofs mount ~/genomics
|
|
577
|
-
cd ~/genomics
|
|
578
|
-
ls -la # See all your granted files
|
|
579
|
-
bcftools view 0xCCe1.../file.vcf | grep "pathogenic" # Instant analysis!
|
|
580
|
-
```
|
|
581
|
-
|
|
582
|
-
**This is true innovation** - a blockchain-native network file system for genomic data! 🚀
|
|
583
|
-
|