@genobank/biofs 2.5.0 → 2.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (100) hide show
  1. package/CHANGELOG.md +2 -0
  2. package/LICENSE +37 -0
  3. package/README.md +2 -1
  4. package/bin/biofs.js +2 -1
  5. package/contracts/ClaraJobNFT.sol +2 -0
  6. package/dist/commands/link-clara.d.ts.map +1 -1
  7. package/dist/commands/link-clara.js +9 -3
  8. package/dist/commands/link-clara.js.map +1 -1
  9. package/dist/commands/tokenize-biosample.d.ts.map +1 -1
  10. package/dist/commands/tokenize-biosample.js +9 -3
  11. package/dist/commands/tokenize-biosample.js.map +1 -1
  12. package/dist/index.js +1 -1
  13. package/docs/AI_AGENT_MCP_MANUAL.md +2 -0
  14. package/docs/USER_MANUAL.md +2 -0
  15. package/docs/technical-details.html +555 -0
  16. package/lib/bionfs-client.js +2 -0
  17. package/lib/consent-manager.js +2 -0
  18. package/lib/consent-prompt.js +2 -0
  19. package/package.json +3 -3
  20. package/presentation/biofs-x402-kite-presentation.html +2 -0
  21. package/scripts/build_annotator_dictionary.py +1 -0
  22. package/scripts/deploy-clara-nft.js +2 -0
  23. package/test-complete-flow.sh +1 -1
  24. package/test.vcf +1 -0
  25. package/test_23andme.txt +1 -0
  26. package/ADMIN_REGISTER_LAB.md +0 -252
  27. package/ANNOTATOR_RECOMMENDATIONS_API.md +0 -490
  28. package/BIOFS_QUICK_START.md +0 -200
  29. package/BIOFS_TOKENIZE_COMPLETE.md +0 -541
  30. package/BIOFS_V1.2.7_PUBLISHED.md +0 -317
  31. package/BIOFS_V1.2.9_FIXES.md +0 -306
  32. package/BIOFS_V1.3.0_DIRECT_GRANT.md +0 -381
  33. package/BIOFS_V1.3.1_GRANTED_FILES.md +0 -328
  34. package/BIONFS_DEPLOYMENT_GUIDE.md +0 -408
  35. package/BIONFS_MOUNT_INTEGRATION_PLAN.md +0 -415
  36. package/BIONFS_PROTOCOL_SPECIFICATION.md +0 -688
  37. package/BIONFS_SEQUENTIAS_INTEGRATION.md +0 -457
  38. package/BIONFT_NFS_PROTOCOL_ARCHITECTURE.md +0 -583
  39. package/BIOOS_CLI_IMPLEMENTATION_COMPLETE.md +0 -422
  40. package/BIOOS_JOB_MANAGEMENT.md +0 -681
  41. package/BUGFIX_V2.1.3.md +0 -72
  42. package/CHANGELOG_V2.1.0.md +0 -386
  43. package/CHANGELOG_V2.2.0.md +0 -284
  44. package/ENHANCEMENT_DOCUMENTATION.md +0 -218
  45. package/GDPR_BIONFS_CONSENT_ARCHITECTURE.md +0 -798
  46. package/GDPR_BIONFS_IMPLEMENTATION_STATUS.md +0 -462
  47. package/INSTALL_V1.2.7_ON_MAC.md +0 -306
  48. package/LABNFT_BIOFS_INTEGRATION.md +0 -396
  49. package/OPENCRAVAT_ANNOTATORS_REFERENCE.md +0 -340
  50. package/OPENCRAVAT_ANNOTATOR_INTELLIGENCE_COMPLETE.md +0 -479
  51. package/QUICK_START_V2.1.0.md +0 -116
  52. package/RELEASE_SUMMARY_V1.2.7.md +0 -479
  53. package/RELEASE_SUMMARY_V1.3.0.md +0 -324
  54. package/SEQUENTIAS_PROTOCOL_ARCHITECTURE.md +0 -880
  55. package/SEQUENTIA_MIGRATION_GUIDE.md +0 -337
  56. package/SESSION_SUMMARY_DIRECT_GRANT.md +0 -321
  57. package/ULTRATHINK_BIONFS_COMPLETE.md +0 -458
  58. package/WEB3FUSE_LICENSE_TOKEN_ANALYSIS.md +0 -494
  59. package/WHERE_PERMISSIONS_LIVE.md +0 -165
  60. package/artifacts/@openzeppelin/contracts/access/Ownable.sol/Ownable.dbg.json +0 -4
  61. package/artifacts/@openzeppelin/contracts/access/Ownable.sol/Ownable.json +0 -63
  62. package/artifacts/@openzeppelin/contracts/interfaces/IERC4906.sol/IERC4906.dbg.json +0 -4
  63. package/artifacts/@openzeppelin/contracts/interfaces/IERC4906.sol/IERC4906.json +0 -328
  64. package/artifacts/@openzeppelin/contracts/token/ERC721/ERC721.sol/ERC721.dbg.json +0 -4
  65. package/artifacts/@openzeppelin/contracts/token/ERC721/ERC721.sol/ERC721.json +0 -357
  66. package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721.sol/IERC721.dbg.json +0 -4
  67. package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721.sol/IERC721.json +0 -296
  68. package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol/IERC721Receiver.dbg.json +0 -4
  69. package/artifacts/@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol/IERC721Receiver.json +0 -45
  70. package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol/ERC721URIStorage.dbg.json +0 -4
  71. package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol/ERC721URIStorage.json +0 -373
  72. package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol/IERC721Metadata.dbg.json +0 -4
  73. package/artifacts/@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol/IERC721Metadata.json +0 -341
  74. package/artifacts/@openzeppelin/contracts/utils/Address.sol/Address.dbg.json +0 -4
  75. package/artifacts/@openzeppelin/contracts/utils/Address.sol/Address.json +0 -10
  76. package/artifacts/@openzeppelin/contracts/utils/Context.sol/Context.dbg.json +0 -4
  77. package/artifacts/@openzeppelin/contracts/utils/Context.sol/Context.json +0 -10
  78. package/artifacts/@openzeppelin/contracts/utils/Strings.sol/Strings.dbg.json +0 -4
  79. package/artifacts/@openzeppelin/contracts/utils/Strings.sol/Strings.json +0 -10
  80. package/artifacts/@openzeppelin/contracts/utils/introspection/ERC165.sol/ERC165.dbg.json +0 -4
  81. package/artifacts/@openzeppelin/contracts/utils/introspection/ERC165.sol/ERC165.json +0 -30
  82. package/artifacts/@openzeppelin/contracts/utils/introspection/IERC165.sol/IERC165.dbg.json +0 -4
  83. package/artifacts/@openzeppelin/contracts/utils/introspection/IERC165.sol/IERC165.json +0 -30
  84. package/artifacts/@openzeppelin/contracts/utils/math/Math.sol/Math.dbg.json +0 -4
  85. package/artifacts/@openzeppelin/contracts/utils/math/Math.sol/Math.json +0 -10
  86. package/artifacts/@openzeppelin/contracts/utils/math/SignedMath.sol/SignedMath.dbg.json +0 -4
  87. package/artifacts/@openzeppelin/contracts/utils/math/SignedMath.sol/SignedMath.json +0 -10
  88. package/artifacts/build-info/1adfbc50ae3e1821260f858a9c689a37.json +0 -1
  89. package/artifacts/contracts/ClaraJobNFT.sol/ClaraJobNFT.dbg.json +0 -4
  90. package/artifacts/contracts/ClaraJobNFT.sol/ClaraJobNFT.json +0 -667
  91. package/cache/solidity-files-cache.json +0 -628
  92. package/check_clara2.js +0 -70
  93. package/check_clara3.js +0 -65
  94. package/config.json +0 -21
  95. package/hardhat.config.js +0 -25
  96. package/jest.config.js +0 -27
  97. package/mint_and_link_all.js +0 -128
  98. package/opencravat_annotators_dictionary.json +0 -2356
  99. package/package.json.working.v1.9.2.20251105_023952 +0 -48
  100. package/test_v2_mint.js +0 -101
@@ -1,880 +0,0 @@
1
- # 🌍 Sequentias Protocol Architecture
2
- ## Decentralized Genomics Network with Global Node Orchestration
3
-
4
- **Vision**: Every GenoBank API/microservice can be replicated as a node in the Sequentias Network, enabling truly decentralized genomics with patient-owned data AND distributed compute.
5
-
6
- ---
7
-
8
- ## 🎯 The Real Problem We're Solving
9
-
10
- ### Current Architecture (Centralized):
11
- ```
12
- User → api.genobank.app (Single Linux Server) → S3
13
- ↑
14
- All jobs execute HERE
15
- (CPU/GPU bottleneck)
16
- ```
17
-
18
- ### Sequentias Protocol Architecture (Decentralized):
19
- ```
20
- User → Sequentias Network → Optimal Node
21
- ↓
22
- Global Registry:
23
- - Mexico Node (CPU, low latency for Mexico users)
24
- - California Node (GPU, for AlphaGenome jobs)
25
- - Europe Node (GDPR compliance, EU data)
26
- - Brazil Node (cached replicas for SA users)
27
-
28
- Each Node = Full GenoBank API + Bioinformatics + Story Protocol
29
- ```
30
-
31
- ---
32
-
33
- ## 🏗️ Three-Layer Architecture: FUSE + BioNFS + Sequentias
34
-
35
- ### Layer 1: FUSE (Client Experience Layer)
36
- **Purpose**: User sees distributed files as if they're local
37
-
38
- ```bash
39
- # User experience (same worldwide)
40
- biofs mount ~/genomics
41
- ls ~/genomics/
42
- # Files appear local, but are actually distributed across global nodes!
43
-
44
- biofs analyze sample.vcf --pipeline alphagenome
45
- # Job automatically routes to optimal node (GPU required? → California)
46
- ```
47
-
48
- **Implementation**: `/home/ubuntu/web3fuse/` (existing codebase)
49
- - Mounts distributed filesystem locally
50
- - Translates file operations to BioNFS protocol calls
51
- - Handles session management (24-hour Web3 sessions)
52
-
53
- ### Layer 2: BioNFS Protocol (Network Communication Layer)
54
- **Purpose**: Standardized protocol for node-to-node communication
55
-
56
- ```
57
- Node-to-Node Communication:
58
- 1. Node Discovery (find available nodes)
59
- 2. Job Submission (send work to optimal node)
60
- 3. Data Replication (cache files at edge)
61
- 4. Result Streaming (return output to client)
62
- ```
63
-
64
- **Implementation**: New BioNFS server (FastAPI + gRPC)
65
- - HTTP/2 for client connections
66
- - gRPC for node-to-node communication
67
- - WebSocket for real-time job updates
68
-
69
- ### Layer 3: Sequentias Network (Orchestration Layer)
70
- **Purpose**: Smart routing, resource allocation, fault tolerance
71
-
72
- ```javascript
73
- // Sequentias Protocol decides:
74
- const optimalNode = sequentias.findNode({
75
- userLocation: "Mexico City",
76
- jobType: "alphagenome", // Needs GPU
77
- dataLocation: "California S3",
78
- constraints: {
79
- maxLatency: 100, // ms
80
- maxCost: 10, // USD
81
- requireGPU: true
82
- }
83
- });
84
-
85
- // Result: Routes to Austin, Texas node
86
- // Why? Middle ground: Has GPU, close to both user and data
87
- ```
88
-
89
- ---
90
-
91
- ## 🌐 Sequentias Node Architecture
92
-
93
- ### What is a Sequentias Node?
94
-
95
- Each node is a **full replica** of current GenoBank infrastructure:
96
-
97
- ```
98
- Sequentias Node (Mexico City):
99
- ├── BioNFS Server (network protocol)
100
- ├── GenoBank APIs
101
- │ ├── api_bioip (tokenization)
102
- │ ├── api_vcf_annotator (OpenCRAVAT)
103
- │ ├── api_alphagenome (variant scoring)
104
- │ ├── api_somos_dao (ancestry)
105
- │ ├── api_clara (GPU variant calling)
106
- │ └── api_trio (family analysis)
107
- ├── Bioinformatics Tools
108
- │ ├── DeepVariant (GPU)
109
- │ ├── OpenCRAVAT (annotation)
110
- │ ├── bcftools, samtools, etc.
111
- ├── AI Models
112
- │ ├── AlphaGenome (local inference)
113
- │ ├── Claude API integration
114
- │ └── Gemini API integration
115
- ├── Vault Storage
116
- │ ├── Local cache (hot data)
117
- │ ├── S3 connection (cold data)
118
- │ └── IPFS pinning (NFT metadata)
119
- └── Story Protocol Client
120
- ├── License verification
121
- ├── IP registration
122
- └── PIL enforcement
123
- ```
124
-
125
- ### Node Capabilities Matrix:
126
-
127
- | Node Type | CPU | GPU | Storage | Use Cases |
128
- |-----------|-----|-----|---------|-----------|
129
- | **Basic** | ✅ | ❌ | 1TB | VCF annotation, ancestry analysis |
130
- | **GPU** | ✅ | ✅ | 5TB | AlphaGenome, DeepVariant, Clara |
131
- | **Storage** | ✅ | ❌ | 50TB | Data replication, edge caching |
132
- | **AI** | ✅ | ✅ | 10TB | Claude analysis, AlphaGenome |
133
-
134
- ---
135
-
136
- ## 🚀 The Hybrid Answer: BOTH FUSE + NFS
137
-
138
- ### Why FUSE Alone Won't Work:
139
- ```
140
- ❌ FUSE is client-side only
141
- - Can't route jobs to different nodes
142
- - No node-to-node communication
143
- - Each client needs full implementation
144
-
145
- ❌ No network protocol
146
- - Nodes can't discover each other
147
- - Can't distribute workload
148
- - No fault tolerance (one node down = job fails)
149
- ```
150
-
151
- ### Why NFS-like Protocol Alone Won't Work:
152
- ```
153
- ❌ Requires mounting on every client
154
- - User must configure NFS mounts
155
- - Complex setup (not "just works")
156
- - Doesn't feel like local files
157
-
158
- ❌ Not suitable for CLI tools
159
- - bcftools, samtools expect local files
160
- - Performance overhead for small operations
161
- - Doesn't integrate with existing workflows
162
- ```
163
-
164
- ### ✅ The Hybrid Solution: FUSE + BioNFS
165
-
166
- ```
167
- ┌─────────────────────────────────────────────────────┐
168
- │ CLIENT SIDE (User's Computer) │
169
- │ │
170
- │ ┌──────────────────────────────────────────────┐ │
171
- │ │ FUSE Layer (web3fuse) │ │
172
- │ │ - Mounts: /mnt/genomics/ │ │
173
- │ │ - User sees files as local │ │
174
- │ │ - Works with bcftools, samtools, IGV │ │
175
- │ └──────────────┬───────────────────────────────┘ │
176
- │ │ │
177
- │ ┌──────────────▼───────────────────────────────┐ │
178
- │ │ BioNFS Client Library │ │
179
- │ │ - Protocol implementation │ │
180
- │ │ - Session management │ │
181
- │ │ - Node discovery │ │
182
- │ └──────────────┬───────────────────────────────┘ │
183
- └─────────────────┼────────────────────────────────────┘
184
- │
185
- BioNFS Protocol
186
- (HTTP/2 + gRPC + WebSocket)
187
- │
188
- ┌─────────────────▼────────────────────────────────────┐
189
- │ SEQUENTIAS NETWORK (Global Distributed Nodes) │
190
- │ │
191
- │ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
192
- │ │ Mexico Node │ │ California │ │ Europe │ │
193
- │ │ (Basic) │ │ (GPU) │ │ (GDPR) │ │
194
- │ │ │ │ │ │ │ │
195
- │ │ BioNFS Server│ │ BioNFS Server│ │ BioNFS │ │
196
- │ │ Annotation │ │ AlphaGenome │ │ Server │ │
197
- │ │ Ancestry │ │ DeepVariant │ │ Storage │ │
198
- │ └──────────────┘ └──────────────┘ └────────────┘ │
199
- │ │
200
- │ ┌─────────────────────────────────────────────────┐ │
201
- │ │ Sequentias Orchestrator (Smart Routing) │ │
202
- │ │ - Find optimal node based on: │ │
203
- │ │ • User location (latency) │ │
204
- │ │ • Job requirements (CPU/GPU) │ │
205
- │ │ • Data location (bandwidth) │ │
206
- │ │ • Cost constraints (budget) │ │
207
- │ │ • Node availability (load balancing) │ │
208
- │ └─────────────────────────────────────────────────┘ │
209
- └────────────────────────────────────────────────────────┘
210
- ```
211
-
212
- ---
213
-
214
- ## 📋 Implementation Roadmap
215
-
216
- ### Week 1-2: FUSE Client Enhancement
217
- **Update `/home/ubuntu/web3fuse/`**
218
-
219
- ```c
220
- // web3fuse with Sequentias support
221
- static int bionfs_fuse_open(const char *path, struct fuse_file_info *fi) {
222
- // 1. Parse IP Asset ID from path
223
- char *ip_id = parse_ip_from_path(path);
224
-
225
- // 2. Query Sequentias registry for optimal node
226
- sequentias_node_t *node = sequentias_find_optimal_node(
227
- current_user_location,
228
- ip_id,
229
- /* requires_gpu */ false
230
- );
231
-
232
- // 3. Connect to node via BioNFS protocol
233
- bionfs_connection_t *conn = bionfs_connect(node->endpoint);
234
-
235
- // 4. Authenticate with session
236
- bionfs_auth(conn, user_session);
237
-
238
- // 5. Open file on remote node
239
- bionfs_file_t *file = bionfs_remote_open(conn, ip_id);
240
- fi->fh = (uint64_t)file;
241
-
242
- return 0;
243
- }
244
-
245
- static int bionfs_fuse_read(const char *path, char *buf, size_t size, off_t offset,
246
- struct fuse_file_info *fi) {
247
- bionfs_file_t *file = (bionfs_file_t *)fi->fh;
248
-
249
- // Stream from remote node (Mexico, California, wherever optimal)
250
- return bionfs_remote_read(file->connection, file->handle, buf, size, offset);
251
- }
252
- ```
253
-
254
- ### Week 3-4: BioNFS Protocol Server
255
- **New: `/home/ubuntu/sequentias_node/bionfs_server/`**
256
-
257
- ```python
258
- from fastapi import FastAPI, WebSocket
259
- from grpc import aio
260
- import sequentias_pb2 # Protocol Buffers
261
-
262
- app = FastAPI()
263
-
264
- class BioNFSServer:
265
- def __init__(self):
266
- self.node_id = os.environ['SEQUENTIAS_NODE_ID']
267
- self.location = os.environ['NODE_LOCATION'] # "Mexico City"
268
- self.has_gpu = torch.cuda.is_available()
269
- self.registry = SequentiasRegistry()
270
-
271
- async def register_with_network(self):
272
- """Register this node with Sequentias Network"""
273
- await self.registry.register_node({
274
- "node_id": self.node_id,
275
- "location": self.location,
276
- "endpoint": "https://mexico.sequentias.io",
277
- "capabilities": {
278
- "cpu_cores": 64,
279
- "gpu": "NVIDIA A100" if self.has_gpu else None,
280
- "storage_tb": 10,
281
- "services": [
282
- "vcf_annotation",
283
- "ancestry_analysis",
284
- "alphagenome" if self.has_gpu else None
285
- ]
286
- },
287
- "operator": os.environ['NODE_OPERATOR_WALLET']
288
- })
289
-
290
- @app.post("/bionfs/v1/execute_job")
291
- async def execute_job(job_spec: JobSpec):
292
- """Execute job on this node"""
293
- # 1. Verify session
294
- session = verify_session(job_spec.session_token)
295
-
296
- # 2. Check if file is cached locally
297
- if not self.cache.has(job_spec.ip_id):
298
- # Fetch from primary vault
299
- await self.replicate_file(job_spec.ip_id)
300
-
301
- # 3. Execute pipeline
302
- if job_spec.pipeline == "alphagenome":
303
- if not self.has_gpu:
304
- return {"error": "This node has no GPU, routing error!"}
305
- result = await run_alphagenome(job_spec.ip_id)
306
- elif job_spec.pipeline == "rare_coding":
307
- result = await run_opencravat(job_spec.ip_id, "rare_coding")
308
-
309
- # 4. Replicate result to user's preferred vault
310
- await self.replicate_result(result, session.wallet)
311
-
312
- # 5. Return presigned URL for download
313
- return {
314
- "status": "success",
315
- "result_ip_id": result.ip_id,
316
- "download_url": self.generate_download_url(result.ip_id)
317
- }
318
-
319
- # gRPC service for node-to-node communication
320
- class NodeCommunication(sequentias_pb2_grpc.NodeServicer):
321
- async def ReplicateFile(self, request, context):
322
- """Another node requesting file replication"""
323
- # Transfer file to requesting node
324
- file_data = await self.vault.get_file(request.ip_id)
325
- return sequentias_pb2.FileData(
326
- ip_id=request.ip_id,
327
- data=file_data,
328
- checksum=hashlib.sha256(file_data).hexdigest()
329
- )
330
- ```
331
-
332
- ### Week 5-6: Sequentias Registry (Smart Contract)
333
- **Deploy on Story Protocol**
334
-
335
- ```solidity
336
- // SPDX-License-Identifier: MIT
337
- pragma solidity ^0.8.20;
338
-
339
- contract SequentiasRegistry {
340
- struct Node {
341
- address operator;
342
- string location; // "Mexico City", "Palo Alto", "Berlin"
343
- string endpoint; // "https://mexico.sequentias.io"
344
- bool hasGPU;
345
- uint256 cpuCores;
346
- uint256 storageTB;
347
- uint256 uptime; // Percentage (9500 = 95.00%)
348
- uint256 jobsCompleted;
349
- bool active;
350
- }
351
-
352
- struct Job {
353
- bytes32 jobId;
354
- address requester;
355
- address executingNode;
356
- string pipeline; // "alphagenome", "rare_coding"
357
- bytes32 ipAssetId;
358
- uint256 startTime;
359
- uint256 endTime;
360
- JobStatus status;
361
- }
362
-
363
- enum JobStatus { Pending, Running, Completed, Failed }
364
-
365
- mapping(address => Node) public nodes;
366
- mapping(bytes32 => Job) public jobs;
367
-
368
- event NodeRegistered(address indexed operator, string location);
369
- event JobRouted(bytes32 indexed jobId, address indexed node);
370
- event JobCompleted(bytes32 indexed jobId, uint256 executionTime);
371
-
372
- function registerNode(
373
- string memory location,
374
- string memory endpoint,
375
- bool hasGPU,
376
- uint256 cpuCores,
377
- uint256 storageTB
378
- ) external {
379
- nodes[msg.sender] = Node({
380
- operator: msg.sender,
381
- location: location,
382
- endpoint: endpoint,
383
- hasGPU: hasGPU,
384
- cpuCores: cpuCores,
385
- storageTB: storageTB,
386
- uptime: 10000, // Start at 100%
387
- jobsCompleted: 0,
388
- active: true
389
- });
390
-
391
- emit NodeRegistered(msg.sender, location);
392
- }
393
-
394
- function findOptimalNode(
395
- string memory userLocation,
396
- bool requiresGPU,
397
- string memory dataLocation
398
- ) external view returns (address) {
399
- // Scoring algorithm:
400
- // 1. Location proximity (30%)
401
- // 2. Resource availability (30%)
402
- // 3. Uptime/reliability (20%)
403
- // 4. Data proximity (20%)
404
-
405
- address bestNode;
406
- uint256 bestScore = 0;
407
-
408
- for (uint i = 0; i < getAllNodes().length; i++) {
409
- address nodeAddr = getAllNodes()[i];
410
- Node memory node = nodes[nodeAddr];
411
-
412
- if (!node.active) continue;
413
- if (requiresGPU && !node.hasGPU) continue;
414
-
415
- uint256 score = calculateNodeScore(
416
- node,
417
- userLocation,
418
- dataLocation
419
- );
420
-
421
- if (score > bestScore) {
422
- bestScore = score;
423
- bestNode = nodeAddr;
424
- }
425
- }
426
-
427
- return bestNode;
428
- }
429
-
430
- function submitJob(
431
- bytes32 jobId,
432
- address nodeOperator,
433
- string memory pipeline,
434
- bytes32 ipAssetId
435
- ) external {
436
- jobs[jobId] = Job({
437
- jobId: jobId,
438
- requester: msg.sender,
439
- executingNode: nodeOperator,
440
- pipeline: pipeline,
441
- ipAssetId: ipAssetId,
442
- startTime: block.timestamp,
443
- endTime: 0,
444
- status: JobStatus.Running
445
- });
446
-
447
- emit JobRouted(jobId, nodeOperator);
448
- }
449
-
450
- function completeJob(bytes32 jobId) external {
451
- Job storage job = jobs[jobId];
452
- require(job.executingNode == nodes[msg.sender].operator, "Not executor");
453
-
454
- job.endTime = block.timestamp;
455
- job.status = JobStatus.Completed;
456
-
457
- // Update node stats
458
- nodes[msg.sender].jobsCompleted++;
459
-
460
- emit JobCompleted(jobId, job.endTime - job.startTime);
461
- }
462
- }
463
- ```
464
-
465
- ### Week 7-8: Orchestration Logic
466
- **New: `/home/ubuntu/sequentias_node/orchestrator/`**
467
-
468
- ```python
469
- from web3 import Web3
470
- from geopy.distance import geodesic
471
-
472
- class SequentiasOrchestrator:
473
- def __init__(self):
474
- self.w3 = Web3(Web3.HTTPProvider(os.environ['STORY_RPC_URL']))
475
- self.registry = self.w3.eth.contract(
476
- address=SEQUENTIAS_REGISTRY_ADDRESS,
477
- abi=SEQUENTIAS_REGISTRY_ABI
478
- )
479
-
480
- async def route_job(self, job_spec: JobSpec) -> Node:
481
- """
482
- Find optimal node for job execution
483
-
484
- Considers:
485
- 1. User location (minimize latency)
486
- 2. Data location (minimize bandwidth)
487
- 3. Resource requirements (CPU/GPU)
488
- 4. Cost constraints (budget)
489
- 5. Node reputation (uptime, success rate)
490
- """
491
-
492
- # Get user's location (from IP or explicitly provided)
493
- user_location = await self.get_user_location(job_spec.user_ip)
494
-
495
- # Get data location (where is the IP Asset stored?)
496
- data_location = await self.get_data_location(job_spec.ip_id)
497
-
498
- # Query registry for available nodes
499
- all_nodes = await self.get_all_active_nodes()
500
-
501
- # Score each node
502
- scored_nodes = []
503
- for node in all_nodes:
504
- score = await self.calculate_node_score(
505
- node=node,
506
- user_location=user_location,
507
- data_location=data_location,
508
- requires_gpu=job_spec.pipeline in ["alphagenome", "deepvariant"],
509
- budget=job_spec.max_cost
510
- )
511
- scored_nodes.append((score, node))
512
-
513
- # Sort by score (highest first)
514
- scored_nodes.sort(reverse=True, key=lambda x: x[0])
515
-
516
- # Return best node
517
- return scored_nodes[0][1]
518
-
519
- async def calculate_node_score(
520
- self,
521
- node: Node,
522
- user_location: Location,
523
- data_location: Location,
524
- requires_gpu: bool,
525
- budget: float
526
- ) -> float:
527
- """
528
- Scoring algorithm:
529
- - Latency score (30%): Distance from user
530
- - Bandwidth score (20%): Distance from data
531
- - Resource score (30%): Has required resources?
532
- - Reliability score (10%): Uptime percentage
533
- - Cost score (10%): Within budget?
534
- """
535
-
536
- # Latency score (closer to user = better)
537
- user_distance = geodesic(
538
- (user_location.lat, user_location.lon),
539
- (node.location.lat, node.location.lon)
540
- ).kilometers
541
- latency_score = max(0, 100 - (user_distance / 100)) # 0-100
542
-
543
- # Bandwidth score (closer to data = better)
544
- data_distance = geodesic(
545
- (data_location.lat, data_location.lon),
546
- (node.location.lat, node.location.lon)
547
- ).kilometers
548
- bandwidth_score = max(0, 100 - (data_distance / 100))
549
-
550
- # Resource score
551
- if requires_gpu and not node.has_gpu:
552
- return 0 # Disqualify immediately
553
- resource_score = 100 if node.has_gpu or not requires_gpu else 0
554
-
555
- # Reliability score
556
- reliability_score = node.uptime / 100 # Convert to 0-100
557
-
558
- # Cost score
559
- estimated_cost = await self.estimate_job_cost(node, requires_gpu)
560
- cost_score = 100 if estimated_cost <= budget else 0
561
-
562
- # Weighted average
563
- total_score = (
564
- latency_score * 0.3 +
565
- bandwidth_score * 0.2 +
566
- resource_score * 0.3 +
567
- reliability_score * 0.1 +
568
- cost_score * 0.1
569
- )
570
-
571
- return total_score
572
-
573
- async def execute_job_on_node(self, node: Node, job_spec: JobSpec):
574
- """Submit job to selected node via BioNFS protocol"""
575
-
576
- async with aiohttp.ClientSession() as session:
577
- # Submit job to node
578
- response = await session.post(
579
- f"{node.endpoint}/bionfs/v1/execute_job",
580
- json={
581
- "session_token": job_spec.session_token,
582
- "ip_id": job_spec.ip_id,
583
- "pipeline": job_spec.pipeline,
584
- "parameters": job_spec.parameters
585
- }
586
- )
587
-
588
- result = await response.json()
589
-
590
- # Register job completion on blockchain
591
- tx = self.registry.functions.completeJob(
592
- jobId=job_spec.job_id
593
- ).build_transaction({
594
- 'from': node.operator,
595
- 'nonce': self.w3.eth.get_transaction_count(node.operator)
596
- })
597
-
598
- # Node operator signs and submits
599
- # ...
600
-
601
- return result
602
- ```
603
-
604
- ---
605
-
606
- ## 🌍 Real-World Usage Scenarios
607
-
608
- ### Scenario 1: Researcher in Mexico Analyzes Local VCF
609
- ```bash
610
- # User in Mexico City
611
- biofs login
612
- biofs mount ~/genomics
613
-
614
- # File is in California S3
615
- ls ~/genomics/granted/
616
- # 0xCCe1.../father.vcf (152 MB, California vault)
617
-
618
- # Analyze with rare_coding pipeline
619
- biofs analyze ~/genomics/granted/0xCCe1.../father.vcf --pipeline rare_coding
620
-
621
- # Behind the scenes:
622
- # 1. Sequentias finds optimal node:
623
- # - Mexico City node (50ms latency, no GPU needed)
624
- # 2. Mexico node caches file from California (one-time transfer)
625
- # 3. Job executes on Mexico node
626
- # 4. Results streamed back to user
627
- # Total time: 30 seconds (vs 5 minutes if run in California)
628
- ```
629
-
630
- ### Scenario 2: Researcher in Palo Alto Needs GPU
631
- ```bash
632
- # User in Palo Alto, California
633
- biofs login
634
- biofs mount ~/genomics
635
-
636
- # Same file (in California S3)
637
- # Analyze with AlphaGenome (needs GPU)
638
- biofs analyze ~/genomics/granted/0xCCe1.../father.vcf --pipeline alphagenome
639
-
640
- # Behind the scenes:
641
- # 1. Sequentias finds optimal node:
642
- # - San Francisco GPU node (30 miles away, has A100)
643
- # 2. SF node already has file cached (same region)
644
- # 3. Job executes on GPU
645
- # 4. Results in <10 minutes (GPU accelerated)
646
- ```
647
-
648
- ### Scenario 3: Patient in Brazil Downloads Own Data
649
- ```bash
650
- # Patient in São Paulo (owns data, no license needed)
651
- biofs login # Authenticate as owner
652
- biofs mount ~/my-genomics
653
-
654
- # Their files appear
655
- ls ~/my-genomics/
656
- # my_exome.vcf (owned file)
657
-
658
- # Download to local
659
- biofs download ~/my-genomics/my_exome.vcf ./local_copy.vcf
660
-
661
- # Behind the scenes:
662
- # 1. Sequentias checks: Brazil node has cached copy?
663
- # - No → Replicate from California vault to Brazil node
664
- # - Yes → Stream from Brazil node
665
- # 2. Future access: Instant (cached in Brazil)
666
- # 3. Latency: 20ms (vs 200ms from California)
667
- ```
668
-
669
- ### Scenario 4: GPU Jobs with Data in Different Regions
670
- ```bash
671
- # User in Tokyo needs GPU for DeepVariant
672
- # Data is in EU (GDPR-compliant storage)
673
-
674
- biofs analyze eu_patient_data.fastq --pipeline deepvariant
675
-
676
- # Behind the scenes:
677
- # 1. Sequentias scoring:
678
- # - Tokyo GPU node: High latency to EU data (disqualified)
679
- # - Singapore GPU node: Medium latency to EU + Tokyo (optimal)
680
- # - EU GPU node: High latency to Tokyo user (suboptimal)
681
- # 2. Routes to Singapore node (middle ground)
682
- # 3. Singapore caches EU data
683
- # 4. Executes with GPU
684
- # 5. Results stream to Tokyo (low latency)
685
- ```
686
-
687
- ---
688
-
689
- ## 🔐 Security & Licensing in Distributed Network
690
-
691
- ### License Token Verification Across Nodes
692
-
693
- ```python
694
- # Every node verifies licenses independently
695
- def verify_license_on_node(ip_id: str, wallet: str) -> bool:
696
- """
697
- Each node queries Story Protocol directly
698
- No central authority needed!
699
- """
700
-
701
- # Query local MongoDB cache first (fast)
702
- local_license = license_cache.find_one({
703
- "ip_id": ip_id,
704
- "receiver": wallet,
705
- "status": "active"
706
- })
707
-
708
- if local_license:
709
- # Verify cache is fresh (< 1 hour old)
710
- if local_license['cached_at'] > time.time() - 3600:
711
- return True
712
-
713
- # If no cache or stale, verify on blockchain
714
- license_token_contract = story_protocol.get_contract("LicenseToken")
715
- owner = license_token_contract.functions.ownerOf(local_license['license_token_id']).call()
716
-
717
- if owner.lower() == wallet.lower():
718
- # Update cache
719
- license_cache.update_one(
720
- {"_id": local_license['_id']},
721
- {"$set": {"cached_at": time.time()}}
722
- )
723
- return True
724
-
725
- return False
726
- ```
727
-
728
- ### Data Replication with License Enforcement
729
-
730
- ```python
731
- # Node can only replicate files it's authorized to serve
732
- async def replicate_file(source_node: str, ip_id: str, requester_wallet: str):
733
- """
734
- Replicate file from source node to this node
735
- License verification at both ends!
736
- """
737
-
738
- # 1. This node verifies requester has license
739
- if not verify_license_on_node(ip_id, requester_wallet):
740
- raise PermissionError("Requester lacks license for this IP Asset")
741
-
742
- # 2. Request file from source node
743
- async with grpc.aio.insecure_channel(source_node) as channel:
744
- stub = sequentias_pb2_grpc.NodeStub(channel)
745
-
746
- # Source node ALSO verifies (defense in depth)
747
- response = await stub.ReplicateFile(
748
- sequentias_pb2.ReplicationRequest(
749
- ip_id=ip_id,
750
- requester_node=THIS_NODE_ID,
751
- requester_wallet=requester_wallet
752
- )
753
- )
754
-
755
- # 3. Verify checksum
756
- if hashlib.sha256(response.data).hexdigest() != response.checksum:
757
- raise IntegrityError("File corrupted during transfer")
758
-
759
- # 4. Store locally with license metadata
760
- await local_vault.store(
761
- ip_id=ip_id,
762
- data=response.data,
763
- license_verified_for=requester_wallet,
764
- replicated_from=source_node,
765
- replicated_at=time.time()
766
- )
767
- ```
768
-
769
- ---
770
-
771
- ## 📊 Performance Comparison
772
-
773
- ### Current Centralized Architecture:
774
- ```
775
- Researcher in Mexico → California Server (200ms latency)
776
- ↓
777
- Download 150GB WGS file (30 minutes on 1 Gbps)
778
- ↓
779
- Analyze locally (2 hours)
780
- ↓
781
- Total: 2.5 hours
782
- ```
783
-
784
- ### Sequentias Distributed Architecture:
785
- ```
786
- Researcher in Mexico → Sequentias Network
787
- ↓
788
- Routes to Mexico City node (20ms latency)
789
- ↓
790
- Mexico node caches file from California (one-time, 30 min)
791
- ↓
792
- Analyze on Mexico node (2 hours, but starts immediately)
793
- ↓
794
- Results stream back (instant)
795
- ↓
796
- Total first run: 2.5 hours (same as centralized)
797
- Total subsequent runs: 2 hours (30min saved, file already cached!)
798
-
799
- Next researcher in Mexico using same file: 2 hours (no download!)
800
- ```
801
-
802
- ### GPU Jobs:
803
- ```
804
- AlphaGenome job from Tokyo on EU data:
805
-
806
- Centralized:
807
- Tokyo → California server → Fetch EU data → GPU analyze → Return
808
- Total: 3 hours (200ms latency + data transfer + compute)
809
-
810
- Sequentias:
811
- Tokyo → Singapore node → Fetch EU data → GPU analyze → Return
812
- Total: 2 hours (50ms latency + optimized data transfer + same compute)
813
- Savings: 33% faster!
814
- ```
815
-
816
- ---
817
-
818
- ## 🎯 THE ANSWER: Build BOTH FUSE + BioNFS
819
-
820
- ### FUSE (Client Layer):
821
- ✅ **Purpose**: User experience - mount files locally
822
- ✅ **Location**: `/home/ubuntu/web3fuse/` (enhance existing)
823
- ✅ **Implementation**: C library with Sequentias integration
824
- ✅ **User sees**: `~/genomics/0xCCe1.../file.vcf` (appears local)
825
- ✅ **Actually**: File streamed from optimal Sequentias node
826
-
827
- ### BioNFS Protocol (Network Layer):
828
- ✅ **Purpose**: Node-to-node communication
829
- ✅ **Location**: `/home/ubuntu/sequentias_node/bionfs_server/` (new)
830
- ✅ **Implementation**: FastAPI + gRPC + WebSocket
831
- ✅ **Handles**: Job routing, data replication, streaming
832
-
833
- ### Sequentias Registry (Orchestration Layer):
834
- ✅ **Purpose**: Smart routing & resource allocation
835
- ✅ **Location**: Story Protocol smart contract (on-chain)
836
- ✅ **Implementation**: Solidity contract + Python orchestrator
837
- ✅ **Decides**: Which node executes which job
838
-
839
- ---
840
-
841
- ## 🚀 Implementation Priority
842
-
843
- ### Month 1: Foundation
844
- - ✅ BioNFS protocol specification (done)
845
- - ✅ Basic FastAPI server
846
- - ✅ FUSE client with network support
847
- - ✅ Simple node registration
848
-
849
- ### Month 2: Distribution
850
- - ✅ Sequentias smart contract
851
- - ✅ Orchestration logic (node scoring)
852
- - ✅ Data replication protocol
853
- - ✅ 3 test nodes (Mexico, California, EU)
854
-
855
- ### Month 3: Production
856
- - ✅ Full node capabilities (all APIs)
857
- - ✅ GPU node setup (Clara, AlphaGenome)
858
- - ✅ Fault tolerance (retry logic)
859
- - ✅ 10+ production nodes globally
860
-
861
- ### Month 4: Optimization
862
- - ✅ Edge caching optimization
863
- - ✅ Cost minimization
864
- - ✅ Performance benchmarks
865
- - ✅ Open node registration (anyone can run a node!)
866
-
867
- ---
868
-
869
- ## 💡 Why This is Revolutionary
870
-
871
- 1. **True Decentralization**: Not just data, but compute too
872
- 2. **Patient Sovereignty**: Data follows patient, not server location
873
- 3. **Cost Optimization**: Jobs route to cheapest available node
874
- 4. **Fault Tolerance**: Node down? Automatically retry elsewhere
875
- 5. **Global Scale**: Add nodes anywhere in the world
876
- 6. **Open Network**: Anyone can run a Sequentias node
877
- 7. **Standards-Based**: Built on IETF NFS v4 + Story Protocol
878
-
879
- This is not just GenoBank CLI. **This is the decentralized genomics network of the future.** 🌍🧬
880
-