@genobank/biofs 1.6.2 โ†’ 1.6.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,2164 @@
1
+ # ๐Ÿš€ BioFS Federated Network - Implementation Plan
2
+
3
+ **Project**: BioFS Node Infrastructure
4
+ **Timeline**: 4 Weeks (3 Phases)
5
+ **Start Date**: October 24, 2025
6
+ **Target Completion**: November 21, 2025
7
+
8
+ ---
9
+
10
+ ## ๐Ÿ“Š Project Overview
11
+
12
+ ### Goal
13
+ Build a federated genomic data network with:
14
+ - **Master Node** (GenoBank.io) - Orchestrator and backup
15
+ - **Lab Nodes** (AUGenomics, UCSF, etc.) - Data providers
16
+ - **Client** (biofs-cli) - Researcher interface
17
+
18
+ ### Success Metrics
19
+ - โœ… Master node can manage 10+ lab nodes
20
+ - โœ… Lab node can serve 100+ researchers
21
+ - โœ… Failover completes in <2 minutes
22
+ - โœ… API response time <500ms (P95)
23
+ - โœ… Data replication lag <1 hour
24
+ - โœ… 99.9% uptime (43 minutes downtime/month allowed)
25
+
26
+ ---
27
+
28
+ ## ๐Ÿ“ฆ Package Structure
29
+
30
+ ```
31
+ biofs/
32
+ โ”œโ”€โ”€ packages/
33
+ โ”‚ โ”œโ”€โ”€ biofs-core/ # Shared library
34
+ โ”‚ โ”‚ โ”œโ”€โ”€ src/
35
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ auth/ # Authentication
36
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ bioip/ # License management
37
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # GDPR consent
38
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ api/ # API client
39
+ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ storage/ # S3 operations
40
+ โ”‚ โ”‚ โ””โ”€โ”€ package.json
41
+ โ”‚ โ”‚
42
+ โ”‚ โ”œโ”€โ”€ biofs/ # Client CLI (existing)
43
+ โ”‚ โ”‚ โ””โ”€โ”€ [current structure]
44
+ โ”‚ โ”‚
45
+ โ”‚ โ””โ”€โ”€ biofs-node/ # NEW - Server daemon
46
+ โ”‚ โ”œโ”€โ”€ src/
47
+ โ”‚ โ”‚ โ”œโ”€โ”€ node/ # Node core
48
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ master.ts
49
+ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ lab.ts
50
+ โ”‚ โ”‚ โ”œโ”€โ”€ api/ # REST API
51
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ routes/
52
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ middleware/
53
+ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ controllers/
54
+ โ”‚ โ”‚ โ”œโ”€โ”€ services/ # Business logic
55
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ anthropic-proxy.ts
56
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ story-proxy.ts
57
+ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ replication.ts
58
+ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ failover.ts
59
+ โ”‚ โ”‚ โ”œโ”€โ”€ monitoring/ # Health checks
60
+ โ”‚ โ”‚ โ”œโ”€โ”€ database/ # PostgreSQL/MongoDB
61
+ โ”‚ โ”‚ โ””โ”€โ”€ config/ # Configuration
62
+ โ”‚ โ”œโ”€โ”€ systemd/ # Service files
63
+ โ”‚ โ”œโ”€โ”€ docker/ # Containerization
64
+ โ”‚ โ””โ”€โ”€ package.json
65
+ โ”‚
66
+ โ””โ”€โ”€ docs/
67
+ โ”œโ”€โ”€ API.md
68
+ โ”œโ”€โ”€ DEPLOYMENT.md
69
+ โ””โ”€โ”€ CONFIGURATION.md
70
+ ```
71
+
72
+ ---
73
+
74
+ # ๐Ÿ“… Phase 1: Core Infrastructure (Week 1-2)
75
+
76
+ **Duration**: October 24 - November 6, 2025 (2 weeks)
77
+ **Goal**: Basic node infrastructure with master/lab modes
78
+
79
+ ---
80
+
81
+ ## Week 1: Foundation (Oct 24-30)
82
+
83
+ ### Day 1-2: Project Setup
84
+
85
+ #### Task 1.1: Create biofs-node package
86
+ ```bash
87
+ mkdir -p packages/biofs-node
88
+ cd packages/biofs-node
89
+ npm init -y
90
+ ```
91
+
92
+ **package.json**:
93
+ ```json
94
+ {
95
+ "name": "@genobank/biofs-node",
96
+ "version": "0.1.0",
97
+ "description": "BioFS Node Server - Master and Lab node daemon",
98
+ "main": "dist/index.js",
99
+ "bin": {
100
+ "biofs-node": "./bin/biofs-node.js"
101
+ },
102
+ "scripts": {
103
+ "build": "tsc",
104
+ "dev": "ts-node src/index.ts",
105
+ "start": "node dist/index.js",
106
+ "test": "jest"
107
+ },
108
+ "dependencies": {
109
+ "@genobank/biofs-core": "^1.0.0",
110
+ "express": "^4.18.0",
111
+ "fastify": "^4.25.0",
112
+ "ws": "^8.14.0",
113
+ "ioredis": "^5.3.0",
114
+ "pg": "^8.11.0",
115
+ "aws-sdk": "^2.1500.0",
116
+ "dotenv": "^16.3.0",
117
+ "commander": "^12.0.0",
118
+ "chalk": "^4.1.2",
119
+ "pino": "^8.16.0",
120
+ "prom-client": "^15.0.0",
121
+ "joi": "^17.11.0"
122
+ },
123
+ "devDependencies": {
124
+ "@types/node": "^20.0.0",
125
+ "@types/express": "^4.17.0",
126
+ "typescript": "^5.3.0",
127
+ "ts-node": "^10.9.0",
128
+ "jest": "^29.7.0"
129
+ }
130
+ }
131
+ ```
132
+
133
+ #### Task 1.2: TypeScript configuration
134
+ ```json
135
+ // tsconfig.json
136
+ {
137
+ "compilerOptions": {
138
+ "target": "ES2020",
139
+ "module": "commonjs",
140
+ "lib": ["ES2020"],
141
+ "outDir": "./dist",
142
+ "rootDir": "./src",
143
+ "strict": true,
144
+ "esModuleInterop": true,
145
+ "skipLibCheck": true,
146
+ "forceConsistentCasingInFileNames": true,
147
+ "resolveJsonModule": true,
148
+ "declaration": true,
149
+ "declarationMap": true,
150
+ "sourceMap": true
151
+ },
152
+ "include": ["src/**/*"],
153
+ "exclude": ["node_modules", "dist"]
154
+ }
155
+ ```
156
+
157
+ #### Task 1.3: Core file structure
158
+ ```bash
159
+ src/
160
+ โ”œโ”€โ”€ index.ts # CLI entry point
161
+ โ”œโ”€โ”€ node/
162
+ โ”‚ โ”œโ”€โ”€ base-node.ts # Abstract base class
163
+ โ”‚ โ”œโ”€โ”€ master-node.ts # Master node implementation
164
+ โ”‚ โ””โ”€โ”€ lab-node.ts # Lab node implementation
165
+ โ”œโ”€โ”€ api/
166
+ โ”‚ โ”œโ”€โ”€ server.ts # HTTP server setup
167
+ โ”‚ โ”œโ”€โ”€ routes/
168
+ โ”‚ โ”‚ โ”œโ”€โ”€ health.ts # Health check endpoints
169
+ โ”‚ โ”‚ โ”œโ”€โ”€ discovery.ts # Node discovery
170
+ โ”‚ โ”‚ โ””โ”€โ”€ files.ts # File serving
171
+ โ”‚ โ””โ”€โ”€ middleware/
172
+ โ”‚ โ”œโ”€โ”€ auth.ts # JWT verification
173
+ โ”‚ โ”œโ”€โ”€ rate-limit.ts # Rate limiting
174
+ โ”‚ โ””โ”€โ”€ cors.ts # CORS configuration
175
+ โ”œโ”€โ”€ config/
176
+ โ”‚ โ”œโ”€โ”€ schema.ts # Configuration validation
177
+ โ”‚ โ”œโ”€โ”€ loader.ts # Load YAML config
178
+ โ”‚ โ””โ”€โ”€ defaults.ts # Default values
179
+ โ”œโ”€โ”€ monitoring/
180
+ โ”‚ โ”œโ”€โ”€ health-checker.ts # Health monitoring
181
+ โ”‚ โ”œโ”€โ”€ metrics.ts # Prometheus metrics
182
+ โ”‚ โ””โ”€โ”€ logger.ts # Structured logging
183
+ โ””โ”€โ”€ types/
184
+ โ”œโ”€โ”€ config.ts # Configuration types
185
+ โ””โ”€โ”€ node.ts # Node types
186
+ ```
187
+
188
+ **Deliverable**: Empty package with correct structure โœ…
189
+
190
+ ---
191
+
192
+ ### Day 3-4: Base Node Implementation
193
+
194
+ #### Task 1.4: Configuration schema
195
+
196
+ **src/types/config.ts**:
197
+ ```typescript
198
+ export type NodeType = 'master' | 'lab';
199
+
200
+ export interface NodeConfig {
201
+ node: {
202
+ type: NodeType;
203
+ id: string;
204
+ name: string;
205
+ wallet: string;
206
+ url: string;
207
+ };
208
+
209
+ api: {
210
+ port: number;
211
+ host: string;
212
+ cors_origins: string[];
213
+ rate_limit: {
214
+ requests_per_minute: number;
215
+ };
216
+ };
217
+
218
+ storage?: {
219
+ bucket: string;
220
+ region: string;
221
+ };
222
+
223
+ master?: {
224
+ url: string;
225
+ registration_token: string;
226
+ };
227
+
228
+ database: {
229
+ type: 'postgresql' | 'mongodb';
230
+ url: string;
231
+ };
232
+
233
+ monitoring: {
234
+ prometheus_enabled: boolean;
235
+ port: number;
236
+ };
237
+ }
238
+
239
+ export interface MasterNodeConfig extends NodeConfig {
240
+ node: NodeConfig['node'] & { type: 'master' };
241
+ labs_managed: string[];
242
+ }
243
+
244
+ export interface LabNodeConfig extends NodeConfig {
245
+ node: NodeConfig['node'] & { type: 'lab' };
246
+ master: {
247
+ url: string;
248
+ registration_token: string;
249
+ };
250
+ storage: {
251
+ bucket: string;
252
+ region: string;
253
+ };
254
+ }
255
+ ```
256
+
257
+ #### Task 1.5: Base node class
258
+
259
+ **src/node/base-node.ts**:
260
+ ```typescript
261
+ import { EventEmitter } from 'events';
262
+ import { NodeConfig } from '../types/config';
263
+ import { Logger } from '../monitoring/logger';
264
+ import { HealthChecker } from '../monitoring/health-checker';
265
+ import { MetricsCollector } from '../monitoring/metrics';
266
+
267
+ export abstract class BaseNode extends EventEmitter {
268
+ protected config: NodeConfig;
269
+ protected logger: Logger;
270
+ protected healthChecker: HealthChecker;
271
+ protected metrics: MetricsCollector;
272
+ protected isRunning: boolean = false;
273
+
274
+ constructor(config: NodeConfig) {
275
+ super();
276
+ this.config = config;
277
+ this.logger = new Logger(config.node.id);
278
+ this.healthChecker = new HealthChecker(this);
279
+ this.metrics = new MetricsCollector(config.monitoring);
280
+ }
281
+
282
+ abstract async start(): Promise<void>;
283
+ abstract async stop(): Promise<void>;
284
+ abstract async getHealth(): Promise<HealthStatus>;
285
+
286
+ async initialize(): Promise<void> {
287
+ this.logger.info('Initializing node', {
288
+ type: this.config.node.type,
289
+ id: this.config.node.id,
290
+ name: this.config.node.name
291
+ });
292
+
293
+ // Start health monitoring
294
+ await this.healthChecker.start();
295
+
296
+ // Start metrics collection
297
+ if (this.config.monitoring.prometheus_enabled) {
298
+ await this.metrics.start();
299
+ }
300
+
301
+ this.emit('initialized');
302
+ }
303
+
304
+ async shutdown(): Promise<void> {
305
+ this.logger.info('Shutting down node');
306
+
307
+ await this.healthChecker.stop();
308
+ await this.metrics.stop();
309
+
310
+ this.isRunning = false;
311
+ this.emit('shutdown');
312
+ }
313
+
314
+ getNodeId(): string {
315
+ return this.config.node.id;
316
+ }
317
+
318
+ getNodeType(): string {
319
+ return this.config.node.type;
320
+ }
321
+ }
322
+
323
+ export interface HealthStatus {
324
+ status: 'healthy' | 'degraded' | 'offline';
325
+ checks: {
326
+ [key: string]: {
327
+ status: 'pass' | 'fail';
328
+ message?: string;
329
+ timestamp: string;
330
+ };
331
+ };
332
+ }
333
+ ```
334
+
335
+ **Deliverable**: Base node class with lifecycle management โœ…
336
+
337
+ ---
338
+
339
+ ### Day 5-7: Node Implementations
340
+
341
+ #### Task 1.6: Master node
342
+
343
+ **src/node/master-node.ts**:
344
+ ```typescript
345
+ import { BaseNode } from './base-node';
346
+ import { MasterNodeConfig } from '../types/config';
347
+ import { LabRegistry } from '../services/lab-registry';
348
+ import { ReplicationManager } from '../services/replication';
349
+
350
+ export class MasterNode extends BaseNode {
351
+ private labRegistry: LabRegistry;
352
+ private replicationManager: ReplicationManager;
353
+
354
+ constructor(config: MasterNodeConfig) {
355
+ super(config);
356
+ this.labRegistry = new LabRegistry(this);
357
+ this.replicationManager = new ReplicationManager(this);
358
+ }
359
+
360
+ async start(): Promise<void> {
361
+ await this.initialize();
362
+
363
+ this.logger.info('Starting Master Node', {
364
+ url: this.config.node.url,
365
+ labs_managed: (this.config as MasterNodeConfig).labs_managed
366
+ });
367
+
368
+ // Start lab registry
369
+ await this.labRegistry.start();
370
+
371
+ // Start replication service
372
+ await this.replicationManager.start();
373
+
374
+ this.isRunning = true;
375
+ this.emit('started');
376
+ }
377
+
378
+ async stop(): Promise<void> {
379
+ await this.labRegistry.stop();
380
+ await this.replicationManager.stop();
381
+ await this.shutdown();
382
+ }
383
+
384
+ async getHealth(): Promise<HealthStatus> {
385
+ return {
386
+ status: 'healthy',
387
+ checks: {
388
+ api: { status: 'pass', timestamp: new Date().toISOString() },
389
+ database: { status: 'pass', timestamp: new Date().toISOString() },
390
+ labs_connected: {
391
+ status: 'pass',
392
+ message: `${this.labRegistry.getConnectedLabCount()} labs online`,
393
+ timestamp: new Date().toISOString()
394
+ }
395
+ }
396
+ };
397
+ }
398
+
399
+ // Master-specific methods
400
+ async registerLab(labInfo: LabRegistrationRequest): Promise<LabRegistrationResponse> {
401
+ return await this.labRegistry.registerLab(labInfo);
402
+ }
403
+
404
+ async getLabStatus(labId: string): Promise<LabStatus> {
405
+ return await this.labRegistry.getLabStatus(labId);
406
+ }
407
+
408
+ async triggerFailover(labId: string): Promise<void> {
409
+ this.logger.warn('Triggering failover', { labId });
410
+ await this.replicationManager.activateFailover(labId);
411
+ }
412
+ }
413
+ ```
414
+
415
+ #### Task 1.7: Lab node
416
+
417
+ **src/node/lab-node.ts**:
418
+ ```typescript
419
+ import { BaseNode } from './base-node';
420
+ import { LabNodeConfig } from '../types/config';
421
+ import { MasterClient } from '../services/master-client';
422
+ import { FileServer } from '../services/file-server';
423
+ import { S3Client } from '@aws-sdk/client-s3';
424
+
425
+ export class LabNode extends BaseNode {
426
+ private masterClient: MasterClient;
427
+ private fileServer: FileServer;
428
+ private s3Client: S3Client;
429
+
430
+ constructor(config: LabNodeConfig) {
431
+ super(config);
432
+
433
+ this.masterClient = new MasterClient(config.master.url, config.master.registration_token);
434
+ this.s3Client = new S3Client({ region: config.storage.region });
435
+ this.fileServer = new FileServer(this.s3Client, config.storage.bucket);
436
+ }
437
+
438
+ async start(): Promise<void> {
439
+ await this.initialize();
440
+
441
+ this.logger.info('Starting Lab Node', {
442
+ url: this.config.node.url,
443
+ bucket: (this.config as LabNodeConfig).storage.bucket,
444
+ master: (this.config as LabNodeConfig).master.url
445
+ });
446
+
447
+ // Register with master
448
+ await this.registerWithMaster();
449
+
450
+ // Start file server
451
+ await this.fileServer.start();
452
+
453
+ // Start heartbeat to master
454
+ this.startHeartbeat();
455
+
456
+ this.isRunning = true;
457
+ this.emit('started');
458
+ }
459
+
460
+ async stop(): Promise<void> {
461
+ await this.fileServer.stop();
462
+ await this.masterClient.disconnect();
463
+ await this.shutdown();
464
+ }
465
+
466
+ async getHealth(): Promise<HealthStatus> {
467
+ return {
468
+ status: 'healthy',
469
+ checks: {
470
+ api: { status: 'pass', timestamp: new Date().toISOString() },
471
+ storage: await this.checkStorageHealth(),
472
+ master_connection: await this.checkMasterConnection()
473
+ }
474
+ };
475
+ }
476
+
477
+ private async registerWithMaster(): Promise<void> {
478
+ this.logger.info('Registering with master node');
479
+
480
+ const registration = await this.masterClient.register({
481
+ node_id: this.config.node.id,
482
+ name: this.config.node.name,
483
+ wallet: this.config.node.wallet,
484
+ url: this.config.node.url,
485
+ storage: (this.config as LabNodeConfig).storage
486
+ });
487
+
488
+ this.logger.info('Registered with master', { registration });
489
+ }
490
+
491
+ private startHeartbeat(): void {
492
+ setInterval(async () => {
493
+ try {
494
+ await this.masterClient.sendHeartbeat({
495
+ node_id: this.config.node.id,
496
+ status: 'healthy',
497
+ timestamp: new Date().toISOString()
498
+ });
499
+ } catch (error) {
500
+ this.logger.error('Heartbeat failed', { error });
501
+ }
502
+ }, 30000); // Every 30 seconds
503
+ }
504
+
505
+ private async checkStorageHealth() {
506
+ // Check S3 connectivity
507
+ // Return status
508
+ }
509
+
510
+ private async checkMasterConnection() {
511
+ // Check master connectivity
512
+ // Return status
513
+ }
514
+ }
515
+ ```
516
+
517
+ **Deliverable**: Working master and lab node classes โœ…
518
+
519
+ ---
520
+
521
+ ## Week 2: API & Registration (Oct 31 - Nov 6)
522
+
523
+ ### Day 8-9: REST API Implementation
524
+
525
+ #### Task 1.8: Health endpoints
526
+
527
+ **src/api/routes/health.ts**:
528
+ ```typescript
529
+ import { FastifyInstance } from 'fastify';
530
+ import { BaseNode } from '../../node/base-node';
531
+
532
+ export async function healthRoutes(fastify: FastifyInstance, node: BaseNode) {
533
+ // Basic health check
534
+ fastify.get('/health', async (request, reply) => {
535
+ const health = await node.getHealth();
536
+
537
+ const statusCode = health.status === 'healthy' ? 200 :
538
+ health.status === 'degraded' ? 503 : 503;
539
+
540
+ return reply.code(statusCode).send(health);
541
+ });
542
+
543
+ // Detailed status
544
+ fastify.get('/status', async (request, reply) => {
545
+ return {
546
+ node: {
547
+ id: node.getNodeId(),
548
+ type: node.getNodeType(),
549
+ uptime: process.uptime(),
550
+ version: process.env.npm_package_version
551
+ },
552
+ health: await node.getHealth(),
553
+ metrics: {
554
+ memory: process.memoryUsage(),
555
+ cpu: process.cpuUsage()
556
+ }
557
+ };
558
+ });
559
+
560
+ // Readiness probe (Kubernetes)
561
+ fastify.get('/ready', async (request, reply) => {
562
+ const health = await node.getHealth();
563
+ return reply.code(health.status === 'healthy' ? 200 : 503).send({ ready: health.status === 'healthy' });
564
+ });
565
+
566
+ // Liveness probe (Kubernetes)
567
+ fastify.get('/live', async (request, reply) => {
568
+ return { alive: true };
569
+ });
570
+ }
571
+ ```
572
+
573
+ #### Task 1.9: Node discovery endpoints
574
+
575
+ **src/api/routes/discovery.ts**:
576
+ ```typescript
577
+ import { FastifyInstance } from 'fastify';
578
+ import { MasterNode } from '../../node/master-node';
579
+
580
+ export async function discoveryRoutes(fastify: FastifyInstance, node: MasterNode) {
581
+ // List all nodes in network
582
+ fastify.get('/api/v1/nodes', async (request, reply) => {
583
+ const nodes = await node.getRegisteredLabs();
584
+ return {
585
+ total: nodes.length,
586
+ nodes: nodes.map(lab => ({
587
+ id: lab.id,
588
+ name: lab.name,
589
+ url: lab.url,
590
+ status: lab.status,
591
+ datasets_count: lab.datasets?.length || 0
592
+ }))
593
+ };
594
+ });
595
+
596
+ // Get specific node details
597
+ fastify.get('/api/v1/nodes/:node_id', async (request, reply) => {
598
+ const { node_id } = request.params as { node_id: string };
599
+ const lab = await node.getLabStatus(node_id);
600
+
601
+ if (!lab) {
602
+ return reply.code(404).send({ error: 'Node not found' });
603
+ }
604
+
605
+ return lab;
606
+ });
607
+
608
+ // Get node health
609
+ fastify.get('/api/v1/nodes/:node_id/health', async (request, reply) => {
610
+ const { node_id } = request.params as { node_id: string };
611
+ const health = await node.getLabHealth(node_id);
612
+
613
+ if (!health) {
614
+ return reply.code(404).send({ error: 'Node not found' });
615
+ }
616
+
617
+ return health;
618
+ });
619
+ }
620
+ ```
621
+
622
+ **Deliverable**: REST API with health and discovery endpoints โœ…
623
+
624
+ ---
625
+
626
+ ### Day 10-12: Node Registration
627
+
628
+ #### Task 1.10: Lab registration API
629
+
630
+ **src/api/routes/registration.ts**:
631
+ ```typescript
632
+ import { FastifyInstance } from 'fastify';
633
+ import { MasterNode } from '../../node/master-node';
634
+
635
+ export async function registrationRoutes(fastify: FastifyInstance, node: MasterNode) {
636
+ // Lab registers with master
637
+ fastify.post('/api/v1/labs/register', {
638
+ schema: {
639
+ body: {
640
+ type: 'object',
641
+ required: ['name', 'wallet', 'storage'],
642
+ properties: {
643
+ name: { type: 'string' },
644
+ wallet: { type: 'string', pattern: '^0x[a-fA-F0-9]{40}$' },
645
+ admin_email: { type: 'string', format: 'email' },
646
+ storage: {
647
+ type: 'object',
648
+ required: ['bucket', 'region'],
649
+ properties: {
650
+ bucket: { type: 'string' },
651
+ region: { type: 'string' },
652
+ size_gb: { type: 'number' }
653
+ }
654
+ },
655
+ sponsored_services: {
656
+ type: 'array',
657
+ items: { type: 'string', enum: ['anthropic', 'story'] }
658
+ },
659
+ replication_enabled: { type: 'boolean' },
660
+ discovery_enabled: { type: 'boolean' }
661
+ }
662
+ }
663
+ }
664
+ }, async (request, reply) => {
665
+ const labInfo = request.body as LabRegistrationRequest;
666
+
667
+ try {
668
+ const registration = await node.registerLab(labInfo);
669
+
670
+ return reply.code(201).send({
671
+ node_id: registration.node_id,
672
+ registration_token: registration.token,
673
+ master_url: node.config.node.url,
674
+ config_template: `${node.config.node.url}/config/${registration.node_id}.yaml`,
675
+ status: 'pending_verification'
676
+ });
677
+ } catch (error) {
678
+ return reply.code(400).send({ error: error.message });
679
+ }
680
+ });
681
+
682
+ // Heartbeat from lab
683
+ fastify.post('/api/v1/labs/:lab_id/heartbeat', async (request, reply) => {
684
+ const { lab_id } = request.params as { lab_id: string };
685
+ const heartbeat = request.body as HeartbeatPayload;
686
+
687
+ await node.updateLabHeartbeat(lab_id, heartbeat);
688
+
689
+ return { acknowledged: true, timestamp: new Date().toISOString() };
690
+ });
691
+ }
692
+ ```
693
+
694
+ #### Task 1.11: Lab registry service
695
+
696
+ **src/services/lab-registry.ts**:
697
+ ```typescript
698
+ import { MasterNode } from '../node/master-node';
699
+
700
+ export class LabRegistry {
701
+ private labs: Map<string, LabInfo> = new Map();
702
+ private node: MasterNode;
703
+
704
+ constructor(node: MasterNode) {
705
+ this.node = node;
706
+ }
707
+
708
+ async start(): Promise<void> {
709
+ // Load existing labs from database
710
+ await this.loadLabs();
711
+
712
+ // Start health monitoring for all labs
713
+ this.startHealthMonitoring();
714
+ }
715
+
716
+ async registerLab(info: LabRegistrationRequest): Promise<LabRegistrationResponse> {
717
+ const node_id = this.generateNodeId(info.name);
718
+ const token = this.generateRegistrationToken();
719
+
720
+ const lab: LabInfo = {
721
+ id: node_id,
722
+ name: info.name,
723
+ wallet: info.wallet,
724
+ admin_email: info.admin_email,
725
+ storage: info.storage,
726
+ sponsored_services: info.sponsored_services || [],
727
+ status: 'pending',
728
+ registered_at: new Date(),
729
+ last_heartbeat: null
730
+ };
731
+
732
+ this.labs.set(node_id, lab);
733
+
734
+ // Save to database
735
+ await this.saveLab(lab);
736
+
737
+ return {
738
+ node_id,
739
+ token,
740
+ status: 'pending_verification'
741
+ };
742
+ }
743
+
744
+ async updateLabHeartbeat(labId: string, heartbeat: HeartbeatPayload): Promise<void> {
745
+ const lab = this.labs.get(labId);
746
+ if (!lab) {
747
+ throw new Error('Lab not found');
748
+ }
749
+
750
+ lab.last_heartbeat = new Date();
751
+ lab.status = heartbeat.status;
752
+
753
+ this.labs.set(labId, lab);
754
+ }
755
+
756
+ getConnectedLabCount(): number {
757
+ return Array.from(this.labs.values()).filter(lab => lab.status === 'online').length;
758
+ }
759
+
760
+ async getLabStatus(labId: string): Promise<LabStatus | null> {
761
+ const lab = this.labs.get(labId);
762
+ return lab ? this.buildLabStatus(lab) : null;
763
+ }
764
+
765
+ private startHealthMonitoring(): void {
766
+ setInterval(() => {
767
+ this.checkLabHealth();
768
+ }, 30000); // Every 30 seconds
769
+ }
770
+
771
+ private async checkLabHealth(): Promise<void> {
772
+ for (const [labId, lab] of this.labs) {
773
+ if (!lab.last_heartbeat) continue;
774
+
775
+ const timeSinceHeartbeat = Date.now() - lab.last_heartbeat.getTime();
776
+
777
+ if (timeSinceHeartbeat > 90000) { // 90 seconds
778
+ if (lab.status === 'online') {
779
+ lab.status = 'degraded';
780
+ this.node.logger.warn('Lab degraded', { labId });
781
+ }
782
+ }
783
+
784
+ if (timeSinceHeartbeat > 180000) { // 3 minutes
785
+ if (lab.status !== 'offline') {
786
+ lab.status = 'offline';
787
+ this.node.logger.error('Lab offline', { labId });
788
+
789
+ // Trigger failover
790
+ await this.node.triggerFailover(labId);
791
+ }
792
+ }
793
+ }
794
+ }
795
+
796
+ private generateNodeId(name: string): string {
797
+ return name.toLowerCase().replace(/[^a-z0-9]/g, '-');
798
+ }
799
+
800
+ private generateRegistrationToken(): string {
801
+ return `biofs_${crypto.randomBytes(32).toString('hex')}`;
802
+ }
803
+ }
804
+ ```
805
+
806
+ **Deliverable**: Complete registration system โœ…
807
+
808
+ ---
809
+
810
+ ### Day 13-14: CLI Commands
811
+
812
+ #### Task 1.12: biofs-node CLI
813
+
814
+ **src/index.ts**:
815
+ ```typescript
816
+ #!/usr/bin/env node
817
+
818
+ import { Command } from 'commander';
819
+ import { loadConfig } from './config/loader';
820
+ import { MasterNode } from './node/master-node';
821
+ import { LabNode } from './node/lab-node';
822
+
823
+ const program = new Command();
824
+
825
+ program
826
+ .name('biofs-node')
827
+ .description('BioFS Node Server - Master and Lab node daemon')
828
+ .version('0.1.0');
829
+
830
+ // Initialize node
831
+ program
832
+ .command('init')
833
+ .description('Initialize a new node')
834
+ .option('--type <type>', 'Node type (master|lab)', 'lab')
835
+ .option('--config <path>', 'Config file path', './biofs-node.yaml')
836
+ .action(async (options) => {
837
+ console.log('๐Ÿ”ง Initializing BioFS node...');
838
+ // Generate config template
839
+ // Save to file
840
+ });
841
+
842
+ // Start node
843
+ program
844
+ .command('start')
845
+ .description('Start the node server')
846
+ .option('--config <path>', 'Config file path', './biofs-node.yaml')
847
+ .option('--daemon', 'Run as daemon')
848
+ .action(async (options) => {
849
+ const config = await loadConfig(options.config);
850
+
851
+ console.log(`๐Ÿš€ Starting ${config.node.type} node: ${config.node.name}`);
852
+
853
+ const node = config.node.type === 'master'
854
+ ? new MasterNode(config)
855
+ : new LabNode(config);
856
+
857
+ await node.start();
858
+
859
+ console.log(`โœ… Node running at ${config.node.url}`);
860
+ console.log(`๐Ÿ“Š Metrics at http://localhost:${config.monitoring.port}`);
861
+
862
+ // Keep process running
863
+ process.on('SIGTERM', async () => {
864
+ console.log('โš ๏ธ Received SIGTERM, shutting down...');
865
+ await node.stop();
866
+ process.exit(0);
867
+ });
868
+ });
869
+
870
+ // Stop node
871
+ program
872
+ .command('stop')
873
+ .description('Stop the node server')
874
+ .action(async () => {
875
+ console.log('โ›” Stopping node...');
876
+ // Send SIGTERM to running process
877
+ });
878
+
879
+ // Status check
880
+ program
881
+ .command('status')
882
+ .description('Check node status')
883
+ .option('--config <path>', 'Config file path', './biofs-node.yaml')
884
+ .action(async (options) => {
885
+ const config = await loadConfig(options.config);
886
+
887
+ // Query health endpoint
888
+ const response = await fetch(`${config.node.url}/health`);
889
+ const health = await response.json();
890
+
891
+ console.log('๐Ÿ“Š Node Status:');
892
+ console.log(JSON.stringify(health, null, 2));
893
+ });
894
+
895
+ // Register lab with master
896
+ program
897
+ .command('register')
898
+ .description('Register lab node with master')
899
+ .requiredOption('--name <name>', 'Lab name')
900
+ .requiredOption('--wallet <address>', 'Lab wallet address')
901
+ .requiredOption('--bucket <uri>', 'S3 bucket URI')
902
+ .requiredOption('--master <url>', 'Master node URL')
903
+ .option('--sponsored-services <services>', 'Comma-separated list (anthropic,story)')
904
+ .action(async (options) => {
905
+ console.log('๐Ÿ“ Registering with master node...');
906
+
907
+ const registration = await fetch(`${options.master}/api/v1/labs/register`, {
908
+ method: 'POST',
909
+ headers: { 'Content-Type': 'application/json' },
910
+ body: JSON.stringify({
911
+ name: options.name,
912
+ wallet: options.wallet,
913
+ storage: {
914
+ bucket: options.bucket,
915
+ region: 'us-east-1'
916
+ },
917
+ sponsored_services: options.sponsoredServices?.split(',') || []
918
+ })
919
+ });
920
+
921
+ const result = await registration.json();
922
+
923
+ console.log('โœ… Registration successful!');
924
+ console.log(` Node ID: ${result.node_id}`);
925
+ console.log(` Token: ${result.registration_token}`);
926
+ console.log('');
927
+ console.log('Next steps:');
928
+ console.log(`1. Download config: curl ${result.config_template} > biofs-node.yaml`);
929
+ console.log('2. Edit config with your settings');
930
+ console.log('3. Start node: biofs-node start');
931
+ });
932
+
933
+ program.parse();
934
+ ```
935
+
936
+ **Deliverable**: Complete CLI for node management โœ…
937
+
938
+ ---
939
+
940
+ ## Phase 1 Deliverables Summary
941
+
942
+ ### โœ… What's Working
943
+ - [x] Master node starts and accepts connections
944
+ - [x] Lab node registers with master
945
+ - [x] Health monitoring with 30-second checks
946
+ - [x] REST API with /health, /status, /api/v1/nodes endpoints
947
+ - [x] Lab registry with automatic failover detection
948
+ - [x] CLI commands: init, start, stop, status, register
949
+ - [x] Configuration via YAML files
950
+ - [x] Structured logging with Pino
951
+ - [x] Prometheus metrics export
952
+
953
+ ### ๐Ÿ“ฆ Packages Published
954
+ - `@genobank/biofs-node@0.1.0`
955
+
956
+ ### ๐Ÿงช Testing Checklist
957
+ ```bash
958
+ # Terminal 1: Start master node
959
+ biofs-node start --config master-node.yaml
960
+
961
+ # Terminal 2: Register lab
962
+ biofs-node register \
963
+ --name "AUGenomics" \
964
+ --wallet 0xAUGenomics... \
965
+ --bucket s3://augenomics-genomics \
966
+ --master http://localhost:8080
967
+
968
+ # Terminal 3: Start lab node
969
+ biofs-node start --config augenomics-node.yaml
970
+
971
+ # Terminal 4: Check health
972
+ curl http://localhost:8080/health
973
+ curl http://localhost:8080/api/v1/nodes
974
+ ```
975
+
976
+ **Phase 1 Success Criteria**: โœ… All nodes start, register, and report health
977
+
978
+ ---
979
+
980
+ # ๐Ÿ“… Phase 2: Sponsored Services (Week 3)
981
+
982
+ **Duration**: November 7-13, 2025 (1 week)
983
+ **Goal**: Anthropic API + Story Protocol proxies with fallback logic
984
+
985
+ ---
986
+
987
+ ## Day 15-16: Anthropic API Proxy
988
+
989
+ ### Task 2.1: Anthropic service
990
+
991
+ **src/services/anthropic-proxy.ts**:
992
+ ```typescript
993
+ import Anthropic from '@anthropic-ai/sdk';
994
+ import { BaseNode } from '../node/base-node';
995
+
996
+ export class AnthropicProxy {
997
+ private node: BaseNode;
998
+ private labClient?: Anthropic;
999
+ private masterClient?: Anthropic;
1000
+
1001
+ constructor(node: BaseNode) {
1002
+ this.node = node;
1003
+
1004
+ // Lab-provided key (optional)
1005
+ if (process.env.LAB_ANTHROPIC_KEY) {
1006
+ this.labClient = new Anthropic({ apiKey: process.env.LAB_ANTHROPIC_KEY });
1007
+ }
1008
+
1009
+ // Master fallback key
1010
+ if (process.env.MASTER_ANTHROPIC_KEY) {
1011
+ this.masterClient = new Anthropic({ apiKey: process.env.MASTER_ANTHROPIC_KEY });
1012
+ }
1013
+ }
1014
+
1015
+ async createMessage(params: {
1016
+ researcher: string;
1017
+ model: string;
1018
+ messages: any[];
1019
+ max_tokens: number;
1020
+ }): Promise<AnthropicResponse> {
1021
+ const { researcher, ...anthropicParams } = params;
1022
+
1023
+ // 1. Try lab key first
1024
+ if (this.labClient) {
1025
+ try {
1026
+ const labQuota = await this.checkLabQuota(researcher);
1027
+ if (labQuota.available > 0) {
1028
+ this.node.logger.info('Using lab Anthropic key', { researcher });
1029
+
1030
+ const response = await this.labClient.messages.create(anthropicParams);
1031
+
1032
+ await this.recordUsage(researcher, 'lab', response);
1033
+
1034
+ return {
1035
+ response,
1036
+ billed_to: 'lab'
1037
+ };
1038
+ }
1039
+ } catch (error) {
1040
+ this.node.logger.warn('Lab Anthropic key failed, falling back', { error });
1041
+ }
1042
+ }
1043
+
1044
+ // 2. Fallback to master key
1045
+ if (this.masterClient) {
1046
+ const masterQuota = await this.checkMasterQuota(researcher);
1047
+ if (masterQuota.available > 0) {
1048
+ this.node.logger.info('Using master Anthropic key', { researcher });
1049
+
1050
+ const response = await this.masterClient.messages.create(anthropicParams);
1051
+
1052
+ await this.recordUsage(researcher, 'master', response);
1053
+
1054
+ return {
1055
+ response,
1056
+ billed_to: 'master'
1057
+ };
1058
+ }
1059
+ }
1060
+
1061
+ // 3. Quota exceeded
1062
+ throw new Error('Anthropic API quota exceeded. Please contact lab admin or upgrade tier.');
1063
+ }
1064
+
1065
+ private async checkLabQuota(researcher: string): Promise<QuotaInfo> {
1066
+ // Query database for lab-level quota
1067
+ const usage = await this.getMonthlyUsage(researcher, 'lab');
1068
+ const limit = await this.getLabQuotaLimit(researcher);
1069
+
1070
+ return {
1071
+ available: limit - usage,
1072
+ used: usage,
1073
+ limit
1074
+ };
1075
+ }
1076
+
1077
+ private async checkMasterQuota(researcher: string): Promise<QuotaInfo> {
1078
+ // Query database for master-level quota
1079
+ const usage = await this.getMonthlyUsage(researcher, 'master');
1080
+ const limit = await this.getMasterQuotaLimit(researcher);
1081
+
1082
+ return {
1083
+ available: limit - usage,
1084
+ used: usage,
1085
+ limit
1086
+ };
1087
+ }
1088
+
1089
+ private async recordUsage(researcher: string, source: 'lab' | 'master', response: any): Promise<void> {
1090
+ const usage = {
1091
+ researcher,
1092
+ source,
1093
+ input_tokens: response.usage.input_tokens,
1094
+ output_tokens: response.usage.output_tokens,
1095
+ model: response.model,
1096
+ timestamp: new Date()
1097
+ };
1098
+
1099
+ // Save to database for billing
1100
+ await this.saveUsageRecord(usage);
1101
+
1102
+ // Emit metrics
1103
+ this.node.metrics.recordAnthropicUsage(usage);
1104
+ }
1105
+ }
1106
+ ```
1107
+
1108
+ ### Task 2.2: Anthropic API routes
1109
+
1110
+ **src/api/routes/anthropic.ts**:
1111
+ ```typescript
1112
+ import { FastifyInstance } from 'fastify';
1113
+ import { LabNode } from '../../node/lab-node';
1114
+ import { AnthropicProxy } from '../../services/anthropic-proxy';
1115
+
1116
+ export async function anthropicRoutes(fastify: FastifyInstance, node: LabNode) {
1117
+ const proxy = new AnthropicProxy(node);
1118
+
1119
+ fastify.post('/api/v1/anthropic/messages', {
1120
+ schema: {
1121
+ headers: {
1122
+ type: 'object',
1123
+ required: ['authorization'],
1124
+ properties: {
1125
+ authorization: { type: 'string' }
1126
+ }
1127
+ },
1128
+ body: {
1129
+ type: 'object',
1130
+ required: ['model', 'messages', 'max_tokens'],
1131
+ properties: {
1132
+ model: { type: 'string' },
1133
+ messages: { type: 'array' },
1134
+ max_tokens: { type: 'number' }
1135
+ }
1136
+ }
1137
+ }
1138
+ }, async (request, reply) => {
1139
+ // Extract researcher from JWT
1140
+ const researcher = request.user.wallet;
1141
+
1142
+ const result = await proxy.createMessage({
1143
+ researcher,
1144
+ ...request.body
1145
+ });
1146
+
1147
+ return {
1148
+ ...result.response,
1149
+ _meta: {
1150
+ billed_to: result.billed_to,
1151
+ quota_remaining: await proxy.getQuotaRemaining(researcher)
1152
+ }
1153
+ };
1154
+ });
1155
+
1156
+ // Check quota
1157
+ fastify.get('/api/v1/anthropic/quota', async (request, reply) => {
1158
+ const researcher = request.user.wallet;
1159
+
1160
+ const labQuota = await proxy.getLabQuota(researcher);
1161
+ const masterQuota = await proxy.getMasterQuota(researcher);
1162
+
1163
+ return {
1164
+ lab: labQuota,
1165
+ master: masterQuota,
1166
+ total_available: labQuota.available + masterQuota.available
1167
+ };
1168
+ });
1169
+ }
1170
+ ```
1171
+
1172
+ **Deliverable**: Anthropic API proxy with lab/master fallback โœ…
1173
+
1174
+ ---
1175
+
1176
+ ## Day 17-18: Story Protocol Proxy
1177
+
1178
+ ### Task 2.3: Story Protocol service
1179
+
1180
+ **src/services/story-proxy.ts**:
1181
+ ```typescript
1182
+ import { ethers } from 'ethers';
1183
+ import { BaseNode } from '../node/base-node';
1184
+
1185
+ export class StoryProtocolProxy {
1186
+ private node: BaseNode;
1187
+ private labExecutor?: ethers.Wallet;
1188
+ private masterExecutor?: ethers.Wallet;
1189
+ private provider: ethers.Provider;
1190
+
1191
+ constructor(node: BaseNode) {
1192
+ this.node = node;
1193
+ this.provider = new ethers.JsonRpcProvider(process.env.STORY_RPC_URL);
1194
+
1195
+ // Lab executor (optional)
1196
+ if (process.env.LAB_EXECUTOR_KEY) {
1197
+ this.labExecutor = new ethers.Wallet(process.env.LAB_EXECUTOR_KEY, this.provider);
1198
+ }
1199
+
1200
+ // Master executor (fallback)
1201
+ if (process.env.MASTER_EXECUTOR_KEY) {
1202
+ this.masterExecutor = new ethers.Wallet(process.env.MASTER_EXECUTOR_KEY, this.provider);
1203
+ }
1204
+ }
1205
+
1206
+ async mintLicenseToken(params: {
1207
+ researcher: string;
1208
+ ip_id: string;
1209
+ license_terms_id: number;
1210
+ receiver: string;
1211
+ }): Promise<StoryProtocolResponse> {
1212
+ const { researcher, ...mintParams } = params;
1213
+
1214
+ // 1. Try lab executor first
1215
+ if (this.labExecutor) {
1216
+ try {
1217
+ const gasQuota = await this.checkLabGasQuota(researcher);
1218
+ if (gasQuota.available > 0) {
1219
+ this.node.logger.info('Using lab executor', { researcher });
1220
+
1221
+ const tx = await this.executeMint(this.labExecutor, mintParams);
1222
+
1223
+ await this.recordGasUsage(researcher, 'lab', tx);
1224
+
1225
+ return {
1226
+ tx_hash: tx.hash,
1227
+ license_token_id: tx.events.find(e => e.event === 'LicenseTokenMinted')?.args?.tokenId,
1228
+ billed_to: 'lab'
1229
+ };
1230
+ }
1231
+ } catch (error) {
1232
+ this.node.logger.warn('Lab executor failed, falling back', { error });
1233
+ }
1234
+ }
1235
+
1236
+ // 2. Fallback to master executor
1237
+ if (this.masterExecutor) {
1238
+ const gasQuota = await this.checkMasterGasQuota(researcher);
1239
+ if (gasQuota.available > 0) {
1240
+ this.node.logger.info('Using master executor', { researcher });
1241
+
1242
+ const tx = await this.executeMint(this.masterExecutor, mintParams);
1243
+
1244
+ await this.recordGasUsage(researcher, 'master', tx);
1245
+
1246
+ return {
1247
+ tx_hash: tx.hash,
1248
+ license_token_id: tx.events.find(e => e.event === 'LicenseTokenMinted')?.args?.tokenId,
1249
+ billed_to: 'master'
1250
+ };
1251
+ }
1252
+ }
1253
+
1254
+ // 3. Gas quota exceeded
1255
+ throw new Error('Story Protocol gas quota exceeded. Please contact lab admin or provide own gas.');
1256
+ }
1257
+
1258
+ private async executeMint(executor: ethers.Wallet, params: any): Promise<any> {
1259
+ // Get contract
1260
+ const contract = new ethers.Contract(
1261
+ process.env.LICENSE_REGISTRY_ADDRESS!,
1262
+ LICENSE_REGISTRY_ABI,
1263
+ executor
1264
+ );
1265
+
1266
+ // Estimate gas
1267
+ const gasEstimate = await contract.mintLicenseTokens.estimateGas(
1268
+ params.ip_id,
1269
+ params.license_terms_id,
1270
+ 1, // amount
1271
+ params.receiver,
1272
+ ethers.ZeroAddress // royalty context
1273
+ );
1274
+
1275
+ // Execute transaction
1276
+ const tx = await contract.mintLicenseTokens(
1277
+ params.ip_id,
1278
+ params.license_terms_id,
1279
+ 1,
1280
+ params.receiver,
1281
+ ethers.ZeroAddress,
1282
+ {
1283
+ gasLimit: gasEstimate * 120n / 100n // 20% buffer
1284
+ }
1285
+ );
1286
+
1287
+ // Wait for confirmation
1288
+ const receipt = await tx.wait();
1289
+
1290
+ return {
1291
+ hash: tx.hash,
1292
+ events: receipt.logs,
1293
+ gas_used: receipt.gasUsed,
1294
+ gas_price: tx.gasPrice
1295
+ };
1296
+ }
1297
+
1298
+ private async checkLabGasQuota(researcher: string): Promise<GasQuotaInfo> {
1299
+ // Query database
1300
+ const usage = await this.getMonthlyGasUsage(researcher, 'lab');
1301
+ const limit = await this.getLabGasLimit(researcher);
1302
+
1303
+ return {
1304
+ available: limit - usage,
1305
+ used: usage,
1306
+ limit
1307
+ };
1308
+ }
1309
+
1310
+ private async checkMasterGasQuota(researcher: string): Promise<GasQuotaInfo> {
1311
+ // Query database
1312
+ const usage = await this.getMonthlyGasUsage(researcher, 'master');
1313
+ const limit = await this.getMasterGasLimit(researcher);
1314
+
1315
+ return {
1316
+ available: limit - usage,
1317
+ used: usage,
1318
+ limit
1319
+ };
1320
+ }
1321
+
1322
+ private async recordGasUsage(researcher: string, source: 'lab' | 'master', tx: any): Promise<void> {
1323
+ const usage = {
1324
+ researcher,
1325
+ source,
1326
+ tx_hash: tx.hash,
1327
+ gas_used: tx.gas_used.toString(),
1328
+ gas_price: tx.gas_price.toString(),
1329
+ cost_eth: ethers.formatEther(tx.gas_used * tx.gas_price),
1330
+ timestamp: new Date()
1331
+ };
1332
+
1333
+ // Save to database for billing
1334
+ await this.saveGasUsageRecord(usage);
1335
+
1336
+ // Emit metrics
1337
+ this.node.metrics.recordGasUsage(usage);
1338
+ }
1339
+ }
1340
+ ```
1341
+
1342
+ **Deliverable**: Story Protocol proxy with gas sponsorship โœ…
1343
+
1344
+ ---
1345
+
1346
+ ## Day 19-21: Quota Management
1347
+
1348
+ ### Task 2.4: Quota tracking database
1349
+
1350
+ **src/database/schema.sql**:
1351
+ ```sql
1352
+ -- Anthropic API usage tracking
1353
+ CREATE TABLE anthropic_usage (
1354
+ id SERIAL PRIMARY KEY,
1355
+ researcher VARCHAR(42) NOT NULL,
1356
+ source VARCHAR(10) NOT NULL, -- 'lab' or 'master'
1357
+ input_tokens INTEGER NOT NULL,
1358
+ output_tokens INTEGER NOT NULL,
1359
+ model VARCHAR(50) NOT NULL,
1360
+ cost_usd DECIMAL(10, 4),
1361
+ timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
1362
+ month VARCHAR(7) NOT NULL, -- '2025-11'
1363
+ INDEX idx_researcher_month (researcher, month),
1364
+ INDEX idx_source (source),
1365
+ INDEX idx_timestamp (timestamp)
1366
+ );
1367
+
1368
+ -- Story Protocol gas usage tracking
1369
+ CREATE TABLE gas_usage (
1370
+ id SERIAL PRIMARY KEY,
1371
+ researcher VARCHAR(42) NOT NULL,
1372
+ source VARCHAR(10) NOT NULL, -- 'lab' or 'master'
1373
+ tx_hash VARCHAR(66) NOT NULL,
1374
+ gas_used BIGINT NOT NULL,
1375
+ gas_price BIGINT NOT NULL,
1376
+ cost_eth DECIMAL(18, 10) NOT NULL,
1377
+ cost_usd DECIMAL(10, 2),
1378
+ timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
1379
+ month VARCHAR(7) NOT NULL,
1380
+ INDEX idx_researcher_month (researcher, month),
1381
+ INDEX idx_source (source),
1382
+ INDEX idx_tx_hash (tx_hash)
1383
+ );
1384
+
1385
+ -- Researcher quota configuration
1386
+ CREATE TABLE researcher_quotas (
1387
+ researcher VARCHAR(42) PRIMARY KEY,
1388
+ lab_id VARCHAR(50) NOT NULL,
1389
+ tier VARCHAR(20) NOT NULL DEFAULT 'free', -- free, basic, premium, vip
1390
+ anthropic_tokens_per_month INTEGER,
1391
+ story_mints_per_month INTEGER,
1392
+ storage_gb INTEGER,
1393
+ bandwidth_gb_per_month INTEGER,
1394
+ multiplier DECIMAL(3, 1) DEFAULT 1.0, -- VIP gets 2.0, etc.
1395
+ created_at TIMESTAMP DEFAULT NOW(),
1396
+ updated_at TIMESTAMP DEFAULT NOW(),
1397
+ INDEX idx_lab_id (lab_id)
1398
+ );
1399
+
1400
+ -- Monthly billing aggregation (materialized view)
1401
+ CREATE MATERIALIZED VIEW monthly_billing AS
1402
+ SELECT
1403
+ lab_id,
1404
+ month,
1405
+ SUM(CASE WHEN source = 'master' THEN cost_usd ELSE 0 END) as master_anthropic_cost,
1406
+ SUM(CASE WHEN source = 'master' THEN cost_eth ELSE 0 END) as master_gas_cost_eth,
1407
+ COUNT(DISTINCT researcher) as active_researchers
1408
+ FROM (
1409
+ SELECT researcher, 'lab' as lab_id, source, cost_usd, 0 as cost_eth, month
1410
+ FROM anthropic_usage
1411
+ UNION ALL
1412
+ SELECT researcher, 'lab' as lab_id, source, 0 as cost_usd, cost_eth, month
1413
+ FROM gas_usage
1414
+ ) combined
1415
+ GROUP BY lab_id, month;
1416
+ ```
1417
+
1418
+ ### Task 2.5: Quota API endpoints
1419
+
1420
+ **src/api/routes/quota.ts**:
1421
+ ```typescript
1422
+ import { FastifyInstance } from 'fastify';
1423
+ import { BaseNode } from '../../node/base-node';
1424
+
1425
+ export async function quotaRoutes(fastify: FastifyInstance, node: BaseNode) {
1426
+ // Get researcher's quota
1427
+ fastify.get('/api/v1/quota', async (request, reply) => {
1428
+ const researcher = request.user.wallet;
1429
+
1430
+ const quota = await node.getResearcherQuota(researcher);
1431
+ const usage = await node.getMonthlyUsage(researcher);
1432
+
1433
+ return {
1434
+ researcher,
1435
+ tier: quota.tier,
1436
+ limits: {
1437
+ anthropic_tokens: quota.anthropic_tokens_per_month,
1438
+ story_mints: quota.story_mints_per_month,
1439
+ storage_gb: quota.storage_gb,
1440
+ bandwidth_gb: quota.bandwidth_gb_per_month
1441
+ },
1442
+ usage: {
1443
+ anthropic_tokens: usage.anthropic_tokens,
1444
+ story_mints: usage.story_mints,
1445
+ storage_gb: usage.storage_gb,
1446
+ bandwidth_gb: usage.bandwidth_gb
1447
+ },
1448
+ remaining: {
1449
+ anthropic_tokens: quota.anthropic_tokens_per_month - usage.anthropic_tokens,
1450
+ story_mints: quota.story_mints_per_month - usage.story_mints,
1451
+ storage_gb: quota.storage_gb - usage.storage_gb,
1452
+ bandwidth_gb: quota.bandwidth_gb_per_month - usage.bandwidth_gb
1453
+ },
1454
+ percentage_used: {
1455
+ anthropic_tokens: (usage.anthropic_tokens / quota.anthropic_tokens_per_month * 100).toFixed(1),
1456
+ story_mints: (usage.story_mints / quota.story_mints_per_month * 100).toFixed(1)
1457
+ }
1458
+ };
1459
+ });
1460
+
1461
+ // Admin: Update researcher quota
1462
+ fastify.put('/api/v1/quota/:researcher', {
1463
+ preHandler: [fastify.authenticate, fastify.requireAdmin]
1464
+ }, async (request, reply) => {
1465
+ const { researcher } = request.params as { researcher: string };
1466
+ const updates = request.body as QuotaUpdates;
1467
+
1468
+ await node.updateResearcherQuota(researcher, updates);
1469
+
1470
+ return { updated: true, researcher, changes: updates };
1471
+ });
1472
+
1473
+ // Get lab-wide billing summary
1474
+ fastify.get('/api/v1/billing/summary', {
1475
+ preHandler: [fastify.authenticate, fastify.requireAdmin]
1476
+ }, async (request, reply) => {
1477
+ const { month } = request.query as { month?: string };
1478
+ const targetMonth = month || new Date().toISOString().slice(0, 7);
1479
+
1480
+ const billing = await node.getLabBillingSummary(targetMonth);
1481
+
1482
+ return billing;
1483
+ });
1484
+ }
1485
+ ```
1486
+
1487
+ **Deliverable**: Complete quota management system โœ…
1488
+
1489
+ ---
1490
+
1491
+ ## Phase 2 Deliverables Summary
1492
+
1493
+ ### โœ… What's Working
1494
+ - [x] Anthropic API proxy with lab/master fallback
1495
+ - [x] Story Protocol executor proxy with gas sponsorship
1496
+ - [x] Quota tracking in PostgreSQL
1497
+ - [x] Usage recording with metrics
1498
+ - [x] Billing aggregation for invoicing
1499
+ - [x] REST API for quota management
1500
+ - [x] Automatic fallback when quota exceeded
1501
+
1502
+ ### ๐Ÿ“Š Billing Report Example
1503
+ ```
1504
+ November 2025 - AUGenomics Lab
1505
+
1506
+ Anthropic API:
1507
+ Lab key used: 1.5M tokens ($45.00)
1508
+ Master fallback: 500k tokens ($15.00)
1509
+ Total: 2.0M tokens ($60.00)
1510
+
1511
+ Story Protocol:
1512
+ Lab executor: 15 mints (0.03 ETH = $75.00)
1513
+ Master fallback: 5 mints (0.01 ETH = $25.00)
1514
+ Total: 20 mints (0.04 ETH = $100.00)
1515
+
1516
+ Total Billable (Master fallback): $40.00
1517
+ ```
1518
+
1519
+ ### ๐Ÿงช Testing Checklist
1520
+ ```bash
1521
+ # Test Anthropic proxy
1522
+ curl -X POST http://localhost:8080/api/v1/anthropic/messages \
1523
+ -H "Authorization: Bearer ${JWT}" \
1524
+ -d '{"model":"claude-3-haiku-20240307","messages":[...],"max_tokens":1024}'
1525
+
1526
+ # Test Story Protocol proxy
1527
+ curl -X POST http://localhost:8080/api/v1/story/mint-license \
1528
+ -H "Authorization: Bearer ${JWT}" \
1529
+ -d '{"ip_id":"0x...","license_terms_id":1,"receiver":"0x..."}'
1530
+
1531
+ # Check quota
1532
+ curl http://localhost:8080/api/v1/quota \
1533
+ -H "Authorization: Bearer ${JWT}"
1534
+ ```
1535
+
1536
+ **Phase 2 Success Criteria**: โœ… Sponsored services work with fallback logic
1537
+
1538
+ ---
1539
+
1540
+ # ๐Ÿ“… Phase 3: Data Replication (Week 4)
1541
+
1542
+ **Duration**: November 14-20, 2025 (1 week)
1543
+ **Goal**: S3 sync, failover mechanism, replica verification
1544
+
1545
+ ---
1546
+
1547
+ ## Day 22-23: S3 Replication Service
1548
+
1549
+ ### Task 3.1: Replication manager
1550
+
1551
+ **src/services/replication.ts**:
1552
+ ```typescript
1553
+ import { S3Client, ListObjectsV2Command, CopyObjectCommand } from '@aws-sdk/client-s3';
1554
+ import { MasterNode } from '../node/master-node';
1555
+
1556
+ export class ReplicationManager {
1557
+ private node: MasterNode;
1558
+ private s3Client: S3Client;
1559
+ private replicationJobs: Map<string, ReplicationJob> = new Map();
1560
+
1561
+ constructor(node: MasterNode) {
1562
+ this.node = node;
1563
+ this.s3Client = new S3Client({ region: process.env.AWS_REGION });
1564
+ }
1565
+
1566
+ async start(): Promise<void> {
1567
+ this.node.logger.info('Starting replication manager');
1568
+
1569
+ // Schedule replication for all labs
1570
+ this.scheduleReplication();
1571
+ }
1572
+
1573
+ private scheduleReplication(): void {
1574
+ // Run every hour
1575
+ setInterval(async () => {
1576
+ await this.replicateAllLabs();
1577
+ }, 3600000); // 1 hour
1578
+
1579
+ // Initial run
1580
+ this.replicateAllLabs();
1581
+ }
1582
+
1583
+ private async replicateAllLabs(): Promise<void> {
1584
+ const labs = await this.node.getLabsWithReplicationEnabled();
1585
+
1586
+ for (const lab of labs) {
1587
+ await this.replicateLab(lab.id);
1588
+ }
1589
+ }
1590
+
1591
+ async replicateLab(labId: string): Promise<void> {
1592
+ const lab = await this.node.getLabStatus(labId);
1593
+ if (!lab) return;
1594
+
1595
+ this.node.logger.info('Starting replication', { labId });
1596
+
1597
+ const job: ReplicationJob = {
1598
+ lab_id: labId,
1599
+ status: 'in_progress',
1600
+ started_at: new Date(),
1601
+ files_total: 0,
1602
+ files_synced: 0,
1603
+ bytes_transferred: 0
1604
+ };
1605
+
1606
+ this.replicationJobs.set(labId, job);
1607
+
1608
+ try {
1609
+ // List objects in lab bucket
1610
+ const objects = await this.listLabObjects(lab.storage.bucket);
1611
+ job.files_total = objects.length;
1612
+
1613
+ // Replicate each file
1614
+ for (const object of objects) {
1615
+ await this.replicateObject(lab, object);
1616
+ job.files_synced++;
1617
+ job.bytes_transferred += object.Size || 0;
1618
+
1619
+ // Log progress every 100 files
1620
+ if (job.files_synced % 100 === 0) {
1621
+ this.node.logger.info('Replication progress', {
1622
+ labId,
1623
+ progress: `${job.files_synced}/${job.files_total}`,
1624
+ bytes: this.formatBytes(job.bytes_transferred)
1625
+ });
1626
+ }
1627
+ }
1628
+
1629
+ job.status = 'completed';
1630
+ job.completed_at = new Date();
1631
+
1632
+ this.node.logger.info('Replication complete', {
1633
+ labId,
1634
+ files: job.files_synced,
1635
+ bytes: this.formatBytes(job.bytes_transferred),
1636
+ duration: job.completed_at.getTime() - job.started_at.getTime()
1637
+ });
1638
+
1639
+ // Save replication record
1640
+ await this.saveReplicationRecord(job);
1641
+
1642
+ } catch (error) {
1643
+ job.status = 'failed';
1644
+ job.error = error.message;
1645
+ this.node.logger.error('Replication failed', { labId, error });
1646
+ }
1647
+ }
1648
+
1649
+ private async listLabObjects(bucket: string): Promise<S3Object[]> {
1650
+ const objects: S3Object[] = [];
1651
+ let continuationToken: string | undefined;
1652
+
1653
+ do {
1654
+ const command = new ListObjectsV2Command({
1655
+ Bucket: bucket,
1656
+ ContinuationToken: continuationToken
1657
+ });
1658
+
1659
+ const response = await this.s3Client.send(command);
1660
+
1661
+ if (response.Contents) {
1662
+ objects.push(...response.Contents);
1663
+ }
1664
+
1665
+ continuationToken = response.NextContinuationToken;
1666
+ } while (continuationToken);
1667
+
1668
+ return objects;
1669
+ }
1670
+
1671
+ private async replicateObject(lab: LabInfo, object: S3Object): Promise<void> {
1672
+ const sourceBucket = lab.storage.bucket;
1673
+ const sourceKey = object.Key!;
1674
+ const destBucket = process.env.MASTER_REPLICA_BUCKET!;
1675
+ const destKey = `replicas/${lab.id}/${sourceKey}`;
1676
+
1677
+ // Check if object already exists and is up-to-date
1678
+ const needsSync = await this.needsReplication(destBucket, destKey, object);
1679
+ if (!needsSync) {
1680
+ return; // Skip, already synced
1681
+ }
1682
+
1683
+ // Copy object
1684
+ const command = new CopyObjectCommand({
1685
+ CopySource: `${sourceBucket}/${sourceKey}`,
1686
+ Bucket: destBucket,
1687
+ Key: destKey,
1688
+ MetadataDirective: 'COPY',
1689
+ TaggingDirective: 'COPY'
1690
+ });
1691
+
1692
+ await this.s3Client.send(command);
1693
+ }
1694
+
1695
+ private async needsReplication(bucket: string, key: string, sourceObject: S3Object): Promise<boolean> {
1696
+ try {
1697
+ const command = new HeadObjectCommand({ Bucket: bucket, Key: key });
1698
+ const existing = await this.s3Client.send(command);
1699
+
1700
+ // Compare ETags (checksums)
1701
+ return existing.ETag !== sourceObject.ETag;
1702
+ } catch (error) {
1703
+ // Object doesn't exist, needs replication
1704
+ return true;
1705
+ }
1706
+ }
1707
+
1708
+ async getReplicationStatus(labId: string): Promise<ReplicationJob | null> {
1709
+ return this.replicationJobs.get(labId) || await this.getLastReplicationRecord(labId);
1710
+ }
1711
+
1712
+ private formatBytes(bytes: number): string {
1713
+ if (bytes === 0) return '0 B';
1714
+ const k = 1024;
1715
+ const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
1716
+ const i = Math.floor(Math.log(bytes) / Math.log(k));
1717
+ return `${(bytes / Math.pow(k, i)).toFixed(2)} ${sizes[i]}`;
1718
+ }
1719
+ }
1720
+ ```
1721
+
1722
+ **Deliverable**: S3 replication service โœ…
1723
+
1724
+ ---
1725
+
1726
+ ## Day 24-25: Failover Mechanism
1727
+
1728
+ ### Task 3.2: Failover service
1729
+
1730
+ **src/services/failover.ts**:
1731
+ ```typescript
1732
+ import { MasterNode } from '../node/master-node';
1733
+ import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
1734
+ import { Readable } from 'stream';
1735
+
1736
+ export class FailoverService {
1737
+ private node: MasterNode;
1738
+ private s3Client: S3Client;
1739
+ private activeFailovers: Map<string, FailoverInfo> = new Map();
1740
+
1741
+ constructor(node: MasterNode) {
1742
+ this.node = node;
1743
+ this.s3Client = new S3Client({ region: process.env.AWS_REGION });
1744
+ }
1745
+
1746
+ async activateFailover(labId: string): Promise<void> {
1747
+ this.node.logger.warn('Activating failover', { labId });
1748
+
1749
+ const lab = await this.node.getLabStatus(labId);
1750
+ if (!lab) {
1751
+ throw new Error('Lab not found');
1752
+ }
1753
+
1754
+ // Check if replication is up-to-date
1755
+ const replicationStatus = await this.node.getReplicationStatus(labId);
1756
+ if (!replicationStatus || replicationStatus.status !== 'completed') {
1757
+ this.node.logger.error('Cannot failover: replication not up-to-date', { labId });
1758
+ throw new Error('Replication not complete');
1759
+ }
1760
+
1761
+ // Activate failover
1762
+ const failover: FailoverInfo = {
1763
+ lab_id: labId,
1764
+ activated_at: new Date(),
1765
+ serving_from: 'master',
1766
+ replica_location: `s3://${process.env.MASTER_REPLICA_BUCKET}/replicas/${labId}/`
1767
+ };
1768
+
1769
+ this.activeFailovers.set(labId, failover);
1770
+
1771
+ // Update lab status
1772
+ await this.node.updateLabStatus(labId, {
1773
+ status: 'offline',
1774
+ failover_active: true
1775
+ });
1776
+
1777
+ // Send notifications
1778
+ await this.notifyLabAdmin(lab, 'Failover activated. Master node serving requests.');
1779
+
1780
+ this.node.logger.info('Failover activated', { labId, replica: failover.replica_location });
1781
+ }
1782
+
1783
+ async deactivateFailover(labId: string): Promise<void> {
1784
+ this.node.logger.info('Deactivating failover', { labId });
1785
+
1786
+ const failover = this.activeFailovers.get(labId);
1787
+ if (!failover) {
1788
+ return; // No active failover
1789
+ }
1790
+
1791
+ // Verify lab is healthy
1792
+ const health = await this.verifyLabHealth(labId);
1793
+ if (!health.healthy) {
1794
+ throw new Error('Lab not healthy, cannot deactivate failover');
1795
+ }
1796
+
1797
+ // Gradual traffic shift back to lab
1798
+ await this.gradualTrafficShift(labId);
1799
+
1800
+ // Deactivate
1801
+ this.activeFailovers.delete(labId);
1802
+
1803
+ await this.node.updateLabStatus(labId, {
1804
+ status: 'online',
1805
+ failover_active: false
1806
+ });
1807
+
1808
+ this.node.logger.info('Failover deactivated', { labId });
1809
+ }
1810
+
1811
+ async serveFileFromReplica(labId: string, filePath: string): Promise<Readable> {
1812
+ const failover = this.activeFailovers.get(labId);
1813
+ if (!failover) {
1814
+ throw new Error('No active failover for this lab');
1815
+ }
1816
+
1817
+ const bucket = process.env.MASTER_REPLICA_BUCKET!;
1818
+ const key = `replicas/${labId}/${filePath}`;
1819
+
1820
+ this.node.logger.info('Serving file from replica', { labId, key });
1821
+
1822
+ const command = new GetObjectCommand({
1823
+ Bucket: bucket,
1824
+ Key: key
1825
+ });
1826
+
1827
+ const response = await this.s3Client.send(command);
1828
+
1829
+ return response.Body as Readable;
1830
+ }
1831
+
1832
+ isFailoverActive(labId: string): boolean {
1833
+ return this.activeFailovers.has(labId);
1834
+ }
1835
+
1836
+ private async verifyLabHealth(labId: string): Promise<{ healthy: boolean; checks: number }> {
1837
+ let successfulChecks = 0;
1838
+ const requiredChecks = 5;
1839
+
1840
+ for (let i = 0; i < requiredChecks; i++) {
1841
+ try {
1842
+ const health = await this.node.checkLabHealth(labId);
1843
+ if (health.status === 'healthy') {
1844
+ successfulChecks++;
1845
+ }
1846
+ } catch (error) {
1847
+ // Check failed
1848
+ }
1849
+
1850
+ await new Promise(resolve => setTimeout(resolve, 10000)); // Wait 10s between checks
1851
+ }
1852
+
1853
+ return {
1854
+ healthy: successfulChecks >= requiredChecks,
1855
+ checks: successfulChecks
1856
+ };
1857
+ }
1858
+
1859
+ private async gradualTrafficShift(labId: string): Promise<void> {
1860
+ // Shift traffic in 20% increments over 5 minutes
1861
+ const increments = [20, 40, 60, 80, 100];
1862
+
1863
+ for (const percentage of increments) {
1864
+ this.node.logger.info('Shifting traffic to lab', { labId, percentage });
1865
+
1866
+ // Update routing weights
1867
+ await this.node.updateRoutingWeight(labId, percentage);
1868
+
1869
+ // Wait 1 minute
1870
+ await new Promise(resolve => setTimeout(resolve, 60000));
1871
+
1872
+ // Verify no errors
1873
+ const errors = await this.node.getLabErrorRate(labId);
1874
+ if (errors > 0.05) { // More than 5% errors
1875
+ this.node.logger.error('High error rate during traffic shift, reverting', { labId, errors });
1876
+ await this.node.updateRoutingWeight(labId, 0);
1877
+ throw new Error('Traffic shift failed due to high error rate');
1878
+ }
1879
+ }
1880
+
1881
+ this.node.logger.info('Traffic shift complete', { labId });
1882
+ }
1883
+
1884
+ private async notifyLabAdmin(lab: LabInfo, message: string): Promise<void> {
1885
+ // Send email notification
1886
+ // Send Slack notification
1887
+ // Log to audit trail
1888
+ }
1889
+ }
1890
+ ```
1891
+
1892
+ **Deliverable**: Automatic failover with health verification โœ…
1893
+
1894
+ ---
1895
+
1896
+ ## Day 26-27: Replication API & Monitoring
1897
+
1898
+ ### Task 3.3: Replication API routes
1899
+
1900
+ **src/api/routes/replication.ts**:
1901
+ ```typescript
1902
+ import { FastifyInstance } from 'fastify';
1903
+ import { MasterNode } from '../../node/master-node';
1904
+
1905
+ export async function replicationRoutes(fastify: FastifyInstance, node: MasterNode) {
1906
+ // Trigger manual replication
1907
+ fastify.post('/api/v1/labs/:lab_id/replicate', {
1908
+ preHandler: [fastify.authenticate, fastify.requireLabAdmin]
1909
+ }, async (request, reply) => {
1910
+ const { lab_id } = request.params as { lab_id: string };
1911
+
1912
+ await node.triggerReplication(lab_id);
1913
+
1914
+ return {
1915
+ triggered: true,
1916
+ lab_id,
1917
+ message: 'Replication started. Check status at /api/v1/labs/:lab_id/replication/status'
1918
+ };
1919
+ });
1920
+
1921
+ // Get replication status
1922
+ fastify.get('/api/v1/labs/:lab_id/replication/status', async (request, reply) => {
1923
+ const { lab_id } = request.params as { lab_id: string };
1924
+
1925
+ const status = await node.getReplicationStatus(lab_id);
1926
+
1927
+ if (!status) {
1928
+ return reply.code(404).send({ error: 'No replication found for this lab' });
1929
+ }
1930
+
1931
+ return status;
1932
+ });
1933
+
1934
+ // Get failover status
1935
+ fastify.get('/api/v1/labs/:lab_id/failover/status', async (request, reply) => {
1936
+ const { lab_id } = request.params as { lab_id: string };
1937
+
1938
+ const failoverActive = node.isFailoverActive(lab_id);
1939
+
1940
+ return {
1941
+ lab_id,
1942
+ failover_active: failoverActive,
1943
+ failover_info: failoverActive ? await node.getFailoverInfo(lab_id) : null
1944
+ };
1945
+ });
1946
+
1947
+ // Manually activate failover (admin only)
1948
+ fastify.post('/api/v1/labs/:lab_id/failover/activate', {
1949
+ preHandler: [fastify.authenticate, fastify.requireMasterAdmin]
1950
+ }, async (request, reply) => {
1951
+ const { lab_id } = request.params as { lab_id: string };
1952
+
1953
+ await node.activateFailover(lab_id);
1954
+
1955
+ return {
1956
+ activated: true,
1957
+ lab_id,
1958
+ message: 'Failover activated. Master node now serving requests.'
1959
+ };
1960
+ });
1961
+
1962
+ // Deactivate failover (admin only)
1963
+ fastify.post('/api/v1/labs/:lab_id/failover/deactivate', {
1964
+ preHandler: [fastify.authenticate, fastify.requireMasterAdmin]
1965
+ }, async (request, reply) => {
1966
+ const { lab_id } = request.params as { lab_id: string };
1967
+
1968
+ await node.deactivateFailover(lab_id);
1969
+
1970
+ return {
1971
+ deactivated: true,
1972
+ lab_id,
1973
+ message: 'Failover deactivated. Lab node now serving requests.'
1974
+ };
1975
+ });
1976
+ }
1977
+ ```
1978
+
1979
+ ### Task 3.4: Monitoring dashboard
1980
+
1981
+ **src/monitoring/dashboard.html** (Grafana-compatible):
1982
+ ```json
1983
+ {
1984
+ "dashboard": {
1985
+ "title": "BioFS Network Health",
1986
+ "panels": [
1987
+ {
1988
+ "title": "Lab Node Status",
1989
+ "type": "stat",
1990
+ "targets": [
1991
+ {
1992
+ "expr": "biofs_lab_status{status='online'}",
1993
+ "legendFormat": "Online Labs"
1994
+ }
1995
+ ]
1996
+ },
1997
+ {
1998
+ "title": "Replication Lag (hours)",
1999
+ "type": "graph",
2000
+ "targets": [
2001
+ {
2002
+ "expr": "biofs_replication_lag_hours",
2003
+ "legendFormat": "{{ lab_id }}"
2004
+ }
2005
+ ]
2006
+ },
2007
+ {
2008
+ "title": "Failover Events",
2009
+ "type": "table",
2010
+ "targets": [
2011
+ {
2012
+ "expr": "increase(biofs_failover_activations_total[24h])"
2013
+ }
2014
+ ]
2015
+ },
2016
+ {
2017
+ "title": "Data Transferred (GB)",
2018
+ "type": "graph",
2019
+ "targets": [
2020
+ {
2021
+ "expr": "sum(rate(biofs_replication_bytes_transferred[5m])) by (lab_id) / 1e9",
2022
+ "legendFormat": "{{ lab_id }}"
2023
+ }
2024
+ ]
2025
+ }
2026
+ ]
2027
+ }
2028
+ }
2029
+ ```
2030
+
2031
+ **Deliverable**: Complete replication monitoring โœ…
2032
+
2033
+ ---
2034
+
2035
+ ## Phase 3 Deliverables Summary
2036
+
2037
+ ### โœ… What's Working
2038
+ - [x] S3 replication from lab to master (hourly)
2039
+ - [x] Incremental sync (only new/modified files)
2040
+ - [x] Automatic failover activation (<2 minutes)
2041
+ - [x] Health verification (5 consecutive checks)
2042
+ - [x] Gradual traffic shifting (20% increments)
2043
+ - [x] Replication status API
2044
+ - [x] Failover control API
2045
+ - [x] Prometheus metrics export
2046
+ - [x] Grafana dashboard
2047
+
2048
+ ### ๐Ÿ“Š Metrics Tracked
2049
+ - Replication lag (hours)
2050
+ - Files synced per lab
2051
+ - Bytes transferred
2052
+ - Failover activations
2053
+ - Traffic routing percentages
2054
+ - Lab health scores
2055
+
2056
+ ### ๐Ÿงช Testing Checklist
2057
+ ```bash
2058
+ # Trigger replication
2059
+ curl -X POST http://localhost:8080/api/v1/labs/augenomics/replicate \
2060
+ -H "Authorization: Bearer ${ADMIN_JWT}"
2061
+
2062
+ # Check status
2063
+ curl http://localhost:8080/api/v1/labs/augenomics/replication/status
2064
+
2065
+ # Simulate lab failure
2066
+ biofs-node stop # On lab node
2067
+
2068
+ # Wait 3 minutes, check failover
2069
+ curl http://localhost:8080/api/v1/labs/augenomics/failover/status
2070
+
2071
+ # Verify master serving files
2072
+ curl http://localhost:8080/api/v1/files/genome1.vcf \
2073
+ -H "Authorization: Bearer ${JWT}"
2074
+
2075
+ # Restore lab
2076
+ biofs-node start # On lab node
2077
+
2078
+ # Deactivate failover
2079
+ curl -X POST http://localhost:8080/api/v1/labs/augenomics/failover/deactivate \
2080
+ -H "Authorization: Bearer ${ADMIN_JWT}"
2081
+ ```
2082
+
2083
+ **Phase 3 Success Criteria**: โœ… Failover completes in <2 minutes, no data loss
2084
+
2085
+ ---
2086
+
2087
+ ## ๐ŸŽฏ Complete Implementation Summary (Phases 1-3)
2088
+
2089
+ ### Timeline Achievement
2090
+ - **Week 1-2**: Core infrastructure โœ…
2091
+ - **Week 3**: Sponsored services โœ…
2092
+ - **Week 4**: Data replication โœ…
2093
+ - **Total**: 4 weeks
2094
+
2095
+ ### Packages Delivered
2096
+ 1. `@genobank/biofs-core@1.0.0` - Shared library
2097
+ 2. `@genobank/biofs@1.6.2` - Client CLI (existing, updated)
2098
+ 3. `@genobank/biofs-node@1.0.0` - Server daemon (NEW)
2099
+
2100
+ ### Features Implemented
2101
+ - โœ… Master node orchestration
2102
+ - โœ… Lab node registration
2103
+ - โœ… Health monitoring with auto-failover
2104
+ - โœ… Anthropic API sponsorship
2105
+ - โœ… Story Protocol gas sponsorship
2106
+ - โœ… Quota management
2107
+ - โœ… S3 data replication
2108
+ - โœ… Automatic failover
2109
+ - โœ… REST API (50+ endpoints)
2110
+ - โœ… Prometheus metrics
2111
+ - โœ… CLI management tools
2112
+
2113
+ ### Production Readiness Checklist
2114
+ - [x] TypeScript with strict mode
2115
+ - [x] Comprehensive error handling
2116
+ - [x] Structured logging (Pino)
2117
+ - [x] Metrics export (Prometheus)
2118
+ - [x] Health endpoints
2119
+ - [x] Database migrations
2120
+ - [x] Configuration validation (Joi)
2121
+ - [x] JWT authentication
2122
+ - [x] Rate limiting
2123
+ - [x] CORS configuration
2124
+ - [x] Systemd service files
2125
+ - [x] Docker containerization
2126
+ - [x] Documentation
2127
+ - [x] API specifications (OpenAPI)
2128
+ - [x] Deployment guide
2129
+
2130
+ ---
2131
+
2132
+ ## ๐Ÿš€ Next Steps (Post-Phase 3)
2133
+
2134
+ ### Phase 4: Advanced Features (Optional, Week 5-6)
2135
+ - [ ] Cross-node license verification
2136
+ - [ ] Researcher roaming (SSO)
2137
+ - [ ] Unified audit trail
2138
+ - [ ] Billing invoice generation
2139
+ - [ ] Node discovery protocol
2140
+ - [ ] Multi-region replicas
2141
+ - [ ] Geographic routing
2142
+
2143
+ ### Immediate Deployment
2144
+ ```bash
2145
+ # 1. Setup master node
2146
+ git clone https://github.com/genobank/biofs-node.git
2147
+ cd biofs-node
2148
+ npm install
2149
+ npm run build
2150
+
2151
+ # 2. Configure
2152
+ cp config/master-node.example.yaml /etc/biofs/master-node.yaml
2153
+ # Edit with your settings
2154
+
2155
+ # 3. Start
2156
+ biofs-node start --config /etc/biofs/master-node.yaml --daemon
2157
+
2158
+ # 4. Verify
2159
+ curl http://localhost:8080/health
2160
+ ```
2161
+
2162
+ ---
2163
+
2164
+ **Ready to start implementation? Which phase should we begin with?**