bulltrackers-module 1.0.763 → 1.0.765

package/functions/computation-system-v2/docs/plans.md

@@ -1,588 +0,0 @@
-# Comprehensive Review & Improvement Suggestions
-
-## 🚨 **CRITICAL ISSUES**
-
-### 1. **Infinite Loop Risk in Conditional Dependencies**
-```javascript
-// framework/core/Manifest.js - Line ~50
-if (config.conditionalDependencies) {
-    for (const condDep of config.conditionalDependencies) {
-        const shouldLoad = condDep.condition({ date: dateStr, config: this.config });
-        if (shouldLoad) {
-            // NO CYCLE DETECTION for conditional deps!
-            const depStatus = dailyStatus.get(condDep.computation.toLowerCase());
-            if (depStatus) {
-                depResults[condDep.computation] = await this.stateRepository.getResult(dateStr, condDep.computation);
-            }
-        }
-    }
-}
-```
-
-**Problem**: Conditional dependencies bypass cycle detection in `Graph.detectCycle()`. If Comp A conditionally depends on Comp B, and Comp B conditionally depends on Comp A, you get an infinite loop.
-
-**Fix**:
-```javascript
-// In ManifestBuilder._processComputation()
-_processComputation(ComputationClass) {
-    const config = ComputationClass.getConfig();
-    
-    // Merge conditional deps into regular deps for graph analysis
-    const allDeps = [...(config.dependencies || [])];
-    if (config.conditionalDependencies) {
-        allDeps.push(...config.conditionalDependencies.map(cd => cd.computation));
-    }
-    
-    return {
-        // ...
-        dependencies: allDeps.map(d => this._normalize(d)),
-        conditionalDependencies: config.conditionalDependencies || [],
-        // ...
-    };
-}
-```
-
-### 2. **Cost Spiral in Worker Pool Retries**
-```javascript
-// framework/execution/RemoteTaskRunner.js - Line ~200
-async _invokeWorkerWithRetry(payload, attempt = 1) {
-    try {
-        return await this._invokeWorker(payload);
-    } catch (e) {
-        if (isRetryable && attempt < this.retries) {
-            // NO COST TRACKING ON RETRIES!
-            return this._invokeWorkerWithRetry(payload, attempt + 1);
-        }
-        throw e;
-    }
-}
-```
-
-**Problem**: Failed workers retry without tracking costs. If 1000 entities fail and each retries 2x, you pay 3x the expected cost with no warning.
-
-**Fix**: Add cost multiplier tracking:
-```javascript
-class RemoteTaskRunner {
-    constructor(config, logger) {
-        this.costMultiplier = 1.0;
-        this.totalRetries = 0;
-    }
-    
-    async _invokeWorkerWithRetry(payload, attempt = 1) {
-        try {
-            return await this._invokeWorker(payload);
-        } catch (e) {
-            if (isRetryable && attempt < this.retries) {
-                this.totalRetries++;
-                this.costMultiplier = 1 + (this.totalRetries / this.stats.totalInvocations);
-                
-                if (this.costMultiplier > 2.0) {
-                    throw new Error(`🚨 RETRY COST SPIRAL: ${this.totalRetries} retries (${this.costMultiplier.toFixed(1)}x cost). Aborting.`);
-                }
-                
-                await new Promise(r => setTimeout(r, delay));
-                return this._invokeWorkerWithRetry(payload, attempt + 1);
-            }
-            throw e;
-        }
-    }
-}
-```
-
-### 3. **Runaway Query Cost in Historical Computations**
-```javascript
-// framework/data/DataFetcher.js - Line ~80
-async fetchForComputation(requires, targetDate, entities = null) {
-    await Promise.all(Object.entries(requires).map(async ([tableName, spec]) => {
-        const data = await this.fetch({
-            lookback: spec.lookback || 0,  // No upper limit!
-            // ...
-        });
-    }));
-}
-```
-
-**Problem**: A computation with `lookback: 365` on a 10TB table scans 365 days. If you have 100 computations with high lookbacks, costs explode.
-
-**Fix**: Add lookback budget:
-```javascript
-// In config
-execution: {
-    maxLookbackDays: 90,  // Hard limit
-    maxLookbackBudgetGB: 100,  // Cumulative budget per run
-}
-
-// In DataFetcher
-async fetch(options) {
-    if (options.lookback > this.config.execution.maxLookbackDays) {
-        throw new Error(`Lookback ${options.lookback} exceeds limit of ${this.config.execution.maxLookbackDays} days`);
-    }
-    
-    // Estimate scan size before executing
-    const estimatedGB = await this._estimateScanSize(options);
-    this.cumulativeScanGB += estimatedGB;
-    
-    if (this.cumulativeScanGB > this.config.execution.maxLookbackBudgetGB) {
-        throw new Error(`Query budget exceeded: ${this.cumulativeScanGB}GB / ${this.config.execution.maxLookbackBudgetGB}GB`);
-    }
-    
-    // Execute...
-}
-```
-
----
-
-## ⚡ **PERFORMANCE BOTTLENECKS**
-
-### 4. **N+1 Dependency Fetches**
-```javascript
-// framework/execution/Orchestrator.js - Line ~450
-async _lazyLoadDependency(dateStr, depName, entityId, preloaded) {
-    if (preloaded[depName] && entityId) return preloaded[depName][entityId];
-    // FETCHES ONE ENTITY AT A TIME!
-    return this.stateRepository.getEntityResult(dateStr, depName, entityId);
-}
-```
-
-**Problem**: If a computation depends on `PopularInvestorProfileMetrics` and processes 10,000 users, this issues 10,000 individual BigQuery queries.
-
-**Fix**: Already partially addressed with `getBatchEntityResults()`, but enforce it:
-```javascript
-async _executeStreaming(entry, dateStr, depResults, previousResult, options) {
-    // ...
-    for await (const batch of batchStream) {
-        const { entityIds } = batch;
-        
-        // FORCE batch prefetch
-        const batchDeps = await this._prefetchBatchDependencies(entry, dateStr, depResults, entityIds);
-        
-        // Remove lazy load fallback to catch bugs
-        await Promise.all(entityIds.map(entityId => limit(async () => {
-            const context = {
-                getDependency: (depName, targetId) => {
-                    const id = targetId || entityId;
-                    if (!batchDeps[depName]?.has(id)) {
-                        throw new Error(`LAZY LOAD DETECTED: ${depName}:${id}. This is a bug - dependencies must be prefetched.`);
-                    }
-                    return batchDeps[depName].get(id);
-                },
-                // ...
-            };
-        })));
-    }
-}
-```
-
-### 5. **Redundant Schema Fetches**
-```javascript
-// framework/data/SchemaRegistry.js - Line ~70
-async getSchema(tableName) {
-    const cached = this.cache.get(tableName);
-    if (cached && !this._isExpired(cached)) {
-        return cached;  // Good!
-    }
-    // But if 100 concurrent requests arrive during cold start...
-    return await this._fetchAndCacheSchema(tableName);  // 100 concurrent fetches!
-}
-```
-
-**Fix**: Coalesce concurrent requests:
-```javascript
-constructor() {
-    this.cache = new Map();
-    this.pendingFetches = new Map();  // NEW
-}
-
-async getSchema(tableName) {
-    const cached = this.cache.get(tableName);
-    if (cached && !this._isExpired(cached)) return cached;
-    
-    // Coalesce concurrent requests
-    if (this.pendingFetches.has(tableName)) {
-        return this.pendingFetches.get(tableName);
-    }
-    
-    const fetchPromise = this._fetchAndCacheSchema(tableName);
-    this.pendingFetches.set(tableName, fetchPromise);
-    
-    try {
-        return await fetchPromise;
-    } finally {
-        this.pendingFetches.delete(tableName);
-    }
-}
-```
-
-### 6. **Inefficient Batch Splitting**
-```javascript
-// framework/data/DataFetcher.js - Line ~200
-async *fetchBatched(options, batchSize = 1000) {
-    let batch = [];
-    let currentEntity = null;
-
-    for await (const row of rowStream) {
-        if (entityField) {
-            const rowEntity = String(row[entityField]);
-            
-            // Splits batch even if at 999 rows (wastes 1 row)
-            if (batch.length >= batchSize && rowEntity !== currentEntity && currentEntity !== null) {
-                yield this._transform(batch, { lookback, dateField, entityField, dataField });
-                batch = [];
-            }
-            currentEntity = rowEntity;
-        }
-        batch.push(row);
-    }
-}
-```
-
-**Problem**: If an entity has 5,000 rows and `batchSize=1000`, you get 5 batches for one entity. If processing is slow, this serializes work.
-
-**Fix**: Add max batch size override:
-```javascript
-async *fetchBatched(options, batchSize = 1000) {
-    const maxBatchSize = batchSize * 10;  // Allow up to 10x for large entities
-    
-    for await (const row of rowStream) {
-        if (entityField) {
-            const rowEntity = String(row[entityField]);
-            
-            // Force split if exceeding max
-            if (batch.length >= maxBatchSize) {
-                yield this._transform(batch, ...);
-                batch = [];
-                currentEntity = null;
-            }
-            
-            // Normal split logic
-            if (batch.length >= batchSize && rowEntity !== currentEntity && currentEntity !== null) {
-                yield this._transform(batch, ...);
-                batch = [];
-            }
-            currentEntity = rowEntity;
-        }
-        batch.push(row);
-    }
-}
-```
-
----
-
-## 🔒 **RELIABILITY ISSUES**
-
-### 7. **Zombie Detection Race Condition**
-```javascript
-// handlers/scheduler.js - Line ~80
-let zombies = [];
-try {
-    zombies = await storageManager.findZombies(ZOMBIE_THRESHOLD_MINUTES);
-    if (zombies.length > 0) {
-        console.log(`DETECTED ${zombies.length} ZOMBIES`);
-    }
-} catch (e) {
-    console.error(`Zombie check failed: ${e.message}`);
-}
-
-// Later: dispatch zombies
-const zombieEntries = zombies.map(z => {...});
-```
-
-**Problem**: If `findZombies()` throws, scheduler continues but `zombieEntries` is empty. Zombies never recover.
-
-**Fix**: Make zombie detection non-blocking:
-```javascript
-async function schedulerHandler(req, res) {
-    // 1. Find due computations (CRITICAL PATH)
-    const dueComputations = findDueComputations(now);
-    
-    // 2. Zombie detection (BEST EFFORT)
-    let zombies = [];
-    try {
-        zombies = await Promise.race([
-            storageManager.findZombies(ZOMBIE_THRESHOLD_MINUTES),
-            new Promise((_, reject) => setTimeout(() => reject(new Error('Timeout')), 5000))
-        ]);
-    } catch (e) {
-        // Log but don't block scheduler
-        console.warn(`⚠️ Zombie detection failed (non-fatal): ${e.message}`);
-    }
-    
-    // Continue with due + zombies...
-}
-```
-
-### 8. **Silent Firestore Write Failures**
-```javascript
-// framework/storage/StorageManager.js - Line ~650
-async _writeToFirestore(dateStr, entry, results, firestoreConfig) {
-    try {
-        // ... batching logic ...
-        return Promise.all(batches.map(b => limit(() => b.commit()))).then(() => ({ docCount }));
-    } catch (error) {
-        if (error.message.includes('Could not load the default credentials')) {
-            console.log('  Skipped: No GCP credentials');
-            return 'skipped';
-        }
-        console.log(`  Error: ${error.message}`);
-        return false;  // Returns false but doesn't throw!
-    }
-}
-```
-
-**Problem**: If Firestore writes fail, `commitResults()` returns `{ firestore: false }` but execution continues. Users think data is saved.
-
-**Fix**:
-```javascript
-async commitResults(dateStr, entry, results, depResultHashes = {}) {
-    const storageConfig = this._resolveStorageConfig(entry);
-    
-    const firestoreTask = async () => {
-        if (!storageConfig.firestore?.enabled) return null;
-        const result = await this._writeToFirestore(...);
-        
-        // ENFORCE: If enabled, must succeed
-        if (result === false || result === 'skipped') {
-            throw new Error(`Firestore write failed for ${entry.name} but was configured as enabled`);
-        }
-        return result;
-    };
-    
-    const [bigqueryResult, firestoreResult] = await Promise.all([gcsTask(), firestoreTask()]);
-    return { bigquery: bigqueryResult, firestore: firestoreResult };
-}
-```
-
----
-
-## 🎯 **NEW FEATURE SUGGESTIONS**
-
-### 9. **Adaptive Batch Sizing**
-```javascript
-// NEW: framework/execution/AdaptiveBatcher.js
-class AdaptiveBatcher {
-    constructor(initialSize = 1000) {
-        this.currentSize = initialSize;
-        this.performanceWindow = [];
-    }
-    
-    recordBatch(entityCount, durationMs) {
-        const throughput = entityCount / (durationMs / 1000);  // entities/sec
-        this.performanceWindow.push({ size: entityCount, throughput });
-        
-        if (this.performanceWindow.length > 10) {
-            this.performanceWindow.shift();
-        }
-        
-        // Adjust batch size based on throughput trend
-        const avgThroughput = this.performanceWindow.reduce((sum, p) => sum + p.throughput, 0) / this.performanceWindow.length;
-        
-        if (avgThroughput < 10) {
-            // Slow processing - reduce batch size for parallelism
-            this.currentSize = Math.max(100, this.currentSize * 0.8);
-        } else if (avgThroughput > 100) {
-            // Fast processing - increase batch size
-            this.currentSize = Math.min(10000, this.currentSize * 1.2);
-        }
-    }
-    
-    getSize() {
-        return Math.round(this.currentSize);
-    }
-}
-
-// In Orchestrator
-this.adaptiveBatcher = new AdaptiveBatcher(config.execution.batchSize);
-
-for await (const batch of batchStream) {
-    const startTime = Date.now();
-    // ... process batch ...
-    this.adaptiveBatcher.recordBatch(batch.entityIds.length, Date.now() - startTime);
-    
-    // Use adaptive size for next batch
-    batchSize = this.adaptiveBatcher.getSize();
-}
-```
-
-### 10. **Query Plan Caching**
-```javascript
-// NEW: framework/data/QueryPlanCache.js
-class QueryPlanCache {
-    constructor(bigquery) {
-        this.cache = new Map();
-        this.bigquery = bigquery;
-    }
-    
-    async getEstimate(query) {
-        const queryHash = crypto.createHash('md5').update(query.sql).digest('hex');
-        
-        if (this.cache.has(queryHash)) {
-            return this.cache.get(queryHash);
-        }
-        
-        // Use BigQuery dry run for cost estimation
-        const [job] = await this.bigquery.createQueryJob({
-            query: query.sql,
-            params: query.params,
-            dryRun: true
-        });
-        
-        const [metadata] = await job.getMetadata();
-        const estimate = {
-            bytesProcessed: parseInt(metadata.statistics.totalBytesProcessed || 0, 10),
-            cacheHit: metadata.statistics.query.cacheHit,
-            estimatedCost: (parseInt(metadata.statistics.totalBytesProcessed || 0, 10) / 1099511627776) * 5
-        };
-        
-        this.cache.set(queryHash, estimate);
-        return estimate;
-    }
-}
-
-// In DataFetcher
-async _execute(query) {
-    const estimate = await this.queryPlanCache.getEstimate(query);
-    
-    if (estimate.estimatedCost > 1.0) {  // $1+ queries
-        this._log('WARN', `Expensive query detected: $${estimate.estimatedCost.toFixed(2)} - ${query.sql.substring(0, 100)}`);
-    }
-    
-    // Execute as normal...
-}
-```
-
-### 11. **Circuit Breaker for Failed Computations**
-```javascript
-// NEW: framework/resilience/CircuitBreaker.js
-class CircuitBreaker {
-    constructor(failureThreshold = 5, resetTimeout = 60000) {
-        this.failures = new Map();
-        this.failureThreshold = failureThreshold;
-        this.resetTimeout = resetTimeout;
-    }
-    
-    recordFailure(computationName) {
-        const state = this.failures.get(computationName) || { count: 0, lastFailure: null };
-        state.count++;
-        state.lastFailure = Date.now();
-        this.failures.set(computationName, state);
-        
-        if (state.count >= this.failureThreshold) {
-            console.warn(`🔴 Circuit breaker OPEN for ${computationName} (${state.count} failures)`);
-        }
-    }
-    
-    recordSuccess(computationName) {
-        this.failures.delete(computationName);
-    }
-    
-    shouldSkip(computationName) {
-        const state = this.failures.get(computationName);
-        if (!state || state.count < this.failureThreshold) return false;
-        
-        // Auto-reset after timeout
-        if (Date.now() - state.lastFailure > this.resetTimeout) {
-            console.log(`🟡 Circuit breaker HALF-OPEN for ${computationName} (attempting reset)`);
-            this.failures.delete(computationName);
-            return false;
-        }
-        
-        return true;
-    }
-}
-
-// In Orchestrator._executeComputation
-if (this.circuitBreaker.shouldSkip(entry.name)) {
-    return { name: entry.name, status: 'circuit_open', reason: 'Too many consecutive failures' };
-}
-
-try {
-    const result = await this._executeGlobal(...);
-    this.circuitBreaker.recordSuccess(entry.name);
-    return result;
-} catch (e) {
-    this.circuitBreaker.recordFailure(entry.name);
-    throw e;
-}
-```
-
----
-
-## 📊 **MONITORING ENHANCEMENTS**
-
-### 12. **Real-Time Cost Dashboard**
-```javascript
-// NEW: Create a separate Cloud Function that queries cost tables
-exports.getCostDashboard = async (req, res) => {
-    const { startDate, endDate } = req.query;
-    
-    const query = `
-        SELECT 
-            computation_name,
-            SUM(estimated_cost_usd) as total_cost,
-            COUNT(*) as runs,
-            AVG(estimated_cost_usd) as avg_cost,
-            MAX(estimated_cost_usd) as max_cost
-        FROM \`${PROJECT}.${DATASET}.computation_costs\`
-        WHERE date BETWEEN @start AND @end
-        GROUP BY computation_name
-        ORDER BY total_cost DESC
-    `;
-    
-    const [rows] = await bigquery.query({ query, params: { start: startDate, end: endDate } });
-    
-    res.json({
-        period: { start: startDate, end: endDate },
-        totalCost: rows.reduce((sum, r) => sum + r.total_cost, 0),
-        computations: rows,
-        alert: rows.some(r => r.total_cost > 10) ? 'HIGH_COST_DETECTED' : null
-    });
-};
-```
-
----
-
-## 🎛️ **CONFIGURATION IMPROVEMENTS**
-
-### 13. **Cost-Aware Scheduling**
-```json
-// In config/bulltrackers.config.js
-{
-  "computations": [
-    {
-      "class": "PopularInvestorRiskMetrics",
-      "schedule": {
-        "frequency": "daily",
-        "time": "02:00",
-        "costBudget": 5.0,  // NEW: Skip if daily cost exceeded
-        "priority": 1  // NEW: Lower priority runs only if budget allows
-      }
-    }
-  ],
-  "execution": {
-    "dailyCostBudget": 50,  // Global budget
-    "costCheckInterval": 3600000  // Re-check every hour
-  }
-}
-```
-
----
-
-## Summary of Critical Fixes Needed
-
-| Issue | Severity | Fix Complexity | Impact |
-|-------|----------|----------------|--------|
-| Conditional dependency cycles | 🔴 CRITICAL | Low | Infinite loops |
-| Worker pool retry cost spiral | 🔴 CRITICAL | Medium | Cost overruns |
-| Unbounded lookback queries | 🔴 CRITICAL | Low | Query cost explosion |
-| N+1 dependency fetches | 🟡 HIGH | Medium | 100x slower execution |
-| Schema fetch stampede | 🟡 HIGH | Low | Cold start failures |
-| Silent Firestore failures | 🟡 HIGH | Low | Data loss |
-| Zombie recovery failures | 🟡 HIGH | Low | Stuck computations |
-
-**Immediate Actions:**
-1. Add conditional dependency cycle detection (1 hour fix)
-2. Add retry cost tracking in RemoteTaskRunner (2 hours)
-3. Add lookback budget limits (1 hour)
-4. Enforce batch dependency prefetching (already partially done, needs enforcement - 2 hours)
-5. Add schema fetch coalescing (1 hour)
-