bulltrackers-module 1.0.737 → 1.0.739

package/functions/computation-system-v2/docs/admin.md

@@ -0,0 +1,91 @@
+# Admin Test Endpoint
+
+## Deploy
+
+```bash
+node deploy.mjs ComputeAdminTest
+```
+
+## Usage Examples
+
+### 1. Check System Status
+
+```bash
+TOKEN=$(gcloud auth print-identity-token --audiences="https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test")
+
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "status"}'
+```
+
+### 2. Analyze What Would Run
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "analyze", "date": "2026-01-25"}'
+```
+
+### 3. Run Full Computation
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "run", "computation": "UserPortfolioSummary", "date": "2026-01-25", "force": true}'
+```
+
+### 4. Run Limited Test
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "run_limited", "computation": "UserPortfolioSummary", "date": "2026-01-25", "limit": 5}'
+```
+
+### 5. Test Specific Entities
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "run", "computation": "UserPortfolioSummary", "date": "2026-01-25", "entityIds": ["user-123", "user-456"], "force": true}'
+```
+
+### 6. Test Worker Directly
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "test_worker", "computation": "UserPortfolioSummary", "date": "2026-01-25", "entityIds": ["user-123"]}'
+```
+
+### 7. Test with Worker Pool Override
+
+```bash
+curl -X POST \
+    "https://europe-west1-stocks-12345.cloudfunctions.net/compute-admin-test" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "run", "computation": "UserPortfolioSummary", "date": "2026-01-25", "useWorkerPool": true, "force": true}'
+```
+
+## Available Actions
+
+| Action | Description |
+|--------|-------------|
+| `status` | List all computations and system status |
+| `analyze` | Check what would run for a given date |
+| `run` | Execute a full computation |
+| `run_limited` | Execute on N random entities (safer for testing) |
+| `test_worker` | Direct test of worker function logic |

package/functions/computation-system-v2/docs/plans.md

@@ -0,0 +1,588 @@
+# Comprehensive Review & Improvement Suggestions
+
+## 🚨 **CRITICAL ISSUES**
+
+### 1. **Infinite Loop Risk in Conditional Dependencies**
+```javascript
+// framework/core/Manifest.js - Line ~50
+if (config.conditionalDependencies) {
+    for (const condDep of config.conditionalDependencies) {
+        const shouldLoad = condDep.condition({ date: dateStr, config: this.config });
+        if (shouldLoad) {
+            // NO CYCLE DETECTION for conditional deps!
+            const depStatus = dailyStatus.get(condDep.computation.toLowerCase());
+            if (depStatus) {
+                depResults[condDep.computation] = await this.stateRepository.getResult(dateStr, condDep.computation);
+            }
+        }
+    }
+}
+```
+
+**Problem**: Conditional dependencies bypass cycle detection in `Graph.detectCycle()`. If Comp A conditionally depends on Comp B, and Comp B conditionally depends on Comp A, you get an infinite loop.
+
+**Fix**:
+```javascript
+// In ManifestBuilder._processComputation()
+_processComputation(ComputationClass) {
+    const config = ComputationClass.getConfig();
+    
+    // Merge conditional deps into regular deps for graph analysis
+    const allDeps = [...(config.dependencies || [])];
+    if (config.conditionalDependencies) {
+        allDeps.push(...config.conditionalDependencies.map(cd => cd.computation));
+    }
+    
+    return {
+        // ...
+        dependencies: allDeps.map(d => this._normalize(d)),
+        conditionalDependencies: config.conditionalDependencies || [],
+        // ...
+    };
+}
+```
+
+### 2. **Cost Spiral in Worker Pool Retries**
+```javascript
+// framework/execution/RemoteTaskRunner.js - Line ~200
+async _invokeWorkerWithRetry(payload, attempt = 1) {
+    try {
+        return await this._invokeWorker(payload);
+    } catch (e) {
+        if (isRetryable && attempt < this.retries) {
+            // NO COST TRACKING ON RETRIES!
+            return this._invokeWorkerWithRetry(payload, attempt + 1);
+        }
+        throw e;
+    }
+}
+```
+
+**Problem**: Failed workers retry without tracking costs. If 1000 entities fail and each retries 2x, you pay 3x the expected cost with no warning.
+
+**Fix**: Add cost multiplier tracking:
+```javascript
+class RemoteTaskRunner {
+    constructor(config, logger) {
+        this.costMultiplier = 1.0;
+        this.totalRetries = 0;
+    }
+    
+    async _invokeWorkerWithRetry(payload, attempt = 1) {
+        try {
+            return await this._invokeWorker(payload);
+        } catch (e) {
+            if (isRetryable && attempt < this.retries) {
+                this.totalRetries++;
+                this.costMultiplier = 1 + (this.totalRetries / this.stats.totalInvocations);
+                
+                if (this.costMultiplier > 2.0) {
+                    throw new Error(`🚨 RETRY COST SPIRAL: ${this.totalRetries} retries (${this.costMultiplier.toFixed(1)}x cost). Aborting.`);
+                }
+                
+                await new Promise(r => setTimeout(r, delay));
+                return this._invokeWorkerWithRetry(payload, attempt + 1);
+            }
+            throw e;
+        }
+    }
+}
+```
+
+### 3. **Runaway Query Cost in Historical Computations**
+```javascript
+// framework/data/DataFetcher.js - Line ~80
+async fetchForComputation(requires, targetDate, entities = null) {
+    await Promise.all(Object.entries(requires).map(async ([tableName, spec]) => {
+        const data = await this.fetch({
+            lookback: spec.lookback || 0,  // No upper limit!
+            // ...
+        });
+    }));
+}
+```
+
+**Problem**: A computation with `lookback: 365` on a 10TB table scans 365 days. If you have 100 computations with high lookbacks, costs explode.
+
+**Fix**: Add lookback budget:
+```javascript
+// In config
+execution: {
+    maxLookbackDays: 90,  // Hard limit
+    maxLookbackBudgetGB: 100,  // Cumulative budget per run
+}
+
+// In DataFetcher
+async fetch(options) {
+    if (options.lookback > this.config.execution.maxLookbackDays) {
+        throw new Error(`Lookback ${options.lookback} exceeds limit of ${this.config.execution.maxLookbackDays} days`);
+    }
+    
+    // Estimate scan size before executing
+    const estimatedGB = await this._estimateScanSize(options);
+    this.cumulativeScanGB += estimatedGB;
+    
+    if (this.cumulativeScanGB > this.config.execution.maxLookbackBudgetGB) {
+        throw new Error(`Query budget exceeded: ${this.cumulativeScanGB}GB / ${this.config.execution.maxLookbackBudgetGB}GB`);
+    }
+    
+    // Execute...
+}
+```
+
+---
+
+## ⚡ **PERFORMANCE BOTTLENECKS**
+
+### 4. **N+1 Dependency Fetches**
+```javascript
+// framework/execution/Orchestrator.js - Line ~450
+async _lazyLoadDependency(dateStr, depName, entityId, preloaded) {
+    if (preloaded[depName] && entityId) return preloaded[depName][entityId];
+    // FETCHES ONE ENTITY AT A TIME!
+    return this.stateRepository.getEntityResult(dateStr, depName, entityId);
+}
+```
+
+**Problem**: If a computation depends on `PopularInvestorProfileMetrics` and processes 10,000 users, this issues 10,000 individual BigQuery queries.
+
+**Fix**: Already partially addressed with `getBatchEntityResults()`, but enforce it:
+```javascript
+async _executeStreaming(entry, dateStr, depResults, previousResult, options) {
+    // ...
+    for await (const batch of batchStream) {
+        const { entityIds } = batch;
+        
+        // FORCE batch prefetch
+        const batchDeps = await this._prefetchBatchDependencies(entry, dateStr, depResults, entityIds);
+        
+        // Remove lazy load fallback to catch bugs
+        await Promise.all(entityIds.map(entityId => limit(async () => {
+            const context = {
+                getDependency: (depName, targetId) => {
+                    const id = targetId || entityId;
+                    if (!batchDeps[depName]?.has(id)) {
+                        throw new Error(`LAZY LOAD DETECTED: ${depName}:${id}. This is a bug - dependencies must be prefetched.`);
+                    }
+                    return batchDeps[depName].get(id);
+                },
+                // ...
+            };
+        })));
+    }
+}
+```
+
+### 5. **Redundant Schema Fetches**
+```javascript
+// framework/data/SchemaRegistry.js - Line ~70
+async getSchema(tableName) {
+    const cached = this.cache.get(tableName);
+    if (cached && !this._isExpired(cached)) {
+        return cached;  // Good!
+    }
+    // But if 100 concurrent requests arrive during cold start...
+    return await this._fetchAndCacheSchema(tableName);  // 100 concurrent fetches!
+}
+```
+
+**Fix**: Coalesce concurrent requests:
+```javascript
+constructor() {
+    this.cache = new Map();
+    this.pendingFetches = new Map();  // NEW
+}
+
+async getSchema(tableName) {
+    const cached = this.cache.get(tableName);
+    if (cached && !this._isExpired(cached)) return cached;
+    
+    // Coalesce concurrent requests
+    if (this.pendingFetches.has(tableName)) {
+        return this.pendingFetches.get(tableName);
+    }
+    
+    const fetchPromise = this._fetchAndCacheSchema(tableName);
+    this.pendingFetches.set(tableName, fetchPromise);
+    
+    try {
+        return await fetchPromise;
+    } finally {
+        this.pendingFetches.delete(tableName);
+    }
+}
+```
+
+### 6. **Inefficient Batch Splitting**
+```javascript
+// framework/data/DataFetcher.js - Line ~200
+async *fetchBatched(options, batchSize = 1000) {
+    let batch = [];
+    let currentEntity = null;
+
+    for await (const row of rowStream) {
+        if (entityField) {
+            const rowEntity = String(row[entityField]);
+            
+            // Splits batch even if at 999 rows (wastes 1 row)
+            if (batch.length >= batchSize && rowEntity !== currentEntity && currentEntity !== null) {
+                yield this._transform(batch, { lookback, dateField, entityField, dataField });
+                batch = [];
+            }
+            currentEntity = rowEntity;
+        }
+        batch.push(row);
+    }
+}
+```
+
+**Problem**: If an entity has 5,000 rows and `batchSize=1000`, you get 5 batches for one entity. If processing is slow, this serializes work.
+
+**Fix**: Add max batch size override:
+```javascript
+async *fetchBatched(options, batchSize = 1000) {
+    const maxBatchSize = batchSize * 10;  // Allow up to 10x for large entities
+    
+    for await (const row of rowStream) {
+        if (entityField) {
+            const rowEntity = String(row[entityField]);
+            
+            // Force split if exceeding max
+            if (batch.length >= maxBatchSize) {
+                yield this._transform(batch, ...);
+                batch = [];
+                currentEntity = null;
+            }
+            
+            // Normal split logic
+            if (batch.length >= batchSize && rowEntity !== currentEntity && currentEntity !== null) {
+                yield this._transform(batch, ...);
+                batch = [];
+            }
+            currentEntity = rowEntity;
+        }
+        batch.push(row);
+    }
+}
+```
+
+---
+
+## 🔒 **RELIABILITY ISSUES**
+
+### 7. **Zombie Detection Race Condition**
+```javascript
+// handlers/scheduler.js - Line ~80
+let zombies = [];
+try {
+    zombies = await storageManager.findZombies(ZOMBIE_THRESHOLD_MINUTES);
+    if (zombies.length > 0) {
+        console.log(`DETECTED ${zombies.length} ZOMBIES`);
+    }
+} catch (e) {
+    console.error(`Zombie check failed: ${e.message}`);
+}
+
+// Later: dispatch zombies
+const zombieEntries = zombies.map(z => {...});
+```
+
+**Problem**: If `findZombies()` throws, scheduler continues but `zombieEntries` is empty. Zombies never recover.
+
+**Fix**: Make zombie detection non-blocking:
+```javascript
+async function schedulerHandler(req, res) {
+    // 1. Find due computations (CRITICAL PATH)
+    const dueComputations = findDueComputations(now);
+    
+    // 2. Zombie detection (BEST EFFORT)
+    let zombies = [];
+    try {
+        zombies = await Promise.race([
+            storageManager.findZombies(ZOMBIE_THRESHOLD_MINUTES),
+            new Promise((_, reject) => setTimeout(() => reject(new Error('Timeout')), 5000))
+        ]);
+    } catch (e) {
+        // Log but don't block scheduler
+        console.warn(`⚠️ Zombie detection failed (non-fatal): ${e.message}`);
+    }
+    
+    // Continue with due + zombies...
+}
+```
+
+### 8. **Silent Firestore Write Failures**
+```javascript
+// framework/storage/StorageManager.js - Line ~650
+async _writeToFirestore(dateStr, entry, results, firestoreConfig) {
+    try {
+        // ... batching logic ...
+        return Promise.all(batches.map(b => limit(() => b.commit()))).then(() => ({ docCount }));
+    } catch (error) {
+        if (error.message.includes('Could not load the default credentials')) {
+            console.log('  Skipped: No GCP credentials');
+            return 'skipped';
+        }
+        console.log(`  Error: ${error.message}`);
+        return false;  // Returns false but doesn't throw!
+    }
+}
+```
+
+**Problem**: If Firestore writes fail, `commitResults()` returns `{ firestore: false }` but execution continues. Users think data is saved.
+
+**Fix**:
+```javascript
+async commitResults(dateStr, entry, results, depResultHashes = {}) {
+    const storageConfig = this._resolveStorageConfig(entry);
+    
+    const firestoreTask = async () => {
+        if (!storageConfig.firestore?.enabled) return null;
+        const result = await this._writeToFirestore(...);
+        
+        // ENFORCE: If enabled, must succeed
+        if (result === false || result === 'skipped') {
+            throw new Error(`Firestore write failed for ${entry.name} but was configured as enabled`);
+        }
+        return result;
+    };
+    
+    const [bigqueryResult, firestoreResult] = await Promise.all([gcsTask(), firestoreTask()]);
+    return { bigquery: bigqueryResult, firestore: firestoreResult };
+}
+```
+
+---
+
+## 🎯 **NEW FEATURE SUGGESTIONS**
+
+### 9. **Adaptive Batch Sizing**
+```javascript
+// NEW: framework/execution/AdaptiveBatcher.js
+class AdaptiveBatcher {
+    constructor(initialSize = 1000) {
+        this.currentSize = initialSize;
+        this.performanceWindow = [];
+    }
+    
+    recordBatch(entityCount, durationMs) {
+        const throughput = entityCount / (durationMs / 1000);  // entities/sec
+        this.performanceWindow.push({ size: entityCount, throughput });
+        
+        if (this.performanceWindow.length > 10) {
+            this.performanceWindow.shift();
+        }
+        
+        // Adjust batch size based on throughput trend
+        const avgThroughput = this.performanceWindow.reduce((sum, p) => sum + p.throughput, 0) / this.performanceWindow.length;
+        
+        if (avgThroughput < 10) {
+            // Slow processing - reduce batch size for parallelism
+            this.currentSize = Math.max(100, this.currentSize * 0.8);
+        } else if (avgThroughput > 100) {
+            // Fast processing - increase batch size
+            this.currentSize = Math.min(10000, this.currentSize * 1.2);
+        }
+    }
+    
+    getSize() {
+        return Math.round(this.currentSize);
+    }
+}
+
+// In Orchestrator
+this.adaptiveBatcher = new AdaptiveBatcher(config.execution.batchSize);
+
+for await (const batch of batchStream) {
+    const startTime = Date.now();
+    // ... process batch ...
+    this.adaptiveBatcher.recordBatch(batch.entityIds.length, Date.now() - startTime);
+    
+    // Use adaptive size for next batch
+    batchSize = this.adaptiveBatcher.getSize();
+}
+```
+
+### 10. **Query Plan Caching**
+```javascript
+// NEW: framework/data/QueryPlanCache.js
+class QueryPlanCache {
+    constructor(bigquery) {
+        this.cache = new Map();
+        this.bigquery = bigquery;
+    }
+    
+    async getEstimate(query) {
+        const queryHash = crypto.createHash('md5').update(query.sql).digest('hex');
+        
+        if (this.cache.has(queryHash)) {
+            return this.cache.get(queryHash);
+        }
+        
+        // Use BigQuery dry run for cost estimation
+        const [job] = await this.bigquery.createQueryJob({
+            query: query.sql,
+            params: query.params,
+            dryRun: true
+        });
+        
+        const [metadata] = await job.getMetadata();
+        const estimate = {
+            bytesProcessed: parseInt(metadata.statistics.totalBytesProcessed || 0, 10),
+            cacheHit: metadata.statistics.query.cacheHit,
+            estimatedCost: (parseInt(metadata.statistics.totalBytesProcessed || 0, 10) / 1099511627776) * 5
+        };
+        
+        this.cache.set(queryHash, estimate);
+        return estimate;
+    }
+}
+
+// In DataFetcher
+async _execute(query) {
+    const estimate = await this.queryPlanCache.getEstimate(query);
+    
+    if (estimate.estimatedCost > 1.0) {  // $1+ queries
+        this._log('WARN', `Expensive query detected: $${estimate.estimatedCost.toFixed(2)} - ${query.sql.substring(0, 100)}`);
+    }
+    
+    // Execute as normal...
+}
+```
+
+### 11. **Circuit Breaker for Failed Computations**
+```javascript
+// NEW: framework/resilience/CircuitBreaker.js
+class CircuitBreaker {
+    constructor(failureThreshold = 5, resetTimeout = 60000) {
+        this.failures = new Map();
+        this.failureThreshold = failureThreshold;
+        this.resetTimeout = resetTimeout;
+    }
+    
+    recordFailure(computationName) {
+        const state = this.failures.get(computationName) || { count: 0, lastFailure: null };
+        state.count++;
+        state.lastFailure = Date.now();
+        this.failures.set(computationName, state);
+        
+        if (state.count >= this.failureThreshold) {
+            console.warn(`🔴 Circuit breaker OPEN for ${computationName} (${state.count} failures)`);
+        }
+    }
+    
+    recordSuccess(computationName) {
+        this.failures.delete(computationName);
+    }
+    
+    shouldSkip(computationName) {
+        const state = this.failures.get(computationName);
+        if (!state || state.count < this.failureThreshold) return false;
+        
+        // Auto-reset after timeout
+        if (Date.now() - state.lastFailure > this.resetTimeout) {
+            console.log(`🟡 Circuit breaker HALF-OPEN for ${computationName} (attempting reset)`);
+            this.failures.delete(computationName);
+            return false;
+        }
+        
+        return true;
+    }
+}
+
+// In Orchestrator._executeComputation
+if (this.circuitBreaker.shouldSkip(entry.name)) {
+    return { name: entry.name, status: 'circuit_open', reason: 'Too many consecutive failures' };
+}
+
+try {
+    const result = await this._executeGlobal(...);
+    this.circuitBreaker.recordSuccess(entry.name);
+    return result;
+} catch (e) {
+    this.circuitBreaker.recordFailure(entry.name);
+    throw e;
+}
+```
+
+---
+
+## 📊 **MONITORING ENHANCEMENTS**
+
+### 12. **Real-Time Cost Dashboard**
+```javascript
+// NEW: Create a separate Cloud Function that queries cost tables
+exports.getCostDashboard = async (req, res) => {
+    const { startDate, endDate } = req.query;
+    
+    const query = `
+        SELECT 
+            computation_name,
+            SUM(estimated_cost_usd) as total_cost,
+            COUNT(*) as runs,
+            AVG(estimated_cost_usd) as avg_cost,
+            MAX(estimated_cost_usd) as max_cost
+        FROM \`${PROJECT}.${DATASET}.computation_costs\`
+        WHERE date BETWEEN @start AND @end
+        GROUP BY computation_name
+        ORDER BY total_cost DESC
+    `;
+    
+    const [rows] = await bigquery.query({ query, params: { start: startDate, end: endDate } });
+    
+    res.json({
+        period: { start: startDate, end: endDate },
+        totalCost: rows.reduce((sum, r) => sum + r.total_cost, 0),
+        computations: rows,
+        alert: rows.some(r => r.total_cost > 10) ? 'HIGH_COST_DETECTED' : null
+    });
+};
+```
+
+---
+
+## 🎛️ **CONFIGURATION IMPROVEMENTS**
+
+### 13. **Cost-Aware Scheduling**
+```json
+// In config/bulltrackers.config.js
+{
+  "computations": [
+    {
+      "class": "PopularInvestorRiskMetrics",
+      "schedule": {
+        "frequency": "daily",
+        "time": "02:00",
+        "costBudget": 5.0,  // NEW: Skip if daily cost exceeded
+        "priority": 1  // NEW: Lower priority runs only if budget allows
+      }
+    }
+  ],
+  "execution": {
+    "dailyCostBudget": 50,  // Global budget
+    "costCheckInterval": 3600000  // Re-check every hour
+  }
+}
+```
+
+---
+
+## Summary of Critical Fixes Needed
+
+| Issue | Severity | Fix Complexity | Impact |
+|-------|----------|----------------|--------|
+| Conditional dependency cycles | 🔴 CRITICAL | Low | Infinite loops |
+| Worker pool retry cost spiral | 🔴 CRITICAL | Medium | Cost overruns |
+| Unbounded lookback queries | 🔴 CRITICAL | Low | Query cost explosion |
+| N+1 dependency fetches | 🟡 HIGH | Medium | 100x slower execution |
+| Schema fetch stampede | 🟡 HIGH | Low | Cold start failures |
+| Silent Firestore failures | 🟡 HIGH | Low | Data loss |
+| Zombie recovery failures | 🟡 HIGH | Low | Stuck computations |
+
+**Immediate Actions:**
+1. Add conditional dependency cycle detection (1 hour fix)
+2. Add retry cost tracking in RemoteTaskRunner (2 hours)
+3. Add lookback budget limits (1 hour)
+4. Enforce batch dependency prefetching (already partially done, needs enforcement - 2 hours)
+5. Add schema fetch coalescing (1 hour)
+