bulltrackers-module 1.0.175 → 1.0.176

package/README.MD

@@ -1,3 +1,1902 @@
-This is the directory for the cloud function entry points in the Bulltrackers Platform
+# BullTrackers Platform - Comprehensive Developer Documentation
 
-Please refer to BullTrackers\Backend\documentation for the full documentation
+## Table of Contents
+
+1. [System Overview](#1-system-overview)
+2. [Architecture & Design Patterns](#2-architecture--design-patterns)
+3. [Core Infrastructure](#3-core-infrastructure)
+4. [Module Reference](#4-module-reference)
+5. [Data Flow & Pipelines](#5-data-flow--pipelines)
+6. [Computation System](#6-computation-system)
+7. [Development Guidelines](#7-development-guidelines)
+8. [Deployment & Operations](#8-deployment--operations)
+
+---
+
+## 1. System Overview
+
+### 1.1 What is BullTrackers?
+
+BullTrackers is a sophisticated financial analytics platform that monitors and analyzes eToro trading portfolios at scale. The system processes millions of user portfolios daily, computing real-time market sentiment signals, risk metrics, and behavioral analytics.
+
+### 1.2 Key Capabilities
+
+- **User Discovery & Monitoring**: Automatically discovers and tracks ~5M+ eToro users
+- **Portfolio Tracking**: Stores daily snapshots of user portfolios (both normal users and high-leverage "speculators")
+- **Trading History**: Captures complete trade history for behavioral analysis
+- **Real-Time Computations**: Runs 100+ mathematical computations daily to generate market insights
+- **Social Sentiment Analysis**: Tracks and analyzes social posts from eToro's platform using AI
+- **Public API**: Exposes computed insights via REST API with schema introspection
+
+### 1.3 Technology Stack
+
+- **Runtime**: Node.js 20+ (Google Cloud Functions Gen 2)
+- **Database**: Google Firestore (NoSQL document store)
+- **Messaging**: Google Cloud Pub/Sub
+- **AI/ML**: Google Gemini API (sentiment analysis)
+- **Proxy Infrastructure**: Google Apps Script (rate limit mitigation)
+- **Package Management**: NPM private packages
+
+---
+
+## 2. Architecture & Design Patterns
+
+### 2.1 Modular Pipe Architecture
+
+The entire codebase follows a **"pipe" pattern** where each module exports stateless functions that receive all dependencies explicitly:
+
+```javascript
+// Example pipe signature
+async function handleRequest(message, context, config, dependencies) {
+    const { logger, db, pubsub } = dependencies;
+    // Function logic here
+}
+```
+
+**Key Benefits**:
+- **Testability**: Easy to mock dependencies
+- **Reusability**: Functions are pure and composable
+- **Maintainability**: Clear separation of concerns
+
+### 2.2 Dependency Injection Pattern
+
+All modules receive dependencies via a `dependencies` object:
+
+```javascript
+const dependencies = {
+    db: firestoreInstance,
+    pubsub: pubsubClient,
+    logger: loggerInstance,
+    headerManager: intelligentHeaderManager,
+    proxyManager: intelligentProxyManager,
+    batchManager: firestoreBatchManager,
+    calculationUtils: { withRetry, loadInstrumentMappings },
+    firestoreUtils: { getBlockCapacities, ... },
+    pubsubUtils: { batchPublishTasks }
+};
+```
+
+### 2.3 Configuration Objects
+
+Each module has its own configuration namespace:
+
+```javascript
+const config = {
+    // Task Engine Config
+    ETORO_API_PORTFOLIO_URL: "https://...",
+    TASK_ENGINE_MAX_USERS_PER_SHARD: 500,
+    
+    // Orchestrator Config
+    discoveryConfig: { normal: {...}, speculator: {...} },
+    updateConfig: { ... },
+    
+    // Computation System Config
+    COMPUTATION_PASS_TO_RUN: "1",
+    resultsCollection: "unified_insights"
+};
+```
+
+### 2.4 Sharded Data Storage
+
+To handle massive scale, user data is **sharded** across multiple Firestore documents:
+
+```
+Collection: NormalUserPortfolios
+├── Block: 0M (users 0-999,999)
+│   └── snapshots/
+│       └── 2025-01-15/
+│           └── parts/
+│               ├── part_0 (users 0-499)
+│               ├── part_1 (users 500-999)
+│               └── ...
+├── Block: 1M (users 1,000,000-1,999,999)
+└── ...
+```
+
+**Why Sharding?**
+- Firestore document size limit: 1MB
+- Each "part" contains ~500 users
+- Enables parallel processing
+
+### 2.5 Batch Processing Pattern
+
+All writes use a **FirestoreBatchManager** to aggregate operations and minimize API calls:
+
+```javascript
+// Instead of 1000 individual writes:
+await batchManager.addToPortfolioBatch(userId, blockId, date, data, userType);
+// ... (accumulates in memory)
+await batchManager.flushBatches(); // Commits in bulk
+```
+
+---
+
+## 3. Core Infrastructure
+
+### 3.1 IntelligentProxyManager
+
+**Purpose**: Manages a pool of Google Apps Script proxies to bypass eToro rate limits.
+
+**Key Features**:
+- **Automatic Proxy Rotation**: Selects available (unlocked) proxies
+- **Rate Limit Detection**: Identifies HTML error pages (DOCTYPE checks)
+- **Exponential Backoff**: Retries failed requests with different proxies
+- **Locking Mechanism**: Temporarily locks proxies that hit rate limits
+
+**Usage**:
+```javascript
+const response = await proxyManager.fetch(url, options);
+// Handles: proxy selection, retries, rate limit detection
+```
+
+**Critical Configuration**:
+```javascript
+proxyUrls: [
+    "https://script.google.com/macros/s/.../exec",
+    // 10-20 different proxy URLs
+],
+proxyLockingEnabled: true,
+PERFORMANCE_DOC_PATH: "system_state/proxy_performance"
+```
+
+### 3.2 IntelligentHeaderManager
+
+**Purpose**: Manages browser headers to mimic real user traffic.
+
+**Key Features**:
+- **Header Rotation**: Uses different User-Agent strings
+- **Performance Tracking**: Records success rates per header
+- **Weighted Selection**: Prefers high-success headers
+- **Firestore Sync**: Persists performance data
+
+**Usage**:
+```javascript
+const { id, header } = await headerManager.selectHeader();
+// Use header in request
+headerManager.updatePerformance(id, wasSuccessful);
+await headerManager.flushPerformanceUpdates();
+```
+
+### 3.3 FirestoreBatchManager
+
+**Purpose**: Aggregates Firestore writes to minimize costs and improve performance.
+
+**Features**:
+- **Portfolio Batching**: Groups user portfolio updates
+- **Trading History Batching**: Aggregates trade history
+- **Timestamp Management**: Tracks last update times
+- **Username Map Caching**: Reduces redundant lookups
+- **10-Minute History Cache**: Prevents duplicate fetches
+
+**Critical Methods**:
+```javascript
+// Add data (accumulates in memory)
+await batchManager.addToPortfolioBatch(userId, blockId, date, data, userType);
+await batchManager.addToTradingHistoryBatch(userId, blockId, date, historyData, userType);
+await batchManager.updateUserTimestamp(userId, userType, instrumentId);
+
+// Commit everything at once
+await batchManager.flushBatches();
+```
+
+**Auto-Flush Logic**:
+```javascript
+_scheduleFlush() {
+    const totalOps = this._estimateBatchSize();
+    if (totalOps >= 400) { 
+        this.flushBatches(); // Immediate flush if near limit
+        return; 
+    }
+    // Otherwise, schedule delayed flush
+    if (!this.batchTimeout) { 
+        this.batchTimeout = setTimeout(
+            () => this.flushBatches(), 
+            FLUSH_INTERVAL_MS
+        ); 
+    }
+}
+```
+
+---
+
+## 4. Module Reference
+
+### 4.1 Orchestrator Module
+
+**Entry Point**: `functions/orchestrator/index.js`
+
+**Purpose**: High-level task coordination (discovery of new users, scheduling updates).
+
+#### 4.1.1 Discovery Orchestration
+
+**Function**: `runDiscoveryOrchestrator(config, deps)`
+
+**Flow**:
+1. Reset proxy locks (clear stale locks from previous runs)
+2. Check if discovery is needed for normal users
+3. Check if discovery is needed for speculators
+4. Generate candidate CIDs (prioritized + random)
+5. Publish discovery tasks to Pub/Sub
+
+**Key Concepts**:
+- **Blocks**: Users are grouped into 1M blocks (e.g., 0M = users 0-999,999)
+- **Target Capacity**: Each block should have ~500 monitored users
+- **Prioritized Discovery**: For speculators, existing normal users holding speculator assets are checked first
+- **Random Discovery**: Random CIDs within blocks are tested
+
+#### 4.1.2 Update Orchestration
+
+**Function**: `runUpdateOrchestrator(config, deps)`
+
+**Flow**:
+1. Reset proxy locks
+2. Query timestamp documents for stale users (normal users: updated >24h ago)
+3. Query speculator blocks for stale users (verified >24h ago AND held assets recently)
+4. Publish update tasks to dispatcher
+
+**Thresholds**:
+```javascript
+const thresholds = {
+    dateThreshold: startOfTodayUTC, // Users updated before today
+    gracePeriodThreshold: 30DaysAgoUTC // Speculators who held assets in last 30 days
+};
+```
+
+### 4.2 Dispatcher Module
+
+**Entry Point**: `functions/dispatcher/index.js`
+
+**Purpose**: Rate-limit task submission to prevent GCP scaling issues.
+
+**Why Needed?**
+- Orchestrator generates 10,000+ tasks instantly
+- Directly publishing causes:
+  - GCP quota errors
+  - Firestore rate limit issues
+  - Cost spikes
+
+**Solution**: Dispatcher batches tasks and publishes with delays:
+
+```javascript
+const config = {
+    batchSize: 50, // Tasks per Pub/Sub message
+    batchDelayMs: 100 // Delay between batches
+};
+```
+
+**Flow**:
+1. Receive batch of tasks from orchestrator
+2. Group into sub-batches of 50 tasks
+3. Publish 1 Pub/Sub message per sub-batch
+4. Sleep 100ms between publishes
+
+### 4.3 Task Engine Module
+
+**Entry Point**: `functions/task-engine/handler_creator.js`
+
+**Purpose**: Worker that executes individual user tasks (discover, verify, update).
+
+#### 4.3.1 Task Types
+
+##### Discover Task
+**Purpose**: Test if random CIDs are public & active users
+
+**Flow**:
+1. Receive batch of CIDs (e.g., [123456, 234567, ...])
+2. POST to eToro Rankings API (returns public users only)
+3. Filter for active users (traded in last 30 days, non-zero exposure)
+4. Apply speculator heuristic (if userType='speculator'):
+   ```javascript
+   const isLikelySpeculator = (
+       (trades > 500) || 
+       (totalTradedInstruments > 50) || 
+       (mediumLeveragePct + highLeveragePct > 50) || 
+       (weeklyDrawdown < -25)
+   );
+   ```
+5. Chain to 'verify' task for active users
+
+**Output**: Publishes 'verify' task with usernames
+
+##### Verify Task
+**Purpose**: Fetch full portfolios to confirm user type
+
+**Flow**:
+1. Receive users from discover task
+2. Sequentially fetch each user's portfolio (to avoid rate limits)
+3. For speculators: Check if they hold target instruments
+4. Write to Firestore:
+   - Block document (`SpeculatorBlocks/{blockId}` or `NormalUserPortfolios/{blockId}`)
+   - Bronze status document (if user is Bronze tier)
+   - Username map shards (for future lookups)
+   - Block count increments
+
+**Output**: Users stored in appropriate collections
+
+##### Update Task
+**Purpose**: Refresh existing user's portfolio & trade history
+
+**Flow**:
+1. Lookup username from cache (or fetch if missing)
+2. **History Fetch** (once per user per invocation):
+   ```javascript
+   if (!batchManager.checkAndSetHistoryFetched(userId)) {
+       // Fetch trade history from eToro
+       // Store in trading_history collection
+   }
+   ```
+3. **Portfolio Fetch** (per instrument for speculators):
+   ```javascript
+   // Normal user: 1 fetch (full portfolio)
+   // Speculator: N fetches (1 per instrument they hold)
+   ```
+4. Accumulate in batch manager
+5. Update timestamp documents
+
+**Critical**: All fetches are **sequential** (concurrency = 1) to avoid AppScript rate limits.
+
+#### 4.3.2 Fallback Mechanism
+
+All API calls use a **two-tier fallback**:
+
+```javascript
+try {
+    // Tier 1: AppScript proxy (preferred)
+    response = await proxyManager.fetch(url, options);
+} catch (proxyError) {
+    // Tier 2: Direct fetch via GCP IPs
+    response = await fetch(url, options);
+}
+```
+
+**Why?**
+- AppScript proxies have better rate limit tolerance
+- Direct fetch used as emergency fallback
+- Only AppScript failures penalize header performance
+
+### 4.4 Computation System Module
+
+**Entry Point**: `functions/computation-system/helpers/computation_pass_runner.js`
+
+**Purpose**: Executes mathematical computations on stored portfolio data.
+
+#### 4.4.1 Architecture
+
+**Manifest Builder** (`computation_manifest_builder.js`):
+- Introspects all calculation classes
+- Validates dependencies
+- Performs topological sort
+- Determines execution passes
+
+**Pass Runner** (`computation_pass_runner.js`):
+- Loads manifest for specific pass
+- Checks data availability
+- Streams portfolio data in chunks
+- Executes computations
+- Writes results to Firestore
+
+#### 4.4.2 Calculation Classes
+
+All calculations extend this pattern:
+
+```javascript
+class MyCalculation {
+    constructor() {
+        this.results = {};
+    }
+    
+    static getMetadata() {
+        return {
+            name: "my-calculation",
+            category: "core", // or 'gem', 'gauss', etc.
+            type: "standard", // or 'meta'
+            isHistorical: false, // true if needs yesterday's data
+            rootDataDependencies: ["portfolio"], // 'insights', 'social', 'history'
+            userType: "all" // 'normal', 'speculator', or 'all'
+        };
+    }
+    
+    static getDependencies() {
+        return ["other-calculation"]; // Names of required computations
+    }
+    
+    static getSchema() {
+        return {
+            "TICKER": {
+                "metric": 0.0,
+                "otherField": "string"
+            }
+        };
+    }
+    
+    async process(context) {
+        const { user, math, computed, previousComputed } = context;
+        
+        // Access current user's portfolio
+        const positions = math.extract.getPositions(
+            user.portfolio.today, 
+            user.type
+        );
+        
+        // Use math primitives
+        for (const pos of positions) {
+            const pnl = math.extract.getNetProfit(pos);
+            const ticker = context.mappings.instrumentToTicker[
+                math.extract.getInstrumentId(pos)
+            ];
+            
+            this.results[ticker] = { pnl };
+        }
+    }
+    
+    async getResult() {
+        return this.results;
+    }
+}
+```
+
+#### 4.4.3 Math Primitives Layer
+
+**File**: `functions/computation-system/layers/math_primitives.js`
+
+**Purpose**: Single source of truth for data extraction and mathematical operations.
+
+**Key Classes**:
+
+##### DataExtractor
+```javascript
+// Schema-agnostic position access
+const positions = DataExtractor.getPositions(portfolio, userType);
+const instrumentId = DataExtractor.getInstrumentId(position);
+const netProfit = DataExtractor.getNetProfit(position);
+const weight = DataExtractor.getPositionWeight(position, userType);
+
+// Speculator-specific
+const leverage = DataExtractor.getLeverage(position);
+const openRate = DataExtractor.getOpenRate(position);
+const stopLoss = DataExtractor.getStopLossRate(position);
+```
+
+##### MathPrimitives
+```javascript
+// Statistical functions
+const avg = MathPrimitives.average(values);
+const median = MathPrimitives.median(values);
+const stdDev = MathPrimitives.standardDeviation(values);
+
+// Risk calculations
+const hitProbability = MathPrimitives.calculateHitProbability(
+    currentPrice, 
+    stopLossPrice, 
+    volatility, 
+    daysAhead
+);
+
+// Monte Carlo simulation
+const futurePrices = MathPrimitives.simulateGBM(
+    currentPrice, 
+    volatility, 
+    days, 
+    simulations
+);
+```
+
+##### HistoryExtractor
+```javascript
+const history = HistoryExtractor.getDailyHistory(user);
+const assets = HistoryExtractor.getTradedAssets(history);
+const summary = HistoryExtractor.getSummary(history);
+```
+
+##### SignalPrimitives
+```javascript
+// Access dependency results
+const metric = SignalPrimitives.getMetric(
+    dependencies, 
+    "calc-name", 
+    "TICKER", 
+    "fieldName"
+);
+
+// Get previous state (for time-series)
+const prevValue = SignalPrimitives.getPreviousState(
+    previousComputed, 
+    "calc-name", 
+    "TICKER", 
+    "fieldName"
+);
+```
+
+#### 4.4.4 Execution Flow
+
+1. **Manifest Generation**:
+   ```javascript
+   const manifest = buildManifest(
+       ["core", "gem"], // Product lines to include
+       calculations // Imported calculation classes
+   );
+   ```
+
+2. **Pass Execution**:
+   ```javascript
+   await runComputationPass(config, dependencies, manifest);
+   ```
+
+3. **Data Streaming**:
+   ```javascript
+   // Stream portfolio data in chunks of 50 users
+   for await (const chunk of streamPortfolioData(...)) {
+       await Promise.all(calcs.map(c => 
+           controller.executor.executePerUser(c, metadata, dateStr, chunk, ...)
+       ));
+   }
+   ```
+
+4. **Result Storage**:
+   ```
+   Collection: unified_insights
+   └── Date: 2025-01-15
+       └── results/
+           └── Category: core
+               └── computations/
+                   └── calculation-name: { data }
+   ```
+
+5. **Status Tracking**:
+   ```javascript
+   // Global status document (NEW)
+   {
+       "2025-01-15": {
+           "calculation-a": true,
+           "calculation-b": false, // Failed or not run
+           ...
+       }
+   }
+   ```
+
+#### 4.4.5 Computation Passes
+
+**Why Passes?**
+- Some calculations depend on others
+- Must execute in dependency order
+- Passes enable parallel execution within each level
+
+**Example**:
+```
+Pass 1: [raw-data-extractors] (no dependencies)
+Pass 2: [aggregators] (depends on Pass 1)
+Pass 3: [signals] (depends on Pass 2)
+```
+
+**Configuration**:
+```javascript
+const config = {
+    COMPUTATION_PASS_TO_RUN: "1", // Set via environment variable
+    // Cloud Scheduler runs 3 separate functions:
+    // - computation-pass-1 (COMPUTATION_PASS_TO_RUN=1)
+    // - computation-pass-2 (COMPUTATION_PASS_TO_RUN=2)
+    // - computation-pass-3 (COMPUTATION_PASS_TO_RUN=3)
+};
+```
+
+### 4.5 Generic API Module
+
+**Entry Point**: `functions/generic-api/index.js`
+
+**Purpose**: REST API for accessing computed insights.
+
+#### 4.5.1 Endpoints
+
+##### GET /
+**Query Parameters**:
+- `computations`: Comma-separated list of calculation names
+- `startDate`: YYYY-MM-DD
+- `endDate`: YYYY-MM-DD
+
+**Example**:
+```
+GET /?computations=sentiment-score,momentum&startDate=2025-01-10&endDate=2025-01-15
+```
+
+**Response**:
+```json
+{
+    "status": "success",
+    "metadata": {
+        "computations": ["sentiment-score", "momentum"],
+        "startDate": "2025-01-10",
+        "endDate": "2025-01-15"
+    },
+    "data": {
+        "2025-01-10": {
+            "sentiment-score": { "AAPL": 0.75, ... },
+            "momentum": { "AAPL": 1.2, ... }
+        },
+        ...
+    }
+}
+```
+
+##### GET /manifest
+**Purpose**: List all available computations with schemas
+
+**Response**:
+```json
+{
+    "status": "success",
+    "summary": {
+        "totalComputations": 127,
+        "schemasAvailable": 127
+    },
+    "manifest": {
+        "calculation-name": {
+            "category": "core",
+            "structure": { "TICKER": { "field": 0 } },
+            "metadata": { ... },
+            "lastUpdated": "2025-01-15T..."
+        }
+    }
+}
+```
+
+##### GET /manifest/:computationName
+**Purpose**: Get detailed schema for specific computation
+
+##### POST /manifest/generate/:computationName
+**Purpose**: Manually trigger schema generation for a calculation
+
+##### GET /structure/:computationName
+**Purpose**: Get example data structure from latest stored result
+
+##### GET /list-computations
+**Purpose**: Simple list of all computation keys
+
+#### 4.5.2 Caching
+
+API responses are cached in-memory for 10 minutes:
+
+```javascript
+const CACHE = {};
+const CACHE_TTL_MS = 10 * 60 * 1000;
+
+// On cache HIT: Return from memory (no DB query)
+// On cache MISS: Query DB, cache response
+```
+
+### 4.6 Maintenance Modules
+
+#### 4.6.1 Speculator Cleanup
+
+**Purpose**: Remove inactive speculators to free up monitoring slots
+
+**Logic**:
+1. Query `PendingSpeculators` for users older than grace period (12 hours)
+2. Query `SpeculatorBlocks` for users who haven't held assets in 30 days
+3. Batch delete stale users
+4. Decrement block counts
+
+#### 4.6.2 Invalid Speculator Handler
+
+**Purpose**: Log CIDs that are private or don't meet speculator criteria
+
+**Why?**
+- Prevents re-discovery of known invalid users
+- Used for analytics (discovery success rate)
+
+#### 4.6.3 Fetch Insights
+
+**Purpose**: Daily fetch of eToro's instrument insights
+
+**Data Fetched**:
+- Market trends
+- Volatility metrics
+- Trading volume
+- Sentiment indicators
+
+**Storage**:
+```
+Collection: daily_instrument_insights
+└── Document: YYYY-MM-DD
+    └── insights: [ {...}, {...}, ... ]
+```
+
+#### 4.6.4 Price Fetcher
+
+**Purpose**: Daily fetch of closing prices
+
+**Storage**:
+```
+Collection: asset_prices
+└── Document: shard_N (N = instrumentId % 40)
+    └── {
+        "10000": {
+            "ticker": "AAPL",
+            "prices": {
+                "2025-01-15": 150.25,
+                "2025-01-14": 149.80,
+                ...
+            }
+        }
+    }
+```
+
+#### 4.6.5 Social Orchestrator & Task Handler
+
+**Purpose**: Fetch and analyze social posts from eToro
+
+**Flow**:
+1. **Orchestrator**: Publishes 1 task per target ticker
+2. **Task Handler**:
+   - Fetches posts since last run (12-hour window)
+   - Deduplicates via `processed_posts` collection
+   - Filters to English posts
+   - Calls Gemini AI for sentiment analysis
+   - Extracts topics (e.g., "FOMC", "Earnings", "CPI")
+   - Stores in `daily_social_insights/{date}/posts/{postId}`
+
+**Gemini Analysis**:
+```javascript
+const prompt = `Analyze this post. Return JSON:
+{
+    "overallSentiment": "Bullish|Bearish|Neutral",
+    "topics": ["FOMC", "Inflation", ...]
+}`;
+
+const result = await geminiModel.generateContent(prompt);
+```
+
+---
+
+## 5. Data Flow & Pipelines
+
+### 5.1 Daily Orchestration Schedule
+
+```
+00:00 UTC - Update Orchestrator (normal users)
+00:30 UTC - Update Orchestrator (speculators)
+01:00 UTC - Discovery Orchestrator (if needed)
+02:00 UTC - Fetch Insights
+02:30 UTC - Fetch Prices
+03:00 UTC - Computation Pass 1
+04:00 UTC - Computation Pass 2
+05:00 UTC - Computation Pass 3
+06:00 UTC - Social Orchestrator
+12:00 UTC - Speculator Cleanup
+```
+
+### 5.2 User Discovery Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ 1. Orchestrator: Check block capacities                     │
+│    - Are there <500 users in any block?                     │
+└────────────────┬────────────────────────────────────────────┘
+                 │ YES
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 2. Orchestrator: Generate candidate CIDs                     │
+│    - Prioritized: Normal users holding speculator assets    │
+│    - Random: Random CIDs within target blocks               │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 3. Dispatcher: Batch tasks (50 per message, 100ms delay)    │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 4. Task Engine (Discover): Test CIDs                         │
+│    - POST to Rankings API                                   │
+│    - Filter for active users                                │
+│    - Apply speculator heuristic (if applicable)             │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 5. Task Engine (Verify): Fetch portfolios                   │
+│    - Confirm user type                                      │
+│    - Check instrument holdings (speculators)                │
+│    - Write to Firestore blocks                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 5.3 User Update Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ 1. Orchestrator: Query timestamp documents                   │
+│    - Normal: lastUpdated < startOfToday                     │
+│    - Speculator: lastVerified < startOfToday                │
+│                  AND lastHeldAsset > 30DaysAgo              │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 2. Dispatcher: Batch update tasks                            │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 3. Task Engine: Prepare tasks                                │
+│    - Lookup usernames from cache                            │
+│    - If missing, bulk fetch from Rankings API               │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 4. Task Engine (Update): Sequential execution                │
+│    - Fetch trade history (once per user)                    │
+│    - Fetch portfolio (1x normal, Nx speculator)             │
+│    - Accumulate in batch manager                            │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 5. Task Engine: Flush batches                                │
+│    - Write all data to Firestore                            │
+│    - Update timestamps                                      │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 5.4 Computation Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ 1. Build Manifest                                            │
+│    - Introspect calculation classes                         │
+│    - Validate dependencies                                  │
+│    - Topological sort                                       │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 2. Load Global Status                                        │
+│    - Check which calcs completed yesterday                  │
+└────────────────┬────────────────────────────────────────────┘
+                 │
+                 ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 3. For each date (from earliest data to yesterday):         │
+│    ┌───────────────────────────────────────────────────┐   │
+│    │ 3a. Check root data availability                   │   │
+│    │     - Portfolio snapshots exist?                   │   │
+│    │     - Insights fetched?                            │   │
+│    │     - Social data available?                       │   │
+│    └────────────────┬──────────────────────────────────┘   │
+│                     │                                        │
+│                     ▼                                        │
+│    ┌───────────────────────────────────────────────────┐   │
+│    │ 3b. Filter calculations                            │   │
+│    │     - Skip if already completed                    │   │
+│    │     - Skip if dependencies not met                 │   │
+│    │     - Skip if root data missing                    │   │
+│    └────────────────┬──────────────────────────────────┘   │
+│                     │                                        │
+│                     ▼                                        │
+│    ┌───────────────────────────────────────────────────┐   │
+│    │ 3c. Stream portfolio data                          │   │
+│    │     - Load in chunks of 50 users                   │   │
+│    │     - Execute computations on each chunk           │   │
+│    └────────────────┬──────────────────────────────────┘   │
+│                     │                                        │
+│                     ▼                                        │
+│    ┌───────────────────────────────────────────────────┐   │
+│    │ 3d. Commit results                                 │   │
+│    │     - Store to unified_insights                    │   │
+│    │     - Update status document                       │   │
+│    └───────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 6. Computation System
+
+### 6.1 Adding a New Calculation
+
+**Step 1**: Create calculation class file
+
+```javascript
+// backend/core/calculations/category/newcalc.js
+
+class MyNewCalculation {
+    constructor() {
+        this.results = {};
+    }
+    
+    static getMetadata() {
+        return {
+            name: "my-new-calc",
+            category: "core",
+            type: "standard",
+            isHistorical: false,
+            rootDataDependencies: ["portfolio"],
+            userType: "all"
+        };
+    }
+    static getDependencies() {
+        return []; // Or list other calculation names
+    }
+    
+    static getSchema() {
+        return {
+            "TICKER": {
+                "myMetric": 0.0,
+                "confidence": 0.0
+            }
+        };
+    }
+    
+    async process(context) {
+        const { user, math, mappings } = context;
+        
+        // Get positions
+        const positions = math.extract.getPositions(
+            user.portfolio.today,
+            user.type
+        );
+        
+        // Process each position
+        for (const pos of positions) {
+            const instId = math.extract.getInstrumentId(pos);
+            const ticker = mappings.instrumentToTicker[instId];
+            
+            if (!ticker) continue;
+            
+            // Your calculation logic here
+            const netProfit = math.extract.getNetProfit(pos);
+            const weight = math.extract.getPositionWeight(pos);
+            
+            this.results[ticker] = {
+                myMetric: netProfit * weight,
+                confidence: weight > 10 ? 0.9 : 0.5
+            };
+        }
+    }
+    
+    async getResult() {
+        return this.results;
+    }
+}
+```
+
+```
+
+**Step 3**: The manifest builder will automatically:
+- Discover your calculation
+- Validate its metadata
+- Determine its execution pass based on dependencies
+- Include it in the manifest
+
+**Step 4**: Deploy and run
+
+```bash
+npm version patch
+npm publish
+# Update calculations package version number in core package.json
+# Deploy cloud function
+Main computation system will automatically discover the new computation.
+```
+
+### 6.2 Using Dependencies
+
+**Example**: Calculation that depends on another
+
+```javascript
+static getDependencies() {
+    return ["sentiment-score", "momentum"];
+}
+
+async process(context) {
+    const { computed, math } = context;
+    
+    // Access dependency results
+    const sentimentScore = math.signals.getMetric(
+        computed,
+        "sentiment-score",
+        "AAPL",
+        "score"
+    );
+    
+    const momentum = math.signals.getMetric(
+        computed,
+        "momentum",
+        "AAPL",
+        "value"
+    );
+    
+    // Combine signals
+    this.results["AAPL"] = {
+        combinedSignal: (sentimentScore * 0.6) + (momentum * 0.4)
+    };
+}
+```
+
+### 6.3 Historical Calculations
+
+**Purpose**: Compare today's data with yesterday's
+
+```javascript
+class PortfolioChangeCalculation {
+    static getMetadata() {
+        return {
+            name: "portfolio-change",
+            isHistorical: true, // Requires yesterday's data
+            // ...
+        };
+    }
+    
+    async process(context) {
+        const { user, math } = context;
+        
+        // Access today's portfolio
+        const todayPositions = math.extract.getPositions(
+            user.portfolio.today,
+            user.type
+        );
+        
+        // Access yesterday's portfolio
+        const yesterdayPositions = math.extract.getPositions(
+            user.portfolio.yesterday,
+            user.type
+        );
+        
+        // Compare positions
+        // ...
+    }
+}
+```
+
+### 6.4 Meta Calculations
+
+**Purpose**: Aggregate across all users (not per-user processing)
+
+```javascript
+class MarketWideSentimentCalculation {
+    static getMetadata() {
+        return {
+            type: "meta", // Not per-user
+            category: "core",
+            rootDataDependencies: ["insights", "social"]
+        };
+    }
+    
+    static getDependencies() {
+        return ["user-level-sentiment"]; // Depends on per-user calc
+    }
+    
+    async process(context) {
+        const { computed, insights, social } = context;
+        
+        // Access results from per-user calculations
+        const userSentiments = computed["user-level-sentiment"];
+        
+        // Access insights data
+        const marketData = insights.today;
+        
+        // Access social data
+        const socialPosts = social.today;
+        
+        // Perform market-wide aggregation
+        const tickers = new Set();
+        for (const ticker in userSentiments) {
+            tickers.add(ticker);
+        }
+        
+        for (const ticker of tickers) {
+            // Aggregate logic
+            this.results[ticker] = {
+                aggregatedSentiment: // ...
+            };
+        }
+    }
+}
+```
+
+### 6.5 Using Time Series
+
+**Purpose**: Track moving averages, trends, etc.
+
+```javascript
+class MovingAverageSentiment {
+    static getDependencies() {
+        return ["sentiment-score"];
+    }
+    
+    async process(context) {
+        const { computed, previousComputed, math } = context;
+        
+        const currentScore = math.signals.getMetric(
+            computed,
+            "sentiment-score",
+            "AAPL",
+            "score"
+        );
+        
+        // Get previous state (EMA parameters)
+        const prevState = math.signals.getPreviousState(
+            previousComputed,
+            "moving-average-sentiment",
+            "AAPL",
+            "_state"
+        );
+        
+        // Update EMA
+        const newState = math.TimeSeries.updateEMAState(
+            currentScore,
+            prevState || { mean: 0, variance: 1 },
+            0.1 // alpha (decay factor)
+        );
+        
+        this.results["AAPL"] = {
+            ema: newState.mean,
+            volatility: Math.sqrt(newState.variance),
+            _state: newState // Store for next day
+        };
+    }
+}
+```
+
+### 6.6 Advanced: Monte Carlo Risk Analysis
+
+```javascript
+class StopLossRiskCalculation {
+    async process(context) {
+        const { user, math, prices } = context;
+        
+        const positions = math.extract.getPositions(
+            user.portfolio.today,
+            user.type
+        );
+        
+        for (const pos of positions) {
+            const instId = math.extract.getInstrumentId(pos);
+            const ticker = context.mappings.instrumentToTicker[instId];
+            
+            // Get current price and stop loss
+            const currentRate = math.extract.getCurrentRate(pos);
+            const stopLoss = math.extract.getStopLossRate(pos);
+            
+            if (!stopLoss || stopLoss === 0) continue;
+            
+            // Get historical prices for volatility calculation
+            const priceHistory = math.priceExtractor.getHistory(
+                prices,
+                ticker
+            );
+            
+            // Calculate historical volatility
+            const returns = [];
+            for (let i = 1; i < priceHistory.length; i++) {
+                const ret = Math.log(
+                    priceHistory[i].price / priceHistory[i-1].price
+                );
+                returns.push(ret);
+            }
+            const volatility = math.compute.standardDeviation(returns) * Math.sqrt(252);
+            
+            // Calculate probability of hitting stop loss in next 3 days
+            const hitProbability = math.compute.calculateHitProbability(
+                currentRate,
+                stopLoss,
+                volatility,
+                3, // days
+                0  // drift (risk-neutral)
+            );
+            
+            // Run Monte Carlo simulation
+            const simulations = math.compute.simulateGBM(
+                currentRate,
+                volatility,
+                3,
+                1000 // number of paths
+            );
+            
+            // Count how many simulations hit stop loss
+            let hitCount = 0;
+            for (const simPrice of simulations) {
+                if (simPrice <= stopLoss) hitCount++;
+            }
+            const empiricalProbability = hitCount / 1000;
+            
+            this.results[ticker] = {
+                theoreticalHitProb: hitProbability,
+                empiricalHitProb: empiricalProbability,
+                currentRate,
+                stopLoss,
+                volatility
+            };
+        }
+    }
+}
+```
+
+---
+
+## 7. Development Guidelines
+
+### 7.1 Code Style
+
+#### 7.1.1 Naming Conventions
+
+```javascript
+// Calculation names: kebab-case
+"sentiment-score"
+"moving-average-momentum"
+
+// File names: snake_case
+my_calculation.js
+data_loader.js
+
+// Class names: PascalCase
+class SentimentScoreCalculation { }
+class IntelligentProxyManager { }
+
+// Functions: camelCase
+async function handleRequest() { }
+function getPositions() { }
+
+// Constants: SCREAMING_SNAKE_CASE
+const MAX_RETRIES = 3;
+const ETORO_API_URL = "https://...";
+```
+
+#### 7.1.2 Logging Standards
+
+```javascript
+// Always use structured logging
+logger.log('INFO', '[ModuleName] Descriptive message', {
+    contextKey: contextValue,
+    errorMessage: error.message // if error
+});
+
+// Log levels
+logger.log('TRACE', 'Very detailed debug info');
+logger.log('INFO', 'General information');
+logger.log('WARN', 'Warning condition');
+logger.log('ERROR', 'Error condition');
+logger.log('SUCCESS', 'Operation completed successfully');
+
+// Task-specific logging pattern
+logger.log('INFO', `[TaskType/${taskId}/${userId}] Message`);
+// Example: [UPDATE/batch-123/456789] Portfolio fetch successful
+```
+
+#### 7.1.3 Error Handling
+
+```javascript
+// Always wrap API calls in try-catch
+try {
+    const response = await proxyManager.fetch(url, options);
+    if (!response.ok) {
+        throw new Error(`API error ${response.status}`);
+    }
+    wasSuccess = true;
+    // Process response
+} catch (error) {
+    logger.log('ERROR', '[Module] Operation failed', {
+        errorMessage: error.message,
+        errorStack: error.stack,
+        context: { userId, url }
+    });
+    wasSuccess = false;
+} finally {
+    // Always update performance tracking
+    if (selectedHeader) {
+        headerManager.updatePerformance(selectedHeader.id, wasSuccess);
+    }
+}
+```
+
+#### 7.1.4 Async/Await Best Practices
+
+```javascript
+// BAD: Uncontrolled concurrency
+const promises = users.map(user => fetchUser(user));
+await Promise.all(promises); // Can overwhelm APIs
+
+// GOOD: Controlled concurrency
+const limit = pLimit(1); // Sequential
+const promises = users.map(user => 
+    limit(() => fetchUser(user))
+);
+await Promise.all(promises);
+
+// GOOD: Batch processing
+for (let i = 0; i < items.length; i += BATCH_SIZE) {
+    const batch = items.slice(i, i + BATCH_SIZE);
+    await processBatch(batch);
+    await sleep(DELAY_MS); // Rate limiting
+}
+```
+
+### 7.2 Testing Strategy
+
+#### 7.2.1 Local Testing Setup
+
+```javascript
+// test-setup.js
+const admin = require('firebase-admin');
+const { pipe } = require('bulltrackers-module');
+
+// Initialize Firebase Admin
+admin.initializeApp({
+    credential: admin.credential.applicationDefault(),
+    projectId: 'your-project-id'
+});
+
+const db = admin.firestore();
+const logger = {
+    log: (level, msg, data) => console.log(`[${level}] ${msg}`, data)
+};
+
+const dependencies = {
+    db,
+    logger,
+    // ... other dependencies
+};
+
+const config = {
+    // Your config
+};
+
+// Test a specific function
+async function testUpdateTask() {
+    const task = {
+        type: 'update',
+        userId: '123456',
+        userType: 'normal'
+    };
+    
+    await pipe.taskEngine.handleUpdate(
+        task,
+        'test-task-id',
+        dependencies,
+        config,
+        'testuser'
+    );
+}
+
+testUpdateTask().catch(console.error);
+```
+
+#### 7.2.2 Unit Test Example
+
+```javascript
+// test/math-primitives.test.js
+const { MathPrimitives } = require('../functions/computation-system/layers/math_primitives');
+
+describe('MathPrimitives', () => {
+    describe('calculateHitProbability', () => {
+        it('should return 0 for invalid inputs', () => {
+            const result = MathPrimitives.calculateHitProbability(
+                0, 100, 0.3, 5
+            );
+            expect(result).toBe(0);
+        });
+        
+        it('should return probability between 0 and 1', () => {
+            const result = MathPrimitives.calculateHitProbability(
+                100, 90, 0.3, 5
+            );
+            expect(result).toBeGreaterThanOrEqual(0);
+            expect(result).toBeLessThanOrEqual(1);
+        });
+        
+        it('should return higher probability for closer barriers', () => {
+            const far = MathPrimitives.calculateHitProbability(
+                100, 80, 0.3, 5
+            );
+            const close = MathPrimitives.calculateHitProbability(
+                100, 95, 0.3, 5
+            );
+            expect(close).toBeGreaterThan(far);
+        });
+    });
+});
+```
+
+#### 7.2.3 Integration Test Pattern
+
+```javascript
+// test/integration/computation-system.test.js
+async function testComputationPipeline() {
+    // 1. Setup test data
+    await setupTestPortfolios();
+    
+    // 2. Build manifest
+    const calculations = require('aiden-shared-calculations-unified');
+    const manifest = pipe.computationSystem.buildManifest(
+        ['core'],
+        calculations
+    );
+    
+    // 3. Run computation
+    const config = {
+        COMPUTATION_PASS_TO_RUN: "1",
+        resultsCollection: 'test_unified_insights'
+    };
+    
+    await pipe.computationSystem.runComputationPass(
+        config,
+        dependencies,
+        manifest
+    );
+    
+    // 4. Verify results
+    const results = await db
+        .collection('test_unified_insights')
+        .doc('2025-01-15')
+        .collection('results')
+        .doc('core')
+        .collection('computations')
+        .doc('test-calculation')
+        .get();
+    
+    expect(results.exists).toBe(true);
+    expect(results.data()).toHaveProperty('AAPL');
+}
+```
+
+### 7.3 Performance Optimization
+
+#### 7.3.1 Firestore Query Optimization
+
+```javascript
+// BAD: Multiple individual reads
+for (const userId of userIds) {
+    const doc = await db.collection('users').doc(userId).get();
+    // Process doc
+}
+
+// GOOD: Batch read (up to 500 docs)
+const docRefs = userIds.map(id => db.collection('users').doc(id));
+const snapshots = await db.getAll(...docRefs);
+snapshots.forEach(doc => {
+    // Process doc
+});
+```
+
+#### 7.3.2 Computation Optimization
+
+```javascript
+// BAD: Repeated calculations
+for (const position of positions) {
+    const instrumentId = DataExtractor.getInstrumentId(position);
+    const ticker = mappings.instrumentToTicker[instrumentId];
+    const netProfit = DataExtractor.getNetProfit(position);
+    // ... repeated work
+}
+
+// GOOD: Cache and reuse
+const tickerCache = new Map();
+for (const position of positions) {
+    const instrumentId = DataExtractor.getInstrumentId(position);
+    
+    let ticker = tickerCache.get(instrumentId);
+    if (!ticker) {
+        ticker = mappings.instrumentToTicker[instrumentId];
+        tickerCache.set(instrumentId, ticker);
+    }
+    
+    // ... use cached ticker
+}
+```
+
+#### 7.3.3 Memory Management
+
+```javascript
+// BAD: Loading all data into memory
+const allUsers = await loadFullDayMap(config, deps, refs);
+// Process 1M+ users at once
+
+// GOOD: Streaming
+for await (const chunk of streamPortfolioData(config, deps, dateStr, refs)) {
+    // Process 50 users at a time
+    await processChunk(chunk);
+    // Chunk is garbage collected after processing
+}
+```
+
+### 7.4 Common Patterns
+
+#### 7.4.1 Retry with Exponential Backoff
+
+```javascript
+async function fetchWithRetry(url, options, maxRetries = 3) {
+    for (let attempt = 1; attempt <= maxRetries; attempt++) {
+        try {
+            const response = await fetch(url, options);
+            if (response.ok) return response;
+            
+            if (attempt < maxRetries) {
+                const backoffMs = Math.pow(2, attempt) * 1000;
+                await sleep(backoffMs);
+            }
+        } catch (error) {
+            if (attempt === maxRetries) throw error;
+        }
+    }
+}
+```
+
+#### 7.4.2 Rate Limiting
+
+```javascript
+class RateLimiter {
+    constructor(maxRequests, windowMs) {
+        this.maxRequests = maxRequests;
+        this.windowMs = windowMs;
+        this.requests = [];
+    }
+    
+    async acquire() {
+        const now = Date.now();
+        
+        // Remove old requests
+        this.requests = this.requests.filter(
+            time => now - time < this.windowMs
+        );
+        
+        // Check if we can proceed
+        if (this.requests.length >= this.maxRequests) {
+            const oldestRequest = this.requests[0];
+            const waitTime = this.windowMs - (now - oldestRequest);
+            await sleep(waitTime);
+            return this.acquire(); // Recursive retry
+        }
+        
+        this.requests.push(now);
+    }
+}
+
+// Usage
+const limiter = new RateLimiter(10, 60000); // 10 requests per minute
+
+for (const task of tasks) {
+    await limiter.acquire();
+    await processTask(task);
+}
+```
+
+#### 7.4.3 Batch Aggregation Pattern
+
+```javascript
+class BatchAggregator {
+    constructor(flushCallback, maxSize = 500, maxWaitMs = 5000) {
+        this.flushCallback = flushCallback;
+        this.maxSize = maxSize;
+        this.maxWaitMs = maxWaitMs;
+        this.items = [];
+        this.timer = null;
+    }
+    
+    add(item) {
+        this.items.push(item);
+        
+        if (this.items.length >= this.maxSize) {
+            this.flush();
+        } else if (!this.timer) {
+            this.timer = setTimeout(() => this.flush(), this.maxWaitMs);
+        }
+    }
+    
+    async flush() {
+        if (this.timer) {
+            clearTimeout(this.timer);
+            this.timer = null;
+        }
+        
+        if (this.items.length === 0) return;
+        
+        const itemsToFlush = this.items;
+        this.items = [];
+        
+        await this.flushCallback(itemsToFlush);
+    }
+}
+
+// Usage
+const aggregator = new BatchAggregator(async (items) => {
+    const batch = db.batch();
+    for (const item of items) {
+        batch.set(item.ref, item.data);
+    }
+    await batch.commit();
+});
+
+// Add items
+aggregator.add({ ref: docRef1, data: data1 });
+aggregator.add({ ref: docRef2, data: data2 });
+// ... auto-flushes when full or after timeout
+```
+
+---
+
+## 8. Deployment & Operations
+
+### 8.1 Deployment Process
+
+#### 8.1.1 Module Updates
+
+```bash
+# 1. Make changes to bulltrackers-module
+cd Backend/Core/bulltrackers-module
+
+# 2. Update version
+npm version patch  # or minor, or major
+
+# 3. Publish to NPM
+npm publish
+
+# 4. Update dependent packages
+cd ../your-cloud-function
+npm install bulltrackers-module@latest
+
+# 5. Deploy cloud function
+gcloud functions deploy your-function \
+    --gen2 \
+    --runtime=nodejs20 \
+    --region=us-central1 \
+    --source=. \
+    --entry-point=yourEntryPoint \
+    --trigger-topic=your-topic
+```
+
+#### 8.1.2 Calculation Updates
+
+```bash
+# 1. Update calculation in aiden-shared-calculations-unified
+cd aiden-shared-calculations-unified
+
+# 2. Version bump
+npm version patch
+
+# 3. Publish
+npm publish
+
+# 4. Update bulltrackers-module dependency
+cd ../bulltrackers-module
+npm install aiden-shared-calculations-unified@latest
+npm version patch
+npm publish
+
+# 5. Redeploy computation functions
+# (They will pick up new calculations automatically)
+```
+
+#### 8.1.3 Environment Variables
+
+```bash
+# Set via Cloud Console or gcloud CLI
+gcloud functions deploy computation-pass-1 \
+    --set-env-vars="COMPUTATION_PASS_TO_RUN=1" \
+    --set-env-vars="FIRESTORE_PROJECT_ID=your-project" \
+    --set-env-vars="ETORO_API_URL=https://..."
+```
+
+### 8.2 Monitoring
+
+#### 8.2.1 Key Metrics
+
+**Cloud Functions**:
+- Invocation count per function
+- Execution time (p50, p95, p99)
+- Error rate
+- Memory usage
+- Active instances
+
+**Firestore**:
+- Read operations per day
+- Write operations per day
+- Document count per collection
+- Storage usage
+
+**Pub/Sub**:
+- Messages published per topic
+- Messages consumed
+- Oldest unacked message age
+- Dead letter queue size
+
+#### 8.2.2 Logging Queries
+
+```
+# Find errors in last 24 hours
+resource.type="cloud_function"
+severity="ERROR"
+timestamp>="2025-01-15T00:00:00Z"
+
+# Find specific user task
+resource.type="cloud_function"
+jsonPayload.message=~"userId.*123456"
+
+# Find rate limit errors
+resource.type="cloud_function"
+jsonPayload.message=~"rate limit|429|too many"
+
+# Computation pass performance
+resource.type="cloud_function"
+resource.labels.function_name="computation-pass-1"
+jsonPayload.message=~"Pass.*finished"
+```
+
+#### 8.2.3 Alerting Rules
+
+```yaml
+# Example alert policy
+displayName: "Task Engine Error Rate High"
+conditions:
+  - displayName: "Error rate > 5%"
+    conditionThreshold:
+      filter: |
+        resource.type="cloud_function"
+        resource.labels.function_name="task-engine"
+        severity="ERROR"
+      aggregations:
+        - alignmentPeriod: 300s
+          perSeriesAligner: ALIGN_RATE
+      comparison: COMPARISON_GT
+      thresholdValue: 0.05
+      duration: 600s
+
+notificationChannels:
+  - "projects/YOUR_PROJECT/notificationChannels/YOUR_CHANNEL"
+```
+
+### 8.3 Cost Optimization
+
+#### 8.3.1 Firestore Costs
+
+**Current Architecture** (Optimized):
+- Sharded writes: ~50 writes per 500 users = 10¢ per 1M users
+- Status tracking: Single global document = $0
+- Batch commits: 1 batch per flush = 10× cost reduction
+
+**Anti-Patterns to Avoid**:
+```javascript
+// BAD: Individual writes
+for (const user of users) {
+    await db.collection('data').doc(user.id).set(data);
+}
+// Cost: N writes = $0.18 per 1000 writes
+
+// GOOD: Batched writes
+const batch = db.batch();
+for (const user of users) {
+    batch.set(db.collection('data').doc(user.id), data);
+}
+await batch.commit();
+// Cost: 1 write = $0.00018
+```
+
+#### 8.3.2 Cloud Function Costs
+
+**Optimization Strategies**:
+- **Memory Allocation**: Use 512MB for most functions (256MB often causes cold starts)
+- **Timeout**: Set realistic timeouts (5min for task engine, 1min for API)
+- **Concurrency**: Limit concurrent instances to avoid quota exhaustion
+
+```yaml
+# function-config.yaml
+availableMemoryMb: 512
+timeout: 300s
+maxInstances: 100
+minInstances: 0  # Set to 1-2 for critical functions
+```
+
+#### 8.3.3 API Call Optimization
+
+**Current Strategy**:
+- AppScript proxies: Free (within Google's generous limits)
+- Direct calls: Fallback only (minimize usage)
+- Batch API calls: 50-100 users per request
+
+**Cost Comparison**:
+```
+AppScript Proxy:
+- Cost: $0
+- Rate Limit: ~100 requests/min per proxy
+- Solution: 10+ proxies = 1000 req/min
+
+Direct GCP IP:
+- Cost: $0
+- Rate Limit: ~20 requests/min (eToro throttles)
+- Solution: Use only as fallback
+```
+
+### 8.4 Disaster Recovery
+
+#### 8.4.1 Backup Strategy
+
+```javascript
+// Automated daily backup (Cloud Scheduler)
+const {Firestore} = require('@google-cloud/firestore');
+
+async function backupFirestore() {
+    const firestore = new Firestore();
+    
+    const bucket = 'gs://your-backup-bucket';
+    const timestamp = new Date().toISOString().split('T')[0];
+    
+    await firestore.export({
+        collectionIds: [
+            'NormalUserPortfolios',
+            'SpeculatorBlocks',
+            'unified_insights'
+        ],
+        outputUriPrefix: `${bucket}/firestore-backups/${timestamp}`
+    });
+}
+```
+
+#### 8.4.2 Data Corruption Recovery
+
+```javascript
+// Rebuild computation results from historical data
+async function rebuildComputations(startDate, endDate) {
+    const config = {
+        COMPUTATION_PASS_TO_RUN: "1",
+        // Force recompute by clearing status
+    };
+    
+    // Clear status for date range
+    const statusRef = db.collection('computation_status').doc('global_status');
+    const updates = {};
+    
+    for (let date = startDate; date <= endDate; date.setDate(date.getDate() + 1)) {
+        const dateStr = date.toISOString().slice(0, 10);
+        // Delete all calculation statuses for this date
+        for (const calcName of allCalculationNames) {
+            updates[`${dateStr}.${calcName}`] = FieldValue.delete();
+        }
+    }
+    
+    await statusRef.update(updates);
+    
+    // Trigger computation passes
+    // They will recompute all dates with missing status
+}
+```
+
+#### 8.4.3 Rollback Procedure
+
+```bash
+# 1. Identify problematic version
+git log --oneline
+
+# 2. Revert to previous version
+git revert <commit-hash>
+
+# 3. Republish packages
+npm version patch
+npm publish
+
+# 4. Redeploy functions
+gcloud functions deploy all-functions --source=.
+
+# 5. Verify in logs
+gcloud logging read "resource.type=cloud_function" --limit=100
+```
+
+---