bulltrackers-module 1.0.741 → 1.0.743

package/functions/computation-system-v2/README.md

@@ -5,148 +5,58 @@
 A generic, BigQuery-driven computation framework that can be applied to any dataset.
 
 **Key Principles:**
-- **Zero hardcoded schemas** - Schemas are discovered dynamically from BigQuery metadata
-- **Pre-query validation** - Queries are validated against cached schemas before execution
-- **Simple computation interface** - Just declare what tables you need
-- **Pass-based execution** - Topological sort determines execution order (Kahn's algorithm)
-- **Hash-based versioning** - Automatic detection of code/dependency changes
+- **Zero hardcoded schemas** - Schemas are discovered dynamically from BigQuery.
+- **Pre-query validation** - Queries are validated against cached schemas before execution.
+- **Pass-based execution** - Topological sort determines execution order (Kahn's algorithm).
+- **Hash-based versioning** - Automatic detection of code/dependency changes to support cached re-runs.
+- **Remote Execution** - Offloads heavy per-entity computations to serverless workers.
 
-## Directory Structure
+## Documentation
 
-```
-computation-system-v2/
-├── framework/                    # GENERIC - Works with any BigQuery data
-│   ├── core/
-│   │   ├── Computation.js       # Base class for all computations
-│   │   ├── Executor.js          # Orchestrates computation execution
-│   │   └── Manifest.js          # Builds & validates computation graph
-│   ├── data/
-│   │   ├── SchemaRegistry.js    # Dynamic schema discovery + caching
-│   │   ├── QueryBuilder.js      # Builds validated SQL queries
-│   │   └── DataFetcher.js       # Executes queries, returns data
-│   ├── persistence/
-│   │   └── ResultWriter.js      # Writes computation results
-│   └── utils/
-│       ├── hash.js              # Code hashing utilities
-│       └── logger.js            # Logging utilities
-│
-├── config/
-│   └── bulltrackers.config.js   # BullTrackers-specific configuration
-│
-├── computations/                 # BullTrackers computations (v2 format)
-│   └── (migrated computations go here)
-│
-└── index.js                      # Main entry point
-```
+The documentation is split into four detailed guides:
+
+### 1. [Architecture Guide](docs/architecture.md)
+*   **Target Audience**: Architects, Senior Engineers
+*   **Contents**: System Philosophy, Core Components (Manifest, Schema Registry), Data Flow, and Remote Execution internals.
+
+### 2. [Developer Guide](docs/developer_guide.md)
+*   **Target Audience**: Data Engineers, Developers writing logic
+*   **Contents**: How to create a new `Computation`, configure dependencies (`requires`), and implement the `process()` method.
+
+### 3. [API Reference](docs/api_reference.md)
+*   **Target Audience**: Developers
+*   **Contents**: Complete API documentation for the `Computation` class, `context` object, and configuration schema. Includes **System Defaults & Fallbacks**.
+
+### 4. [Operations Manual](docs/operations.md)
+*   **Target Audience**: DevOps, SREs, On-call Support
+*   **Contents**: Deployment instructions, Management API usage (`status`, `analyze`, `run`), Monitoring guide, and Troubleshooting common issues.
 
 ## Quick Start
 
-### 1. Define a Computation
+### Define a Computation
 
 ```javascript
 const { Computation } = require('./framework/core/Computation');
 
-class MyComputation extends Computation {
+class UserPortfolioSummary extends Computation {
     static getConfig() {
         return {
-            name: 'MyComputation',
-            requires: {
-                'my_table': { 
-                    lookback: 0,      // 0 = today only, N = last N days
-                    mandatory: true   // If true, computation won't run without this data
-                }
-            },
-            dependencies: [],  // Other computations this depends on
-            type: 'global'     // 'global' or 'per-entity'
+            name: 'UserPortfolioSummary',
+            type: 'per-entity',
+            requires: { 'orders': { lookback: 7, mandatory: true } }
         };
     }
 
     async process(context) {
-        const data = context.data['my_table'];
         // Your logic here
-        return { result: 'computed value' };
+        return { totalValue: 1000 };
     }
 }
 ```
 
-### 2. Register in Config
-
-```javascript
-// config/bulltrackers.config.js
-module.exports = {
-    bigquery: {
-        projectId: 'your-project',
-        dataset: 'your_dataset'
-    },
-    tables: {
-        'my_table': {
-            dateField: 'date',
-            entityField: 'user_id'
-        }
-    },
-    computations: [
-        require('./computations/MyComputation')
-    ]
-};
-```
-
-### 3. Run
-
-```javascript
-const { execute } = require('./computation-system-v2');
-await execute({ date: '2026-01-24', config: require('./config/bulltrackers.config') });
-```
-
-## Key Concepts
-
-### Schema Registry
-
-The `SchemaRegistry` fetches table schemas from BigQuery's `INFORMATION_SCHEMA` and caches them:
-
-```javascript
-const schema = await schemaRegistry.getSchema('my_table');
-// Returns: [{ name: 'id', type: 'INT64' }, { name: 'date', type: 'DATE' }, ...]
-```
-
-Before any query is executed, it's validated against the cached schema to prevent runtime errors.
-
-### Query Validation
-
-```javascript
-// This will throw BEFORE hitting BigQuery if 'nonexistent_column' doesn't exist
-const query = queryBuilder.build({
-    table: 'my_table',
-    select: ['id', 'nonexistent_column'],  // Error: column doesn't exist
-    where: { date: '2026-01-24' }
-});
-```
-
-### Pass System
-
-Computations are automatically organized into "passes" based on dependencies:
+### Run It
 
+```bash
+# Run locally (Dry Run)
+node index.js execute --computation UserPortfolioSummary --date 2026-01-26 --dry-run
 ```
-Pass 1: Computations with no dependencies (can run in parallel)
-Pass 2: Computations depending only on Pass 1 results
-Pass 3: Computations depending on Pass 1 or 2 results
-...
-```
-
-### Hash-Based Versioning
-
-Each computation gets a unique hash based on:
-- Its own code
-- The code of any shared utilities it uses
-- The hashes of its dependencies
-
-When a hash changes, the computation re-runs. When hashes match, results are reused.
-
-## Comparison with v1
-
-| Aspect | v1 | v2 |
-|--------|----|----|
-| Schema definition | Hardcoded in `bigquery_utils.js` | Dynamic from `INFORMATION_SCHEMA` |
-| Query building | Per-table functions | Single generic builder with validation |
-| Data loading | Complex routing through multiple layers | Direct fetch with simple interface |
-| Computation declaration | Multiple overlapping fields | Single `requires` object |
-| Domain coupling | Deeply embedded | Configuration-only |

package/functions/computation-system-v2/config/bulltrackers.config.js

@@ -300,7 +300,7 @@ module.exports = {
         // Service account for OIDC authentication when invoking Dispatcher
         // This SA needs roles/cloudfunctions.invoker on the Dispatcher function
         serviceAccountEmail: process.env.CLOUD_TASKS_SA_EMAIL || 
-            'computation-scheduler@stocks-12345.iam.gserviceaccount.com'
+            '879684846540-compute@developer.gserviceaccount.com'
     },
     
     // =========================================================================

package/functions/computation-system-v2/docs/api_reference.md

@@ -0,0 +1,118 @@
+# API Reference
+
+## 1. `Computation` Class
+
+The base class for all computations.
+
+### Static Methods
+
+#### `getConfig()`
+Must be implemented by the subclass. Returns the configuration object.
+*   **Returns**: `Object` (See Configuration Schema below)
+
+#### `validateConfig()`
+Internal method used by the Manifest Builder to verify the config.
+*   **Returns**: `{ valid: boolean, errors: string[] }`
+
+#### `getSchema()`
+Optional. Define a hardcoded schema if BigQuery discovery is not used (Legacy support).
+
+### Instance Methods
+
+#### `async process(context)`
+**Required**. The main execution logic.
+*   **Args**: `context` (Object)
+*   **Returns**: `Object` (The result to be saved)
+
+#### `log(level, message)`
+Structured logging helper.
+*   **Args**:
+    *   `level`: 'INFO', 'WARN', 'ERROR', 'DEBUG'
+    *   `message`: String
+
+---
+
+## 2. `Context` Object
+
+The object passed to `toprocess()`.
+
+| Property | Type | Description |
+| :--- | :--- | :--- |
+| `date` | String | The target date of execution (YYYY-MM-DD). |
+| `entityId` | String | The ID of the specific entity being processed. |
+| `data` | Object | Map of table names to data (Rows or Arrays of Rows). |
+| `previousResult` | Object | The result of *this* computation from the previous date (if `isHistorical: true`). |
+| `config` | Object | A safe subset of the global configuration. |
+| `references` | Object | Cached reference data (e.g., global mappings). |
+
+### Methods
+
+#### `getDependency(computationName, [entityId])`
+Retrieves the result of a dependency.
+*   If `entityId` is omitted, it defaults to the current `context.entityId`.
+*   Use `_global` as `entityId` to fetch Global computation results.
+
+---
+
+## 3. Configuration Schema (`bulltrackers.config.js`)
+
+```javascript
+module.exports = {
+    project: 'gcp-project-id',
+    
+    bigquery: {
+        projectId: 'gcp-project-id',
+        dataset: 'dataset_name',
+        location: 'US',
+        cacheTTLMs: 3600000 // 1 hour
+    },
+    
+    workerPool: {
+        enabled: true,
+        workerUrl: 'https://...',
+        concurrency: 100
+    },
+    
+    tables: {
+        'table_name': {
+            entityField: 'user_id',
+            dateField: 'date'
+        }
+    }
+};
+```
+
+---
+
+## 4. System Defaults & Fallbacks
+
+The system uses the following default values if not explicitly configured:
+
+### scheduling (`ScheduleValidator.js`)
+| Parameter | Default Value | Description |
+| :--- | :--- | :--- |
+| `frequency` | `'daily'` | Default schedule frequency. |
+| `time` | `'02:00'` | Default execution time (UTC). |
+| `timezone` | `'UTC'` | Default timezone. |
+| `dependencyGapMinutes` | `15` | Minimum safe gap between dependent computations. |
+
+### Execution (`Orchestrator.js`)
+| Parameter | Default Value | Description |
+| :--- | :--- | :--- |
+| `batchSize` | `1000` | Number of entities per batch in Streaming/Remote mode. |
+| `entityConcurrency` | `20` | Concurrent entities processed *per batch* in Local mode. |
+
+### Remote Worker Pool (`RemoteTaskRunner.js`)
+| Parameter | Default Value | Description |
+| :--- | :--- | :--- |
+| `concurrency` | `100` | Max concurrent Worker Functions invoked. |
+| `timeout` | `60000` (60s) | HTTP timeout for worker invocation. |
+| `retries` | `2` | Number of retries for transient failures. |
+| `circuitBreaker.failureThreshold` | `0.30` (30%) | Failure rate needed to trip the circuit. |
+| `circuitBreaker.minInvocations` | `20` | Minimum calls before Circuit Breaker activates. |
+
+### BigQuery
+| Parameter | Default Value | Description |
+| :--- | :--- | :--- |
+| `location` | `'US'` | Default BigQuery location. |
+| `cacheTTLMs` | `3600000` (1h) | Duration to cache schemas in memory. |

package/functions/computation-system-v2/docs/architecture.md

@@ -1,59 +1,103 @@
+# System Architecture
+
+## 1. System Philosophy
+
+The Computation System v2 is designed to be a generic, configuration-driven Directed Acyclic Graph (DAG) executor that operates directly on BigQuery data. It departs from traditional ETL pipelines by adhering to five core principles:
+
+1.  **Zero Hardcoded Schemas**: The system never defines schemas in code. It dynamically discovers them from BigQuery's `INFORMATION_SCHEMA` at runtime. This eliminates the "schema drift" problem where code falls out of sync with the database.
+2.  **Pre-Query Validation**: Every SQL query is validated against the cached schema *before* it is sent to BigQuery. This prevents expensive runtime failures and SQL errors.
+3.  **Pass-Based Execution**: Computations are automatically organized into "passes" (waves) based on their dependencies using Kahn's algorithm.
+4.  **Hash-Based Versioning**: The system tracks the "Identity" of every computation result using a cryptographic hash of:
+    *   The Computation Code
+    *   Shared Utility "Layers"
+    *   Business Logic "Rules"
+    *   Dependency Result Hashes
+5.  **Hybrid Execution**: Light computations run globally in-memory, while heavy per-entity computations are offloaded to a serverless worker pool (Remote Task Runner).
+
+---
+
+## 2. Core Components
+
+### 2.1. The Manifest Builder (`framework/core/Manifest.js`)
+The Manifest is the blueprint of the system. At startup, it:
+1.  **Loads Configs**: Reads static `getConfig()` from all Computation classes.
+2.  **Builds Graph**: Constructs the dependency graph.
+3.  **Detects Cycles**: Throws an error if A -> B -> A.
+4.  **Calculates Passes**: Uses Topological Sort to assign a `pass` number (0, 1, 2...) to each computation.
+5.  **Generates Hashes**: Computes the intrinsic version hash for change detection.
+
+### 2.2. Schema Registry (`framework/data/SchemaRegistry.js`)
+The bridge between code and data.
+*   **Dynamic Discovery**: Fetches column definitions, types, and nullability from BigQuery.
+*   **Caching**: Caches schemas in memory (default TTL: 1 hour) to reduce latency.
+*   **Request Coalescing**: Prevents "Thundering Herd" issues by merging simultaneous requests for the same table schema.
+
+### 2.3. The Orchestrator (`framework/execution/Orchestrator.js`)
+The central nervous system. It creates the execution plan for a given date.
+*   **Dependency Resolution**: Ensures pre-requisite data is loaded.
+*   **Execution Strategy**: Decides whether to run a computation locally or offload it to the Remote Task Runner.
+*   **Streaming**: For large datasets, it streams data in batches (default: 1000 entities) to avoid Out-Of-Memory (OOM) crashes.
+
+---
+
+## 3. Data Flow
+
 ```mermaid
 graph TD
-    Root((System))
-
-    %% Subgraph: Scheduling & Control
-    subgraph Control_Plane [Control Plane]
-        Cron((Timer)) -->|Every Minute| Scheduler[Scheduler Handler]
-        Scheduler -->|Find Due & Zombies| StateRepo[(State DB)]
-        Scheduler -->|Dispatch Task| CloudTasks[Cloud Tasks Queue]
-        CloudTasks -->|HTTP POST w/ Backoff| Dispatcher[Dispatcher Handler]
-        Dispatcher -->|Run Computation| Orchestrator[Orchestrator]
-        Orchestrator -->|Return Status| Dispatcher
-        Dispatcher -.->|Blocked| Return503[503 Retry]
-        Return503 -.-> CloudTasks
-        Dispatcher -.->|Success / Skipped| Return200[200 OK]
+    User[User/Scheduler] -->|Trigger| Dispatcher
+    Dispatcher -->|POST| Orchestrator
+    
+    subgraph "Phase 1: Preparation"
+        Orchestrator -->|Build| Manifest
+        Orchestrator -->|Check Status| StateDB[(State Repository)]
+        StateDB -->|Run/Skip/ReRun?| Orchestrator
     end
-
-    %% Subgraph: Execution
-    subgraph Execution_Core [Execution Core]
-        Orchestrator --> Manifest[Manifest Builder]
-        Orchestrator -->|Check Hashes & Deps| StateRepo
-        Orchestrator -->|Fetch Data| BigQuery[(BigQuery)]
-        Orchestrator --> ExecMode{Mode?}
-
-        ExecMode -->|Global / Light| LocalExec[Local Execution]
-        LocalExec --> Logic[Computation Logic]
-        Logic --> LocalExec
-
-        ExecMode -->|Per-Entity / Heavy| RemoteRunner[Remote Task Runner]
-        RemoteRunner -->|Upload Context| GCS[(Cloud Storage)]
-        RemoteRunner --> Worker[Worker Handler]
-        Worker -->|Download Context| GCS
-        Worker -->|Execute| Logic
-        Worker -->|Return Result| RemoteRunner
+    
+    subgraph "Phase 2: Execution"
+        Orchestrator -->|Fetch Data| DataFetcher
+        DataFetcher -->|Validate| SchemaRegistry
+        SchemaRegistry -->|Meta| BigQuery[(BigQuery)]
+        DataFetcher -->|Query| BigQuery
+        
+        Orchestrator -->|Pass 0| C1[Computation A]
+        Orchestrator -->|Pass 1| C2[Computation B]
+        
+        C1 -->|Result| Storage[Storage Manager]
+        Storage -->|Persist| BigQuery
     end
+```
 
-    %% Subgraph: Persistence
-    subgraph Persistence [Persistence Layer]
-        LocalExec -->|Commit Results| StateRepo
-        RemoteRunner -->|Commit Batch| StateRepo
-    end
+---
+
+## 4. Execution Modes
+
+### 4.1. Global Mode (Local Isolation)
+Used for: Aggregations, summaries, or light computations.
+*   **Flow**: Fetches *all* required data into memory -> Runs logic -> Writes single result.
+*   **Concurrency**: Parallelized by `p-limit` locally.
+
+### 4.2. Per-Entity Mode (Remote Offload)
+Used for: Complex logic (e.g., Portfolio Calculations) requiring isolation.
+*   **Orchestrator**: Fetches data for a batch (e.g., 100 users).
+*   **Storage**: Uploads context (Data + Rules + Config) to Cloud Storage (GCS).
+*   **Worker Pool**: Invokes stateless Cloud Functions to process the batch.
+*   **Circuit Breaker**: Stops execution if failure rate exceeds threshold (default: 30%).
 
-    %% Single-root anchoring (critical)
-    Root --> Cron
-    Root -.-> Orchestrator
-    Root -.-> LocalExec
-    Root -.-> RemoteRunner
-
-    %% Styling
-    classDef plain fill:#ffffff,stroke:#333,stroke-width:1px;
-    classDef db fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
-    classDef logic fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
-    classDef queue fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
-
-    class Cron,Scheduler,Dispatcher,Orchestrator,Manifest,LocalExec,RemoteRunner,Worker plain;
-    class StateRepo,BigQuery,GCS db;
-    class Logic logic;
-    class CloudTasks queue;
+---
+
+## 5. Versioning & Hashing
+
+How does the system know when to re-run a computation?
+
+The **Composite Hash** is calculated as:
 ```
+Hash = SHA256(
+    Code Body +
+    Epoch Version +
+    Shared Layer Hashes +
+    Rule Module Hashes +
+    Dependency Result Hashes
+)
+```
+
+If this hash matches the `resultHash` stored in the State Repository for a given date, the computation is **SKIPPED** (unless forced). This ensures idempotency and avoids unnecessary processing costs.

package/functions/computation-system-v2/docs/developer_guide.md

@@ -0,0 +1,125 @@
+# Developer Guide
+
+This guide explains how to create, configure, and test new Computations in the v2 framework.
+
+## 1. Creating a Computation
+
+All computations must extend the base `Computation` class.
+
+### Basic Template
+
+```javascript
+const { Computation } = require('../../framework/core/Computation');
+
+class UserDailyActive extends Computation {
+    static getConfig() {
+        return {
+            name: 'UserDailyActive',
+            type: 'per-entity', // or 'global'
+            requires: {
+                // Table Dependencies (Data)
+                'user_logins': { 
+                    lookback: 0,      // 0 = today only
+                    mandatory: true
+                }
+            },
+            dependencies: [
+                // Computation Dependencies (Prerequisites)
+                // 'UserSessionSummary' 
+            ]
+        };
+    }
+
+    async process(context) {
+        const { date, entityId, data } = context;
+        
+        // Access pre-fetched data
+        const logins = data['user_logins'];
+        
+        if (!logins || logins.length === 0) {
+            return null; // Return null to skip saving result
+        }
+
+        return {
+            date,
+            userId: entityId,
+            loginCount: logins.length,
+            lastLoginTime: logins[logins.length - 1].timestamp
+        };
+    }
+}
+module.exports = UserDailyActive;
+```
+
+---
+
+## 2. Configuration Options (`getConfig`)
+
+| Field | Type | Description |
+| :--- | :--- | :--- |
+| `name` | String | **Required**. Unique name of the computation. |
+| `type` | String | `'global'` (runs once) or `'per-entity'` (runs for each entity). |
+| `requires` | Object | Map of table names to requirements. |
+| `dependencies` | Array | List of other computation names that must run *before* this one. |
+| `schedule` | Object | `{ frequency: 'daily', time: '02:00' }`. Defaults to daily/02:00. |
+| `ttlDays` | Number | How long to keep results in State DB (default: 365). |
+| `isHistorical`| Boolean| If `true`, `context.previousResult` will contain yesterday's output. |
+
+### The `requires` Object
+
+```javascript
+requires: {
+    'orders': {
+        lookback: 7,       // Fetch last 7 days of data
+        mandatory: false,  // If true, crashes if table is empty/missing
+        dateField: 'created_at' // Optional override
+    }
+}
+```
+
+---
+
+## 3. The `process(context)` Method
+
+The `process` method is the heart of your logic. It receives a `context` object:
+
+### `context.data`
+Contains the raw data fetched from BigQuery.
+*   **Per-Entity**: `data['table_name']` is the row (or array of rows) *specifically for that entity*.
+*   **Global**: `data['table_name']` is the entire dataset fetched.
+
+### `context.getDependency(name, [entityId])`
+Access results from previous computations.
+```javascript
+// In a per-entity computation, getting its own dependency:
+const sessionStats = context.getDependency('UserSessionSummary');
+
+// Getting a global dependency result from a per-entity computation:
+const globalSettings = context.getDependency('GlobalSettings', '_global');
+```
+
+### `context.rules`
+Access to shared business logic modules (if configured).
+
+---
+
+## 4. Testing & Validation
+
+### Local Dry Run
+You can run your computation locally without writing to BigQuery/Firestore by using the dry-run flag.
+
+```bash
+# Run a specific computation for a specific date
+node index.js execute \
+  --computation UserDailyActive \
+  --date 2026-01-26 \
+  --dry-run \
+  --entityId user_12345
+```
+
+### Logging
+Use `this.log(level, message)` instead of `console.log`. This ensures logs are properly tagged with the computation name and entity ID in Cloud Logging.
+
+```javascript
+this.log('INFO', `Processing user ${entityId} with ${logins.length} logins`);
+```

package/functions/computation-system-v2/docs/operations.md

@@ -0,0 +1,99 @@
+# Operations Manual
+
+This guide covers deployment, daily operations, monitoring, and troubleshooting.
+
+## 1. Deployment
+
+The system is deployed as a Google Cloud Function (Gen 2).
+
+### Deploy Command
+```bash
+# Navigate to the function directory
+cd functions/computation-system-v2
+
+# Deploy using the included script
+node deploy.mjs ComputeSystem
+```
+
+**Note**: Ensure you have the `gcloud` CLI installed and authenticated.
+
+---
+
+## 2. Management API
+
+The system exposes an HTTP endpoint for manual administration.
+
+### Setup (One-time)
+```bash
+# Get Function URL
+FUNCTION_URL=$(gcloud functions describe compute-system --region=europe-west1 --format="value(serviceConfig.uri)")
+
+# Get Auth Token
+TOKEN=$(gcloud auth print-identity-token)
+```
+
+### Common Actions
+
+#### Check Status
+Lists all registered computations and their hash status.
+```bash
+curl -X POST "$FUNCTION_URL" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "status"}'
+```
+
+#### Analyze a Date
+Simulates a run for a valid date, showing what would run, skip, or be blocked.
+```bash
+curl -X POST "$FUNCTION_URL" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "analyze", "date": "2026-01-26"}'
+```
+
+#### Manually Trigger (Force Run)
+Forces a computation to run, ignoring hash checks.
+```bash
+curl -X POST "$FUNCTION_URL" \
+    -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"action": "run", "computation": "UserDailyActive", "date": "2026-01-26", "force": true}'
+```
+
+---
+
+## 3. Monitoring
+
+The Orchestrator logs a structured **Execution Summary** at the end of every run.
+
+### Understanding Statuses
+
+| Status | Icon | Description | Action Required |
+| :--- | :--- | :--- | :--- |
+| `completed` | ✅ | Computation finished successfully. | None |
+| `skipped` | ⏭️ | Result hash matches previous run (Up-to-date). | None |
+| `blocked` | ⛔ | A dependency failed or hasn't run yet. | Fix dependency then re-run. |
+| `impossible` | 🚫 | A mandatory requirement (table) is empty. | Check upstream ETL. |
+| `failed` | ❌ | Runtime error threw an exception. | Check logs for stack trace. |
+
+### Circuit Breaker (Remote Mode)
+If you see `CIRCUIT BREAKER TRIPPED` in the logs:
+1.  **Stop**: The system automatically aborted to prevent wasting money on failing workers.
+2.  **Investigate**: Look for "Worker failed" logs to see the root cause (e.g., Syntax Error in calculation).
+3.  **Fix**: Deploy the fix.
+4.  **Retry**: The circuit resets automatically on the next run.
+
+---
+
+## 4. Troubleshooting
+
+### "Thundering Herd" on Startup
+**Symptom**: Massive latency on the first few requests.
+**Cause**: Cold start + filling Schema Cache.
+**Fix**: The system now uses **Request Coalescing**. If it persists, increase `concurrency` in `workerPool` config.
+
+### "Zombie" Computations
+**Symptom**: A computation stuck in "Running" state for hours.
+**Cause**: Function timeout or crash before writing status.
+**Fix**: Run the `status` command. The system automatically detects stale locks (> 1 hour) and allows re-execution.

package/functions/computation-system-v2/handlers/scheduler.js

@@ -55,8 +55,10 @@ async function schedulerHandler(req, res) {
         let zombies = [];
         try {
             zombies = await storageManager.findZombies(ZOMBIE_THRESHOLD_MINUTES);
+            // New Code
             if (zombies.length > 0) {
-                console.log(`[Scheduler] DETECTED ${zombies.length} ZOMBIES: ${zombies.map(z => z.name).join(', ')}`);
+                const zombieDetails = zombies.map(z => `${z.name} [${z.date}]`).join(', ');
+                console.log(`[Scheduler] DETECTED ${zombies.length} ZOMBIES: ${zombieDetails}`);
             }
         } catch (e) {
             console.error(`[Scheduler] Zombie check failed: ${e.message}`);
@@ -136,10 +138,14 @@ function isScheduleDue(schedule, currentTime, dayOfWeek, dayOfMonth) {
 
 async function dispatchComputations(computations, defaultDate, scheduledTime) {
     const limit = pLimit(CLOUD_TASKS_CONCURRENCY);
-    const { projectId, location, queueName, dispatcherUrl } = config.cloudTasks;
+    const { projectId, location, queueName, dispatcherUrl, serviceAccountEmail } = config.cloudTasks; // Ensure serviceAccountEmail is destructured
     const queuePath = tasksClient.queuePath(projectId, location, queueName);
     const timeSlot = formatTimeCompact(scheduledTime);
     
+    // Log the configuration ONCE at the start of dispatch to verify
+    console.log(`[Scheduler] Dispatching to Queue: ${queuePath}`);
+    console.log(`[Scheduler] Using OIDC Service Account: ${serviceAccountEmail}`);
+    
     const tasks = computations.map(entry => limit(async () => {
         try {
             // Determine date: Zombies use their original stuck date, normal tasks use today
@@ -170,7 +176,7 @@ async function dispatchComputations(computations, defaultDate, scheduledTime) {
                     headers: { 'Content-Type': 'application/json' },
                     body: Buffer.from(JSON.stringify(taskPayload)).toString('base64'),
                     oidcToken: {
-                        serviceAccountEmail: config.cloudTasks.serviceAccountEmail,
+                        serviceAccountEmail: serviceAccountEmail, // Use the destructured variable
                         audience: dispatcherUrl
                     }
                 },
@@ -186,9 +192,26 @@ async function dispatchComputations(computations, defaultDate, scheduledTime) {
             };
             
         } catch (error) {
-            if (error.code === 6) { // ALREADY_EXISTS
+            // EXISTING HANDLE: Duplicate tasks
+            if (error.code === 6) { 
                 return { computation: entry.originalName, status: 'skipped', reason: 'duplicate' };
             }
+
+            // NEW HANDLE: NOT_FOUND (Configuration Errors)
+            if (error.code === 5) {
+                console.error(`[Scheduler] 🚨 CONFIGURATION ERROR: 5 NOT_FOUND`);
+                console.error(`[Scheduler] Check 1: Does Queue exist? "${queuePath}"`);
+                console.error(`[Scheduler] Check 2: Does Service Account exist? "${serviceAccountEmail}"`);
+                console.error(`[Scheduler] Raw Error: ${error.message}`);
+                
+                return { 
+                    computation: entry.originalName, 
+                    status: 'error', 
+                    error: `Configuration Error: Queue or Service Account not found. (${error.message})`
+                };
+            }
+
+            // General Errors
             console.error(`[Scheduler] Failed to dispatch ${entry.originalName}:`, error.message);
             return { computation: entry.originalName, status: 'error', error: error.message };
         }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bulltrackers-module",
-  "version": "1.0.741",
+  "version": "1.0.743",
   "description": "Helper Functions for Bulltrackers.",
   "main": "index.js",
   "files": [