bulltrackers-module 1.0.751 → 1.0.753

package/functions/computation-system-v2/scripts/sh.md

@@ -0,0 +1,115 @@
+Here is the comprehensive documentation for the **Admin Test Endpoint (`compute-admin-test`)**. This guide details the supported actions, available overrides, and their specific impacts on your infrastructure.
+
+### **Overview**
+
+The Admin Test Endpoint is a privileged tool designed for **safe, isolated testing** of computations in production. It allows you to trigger runs manually, bypass scheduling locks, and—most importantly—**divert execution to test infrastructure** (custom tables or worker pools) to prevent polluting production data.
+
+---
+
+### **Global Overrides (Infrastructure Testing)**
+
+These parameters can be applied to **`run`** and **`run_limited`** actions to redirect the execution flow.
+
+| Parameter | Type | Description | Impact |
+| --- | --- | --- | --- |
+| **`outputTable`** | `string` | The BigQuery table where results will be written. | **Data Diversion**: Instead of writing to `computation_results_v3`, the Orchestrator writes to this table. Useful for validating logic without affecting production dashboards. |
+| **`workerUrl`** | `string` | The URL of the worker Cloud Function to invoke. | **Traffic Diversion**: If the worker pool is used, the Orchestrator will send HTTP requests to this URL instead of the production worker. |
+| **`useWorkerPool`** | `boolean` | Force enable/disable the remote worker pool. | **Execution Strategy**: <br>
+
+<br>`true`: Forces remote execution (even for small batches).<br>
+
+<br>`false`: Forces local execution (inside the Admin function).<br>
+
+<br>`undefined`: Uses system default config. |
+
+---
+
+### **Supported Actions**
+
+#### **1. `status**`
+
+Returns the current system manifest, listing all registered computations and their schedules.
+
+* **Configurables:** None.
+* **Data Written:** None.
+* **Impact:** Read-only. Low impact.
+* **Cloud Functions:** `compute-admin-test` (Local).
+
+#### **2. `analyze**`
+
+Runs the "Scheduler Logic" for a specific date to determine what *would* run, without actually running it. Checks dependencies, hash changes, and locks.
+
+* **Configurables:**
+* `date` (YYYY-MM-DD): The target date to analyze.
+
+
+* **Data Written:** None.
+* **Impact:** Read-only. Low impact.
+* **Cloud Functions:** `compute-admin-test` (Local).
+
+#### **3. `run**`
+
+Executes a full computation for the specified date. This is the primary tool for manual triggers and backfills.
+
+* **Configurables:**
+* `computation` (Required): Name of the computation (e.g., `UserPortfolioSummary`).
+* `date`: Target date (Default: Today).
+* `entityIds` (Array): Run only for these specific entities.
+* `force` (Boolean): If `true`, runs even if the code/data hasn't changed (Default: `true` for testing).
+* `dryRun` (Boolean): If `true`, computes results but **does not write to BigQuery**.
+* *Plus Global Overrides (`outputTable`, `workerUrl`, `useWorkerPool`)*.
+
+
+* **Data Written:**
+* **Default:** Writes to `config.resultStore.table` (Production).
+* **With `outputTable`:** Writes to the specified custom table.
+* **With `dryRun`:** No data written.
+
+
+* **Impact:** High. Can trigger heavy BigQuery queries and write large amounts of data.
+* **Cloud Functions:**
+* **Local Mode:** `compute-admin-test` processes all logic.
+* **Worker Pool Mode:** `compute-admin-test` acts as the Orchestrator; `computation-worker` (or `workerUrl`) executes the logic.
+
+
+
+#### **4. `run_limited**`
+
+A safer version of `run` that automatically fetches a small sample of entities (e.g., 5 random users) and runs the computation only for them.
+
+* **Configurables:**
+* `computation` (Required): Name of computation.
+* `limit` (Integer): Number of entities to test (Default: 10).
+* *Plus Global Overrides (`outputTable`, `workerUrl`, `useWorkerPool`)*.
+
+
+* **Data Written:** Same behavior as `run`, but only for the sampled entities.
+* **Impact:** Moderate. Executes real logic but on a strictly limited scope.
+* **Cloud Functions:** Same as `run`.
+
+#### **5. `test_worker**`
+
+Directly tests the **logic** of a worker execution locally within the Admin function. This bypasses the Orchestrator's batching and storage logic, simulating exactly what happens inside a single worker instance.
+
+* **Configurables:**
+* `computation` (Required): Name of computation.
+* `entityIds` (Required Array): Must provide at least one ID to test.
+* `date`: Target date.
+
+
+* **Data Written:** **None**. The result is returned directly in the HTTP response body for inspection.
+* **Impact:** Low. Fetches data for 1 entity and runs logic in-memory.
+* **Cloud Functions:** `compute-admin-test` (Local only).
+
+---
+
+### **Summary of Data Flow**
+
+| Scenario | Execution Location | Data Destination |
+| --- | --- | --- |
+| **Standard Run** | `compute-admin-test` (Local) | `computation_results_v3` (Prod) |
+| **Standard Run + Worker Pool** | `computation-worker` (Remote) | `computation_results_v3` (Prod) |
+| **Run + `outputTable**` | `compute-admin-test` (Local) | **`YOUR_CUSTOM_TABLE`** |
+| **Run + `workerUrl**` | **`worker-test`** (Remote) | `computation_results_v3` (Prod) |
+| **Run + `dryRun**` | `compute-admin-test` (Local) | *None* |
+| **Run + `outputTable` + `workerUrl**` | **`worker-test`** (Remote) | **`YOUR_CUSTOM_TABLE`** |

package/functions/computation-system-v2/scripts/test-admin.sh

@@ -95,6 +95,7 @@ while true; do
     echo "5) Test Specific Entities (run with entityIds)"
     echo "6) Test Worker Logic Directly (test_worker)"
     echo "7) Test Worker Pool Offloading (run with useWorkerPool)"
+    echo "8) Advanced Infrastructure Test (Custom Table/Worker)"
     echo "q) Quit"
     
     read -p "Enter choice: " choice
@@ -167,6 +168,45 @@ while true; do
             echo -e "${YELLOW}Running with Worker Pool enabled...${NC}"
             run_test "{\"action\": \"run\", \"computation\": \"$COMP_NAME\", \"date\": \"$TARGET_DATE\", \"useWorkerPool\": true, \"force\": true}"
             ;;
+
+        8)
+            # Advanced Infrastructure Test (Custom Table & Worker)
+            echo -e "${YELLOW}--- Infrastructure Integration Test ---${NC}"
+            ask_var "Enter Computation Name" "UserPortfolioSummary" "COMP_NAME"
+            ask_var "Enter Date" "$DEFAULT_DATE" "TARGET_DATE"
+            ask_var "Enter Output Table Name" "computation_results_test" "OUT_TABLE"
+            ask_var "Enter Custom Worker URL (optional)" "" "WORKER_URL"
+            ask_var "Use Worker Pool? (true/false)" "true" "USE_POOL"
+            ask_var "Entity Limit (0 for Full Run)" "10" "LIMIT_NUM"
+
+            # Determine Action (run vs run_limited)
+            if [ "$LIMIT_NUM" -gt 0 ]; then
+                ACTION="run_limited"
+                LIMIT_PART=", \"limit\": $LIMIT_NUM"
+            else
+                ACTION="run"
+                LIMIT_PART=", \"force\": true"
+            fi
+
+            # Build Optional JSON Parts
+            TABLE_PART=""
+            if [ ! -z "$OUT_TABLE" ]; then
+                TABLE_PART=", \"outputTable\": \"$OUT_TABLE\""
+            fi
+
+            WORKER_PART=""
+            if [ ! -z "$WORKER_URL" ]; then
+                WORKER_PART=", \"workerUrl\": \"$WORKER_URL\""
+            fi
+
+            echo -e "${YELLOW}Running ${ACTION} on ${OUT_TABLE}...${NC}"
+            
+            # Construct Payload
+            # We use loose concatenation here which works fine for JSON string building in bash
+            JSON="{\"action\": \"$ACTION\", \"computation\": \"$COMP_NAME\", \"date\": \"$TARGET_DATE\", \"useWorkerPool\": $USE_POOL $LIMIT_PART $TABLE_PART $WORKER_PART}"
+            
+            run_test "$JSON"
+            ;;
             
         q|Q)
             echo "Exiting."

package/functions/orchestrator/index.js

@@ -16,6 +16,9 @@ const QUEUE_NAME = process.env.ORCHESTRATOR_QUEUE || 'task-engine-queue';
 const LOCATION = process.env.GCP_REGION || 'europe-west1'; 
 const PROJECT = process.env.GCP_PROJECT_ID;
 
+// --- FEATURE FLAG: Disable Normal/Speculator Users ---
+const ENABLE_LEGACY_USERS = process.env.ENABLE_LEGACY_USERS === 'true';
+
 /**
  * ENTRY POINT: HTTP Handler for Workflow Interaction
  */
@@ -33,6 +36,14 @@ async function handleOrchestratorHttp(req, res, dependencies, config) {
                 throw new Error("Missing userType or date for PLAN action");
             }
             
+            // --- NEW: Block Legacy Users if Disabled ---
+            if ((userType === 'normal' || userType === 'speculator') && !ENABLE_LEGACY_USERS) {
+                const msg = `[Orchestrator] SKIPPING PLAN for '${userType}': ENABLE_LEGACY_USERS is false.`;
+                logger.log('WARN', msg);
+                // Return 200 to prevent retry loops in workflows
+                return res.status(200).send({ status: 'skipped', message: msg });
+            }
+
             // Determine self-URL for callback (Cloud Task needs to call this function back)
             // We use the env var passed by GCF (FUNCTION_URI) or construct it manually
             const orchestratorUrl = orchestratorUrlOverride || 
@@ -47,6 +58,14 @@ async function handleOrchestratorHttp(req, res, dependencies, config) {
             if (!planId || !windowId) {
                 throw new Error("Missing planId or windowId for EXECUTE_WINDOW action");
             }
+            
+            // --- NEW: Block Legacy Users if Disabled (Double Check) ---
+            if ((userType === 'normal' || userType === 'speculator') && !ENABLE_LEGACY_USERS) {
+                 const msg = `[Orchestrator] SKIPPING EXECUTE_WINDOW for '${userType}': ENABLE_LEGACY_USERS is false.`;
+                 logger.log('WARN', msg);
+                 return res.status(200).send({ status: 'skipped', message: msg });
+            }
+
             const result = await executeUpdateWindow(planId, windowId, userType, config, dependencies);
             res.status(200).send(result);
 
@@ -227,8 +246,13 @@ async function runDiscoveryOrchestrator(config, deps) {
     const { logger, firestoreUtils } = deps;
     logger.log('INFO', '🚀 Discovery Orchestrator triggered...');
     await firestoreUtils.resetProxyLocks(deps, config);
-    if (isUserTypeEnabled('normal', config.enabledUserTypes)) await runDiscovery('normal', config.discoveryConfig.normal, config, deps);
-    if (isUserTypeEnabled('speculator', config.enabledUserTypes)) await runDiscovery('speculator', config.discoveryConfig.speculator, config, deps);
+    
+    if (ENABLE_LEGACY_USERS) {
+        if (isUserTypeEnabled('normal', config.enabledUserTypes)) await runDiscovery('normal', config.discoveryConfig.normal, config, deps);
+        if (isUserTypeEnabled('speculator', config.enabledUserTypes)) await runDiscovery('speculator', config.discoveryConfig.speculator, config, deps);
+    } else {
+        logger.log('INFO', 'Discovery skipped for legacy users (normal/speculator) because ENABLE_LEGACY_USERS is false.');
+    }
 }
 
 async function runUpdateOrchestrator(config, deps) {
@@ -237,8 +261,13 @@ async function runUpdateOrchestrator(config, deps) {
     await firestoreUtils.resetProxyLocks(deps, config);
     const enabledTypes = config.enabledUserTypes || [];
 
-    if (isUserTypeEnabled('normal', enabledTypes)) await runUpdates('normal', config.updateConfig, config, deps);
-    if (isUserTypeEnabled('speculator', enabledTypes)) await runUpdates('speculator', config.updateConfig, config, deps);
+    if (ENABLE_LEGACY_USERS) {
+        if (isUserTypeEnabled('normal', enabledTypes)) await runUpdates('normal', config.updateConfig, config, deps);
+        if (isUserTypeEnabled('speculator', enabledTypes)) await runUpdates('speculator', config.updateConfig, config, deps);
+    } else {
+        logger.log('INFO', 'Updates skipped for legacy users (normal/speculator) because ENABLE_LEGACY_USERS is false.');
+    }
+
     if (isUserTypeEnabled('popular_investor', enabledTypes)) {
         const piConfig = { ...config.updateConfig, popularInvestorRankingsCollection: config.updateConfig.popularInvestorRankingsCollection || 'popular_investor_rankings' };
         await runUpdates('popular_investor', piConfig, config, deps);

package/functions/task-engine/tests/test-task-engine.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+# ==============================================================================
+# BULLTRACKERS TASK ENGINE END-TO-END TESTER
+# This script triggers the Orchestrator to plan an immediate execution window.
+# ==============================================================================
+
+# --- CONFIGURATION ---
+FUNCTION_NAME="orchestrator-http"
+REGION="europe-west1"
+DATE=$(date +%Y-%m-%d) # Defaults to today
+USER_TYPE="normal"     # Options: normal, speculator, popular_investor
+WINDOWS=1              # 1 window = immediate execution (0s delay)
+
+# --- 1. FETCH URL DYNAMICALLY ---
+echo "🔍 Fetching URL for function: $FUNCTION_NAME ($REGION)..."
+
+# Try Gen 2 (Cloud Run) URL first
+URL=$(gcloud functions describe $FUNCTION_NAME --region=$REGION --format='value(serviceConfig.uri)' 2>/dev/null)
+
+# Fallback to Gen 1 if empty
+if [ -z "$URL" ]; then
+    URL=$(gcloud functions describe $FUNCTION_NAME --region=$REGION --format='value(httpsTrigger.url)' 2>/dev/null)
+fi
+
+if [ -z "$URL" ]; then
+    echo "❌ Error: Could not find URL for function '$FUNCTION_NAME'. Check if it is deployed."
+    exit 1
+fi
+
+echo "✅ Target URL: $URL"
+
+# --- 2. GET AUTH TOKEN ---
+echo "🔑 Generating Identity Token..."
+TOKEN=$(gcloud auth print-identity-token)
+
+if [ -z "$TOKEN" ]; then
+    echo "❌ Error: Could not generate token. Run 'gcloud auth login' first."
+    exit 1
+fi
+
+# --- 3. SEND REQUEST ---
+echo "🚀 Triggering Plan for $USER_TYPE on $DATE ($WINDOWS window)..."
+
+RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$URL" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "{
+    \"action\": \"PLAN\",
+    \"userType\": \"$USER_TYPE\",
+    \"date\": \"$DATE\",
+    \"windows\": $WINDOWS
+  }")
+
+# --- 4. PARSE RESPONSE ---
+HTTP_BODY=$(echo "$RESPONSE" | head -n -1)
+HTTP_CODE=$(echo "$RESPONSE" | tail -n 1)
+
+if [ "$HTTP_CODE" -eq 200 ]; then
+    echo ""
+    echo "✅ SUCCESS (HTTP 200)"
+    echo "---------------------------------------------------"
+    echo "$HTTP_BODY" | python3 -m json.tool 2>/dev/null || echo "$HTTP_BODY"
+    echo "---------------------------------------------------"
+    echo "👉 Monitor 'task-engine-queue' in Cloud Tasks Console."
+    echo "👉 Check Logs Explorer for 'Orchestrator' and 'Dispatcher'."
+else
+    echo ""
+    echo "❌ FAILED (HTTP $HTTP_CODE)"
+    echo "---------------------------------------------------"
+    echo "$HTTP_BODY"
+    echo "---------------------------------------------------"
+fi

package/index.js

@@ -1,6 +1,7 @@
 /**
  * @fileoverview Main entry point for the Bulltrackers shared module.
  * CLEANED: Removed legacy V1 Dispatcher/Computation references.
+ * UPDATED: Re-integrated Dispatcher (Task Throttler).
  */
 
 // Core utilities
@@ -21,7 +22,6 @@ const { checkDiscoveryNeed, getDiscoveryCandidates, dispatchDiscovery } = requir
 const { getUpdateTargets, dispatchUpdates }                             = require('./functions/orchestrator/helpers/update_helpers');
 
 // --- COMPUTATION SYSTEM V2 (The new standard) ---
-// We import the WHOLE package now, which includes handlers AND ManifestBuilder
 const computationSystemV2 = require('./functions/computation-system-v2/index');
 
 // Task Engine
@@ -30,6 +30,9 @@ const { handleDiscover }                                  = require('./functions
 const { handleVerify }                                    = require('./functions/task-engine/helpers/verify_helpers');
 const { handleUpdate }                                    = require('./functions/task-engine/helpers/update_helpers');
 
+// Dispatcher (Task Throttler)
+const { handleRequest: dispatcherRequest }                = require('./functions/dispatcher/index');
+
 const { createApiV2App } = require('./functions/api-v2/index');
 
 // Maintenance & Backfills
@@ -76,7 +79,9 @@ const orchestrator = {
   dispatchUpdates,
 };
 
-// [REMOVED] Legacy 'dispatcher' pipe. It is replaced by computationSystemV2.
+const dispatcher = {
+  handleRequest: dispatcherRequest
+};
 
 const taskEngine = {
   handleRequest: taskRequest,
@@ -85,7 +90,7 @@ const taskEngine = {
   handleUpdate,
 };
 
-const computationSystem = computationSystemV2; // Direct mapping
+const computationSystem = computationSystemV2; 
 
 const api = {
   createApiV2App, 
@@ -119,5 +124,5 @@ const alertSystem = {
 };
 
 module.exports = {
-  pipe: { core, orchestrator, taskEngine, computationSystem, api, maintenance, proxy, alertSystem },
+  pipe: { core, orchestrator, dispatcher, taskEngine, computationSystem, api, maintenance, proxy, alertSystem },
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bulltrackers-module",
-  "version": "1.0.751",
+  "version": "1.0.753",
   "description": "Helper Functions for Bulltrackers.",
   "main": "index.js",
   "files": [