bulltrackers-module 1.0.751 → 1.0.752

package/functions/computation-system-v2/scripts/sh.md

@@ -0,0 +1,115 @@
+Here is the comprehensive documentation for the **Admin Test Endpoint (`compute-admin-test`)**. This guide details the supported actions, available overrides, and their specific impacts on your infrastructure.
+
+### **Overview**
+
+The Admin Test Endpoint is a privileged tool designed for **safe, isolated testing** of computations in production. It allows you to trigger runs manually, bypass scheduling locks, and—most importantly—**divert execution to test infrastructure** (custom tables or worker pools) to prevent polluting production data.
+
+---
+
+### **Global Overrides (Infrastructure Testing)**
+
+These parameters can be applied to **`run`** and **`run_limited`** actions to redirect the execution flow.
+
+| Parameter | Type | Description | Impact |
+| --- | --- | --- | --- |
+| **`outputTable`** | `string` | The BigQuery table where results will be written. | **Data Diversion**: Instead of writing to `computation_results_v3`, the Orchestrator writes to this table. Useful for validating logic without affecting production dashboards. |
+| **`workerUrl`** | `string` | The URL of the worker Cloud Function to invoke. | **Traffic Diversion**: If the worker pool is used, the Orchestrator will send HTTP requests to this URL instead of the production worker. |
+| **`useWorkerPool`** | `boolean` | Force enable/disable the remote worker pool. | **Execution Strategy**: <br>
+
+<br>`true`: Forces remote execution (even for small batches).<br>
+
+<br>`false`: Forces local execution (inside the Admin function).<br>
+
+<br>`undefined`: Uses system default config. |
+
+---
+
+### **Supported Actions**
+
+#### **1. `status**`
+
+Returns the current system manifest, listing all registered computations and their schedules.
+
+* **Configurables:** None.
+* **Data Written:** None.
+* **Impact:** Read-only. Low impact.
+* **Cloud Functions:** `compute-admin-test` (Local).
+
+#### **2. `analyze**`
+
+Runs the "Scheduler Logic" for a specific date to determine what *would* run, without actually running it. Checks dependencies, hash changes, and locks.
+
+* **Configurables:**
+* `date` (YYYY-MM-DD): The target date to analyze.
+
+
+* **Data Written:** None.
+* **Impact:** Read-only. Low impact.
+* **Cloud Functions:** `compute-admin-test` (Local).
+
+#### **3. `run**`
+
+Executes a full computation for the specified date. This is the primary tool for manual triggers and backfills.
+
+* **Configurables:**
+* `computation` (Required): Name of the computation (e.g., `UserPortfolioSummary`).
+* `date`: Target date (Default: Today).
+* `entityIds` (Array): Run only for these specific entities.
+* `force` (Boolean): If `true`, runs even if the code/data hasn't changed (Default: `true` for testing).
+* `dryRun` (Boolean): If `true`, computes results but **does not write to BigQuery**.
+* *Plus Global Overrides (`outputTable`, `workerUrl`, `useWorkerPool`)*.
+
+
+* **Data Written:**
+* **Default:** Writes to `config.resultStore.table` (Production).
+* **With `outputTable`:** Writes to the specified custom table.
+* **With `dryRun`:** No data written.
+
+
+* **Impact:** High. Can trigger heavy BigQuery queries and write large amounts of data.
+* **Cloud Functions:**
+* **Local Mode:** `compute-admin-test` processes all logic.
+* **Worker Pool Mode:** `compute-admin-test` acts as the Orchestrator; `computation-worker` (or `workerUrl`) executes the logic.
+
+
+
+#### **4. `run_limited**`
+
+A safer version of `run` that automatically fetches a small sample of entities (e.g., 5 random users) and runs the computation only for them.
+
+* **Configurables:**
+* `computation` (Required): Name of computation.
+* `limit` (Integer): Number of entities to test (Default: 10).
+* *Plus Global Overrides (`outputTable`, `workerUrl`, `useWorkerPool`)*.
+
+
+* **Data Written:** Same behavior as `run`, but only for the sampled entities.
+* **Impact:** Moderate. Executes real logic but on a strictly limited scope.
+* **Cloud Functions:** Same as `run`.
+
+#### **5. `test_worker**`
+
+Directly tests the **logic** of a worker execution locally within the Admin function. This bypasses the Orchestrator's batching and storage logic, simulating exactly what happens inside a single worker instance.
+
+* **Configurables:**
+* `computation` (Required): Name of computation.
+* `entityIds` (Required Array): Must provide at least one ID to test.
+* `date`: Target date.
+
+
+* **Data Written:** **None**. The result is returned directly in the HTTP response body for inspection.
+* **Impact:** Low. Fetches data for 1 entity and runs logic in-memory.
+* **Cloud Functions:** `compute-admin-test` (Local only).
+
+---
+
+### **Summary of Data Flow**
+
+| Scenario | Execution Location | Data Destination |
+| --- | --- | --- |
+| **Standard Run** | `compute-admin-test` (Local) | `computation_results_v3` (Prod) |
+| **Standard Run + Worker Pool** | `computation-worker` (Remote) | `computation_results_v3` (Prod) |
+| **Run + `outputTable**` | `compute-admin-test` (Local) | **`YOUR_CUSTOM_TABLE`** |
+| **Run + `workerUrl**` | **`worker-test`** (Remote) | `computation_results_v3` (Prod) |
+| **Run + `dryRun**` | `compute-admin-test` (Local) | *None* |
+| **Run + `outputTable` + `workerUrl**` | **`worker-test`** (Remote) | **`YOUR_CUSTOM_TABLE`** |

package/functions/computation-system-v2/scripts/test-admin.sh

@@ -95,6 +95,7 @@ while true; do
     echo "5) Test Specific Entities (run with entityIds)"
     echo "6) Test Worker Logic Directly (test_worker)"
     echo "7) Test Worker Pool Offloading (run with useWorkerPool)"
+    echo "8) Advanced Infrastructure Test (Custom Table/Worker)"
     echo "q) Quit"
     
     read -p "Enter choice: " choice
@@ -167,6 +168,45 @@ while true; do
             echo -e "${YELLOW}Running with Worker Pool enabled...${NC}"
             run_test "{\"action\": \"run\", \"computation\": \"$COMP_NAME\", \"date\": \"$TARGET_DATE\", \"useWorkerPool\": true, \"force\": true}"
             ;;
+
+        8)
+            # Advanced Infrastructure Test (Custom Table & Worker)
+            echo -e "${YELLOW}--- Infrastructure Integration Test ---${NC}"
+            ask_var "Enter Computation Name" "UserPortfolioSummary" "COMP_NAME"
+            ask_var "Enter Date" "$DEFAULT_DATE" "TARGET_DATE"
+            ask_var "Enter Output Table Name" "computation_results_test" "OUT_TABLE"
+            ask_var "Enter Custom Worker URL (optional)" "" "WORKER_URL"
+            ask_var "Use Worker Pool? (true/false)" "true" "USE_POOL"
+            ask_var "Entity Limit (0 for Full Run)" "10" "LIMIT_NUM"
+
+            # Determine Action (run vs run_limited)
+            if [ "$LIMIT_NUM" -gt 0 ]; then
+                ACTION="run_limited"
+                LIMIT_PART=", \"limit\": $LIMIT_NUM"
+            else
+                ACTION="run"
+                LIMIT_PART=", \"force\": true"
+            fi
+
+            # Build Optional JSON Parts
+            TABLE_PART=""
+            if [ ! -z "$OUT_TABLE" ]; then
+                TABLE_PART=", \"outputTable\": \"$OUT_TABLE\""
+            fi
+
+            WORKER_PART=""
+            if [ ! -z "$WORKER_URL" ]; then
+                WORKER_PART=", \"workerUrl\": \"$WORKER_URL\""
+            fi
+
+            echo -e "${YELLOW}Running ${ACTION} on ${OUT_TABLE}...${NC}"
+            
+            # Construct Payload
+            # We use loose concatenation here which works fine for JSON string building in bash
+            JSON="{\"action\": \"$ACTION\", \"computation\": \"$COMP_NAME\", \"date\": \"$TARGET_DATE\", \"useWorkerPool\": $USE_POOL $LIMIT_PART $TABLE_PART $WORKER_PART}"
+            
+            run_test "$JSON"
+            ;;
             
         q|Q)
             echo "Exiting."

package/index.js

@@ -1,6 +1,7 @@
 /**
  * @fileoverview Main entry point for the Bulltrackers shared module.
  * CLEANED: Removed legacy V1 Dispatcher/Computation references.
+ * UPDATED: Re-integrated Dispatcher (Task Throttler).
  */
 
 // Core utilities
@@ -21,7 +22,6 @@ const { checkDiscoveryNeed, getDiscoveryCandidates, dispatchDiscovery } = requir
 const { getUpdateTargets, dispatchUpdates }                             = require('./functions/orchestrator/helpers/update_helpers');
 
 // --- COMPUTATION SYSTEM V2 (The new standard) ---
-// We import the WHOLE package now, which includes handlers AND ManifestBuilder
 const computationSystemV2 = require('./functions/computation-system-v2/index');
 
 // Task Engine
@@ -30,6 +30,9 @@ const { handleDiscover }                                  = require('./functions
 const { handleVerify }                                    = require('./functions/task-engine/helpers/verify_helpers');
 const { handleUpdate }                                    = require('./functions/task-engine/helpers/update_helpers');
 
+// Dispatcher (Task Throttler)
+const { handleRequest: dispatcherRequest }                = require('./functions/dispatcher/index');
+
 const { createApiV2App } = require('./functions/api-v2/index');
 
 // Maintenance & Backfills
@@ -76,7 +79,9 @@ const orchestrator = {
   dispatchUpdates,
 };
 
-// [REMOVED] Legacy 'dispatcher' pipe. It is replaced by computationSystemV2.
+const dispatcher = {
+  handleRequest: dispatcherRequest
+};
 
 const taskEngine = {
   handleRequest: taskRequest,
@@ -85,7 +90,7 @@ const taskEngine = {
   handleUpdate,
 };
 
-const computationSystem = computationSystemV2; // Direct mapping
+const computationSystem = computationSystemV2; 
 
 const api = {
   createApiV2App, 
@@ -119,5 +124,5 @@ const alertSystem = {
 };
 
 module.exports = {
-  pipe: { core, orchestrator, taskEngine, computationSystem, api, maintenance, proxy, alertSystem },
+  pipe: { core, orchestrator, dispatcher, taskEngine, computationSystem, api, maintenance, proxy, alertSystem },
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bulltrackers-module",
-  "version": "1.0.751",
+  "version": "1.0.752",
   "description": "Helper Functions for Bulltrackers.",
   "main": "index.js",
   "files": [