bulltrackers-module 1.0.781 → 1.0.783

package/functions/computation-system-v3/DOCS/Computations.MD

@@ -0,0 +1,235 @@
+---
+
+# V3 Computation Development Guide
+
+## 1. Core Philosophy
+
+The V3 architecture is based on **Functional Composition**. A computation is a "Recipe" consisting of:
+
+1. **Config:** Metadata, data requirements, and dependency graph definitions.
+2. **Process:** A stateless, pure function that transforms inputs into outputs.
+
+**System Constraints:**
+
+* **Stateless:** Computations must not maintain internal state between runs.
+* **Injection-Based:** Do not use `require()` for system libraries. All utilities are injected into the context.
+* **Isolated:** Computations cannot directly access the database or network. They must request data via `config.requires`.
+
+---
+
+## 2. File Structure
+
+Every computation is a Node.js module that must export exactly two properties: `config` and `process`.
+
+```javascript
+/**
+ * Standard V3 Recipe Structure
+ */
+module.exports = {
+    config: {
+        // ... Metadata and Data Requirements
+    },
+    process: ({ data, lib, entityId, date, log }) => {
+        // ... Business Logic
+        return { /* Result Object */ };
+    }
+};
+
+```
+
+---
+
+## 3. Configuration Schema (`config`)
+
+The `config` object defines how the system prepares the execution environment.
+
+| Field | Type | Required | Description |
+| --- | --- | --- | --- |
+| `name` | `string` | **Yes** | Unique CamelCase identifier (e.g., `UserVolatilityMetrics`). |
+| `type` | `string` | **Yes** | `'per-entity'` (runs batch-wise per user/symbol) or `'global'` (runs once). |
+| `dependencies` | `string[]` | No | Array of other computation names that must finish before this one runs. |
+| `requires` | `Object` | No | Data fetching specification (see below). |
+| `storage` | `Object` | No | Output destinations (defaults to BigQuery enabled). |
+
+### 3.1 Data Requirements (`requires`)
+
+Define what data the `BigQueryAdapter` must fetch. Keys correspond to table names or source identifiers.
+
+```javascript
+requires: {
+    // Example: Fetch raw portfolio snapshots
+    "portfolio_snapshots": {
+        fields: ["user_id", "timestamp", "equity", "positions"], // Columns to select
+        lookback: 30,       // Days of history needed (0 = current date only)
+        mandatory: true,    // If true, blocks execution if data is missing
+        filter: {           // SQL WHERE clause equivalents
+            "status": "active"
+        }
+    },
+    // Example: Fetch a metric from a dependency
+    "UserRiskScore": {
+        type: "metric",     // Special type for previous computation results
+        source: "UserRiskScore",
+        lookback: 0
+    }
+}
+
+```
+
+---
+
+## 4. The Process Function (`process`)
+
+The logic kernel. It receives a strictly defined `context` object.
+
+**Signature:**
+`(context) => Object | null`
+
+### 4.1 The Context Object
+
+| Property | Description |
+| --- | --- |
+| `data` | Object containing arrays of rows keyed by the `requires` alias. |
+| `lib` | The Standard Library (see Section 5). |
+| `entityId` | The ID of the current entity being processed (if `type: 'per-entity'`). |
+| `date` | The target execution date string (`YYYY-MM-DD`). |
+| `log` | Helper function for structured logging. |
+
+### 4.2 Return Value
+
+* **Success:** Return a flat JSON object. Keys become columns in the results table.
+* **Skip:** Return `null` if the entity should be skipped (e.g., insufficient history).
+* **Failure:** Throw an `Error` to mark the entity as failed (will be logged in `computation_errors`).
+
+---
+
+## 5. The Standard Library (`lib`)
+
+Do not write custom math or data parsing logic if a library function exists. The `lib` object is auto-injected.
+
+**Available Modules:**
+
+* **`lib.math`**: `mean`, `standardDeviation`, `round`, `clamp`, `percentileRank`.
+* **`lib.finance`**: `calculateSharpeRatio`, `calculateMaxDrawdown`, `downsideDeviation`.
+* **`lib.time`**: `parseDate`, `formatDate`, `addDays`, `subtractDays`, `getDateRange`.
+* **`lib.portfolio`**: `extractPositions`, `calculateTotalValue`, `calculateSectorExposure`, `groupBySector`.
+* **`lib.trades`**: `calculateTradeStats`, `filterByDateRange`, `groupByInstrument`.
+* **`lib.social`**: `getPostEngagement`, `getRecentPosts`, `calculateEngagement`.
+* **`lib.utils`**: General helpers.
+
+The above is not exhaustive. Review the current lib to see available functions.
+
+---
+
+## 6. Rules: Do's and Don'ts
+
+### DO:
+
+1. **Use `lib` functions:** Always prefer `lib.math.mean(arr)` over writing a loop to sum values.
+2. **Handle missing data:** Check if `data.source_table` exists and has length before accessing index 0.
+3. **Return JSON:** Ensure the output is serializable (no Functions or undefined values).
+4. **Use `entityId`:** In `'per-entity'` mode, use `entityId` to correlate data if manually filtering (though the adapter usually handles this).
+
+### DON'T:
+
+1. **NO `require()`:** Never import external modules. All dependencies must be in `lib`.
+2. **NO `this`:** Never rely on class instance context.
+3. **NO Side Effects:** Do not attempt to write to files, call external APIs, or modify the global scope.
+4. **NO Hardcoded Dates:** Always use `ctx.date` relative to the execution.
+
+---
+
+## 7. Example Computation
+
+**Goal:** Calculate the 30-day volatility of a user's portfolio.
+
+```javascript
+/**
+ * Computation: UserVolatilityMetrics
+ * Calculates annualized volatility based on daily equity changes.
+ */
+module.exports = {
+    // 1. CONFIGURATION
+    config: {
+        name: "UserVolatilityMetrics",
+        type: "per-entity", // Batched execution per user
+        dependencies: [],   // No upstream dependencies
+        requires: {
+            // Requirement 1: Daily portfolio history
+            "portfolio_daily_history": {
+                fields: ["date", "equity_usd"],
+                lookback: 30, // Need 30 days of history
+                mandatory: true
+            }
+        }
+    },
+
+    // 2. PROCESS LOGIC
+    process: ({ data, lib, entityId, date }) => {
+        // Access fetched data
+        const history = data.portfolio_daily_history || [];
+
+        // Validation: Need at least 2 points to calculate volatility
+        if (history.length < 2) {
+            return null; // Skip this entity
+        }
+
+        // Sort by date (ascending) using lib.time
+        // Note: Adapter usually returns sorted, but safety is good
+        history.sort((a, b) => new Date(a.date) - new Date(b.date));
+
+        // Calculate Daily Returns
+        const returns = [];
+        for (let i = 1; i < history.length; i++) {
+            const current = history[i].equity_usd;
+            const prev = history[i - 1].equity_usd;
+            
+            if (prev > 0) {
+                const dailyReturn = (current - prev) / prev;
+                returns.push(dailyReturn);
+            }
+        }
+
+        // Logic: Use Standard Library for Math
+        const stdev = lib.math.standardDeviation(returns);
+        const meanReturn = lib.math.mean(returns);
+
+        // Annualize (assuming 252 trading days)
+        const annualizedVolatility = stdev * Math.sqrt(252);
+
+        // Return Result
+        return {
+            calculation_date: date,
+            data_points: returns.length,
+            daily_mean_return: lib.math.round(meanReturn, 6),
+            daily_stdev: lib.math.round(stdev, 6),
+            annualized_volatility: lib.math.round(annualizedVolatility, 4),
+            is_high_risk: annualizedVolatility > 0.40 // Simple logic
+        };
+    }
+};
+
+```
+
+---
+
+## 8. Development & Testing Procedure
+
+1. **Create File:** Create `computations/MyNewComputation.js`.
+2. **Define Config:** Determine inputs in `requires`.
+3. **Draft Logic:** Write the `process` function using `lib`.
+4. **Unit Test:**
+* Create `tests/unit/MyNewComputation.test.js`.
+* Use `createTestContext` to inject mock data.
+* Assert the output object.
+
+
+5. **Integration Test (Local):**
+* Use `framework/debug-local.js`.
+* Modify the script to target your new computation.
+* Run `node framework/debug-local.js [YYYY-MM-DD] MyNewComputation`.
+
+5. **Schema Verification:**
+* Use `framework/bin/fetch-fixtures.js`
+* This gives the real data from big query, taken from the tables defined in configs, stored into a fixtures directory
+* Run this to allow computations to be tested on real data, locally stored, mocks can be directed to use the local fixtures.

package/functions/computation-system-v3/DOCS/VerificationProtocol.md

@@ -0,0 +1,54 @@
+### The Protocol Documentation
+
+```markdown
+# V3 Deployment Verification Protocol
+
+Before merging to `main` or deploying to Cloud Functions, all changes must pass the **Verification Protocol**. This protocol runs tests in a specific dependency order to fail fast and ensure system integrity.
+
+## 🏃 Running the Protocol
+
+From the project root:
+
+```bash
+# Make the script executable (first time only)
+chmod +x framework/bin/verify-deployment.js
+
+# Run the full protocol
+node framework/bin/verify-deployment.js
+
+```
+
+## 🏗 Protocol Tiers
+
+The script executes test suites in the following order. If a tier fails, the script **aborts immediately**.
+
+### Tier 1: Unit Logic (`framework/tests/unit/`)
+
+* **What it tests:** Individual recipes (e.g., `UserPortfolioMetrics`), math libraries, and utility functions.
+* **Speed:** Very Fast (< 2s).
+* **Goal:** Ensure the *business logic* is correct before checking if the system can run it.
+* **Example Source:** `UserPortfolioMetrics.test.js`
+
+### Tier 2: Planning & Integrity (`framework/tests/planning/`)
+
+* **What it tests:** `Manifest.js`, `TaskScheduler`, and Hashing logic.
+* **Speed:** Fast (~5s).
+* **Goal:** Ensure the *Dependency Graph (DAG)* constructs correctly and the scheduler detects changes accurately.
+* **Example Source:** `test-dispatcher-decision.js`, `test-planner-prediction.js`
+
+### Tier 3: System Simulation (`framework/tests/system/`)
+
+* **What it tests:** The `Coordinator`, `CloudTasks` mocks, and end-to-end flows like `HappyPath` and `CascadingDepth`.
+* **Speed:** Slower (10s+).
+* **Goal:** Ensure the V3 Core acts correctly (dispatching, retrying, state updates) when running the logic.
+* **Example Source:** `HappyPath.test.js`, `CascadingDepth.test.js`
+
+## 🚨 Troubleshooting Failures
+
+* **Unit Failure:** You likely broke a calculation. Check the specific recipe file.
+* **Planning Failure:** You likely introduced a circular dependency or invalid config in a recipe's `index.js`.
+* **System Failure:** The Core infrastructure (Coordinator/Worker) is likely broken, or your mocks are out of sync with reality.
+
+```
+
+```

package/functions/computation-system-v3/DOCS.MD

@@ -0,0 +1,144 @@
+---
+
+# Computation System V3: Debugging & Testing Toolkit
+
+## Overview
+
+System V3 is designed around the principle of **"Trivial Testability."** Unlike V2, you do not need to deploy to Google Cloud, check logs in the console, or wait for scheduled runs to verify your logic.
+
+We provide a **Local Test Harness** that completely mimics the production environment (Coordinator, Storage, State, and Cloud Tasks) directly on your laptop, using real data snapshots.
+
+---
+
+## 🛠 Core CLI Tools
+
+### 1. The Fixture Fetcher
+
+**Script:** `node framework/bin/fetch-fixtures.js [Date]`
+
+This tool connects to BigQuery and downloads a "slice" of production data relevant to the specific date you want to test. It handles lookback windows automatically (e.g., fetching 30 days of price history for a specific date).
+
+* **Usage:**
+```bash
+# Fetch data required to run calculations for Feb 2nd, 2026
+node framework/bin/fetch-fixtures.js 2026-02-02
+
+```
+
+
+* **What it does:**
+* Reads `config/bulltrackers.config.js` to find all table definitions.
+* Executes smart queries (using `DATE_SUB` logic) to get relevant history.
+* Saves data as JSON files in `tests/fixtures/`.
+* **Note:** If no date is provided, it defaults to *today*.
+
+
+
+### 2. The Local Harness
+
+**Script:** `node framework/debug-local.js [Date] [ComputationName]`
+
+This is the main runner. It initializes the V3 Core with **Local Adapters** instead of Cloud Adapters. It reads from your local JSON fixtures and writes results to a local folder.
+
+* **Usage:**
+```bash
+# Run the BehavioralAnomaly recipe for the date we just fetched
+node framework/debug-local.js 2026-02-02 BehavioralAnomaly
+
+```
+
+
+* **What it does:**
+* **Mocks BigQuery:** Reads from `tests/fixtures/{table}.json`.
+* **Mocks Cloud Tasks:** Executes "worker" tasks immediately in-memory (using `setImmediate`).
+* **Mocks State:** Writes status updates to `debug-out/state/`.
+* **Mocks Storage:** Writes final JSON results to `debug-out/results/`.
+
+
+
+---
+
+## 📂 Directory Structure & Outputs
+
+When you run the tools, they interact with these specific directories:
+
+```text
+computation-system-v3/
+├── computations/               # [INPUT] Your Recipe Logic (e.g., BehavioralAnomaly.js)
+│
+├── tests/
+│   └── fixtures/               # [INPUT] Data snapshot from BigQuery
+│       ├── portfolio_snapshots.json
+│       ├── asset_prices.json
+│       └── ...
+│
+├── debug-out/                  # [OUTPUT] Generated by debug-local.js
+│   ├── state/
+│   │   └── computation_state.json  # Tracks status (running -> completed)
+│   │
+│   └── results/                    # The Final JSON Output
+│       └── BehavioralAnomaly_2026-02-02_{timestamp}.json
+│
+└── framework/
+    ├── bin/fetch-fixtures.js   # Tool 1
+    └── debug-local.js          # Tool 2
+
+```
+
+---
+
+## 🚀 Workflow: How to Develop & Test a Recipe
+
+Follow this loop when building or fixing a computation.
+
+### Step 1: Get Real Data
+
+Don't guess what the data looks like. Download a real snapshot.
+
+```bash
+node framework/bin/fetch-fixtures.js 2023-10-25
+
+```
+
+*Tip: Pick a date where you know "interesting" things happened (e.g., a market crash or a user anomaly).*
+
+### Step 2: Write/Modify Your Recipe
+
+Edit your file in `computations/`. Remember, V3 recipes must be **stateless**.
+
+```javascript
+exports.process = ({ data, lib }) => {
+    // ... your logic ...
+};
+
+```
+
+### Step 3: Run Locally
+
+Execute the harness. It runs instantly (no cloud latency).
+
+```bash
+node framework/debug-local.js 2023-10-25 MyRecipeName
+
+```
+
+### Step 4: Verify Output
+
+Open the generated file in `debug-out/results/`.
+
+* Does the JSON structure match what the frontend expects?
+* Are the calculations correct?
+* Did it handle edge cases (nulls, empty arrays)?
+
+---
+
+## 🧠 Best Practices
+
+1. **Trust the Fixtures:**
+The `FileBasedDataAdapter` is smart. It filters data exactly like the production system. If your local run works but production fails, it's likely because your local fixtures are stale. Re-run `fetch-fixtures.js`.
+2. **Commit Small Fixtures:**
+While `fetch-fixtures.js` grabs live data, you should keep a small set of "Golden Fixtures" in `tests/unit/data` for your automated unit tests (Jest/Mocha), so CI/CD can run without BigQuery access.
+3. **Check the "State":**
+If a run seems to hang or not finish, check `debug-out/state/computation_state.json`. It tells you if the system thinks it's `running`, `failed`, or `completed`.
+4. **Simulate "Dry Runs":**
+The harness runs in "Live Mode" by default (writing results). To test logic without writing, you can modify `debug-local.js` to pass `dryRun: true` to the `harness.run()` command.

package/functions/computation-system-v3/MemoryAdapters.js

@@ -0,0 +1,110 @@
+/**
+ * @fileoverview Memory Adapters for Hybrid Testing
+ * FIX: MemoryCloudTasks now generates unique IDs per date to prevent false deduplication.
+ */
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+
+class LocalDiskStorage {
+    constructor(baseDir) {
+        this.baseDir = baseDir;
+        if (!fs.existsSync(baseDir)) fs.mkdirSync(baseDir, { recursive: true });
+    }
+
+    async saveResults({ entry, date, data }) {
+        const filename = `${entry.name.toLowerCase()}-${date}.json`;
+        const filePath = path.join(this.baseDir, filename);
+        let existing = {};
+        if (fs.existsSync(filePath)) {
+            try { existing = JSON.parse(fs.readFileSync(filePath, 'utf8')); } catch (e) {}
+        }
+        const merged = { ...existing, ...data };
+        fs.writeFileSync(filePath, JSON.stringify(merged, null, 2));
+        console.log(`[Storage] Saving results to ${filePath} (Entities: ${Object.keys(merged).length})`);
+    }
+
+    readResults(computationName, date) {
+        const filename = `${computationName.toLowerCase()}-${date}.json`;
+        const filePath = path.join(this.baseDir, filename);
+        if (fs.existsSync(filePath)) {
+            try { return JSON.parse(fs.readFileSync(filePath, 'utf8')); } catch (e) { }
+        }
+        return {};
+    }
+}
+
+class MemoryStateRepo {
+    constructor(storage) { this.storage = storage; }
+
+    async getDailyStatus(date) {
+        const files = fs.readdirSync(this.storage.baseDir);
+        const map = new Map();
+        files.forEach(f => {
+            if (f.includes(date)) {
+                const name = f.split('-')[0];
+                map.set(name, { status: 'completed', hash: 'MOCK_HASH', timestamp: Date.now() });
+            }
+        });
+        return map;
+    }
+
+    async getResult(date, computationName) { return this.storage.readResults(computationName, date); }
+    
+    async getBatchEntityResults(date, computationName, entityIds) {
+        const allResults = this.storage.readResults(computationName, date);
+        const subset = {};
+        if (Array.isArray(entityIds)) {
+            entityIds.forEach(id => { if (allResults[id]) subset[id] = allResults[id]; });
+        }
+        return subset;
+    }
+    
+    async updateStatus() { return true; }
+    async findZombies() { return []; }
+    async claimZombie() {}
+}
+
+class MemoryCloudTasks {
+    constructor() {
+        this.queue = [];
+        this.history = [];
+    }
+    
+    getQueuePath() { return 'projects/test/locations/test/queues/test'; }
+    
+    async createTask(parent, payload, options = {}) {
+        // [FIX] Deduplication check: Only drop if EXACT name exists
+        if (options.name && this.queue.some(t => t.name === options.name)) {
+            return { status: 'exists' };
+        }
+        
+        const queuedTask = { 
+            ...payload, 
+            id: options.name || `task-${Date.now()}-${Math.random()}`, 
+            name: options.name,
+            targetDate: payload.targetDate || payload.date
+        };
+        
+        this.queue.push(queuedTask);
+        this.history.push(queuedTask);
+        return { name: queuedTask.id };
+    }
+
+    async scheduleTask(t) { 
+        // [FIX] Generate a UNIQUE name per date to prevent false deduplication
+        // Real adapter uses: root-{name}-{date}-{hash}-{random}
+        const suffix = crypto.randomBytes(3).toString('hex');
+        const dateStr = t.targetDate || t.date || 'nodate';
+        const uniqueName = `scheduled-${t.name}-${dateStr}-${suffix}`;
+        console.log(`[CloudTasks] Scheduling Task: ${uniqueName}`);
+        
+        return this.createTask(null, t, { name: uniqueName }); 
+    }
+
+    getTasks() { return this.queue; }
+    clear() { this.queue = []; }
+    async listAndClean() { return { deleted: 0, activeKeys: new Set() }; }
+}
+
+module.exports = { LocalDiskStorage, MemoryStateRepo, MemoryCloudTasks };