bulltrackers-module 1.0.763 → 1.0.765

package/functions/computation-system-v2/docs/architecture.md

@@ -1,103 +0,0 @@
-# System Architecture
-
-## 1. System Philosophy
-
-The Computation System v2 is designed to be a generic, configuration-driven Directed Acyclic Graph (DAG) executor that operates directly on BigQuery data. It departs from traditional ETL pipelines by adhering to five core principles:
-
-1.  **Zero Hardcoded Schemas**: The system never defines schemas in code. It dynamically discovers them from BigQuery's `INFORMATION_SCHEMA` at runtime. This eliminates the "schema drift" problem where code falls out of sync with the database.
-2.  **Pre-Query Validation**: Every SQL query is validated against the cached schema *before* it is sent to BigQuery. This prevents expensive runtime failures and SQL errors.
-3.  **Pass-Based Execution**: Computations are automatically organized into "passes" (waves) based on their dependencies using Kahn's algorithm.
-4.  **Hash-Based Versioning**: The system tracks the "Identity" of every computation result using a cryptographic hash of:
-    *   The Computation Code
-    *   Shared Utility "Layers"
-    *   Business Logic "Rules"
-    *   Dependency Result Hashes
-5.  **Hybrid Execution**: Light computations run globally in-memory, while heavy per-entity computations are offloaded to a serverless worker pool (Remote Task Runner).
-
----
-
-## 2. Core Components
-
-### 2.1. The Manifest Builder (`framework/core/Manifest.js`)
-The Manifest is the blueprint of the system. At startup, it:
-1.  **Loads Configs**: Reads static `getConfig()` from all Computation classes.
-2.  **Builds Graph**: Constructs the dependency graph.
-3.  **Detects Cycles**: Throws an error if A -> B -> A.
-4.  **Calculates Passes**: Uses Topological Sort to assign a `pass` number (0, 1, 2...) to each computation.
-5.  **Generates Hashes**: Computes the intrinsic version hash for change detection.
-
-### 2.2. Schema Registry (`framework/data/SchemaRegistry.js`)
-The bridge between code and data.
-*   **Dynamic Discovery**: Fetches column definitions, types, and nullability from BigQuery.
-*   **Caching**: Caches schemas in memory (default TTL: 1 hour) to reduce latency.
-*   **Request Coalescing**: Prevents "Thundering Herd" issues by merging simultaneous requests for the same table schema.
-
-### 2.3. The Orchestrator (`framework/execution/Orchestrator.js`)
-The central nervous system. It creates the execution plan for a given date.
-*   **Dependency Resolution**: Ensures pre-requisite data is loaded.
-*   **Execution Strategy**: Decides whether to run a computation locally or offload it to the Remote Task Runner.
-*   **Streaming**: For large datasets, it streams data in batches (default: 1000 entities) to avoid Out-Of-Memory (OOM) crashes.
-
----
-
-## 3. Data Flow
-
-```mermaid
-graph TD
-    User[User/Scheduler] -->|Trigger| Dispatcher
-    Dispatcher -->|POST| Orchestrator
-    
-    subgraph "Phase 1: Preparation"
-        Orchestrator -->|Build| Manifest
-        Orchestrator -->|Check Status| StateDB[(State Repository)]
-        StateDB -->|Run/Skip/ReRun?| Orchestrator
-    end
-    
-    subgraph "Phase 2: Execution"
-        Orchestrator -->|Fetch Data| DataFetcher
-        DataFetcher -->|Validate| SchemaRegistry
-        SchemaRegistry -->|Meta| BigQuery[(BigQuery)]
-        DataFetcher -->|Query| BigQuery
-        
-        Orchestrator -->|Pass 0| C1[Computation A]
-        Orchestrator -->|Pass 1| C2[Computation B]
-        
-        C1 -->|Result| Storage[Storage Manager]
-        Storage -->|Persist| BigQuery
-    end
-```
-
----
-
-## 4. Execution Modes
-
-### 4.1. Global Mode (Local Isolation)
-Used for: Aggregations, summaries, or light computations.
-*   **Flow**: Fetches *all* required data into memory -> Runs logic -> Writes single result.
-*   **Concurrency**: Parallelized by `p-limit` locally.
-
-### 4.2. Per-Entity Mode (Remote Offload)
-Used for: Complex logic (e.g., Portfolio Calculations) requiring isolation.
-*   **Orchestrator**: Fetches data for a batch (e.g., 100 users).
-*   **Storage**: Uploads context (Data + Rules + Config) to Cloud Storage (GCS).
-*   **Worker Pool**: Invokes stateless Cloud Functions to process the batch.
-*   **Circuit Breaker**: Stops execution if failure rate exceeds threshold (default: 30%).
-
----
-
-## 5. Versioning & Hashing
-
-How does the system know when to re-run a computation?
-
-The **Composite Hash** is calculated as:
-```
-Hash = SHA256(
-    Code Body +
-    Epoch Version +
-    Shared Layer Hashes +
-    Rule Module Hashes +
-    Dependency Result Hashes
-)
-```
-
-If this hash matches the `resultHash` stored in the State Repository for a given date, the computation is **SKIPPED** (unless forced). This ensures idempotency and avoids unnecessary processing costs.

package/functions/computation-system-v2/docs/developer_guide.md

@@ -1,125 +0,0 @@
-# Developer Guide
-
-This guide explains how to create, configure, and test new Computations in the v2 framework.
-
-## 1. Creating a Computation
-
-All computations must extend the base `Computation` class.
-
-### Basic Template
-
-```javascript
-const { Computation } = require('../../framework/core/Computation');
-
-class UserDailyActive extends Computation {
-    static getConfig() {
-        return {
-            name: 'UserDailyActive',
-            type: 'per-entity', // or 'global'
-            requires: {
-                // Table Dependencies (Data)
-                'user_logins': { 
-                    lookback: 0,      // 0 = today only
-                    mandatory: true
-                }
-            },
-            dependencies: [
-                // Computation Dependencies (Prerequisites)
-                // 'UserSessionSummary' 
-            ]
-        };
-    }
-
-    async process(context) {
-        const { date, entityId, data } = context;
-        
-        // Access pre-fetched data
-        const logins = data['user_logins'];
-        
-        if (!logins || logins.length === 0) {
-            return null; // Return null to skip saving result
-        }
-
-        return {
-            date,
-            userId: entityId,
-            loginCount: logins.length,
-            lastLoginTime: logins[logins.length - 1].timestamp
-        };
-    }
-}
-module.exports = UserDailyActive;
-```
-
----
-
-## 2. Configuration Options (`getConfig`)
-
-| Field | Type | Description |
-| :--- | :--- | :--- |
-| `name` | String | **Required**. Unique name of the computation. |
-| `type` | String | `'global'` (runs once) or `'per-entity'` (runs for each entity). |
-| `requires` | Object | Map of table names to requirements. |
-| `dependencies` | Array | List of other computation names that must run *before* this one. |
-| `schedule` | Object | `{ frequency: 'daily', time: '02:00' }`. Defaults to daily/02:00. |
-| `ttlDays` | Number | How long to keep results in State DB (default: 365). |
-| `isHistorical`| Boolean| If `true`, `context.previousResult` will contain yesterday's output. |
-
-### The `requires` Object
-
-```javascript
-requires: {
-    'orders': {
-        lookback: 7,       // Fetch last 7 days of data
-        mandatory: false,  // If true, crashes if table is empty/missing
-        dateField: 'created_at' // Optional override
-    }
-}
-```
-
----
-
-## 3. The `process(context)` Method
-
-The `process` method is the heart of your logic. It receives a `context` object:
-
-### `context.data`
-Contains the raw data fetched from BigQuery.
-*   **Per-Entity**: `data['table_name']` is the row (or array of rows) *specifically for that entity*.
-*   **Global**: `data['table_name']` is the entire dataset fetched.
-
-### `context.getDependency(name, [entityId])`
-Access results from previous computations.
-```javascript
-// In a per-entity computation, getting its own dependency:
-const sessionStats = context.getDependency('UserSessionSummary');
-
-// Getting a global dependency result from a per-entity computation:
-const globalSettings = context.getDependency('GlobalSettings', '_global');
-```
-
-### `context.rules`
-Access to shared business logic modules (if configured).
-
----
-
-## 4. Testing & Validation
-
-### Local Dry Run
-You can run your computation locally without writing to BigQuery/Firestore by using the dry-run flag.
-
-```bash
-# Run a specific computation for a specific date
-node index.js execute \
-  --computation UserDailyActive \
-  --date 2026-01-26 \
-  --dry-run \
-  --entityId user_12345
-```
-
-### Logging
-Use `this.log(level, message)` instead of `console.log`. This ensures logs are properly tagged with the computation name and entity ID in Cloud Logging.
-
-```javascript
-this.log('INFO', `Processing user ${entityId} with ${logins.length} logins`);
-```

package/functions/computation-system-v2/docs/operations.md

@@ -1,99 +0,0 @@
-# Operations Manual
-
-This guide covers deployment, daily operations, monitoring, and troubleshooting.
-
-## 1. Deployment
-
-The system is deployed as a Google Cloud Function (Gen 2).
-
-### Deploy Command
-```bash
-# Navigate to the function directory
-cd functions/computation-system-v2
-
-# Deploy using the included script
-node deploy.mjs ComputeSystem
-```
-
-**Note**: Ensure you have the `gcloud` CLI installed and authenticated.
-
----
-
-## 2. Management API
-
-The system exposes an HTTP endpoint for manual administration.
-
-### Setup (One-time)
-```bash
-# Get Function URL
-FUNCTION_URL=$(gcloud functions describe compute-system --region=europe-west1 --format="value(serviceConfig.uri)")
-
-# Get Auth Token
-TOKEN=$(gcloud auth print-identity-token)
-```
-
-### Common Actions
-
-#### Check Status
-Lists all registered computations and their hash status.
-```bash
-curl -X POST "$FUNCTION_URL" \
-    -H "Authorization: Bearer $TOKEN" \
-    -H "Content-Type: application/json" \
-    -d '{"action": "status"}'
-```
-
-#### Analyze a Date
-Simulates a run for a valid date, showing what would run, skip, or be blocked.
-```bash
-curl -X POST "$FUNCTION_URL" \
-    -H "Authorization: Bearer $TOKEN" \
-    -H "Content-Type: application/json" \
-    -d '{"action": "analyze", "date": "2026-01-26"}'
-```
-
-#### Manually Trigger (Force Run)
-Forces a computation to run, ignoring hash checks.
-```bash
-curl -X POST "$FUNCTION_URL" \
-    -H "Authorization: Bearer $TOKEN" \
-    -H "Content-Type: application/json" \
-    -d '{"action": "run", "computation": "UserDailyActive", "date": "2026-01-26", "force": true}'
-```
-
----
-
-## 3. Monitoring
-
-The Orchestrator logs a structured **Execution Summary** at the end of every run.
-
-### Understanding Statuses
-
-| Status | Icon | Description | Action Required |
-| :--- | :--- | :--- | :--- |
-| `completed` | ✅ | Computation finished successfully. | None |
-| `skipped` | ⏭️ | Result hash matches previous run (Up-to-date). | None |
-| `blocked` | ⛔ | A dependency failed or hasn't run yet. | Fix dependency then re-run. |
-| `impossible` | 🚫 | A mandatory requirement (table) is empty. | Check upstream ETL. |
-| `failed` | ❌ | Runtime error threw an exception. | Check logs for stack trace. |
-
-### Circuit Breaker (Remote Mode)
-If you see `CIRCUIT BREAKER TRIPPED` in the logs:
-1.  **Stop**: The system automatically aborted to prevent wasting money on failing workers.
-2.  **Investigate**: Look for "Worker failed" logs to see the root cause (e.g., Syntax Error in calculation).
-3.  **Fix**: Deploy the fix.
-4.  **Retry**: The circuit resets automatically on the next run.
-
----
-
-## 4. Troubleshooting
-
-### "Thundering Herd" on Startup
-**Symptom**: Massive latency on the first few requests.
-**Cause**: Cold start + filling Schema Cache.
-**Fix**: The system now uses **Request Coalescing**. If it persists, increase `concurrency` in `workerPool` config.
-
-### "Zombie" Computations
-**Symptom**: A computation stuck in "Running" state for hours.
-**Cause**: Function timeout or crash before writing status.
-**Fix**: Run the `status` command. The system automatically detects stale locks (> 1 hour) and allows re-execution.