bulltrackers-module 1.0.733 → 1.0.734

package/functions/computation-system-v2/README.md

@@ -0,0 +1,152 @@
+# Computation System v2
+
+## Overview
+
+A generic, BigQuery-driven computation framework that can be applied to any dataset.
+
+**Key Principles:**
+- **Zero hardcoded schemas** - Schemas are discovered dynamically from BigQuery metadata
+- **Pre-query validation** - Queries are validated against cached schemas before execution
+- **Simple computation interface** - Just declare what tables you need
+- **Pass-based execution** - Topological sort determines execution order (Kahn's algorithm)
+- **Hash-based versioning** - Automatic detection of code/dependency changes
+
+## Directory Structure
+
+```
+computation-system-v2/
+├── framework/                    # GENERIC - Works with any BigQuery data
+│   ├── core/
+│   │   ├── Computation.js       # Base class for all computations
+│   │   ├── Executor.js          # Orchestrates computation execution
+│   │   └── Manifest.js          # Builds & validates computation graph
+│   ├── data/
+│   │   ├── SchemaRegistry.js    # Dynamic schema discovery + caching
+│   │   ├── QueryBuilder.js      # Builds validated SQL queries
+│   │   └── DataFetcher.js       # Executes queries, returns data
+│   ├── persistence/
+│   │   └── ResultWriter.js      # Writes computation results
+│   └── utils/
+│       ├── hash.js              # Code hashing utilities
+│       └── logger.js            # Logging utilities
+│
+├── config/
+│   └── bulltrackers.config.js   # BullTrackers-specific configuration
+│
+├── computations/                 # BullTrackers computations (v2 format)
+│   └── (migrated computations go here)
+│
+└── index.js                      # Main entry point
+```
+
+## Quick Start
+
+### 1. Define a Computation
+
+```javascript
+const { Computation } = require('./framework/core/Computation');
+
+class MyComputation extends Computation {
+    static getConfig() {
+        return {
+            name: 'MyComputation',
+            requires: {
+                'my_table': { 
+                    lookback: 0,      // 0 = today only, N = last N days
+                    mandatory: true   // If true, computation won't run without this data
+                }
+            },
+            dependencies: [],  // Other computations this depends on
+            type: 'global'     // 'global' or 'per-entity'
+        };
+    }
+
+    async process(context) {
+        const data = context.data['my_table'];
+        // Your logic here
+        return { result: 'computed value' };
+    }
+}
+```
+
+### 2. Register in Config
+
+```javascript
+// config/bulltrackers.config.js
+module.exports = {
+    bigquery: {
+        projectId: 'your-project',
+        dataset: 'your_dataset'
+    },
+    tables: {
+        'my_table': {
+            dateField: 'date',
+            entityField: 'user_id'
+        }
+    },
+    computations: [
+        require('./computations/MyComputation')
+    ]
+};
+```
+
+### 3. Run
+
+```javascript
+const { execute } = require('./computation-system-v2');
+await execute({ date: '2026-01-24', config: require('./config/bulltrackers.config') });
+```
+
+## Key Concepts
+
+### Schema Registry
+
+The `SchemaRegistry` fetches table schemas from BigQuery's `INFORMATION_SCHEMA` and caches them:
+
+```javascript
+const schema = await schemaRegistry.getSchema('my_table');
+// Returns: [{ name: 'id', type: 'INT64' }, { name: 'date', type: 'DATE' }, ...]
+```
+
+Before any query is executed, it's validated against the cached schema to prevent runtime errors.
+
+### Query Validation
+
+```javascript
+// This will throw BEFORE hitting BigQuery if 'nonexistent_column' doesn't exist
+const query = queryBuilder.build({
+    table: 'my_table',
+    select: ['id', 'nonexistent_column'],  // Error: column doesn't exist
+    where: { date: '2026-01-24' }
+});
+```
+
+### Pass System
+
+Computations are automatically organized into "passes" based on dependencies:
+
+```
+Pass 1: Computations with no dependencies (can run in parallel)
+Pass 2: Computations depending only on Pass 1 results
+Pass 3: Computations depending on Pass 1 or 2 results
+...
+```
+
+### Hash-Based Versioning
+
+Each computation gets a unique hash based on:
+- Its own code
+- The code of any shared utilities it uses
+- The hashes of its dependencies
+
+When a hash changes, the computation re-runs. When hashes match, results are reused.
+
+## Comparison with v1
+
+| Aspect | v1 | v2 |
+|--------|----|----|
+| Schema definition | Hardcoded in `bigquery_utils.js` | Dynamic from `INFORMATION_SCHEMA` |
+| Query building | Per-table functions | Single generic builder with validation |
+| Data loading | Complex routing through multiple layers | Direct fetch with simple interface |
+| Computation declaration | Multiple overlapping fields | Single `requires` object |
+| Domain coupling | Deeply embedded | Configuration-only |