bulltrackers-module 1.0.763 → 1.0.765

package/functions/computation-system-v2/docs/Agents.MD

@@ -0,0 +1,964 @@
+# **Computation System v2**
+
+Developer Manual
+
+_A Domain-Agnostic DAG Framework for Data Computations_
+
+# Table of Contents
+
+This document covers:
+
+- System Overview & Architecture
+- Creating New Computations
+- Configuration Reference (getConfig)
+- Data Access & Context
+- Business Rules System
+- Testing & Debugging
+- Best Practices & Common Patterns
+
+# 1\. System Overview
+
+The Computation System v2 is a domain-agnostic Directed Acyclic Graph (DAG) framework designed for managing complex data transformations and calculations. While the current implementation includes finance-specific business rules for the BullTrackers analytics platform, the framework itself is completely generic and can be adapted to any domain.
+
+## 1.1 Core Principles
+
+- Domain Agnostic: The framework has no built-in knowledge of finance or any specific domain
+- Dependency-Driven: Computations declare their dependencies; the system manages execution order
+- Smart Caching: Only re-runs when code, data, or dependencies change
+- Scalable: Supports both in-memory and distributed (worker pool) execution
+- Resilient: Automatic checkpointing and zombie detection for long-running tasks
+
+## 1.2 Key Concepts
+
+### Computations
+
+A computation is a single unit of data processing. It:
+
+- Declares what data it needs (requirements)
+- Declares which other computations it depends on (dependencies)
+- Implements a process() method containing the transformation logic
+- Produces results that can be stored and used by other computations
+
+### DAG (Directed Acyclic Graph)
+
+The system automatically builds a dependency graph from computation declarations. This ensures:
+
+- No circular dependencies (acyclic)
+- Correct execution order (topological sort)
+- Parallel execution of independent computations (passes)
+
+### Passes
+
+Computations are organized into execution passes based on their dependencies:
+
+- Pass 1: Computations with no dependencies (root nodes)
+- Pass 2: Computations that only depend on Pass 1
+- Pass N: Computations whose longest dependency chain is N-1
+
+All computations within a pass can run in parallel.
+
+### Business Rules
+
+Business rules are domain-specific functions that contain the actual business logic. They are:
+
+- Organized into modules (e.g., portfolio, metrics, rankings)
+- Automatically injected into computation contexts
+- Version-tracked (changes trigger re-runs)
+- Completely optional - framework works without them
+
+# 2\. Creating New Computations
+
+## 2.1 Basic Structure
+
+Every computation follows this template:
+
+const { Computation } = require('../framework');
+
+class MyComputation extends Computation {
+
+static getConfig() {
+
+return {
+
+name: 'MyComputation',
+
+type: 'per-entity',
+
+category: 'analytics',
+
+requires: { /\* data requirements \*/ },
+
+dependencies: \[\]
+
+};
+
+}
+
+async process(context) {
+
+const { data, entityId, rules } = context;
+
+// Your logic here
+
+this.setResult(entityId, result);
+
+}
+
+}
+
+module.exports = MyComputation;
+
+## 2.2 File Location
+
+Save your computation file in:
+
+computations/YourComputationName.js
+
+The framework automatically discovers and loads all .js files in this directory (except those starting with underscore).
+
+## 2.3 Computation Types
+
+There are two computation types:
+
+### Global Computations
+
+Run once per date, processing all data together.
+
+type: 'global'
+
+Use cases: System-wide aggregations, rankings, summaries that need the full dataset.
+
+async process(context) {
+
+const { data } = context;
+
+// Process all data at once
+
+const result = analyzeAllData(data);
+
+this.setGlobalResult(result);
+
+}
+
+### Per-Entity Computations
+
+Run once for each entity (user, asset, etc.), enabling massive parallelization.
+
+type: 'per-entity'
+
+Use cases: User-specific calculations, per-asset analytics, individual reports.
+
+async process(context) {
+
+const { data, entityId, rules } = context;
+
+// Process one entity's data
+
+const result = calculateForEntity(data, entityId);
+
+this.setResult(entityId, result);
+
+}
+
+# 3\. Configuration Reference (getConfig)
+
+The getConfig() method returns an object defining the computation's behavior.
+
+## 3.1 Required Fields
+
+| **Field** | **Type** | **Description** | **Example** |
+| --- | --- | --- | --- |
+| name | string | Unique identifier for the computation | 'UserPortfolioSummary' |
+| requires | object | Data requirements specification | See section 3.3 |
+| type | string | Execution mode: 'global' or 'per-entity' | 'per-entity' |
+
+## 3.2 Optional Fields
+
+| **Field** | **Type** | **Default** | **Description** |
+| --- | --- | --- | --- |
+| category | string | 'default' | Organizational category |
+| dependencies | array | \[\] | List of computation names this depends on |
+| conditionalDependencies | array | \[\] | Dependencies loaded only when condition is true |
+| schedule | object\|string | 'daily' | When to run: 'daily', 'weekly', 'monthly', or custom |
+| isHistorical | boolean | false | Requires continuous daily execution (no gaps) |
+| isTest | boolean | false | Always re-runs on today's date (for testing) |
+| storage | object | see 3.8 | Where to store results (BigQuery, Firestore) |
+| ttlDays | number | null | Auto-delete results after N days |
+
+## 3.3 Data Requirements (requires)
+
+The requires object declares which BigQuery tables the computation needs:
+
+requires: {
+
+'portfolio_snapshots': {
+
+lookback: 30, // Days of historical data
+
+mandatory: true, // Fail if missing
+
+fields: \['user_id', 'portfolio_data'\],
+
+filter: { user_type: 'POPULAR_INVESTOR' }
+
+},
+
+'asset_prices': {
+
+lookback: 0, // Today only
+
+mandatory: false // Optional data
+
+}
+
+}
+
+### Requirement Fields
+
+| **Field** | **Type** | **Default** | **Description** |
+| --- | --- | --- | --- |
+| lookback | number | 0   | How many days of historical data to fetch (0 = today only) |
+| mandatory | boolean | false | If true, computation fails if data is missing |
+| fields | array | null | Specific columns to fetch (null = all columns) |
+| filter | object | {}  | WHERE clause conditions to apply |
+
+## 3.4 Dependencies
+
+Dependencies are other computations that must complete first:
+
+dependencies: \['UserPortfolioSummary', 'AssetPriceAnalysis'\]
+
+The framework:
+
+- Ensures dependencies run before this computation
+- Detects circular dependencies at startup
+- Re-runs this computation if a dependency's data changes
+- Makes dependency results available via context.getDependency()
+
+## 3.5 Conditional Dependencies
+
+For dependencies that are only needed under certain conditions:
+
+conditionalDependencies: \[
+
+{
+
+computation: 'WeekendAnalysis',
+
+condition: ({ date }) => {
+
+const day = new Date(date).getUTCDay();
+
+return day === 0 || day === 6; // Weekend
+
+}
+
+}
+
+\]
+
+The condition function receives { date, config } and returns true/false.
+
+## 3.6 Scheduling
+
+Control when the computation runs:
+
+// Simple schedules
+
+schedule: 'daily' // Every day
+
+schedule: 'weekly' // Every Sunday
+
+schedule: 'monthly' // First of month
+
+// Advanced schedule
+
+schedule: {
+
+frequency: 'weekly',
+
+time: '02:00', // UTC time
+
+dayOfWeek: 1, // Monday (0=Sunday)
+
+timezone: 'UTC'
+
+}
+
+## 3.7 Historical Mode
+
+For computations that must run continuously without gaps:
+
+isHistorical: true
+
+This enables:
+
+- Access to previous day's results via context.previousResult
+- Re-run if there's a gap in the daily sequence
+- Version continuity checks (re-runs if code changed mid-sequence)
+
+Use cases: Running balances, cumulative metrics, time-series that depend on previous state.
+
+## 3.8 Storage Configuration
+
+Control where results are stored:
+
+storage: {
+
+bigquery: true, // Default: Always store in BigQuery
+
+firestore: {
+
+enabled: true,
+
+path: 'users/{entityId}/summary',
+
+merge: true,
+
+includeMetadata: true
+
+}
+
+}
+
+Path variables: {entityId}, {date}, {computationName}, {category}
+
+# 4\. Data Access & Context
+
+The context object passed to process() contains everything the computation needs.
+
+## 4.1 Context Properties
+
+| **Property** | **Type** | **Description** |
+| --- | --- | --- |
+| data | object | Fetched table data based on requires configuration |
+| entityId | string | Current entity ID (per-entity computations only) |
+| date | string | Target date in YYYY-MM-DD format |
+| rules | object | Injected business rules modules |
+| references | object | Global reference data (lookups, mappings) |
+| getDependency | function | Fetch dependency computation results |
+| previousResult | any | Yesterday's result (isHistorical only) |
+| computation | object | Metadata about this computation |
+| config | object | System configuration (read-only) |
+
+## 4.2 Accessing Table Data
+
+Data is organized by table name, then by entity ID (for entity-keyed tables):
+
+async process(context) {
+
+const { data, entityId } = context;
+
+// Per-entity table: Get current entity's data
+
+const portfolio = data\['portfolio_snapshots'\];
+
+// portfolio is already filtered to this entityId
+
+// Global table: All entities
+
+const prices = data\['asset_prices'\];
+
+// prices = { instrumentId1: priceData, ... }
+
+}
+
+### Data Structures
+
+With lookback = 0 (today only):
+
+data\['table_name'\] = {
+
+'entity1': { row data },
+
+'entity2': { row data }
+
+}
+
+With lookback > 0 (historical data):
+
+data\['table_name'\] = {
+
+'entity1': \[
+
+{ date: '2026-01-01', ...data },
+
+{ date: '2026-01-02', ...data }
+
+\]
+
+}
+
+## 4.3 Using getDependency()
+
+Access results from dependency computations:
+
+async process(context) {
+
+const { getDependency, entityId } = context;
+
+// Get this entity's result from dependency
+
+const summary = getDependency('UserPortfolioSummary');
+
+// Or explicitly specify entity ID
+
+const summary = getDependency('UserPortfolioSummary', entityId);
+
+// Cross-entity lookup
+
+const otherUser = getDependency('UserPortfolioSummary', 'user-456');
+
+}
+
+## 4.4 Reference Data
+
+Reference data is loaded once and shared across all entities:
+
+async process(context) {
+
+const { references } = context;
+
+// Access pre-loaded lookup tables
+
+const sectorMap = references\['sector_mappings'\];
+
+const sector = sectorMap\['AAPL'\];
+
+}
+
+Configure in bulltrackers.config.js:
+
+referenceData: \['sector_mappings', 'ticker_mappings'\]
+
+# 5\. Business Rules System
+
+Business rules separate domain logic from computation orchestration. They are completely optional - the framework works without them.
+
+## 5.1 Using Rules
+
+Rules are automatically injected into the computation context:
+
+async process(context) {
+
+const { data, rules, entityId } = context;
+
+// Extract data using rules
+
+const portfolioData = rules.portfolio.extractPortfolioData(
+
+data\['portfolio_snapshots'\]
+
+);
+
+const positions = rules.portfolio.extractPositions(portfolioData);
+
+// Calculate metrics using rules
+
+const totalValue = rules.portfolio.calculateTotalValue(positions);
+
+const exposure = rules.portfolio.getExposure(portfolioData);
+
+this.setResult(entityId, { totalValue, exposure });
+
+}
+
+## 5.2 Available Rule Modules (BullTrackers)
+
+The current implementation includes finance-specific rules:
+
+| **Module** | **Purpose** | **Example Functions** |
+| --- | --- | --- |
+| rules.portfolio | Portfolio data extraction and analysis | extractPositions, calculateTotalValue, groupBySector |
+| rules.metrics | Financial metrics calculations | calculateSharpeRatio, calculateMaxDrawdown, calculateVaR |
+| rules.rankings | PI rankings data | getRiskScore, getAUM, calculateQualityScore |
+| rules.trades | Trade history analysis | extractTrades, calculateTradeStats, filterProfitable |
+| rules.social | Social engagement metrics | extractPosts, calculateEngagement, getAverageRating |
+| rules.instruments | Asset and price data | getPrice, calculateSentimentScore, buildTickerMaps |
+
+## 5.3 Creating Custom Rules
+
+To add new business rules:
+
+- Create a new file in rules/ directory
+
+// rules/mymodule.js
+
+function calculateSomething(data) {
+
+// Your logic
+
+return result;
+
+}
+
+module.exports = { calculateSomething };
+
+- Register in rules/index.js
+
+const mymodule = require('./mymodule');
+
+module.exports = {
+
+portfolio,
+
+metrics,
+
+mymodule // Add here
+
+};
+
+- Use in computations
+
+const result = rules.mymodule.calculateSomething(data);
+
+## 5.4 Rule Versioning
+
+Rules are automatically version-tracked:
+
+- Each rule function is hashed based on its source code
+- Computations using a rule include its hash in their version
+- Changing a rule triggers re-runs of dependent computations
+- Only rules actually used (detected via code analysis) affect versioning
+
+This ensures computation results stay fresh when business logic evolves.
+
+# 6\. Testing & Debugging
+
+## 6.1 Using the Admin Test Endpoint
+
+The system includes a protected admin endpoint for testing:
+
+curl -X POST <https://REGION-PROJECT.cloudfunctions.net/compute-admin-test> \\
+
+\-H "Authorization: Bearer \$(gcloud auth print-identity-token)" \\
+
+\-H "Content-Type: application/json" \\
+
+\-d '{"action": "run", "computation": "MyComputation"}'
+
+### Available Actions
+
+| **Action** | **Purpose** | **Parameters** |
+| --- | --- | --- |
+| status | List all computations | None |
+| analyze | Check what would run for a date | date |
+| run | Execute a computation | computation, date, force, dryRun |
+| run_limited | Run on sample entities only | computation, limit |
+| test_worker | Test worker logic locally | computation, entityIds |
+
+## 6.2 Development Workflow
+
+- Create your computation file
+- Deploy to Cloud Functions
+- Test with admin endpoint
+
+{
+
+"action": "run_limited",
+
+"computation": "MyComputation",
+
+"limit": 5,
+
+"dryRun": true
+
+}
+
+- Verify results in BigQuery
+- Iterate as needed
+
+## 6.3 Testing Strategies
+
+### Test Mode
+
+Mark computations as test to always re-run on today:
+
+isTest: true
+
+Perfect for development - runs every time without manual force flags.
+
+### Dry Run
+
+Execute logic without saving results:
+
+{
+
+"action": "run",
+
+"computation": "MyComputation",
+
+"dryRun": true
+
+}
+
+### Entity Filtering
+
+Test on specific entities:
+
+{
+
+"action": "run",
+
+"computation": "MyComputation",
+
+"entityIds": \["user-123", "user-456"\]
+
+}
+
+## 6.4 Debugging Tips
+
+- Use this.log() for logging (respects computation name prefix)
+
+this.log('INFO', 'Processing entity: ' + entityId);
+
+- Check BigQuery for stored results
+- Verify dependencies completed successfully
+- Use analyze action to understand scheduling
+- Review Cloud Function logs for errors
+
+# 7\. Best Practices & Common Patterns
+
+## 7.1 Computation Design
+
+### DO: Keep Computations Simple
+
+Computations should be thin orchestration layers:
+
+// GOOD: Simple orchestration
+
+async process(context) {
+
+const { data, rules, entityId } = context;
+
+const portfolio = rules.portfolio.extractPortfolioData(data);
+
+const positions = rules.portfolio.extractPositions(portfolio);
+
+const summary = rules.portfolio.summarize(positions);
+
+this.setResult(entityId, summary);
+
+}
+
+### DON'T: Implement Complex Logic
+
+// BAD: Complex logic in computation
+
+async process(context) {
+
+const { data } = context;
+
+// 100 lines of complex calculation logic here...
+
+// This should be in business rules instead!
+
+}
+
+## 7.2 Dependency Management
+
+### DO: Declare All Dependencies
+
+Always list computations you depend on:
+
+dependencies: \['UserPortfolioSummary', 'AssetPriceAnalysis'\]
+
+### DON'T: Create Circular Dependencies
+
+The framework will detect and reject these at startup:
+
+// A depends on B
+
+// B depends on C
+
+// C depends on A <- CIRCULAR! Will fail
+
+## 7.3 Data Requirements
+
+### DO: Use Lookback Wisely
+
+Only request the data you need:
+
+requires: {
+
+'portfolio_snapshots': {
+
+lookback: 30 // Only if you need 30 days
+
+}
+
+}
+
+Note: The system has a MAX_LOOKBACK_DAYS limit (currently 30) to prevent cost spirals.
+
+### DO: Use Field Selection
+
+Reduce data transfer by selecting only needed columns:
+
+requires: {
+
+'portfolio_snapshots': {
+
+fields: \['user_id', 'portfolio_data', 'date'\]
+
+}
+
+}
+
+## 7.4 Error Handling
+
+### DO: Handle Missing Data Gracefully
+
+async process(context) {
+
+const { data, entityId } = context;
+
+const portfolio = data\['portfolio_snapshots'\];
+
+if (!portfolio) {
+
+this.log('WARN', 'No portfolio data for ' + entityId);
+
+return; // Skip this entity
+
+}
+
+// Process...
+
+}
+
+### DO: Use Mandatory for Critical Data
+
+requires: {
+
+'portfolio_snapshots': {
+
+mandatory: true // Fail if missing
+
+}
+
+}
+
+## 7.5 Performance
+
+### DO: Use Per-Entity for Scalability
+
+For user/entity-specific calculations, always use per-entity mode:
+
+type: 'per-entity'
+
+Benefits:
+
+- Automatic parallelization
+- Memory-efficient streaming
+- Worker pool support
+- Checkpointing and resume
+
+### DON'T: Load Everything Into Memory
+
+The framework handles batching - don't try to load all entities yourself.
+
+## 7.6 Common Patterns
+
+### Pattern: Aggregation
+
+// Create per-entity computation first
+
+class UserMetrics extends Computation {
+
+static getConfig() {
+
+return { name: 'UserMetrics', type: 'per-entity', ... };
+
+}
+
+}
+
+// Then create global aggregation
+
+class SystemSummary extends Computation {
+
+static getConfig() {
+
+return {
+
+name: 'SystemSummary',
+
+type: 'global',
+
+dependencies: \['UserMetrics'\]
+
+};
+
+}
+
+async process(context) {
+
+const userMetrics = context.getDependency('UserMetrics');
+
+// Aggregate all user results
+
+}
+
+}
+
+### Pattern: Time Series
+
+class RunningBalance extends Computation {
+
+static getConfig() {
+
+return {
+
+name: 'RunningBalance',
+
+type: 'per-entity',
+
+isHistorical: true,
+
+requires: { 'transactions': { lookback: 1 } }
+
+};
+
+}
+
+async process(context) {
+
+const { previousResult, data, entityId } = context;
+
+const prevBalance = previousResult?.balance || 0;
+
+const todayChange = calculateChange(data);
+
+this.setResult(entityId, {
+
+balance: prevBalance + todayChange
+
+});
+
+}
+
+}
+
+# Appendix A: Quick Reference
+
+## Computation Template
+
+const { Computation } = require('../framework');
+
+class MyComputation extends Computation {
+
+static getConfig() {
+
+return {
+
+name: 'MyComputation',
+
+type: 'per-entity',
+
+category: 'analytics',
+
+requires: {
+
+'table_name': {
+
+lookback: 0,
+
+mandatory: true
+
+}
+
+},
+
+dependencies: \[\],
+
+schedule: 'daily'
+
+};
+
+}
+
+async process(context) {
+
+const { data, entityId, rules } = context;
+
+// Your logic here
+
+const result = {};
+
+this.setResult(entityId, result);
+
+}
+
+}
+
+module.exports = MyComputation;
+
+## Common Context Properties
+
+context.data // Fetched table data
+
+context.entityId // Current entity (per-entity only)
+
+context.date // Target date (YYYY-MM-DD)
+
+context.rules // Business rules modules
+
+context.references // Global reference data
+
+context.getDependency(name, id) // Get dependency result
+
+context.previousResult // Yesterday (isHistorical only)
+
+## Useful Helper Methods
+
+this.setResult(entityId, result) // Save result
+
+this.setGlobalResult(result) // Save (global only)
+
+this.log(level, message) // Log message
+
+this.get(obj, 'path.to.prop', default) // Safe access
+
+# Appendix B: Framework Architecture
+
+The system is built on a layered architecture that separates concerns:
+
+## Layer 1: Core Framework
+
+- Computation base class
+- Manifest builder (DAG construction)
+- Orchestrator (execution engine)
+- Rules registry (business logic injection)
+
+## Layer 2: Data Layer
+
+- Schema registry (dynamic schema discovery)
+- Query builder (SQL generation)
+- Data fetcher (BigQuery interface)
+- State repository (result caching)
+
+## Layer 3: Storage Layer
+
+- Storage manager (BigQuery & Firestore)
+- Checkpointer (resume capability)
+- Lineage tracker (data provenance)
+
+## Layer 4: Execution Layer
+
+- Task runner (middleware chain)
+- Remote task runner (worker pool)
+- Profiler middleware (performance tracking)
+- Cost tracker middleware (billing)
+
+## Layer 5: Scheduling Layer
+
+- Scheduler (Cloud Tasks integration)
+- Watchdog (zombie detection)
+- Dispatcher (request routing)
+
+All layers are domain-agnostic. Business logic lives entirely in the rules modules and computation implementations.
+
+# End of Developer Manual
+
+_For additional support, refer to the source code documentation or contact the development team._