npm - bulltrackers-module - Versions diffs - 1.0.770 → 1.0.772 - Mend

bulltrackers-module 1.0.770 → 1.0.772

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (14) hide show

package/functions/computation-system-v2/docs/Agents.MD DELETED Viewed

@@ -1,964 +0,0 @@
-# **Computation System v2**
-Developer Manual
-_A Domain-Agnostic DAG Framework for Data Computations_
-# Table of Contents
-This document covers:
-- System Overview & Architecture
-- Creating New Computations
-- Configuration Reference (getConfig)
-- Data Access & Context
-- Business Rules System
-- Testing & Debugging
-- Best Practices & Common Patterns
-# 1\. System Overview
-The Computation System v2 is a domain-agnostic Directed Acyclic Graph (DAG) framework designed for managing complex data transformations and calculations. While the current implementation includes finance-specific business rules for the BullTrackers analytics platform, the framework itself is completely generic and can be adapted to any domain.
-## 1.1 Core Principles
-- Domain Agnostic: The framework has no built-in knowledge of finance or any specific domain
-- Dependency-Driven: Computations declare their dependencies; the system manages execution order
-- Smart Caching: Only re-runs when code, data, or dependencies change
-- Scalable: Supports both in-memory and distributed (worker pool) execution
-- Resilient: Automatic checkpointing and zombie detection for long-running tasks
-## 1.2 Key Concepts
-### Computations
-A computation is a single unit of data processing. It:
-- Declares what data it needs (requirements)
-- Declares which other computations it depends on (dependencies)
-- Implements a process() method containing the transformation logic
-- Produces results that can be stored and used by other computations
-### DAG (Directed Acyclic Graph)
-The system automatically builds a dependency graph from computation declarations. This ensures:
-- No circular dependencies (acyclic)
-- Correct execution order (topological sort)
-- Parallel execution of independent computations (passes)
-### Passes
-Computations are organized into execution passes based on their dependencies:
-- Pass 1: Computations with no dependencies (root nodes)
-- Pass 2: Computations that only depend on Pass 1
-- Pass N: Computations whose longest dependency chain is N-1
-All computations within a pass can run in parallel.
-### Business Rules
-Business rules are domain-specific functions that contain the actual business logic. They are:
-- Organized into modules (e.g., portfolio, metrics, rankings)
-- Automatically injected into computation contexts
-- Version-tracked (changes trigger re-runs)
-- Completely optional - framework works without them
-# 2\. Creating New Computations
-## 2.1 Basic Structure
-Every computation follows this template:
-const { Computation } = require('../framework');
-class MyComputation extends Computation {
-static getConfig() {
-return {
-name: 'MyComputation',
-type: 'per-entity',
-category: 'analytics',
-requires: { /\* data requirements \*/ },
-dependencies: \[\]
-};
-}
-async process(context) {
-const { data, entityId, rules } = context;
-// Your logic here
-this.setResult(entityId, result);
-}
-}
-module.exports = MyComputation;
-## 2.2 File Location
-Save your computation file in:
-computations/YourComputationName.js
-The framework automatically discovers and loads all .js files in this directory (except those starting with underscore).
-## 2.3 Computation Types
-There are two computation types:
-### Global Computations
-Run once per date, processing all data together.
-type: 'global'
-Use cases: System-wide aggregations, rankings, summaries that need the full dataset.
-async process(context) {
-const { data } = context;
-// Process all data at once
-const result = analyzeAllData(data);
-this.setGlobalResult(result);
-}
-### Per-Entity Computations
-Run once for each entity (user, asset, etc.), enabling massive parallelization.
-type: 'per-entity'
-Use cases: User-specific calculations, per-asset analytics, individual reports.
-async process(context) {
-const { data, entityId, rules } = context;
-// Process one entity's data
-const result = calculateForEntity(data, entityId);
-this.setResult(entityId, result);
-}
-# 3\. Configuration Reference (getConfig)
-The getConfig() method returns an object defining the computation's behavior.
-## 3.1 Required Fields
-| **Field** | **Type** | **Description** | **Example** |
-| --- | --- | --- | --- |
-| name | string | Unique identifier for the computation | 'UserPortfolioSummary' |
-| requires | object | Data requirements specification | See section 3.3 |
-| type | string | Execution mode: 'global' or 'per-entity' | 'per-entity' |
-## 3.2 Optional Fields
-| **Field** | **Type** | **Default** | **Description** |
-| --- | --- | --- | --- |
-| category | string | 'default' | Organizational category |
-| dependencies | array | \[\] | List of computation names this depends on |
-| conditionalDependencies | array | \[\] | Dependencies loaded only when condition is true |
-| schedule | object\|string | 'daily' | When to run: 'daily', 'weekly', 'monthly', or custom |
-| isHistorical | boolean | false | Requires continuous daily execution (no gaps) |
-| isTest | boolean | false | Always re-runs on today's date (for testing) |
-| storage | object | see 3.8 | Where to store results (BigQuery, Firestore) |
-| ttlDays | number | null | Auto-delete results after N days |
-## 3.3 Data Requirements (requires)
-The requires object declares which BigQuery tables the computation needs:
-requires: {
-'portfolio_snapshots': {
-lookback: 30, // Days of historical data
-mandatory: true, // Fail if missing
-fields: \['user_id', 'portfolio_data'\],
-filter: { user_type: 'POPULAR_INVESTOR' }
-},
-'asset_prices': {
-lookback: 0, // Today only
-mandatory: false // Optional data
-}
-}
-### Requirement Fields
-| **Field** | **Type** | **Default** | **Description** |
-| --- | --- | --- | --- |
-| lookback | number | 0   | How many days of historical data to fetch (0 = today only) |
-| mandatory | boolean | false | If true, computation fails if data is missing |
-| fields | array | null | Specific columns to fetch (null = all columns) |
-| filter | object | {}  | WHERE clause conditions to apply |
-## 3.4 Dependencies
-Dependencies are other computations that must complete first:
-dependencies: \['UserPortfolioSummary', 'AssetPriceAnalysis'\]
-The framework:
-- Ensures dependencies run before this computation
-- Detects circular dependencies at startup
-- Re-runs this computation if a dependency's data changes
-- Makes dependency results available via context.getDependency()
-## 3.5 Conditional Dependencies
-For dependencies that are only needed under certain conditions:
-conditionalDependencies: \[
-{
-computation: 'WeekendAnalysis',
-condition: ({ date }) => {
-const day = new Date(date).getUTCDay();
-return day === 0 || day === 6; // Weekend
-}
-}
-\]
-The condition function receives { date, config } and returns true/false.
-## 3.6 Scheduling
-Control when the computation runs:
-// Simple schedules
-schedule: 'daily' // Every day
-schedule: 'weekly' // Every Sunday
-schedule: 'monthly' // First of month
-// Advanced schedule
-schedule: {
-frequency: 'weekly',
-time: '02:00', // UTC time
-dayOfWeek: 1, // Monday (0=Sunday)
-timezone: 'UTC'
-}
-## 3.7 Historical Mode
-For computations that must run continuously without gaps:
-isHistorical: true
-This enables:
-- Access to previous day's results via context.previousResult
-- Re-run if there's a gap in the daily sequence
-- Version continuity checks (re-runs if code changed mid-sequence)
-Use cases: Running balances, cumulative metrics, time-series that depend on previous state.
-## 3.8 Storage Configuration
-Control where results are stored:
-storage: {
-bigquery: true, // Default: Always store in BigQuery
-firestore: {
-enabled: true,
-path: 'users/{entityId}/summary',
-merge: true,
-includeMetadata: true
-}
-}
-Path variables: {entityId}, {date}, {computationName}, {category}
-# 4\. Data Access & Context
-The context object passed to process() contains everything the computation needs.
-## 4.1 Context Properties
-| **Property** | **Type** | **Description** |
-| --- | --- | --- |
-| data | object | Fetched table data based on requires configuration |
-| entityId | string | Current entity ID (per-entity computations only) |
-| date | string | Target date in YYYY-MM-DD format |
-| rules | object | Injected business rules modules |
-| references | object | Global reference data (lookups, mappings) |
-| getDependency | function | Fetch dependency computation results |
-| previousResult | any | Yesterday's result (isHistorical only) |
-| computation | object | Metadata about this computation |
-| config | object | System configuration (read-only) |
-## 4.2 Accessing Table Data
-Data is organized by table name, then by entity ID (for entity-keyed tables):
-async process(context) {
-const { data, entityId } = context;
-// Per-entity table: Get current entity's data
-const portfolio = data\['portfolio_snapshots'\];
-// portfolio is already filtered to this entityId
-// Global table: All entities
-const prices = data\['asset_prices'\];
-// prices = { instrumentId1: priceData, ... }
-}
-### Data Structures
-With lookback = 0 (today only):
-data\['table_name'\] = {
-'entity1': { row data },
-'entity2': { row data }
-}
-With lookback > 0 (historical data):
-data\['table_name'\] = {
-'entity1': \[
-{ date: '2026-01-01', ...data },
-{ date: '2026-01-02', ...data }
-\]
-}
-## 4.3 Using getDependency()
-Access results from dependency computations:
-async process(context) {
-const { getDependency, entityId } = context;
-// Get this entity's result from dependency
-const summary = getDependency('UserPortfolioSummary');
-// Or explicitly specify entity ID
-const summary = getDependency('UserPortfolioSummary', entityId);
-// Cross-entity lookup
-const otherUser = getDependency('UserPortfolioSummary', 'user-456');
-}
-## 4.4 Reference Data
-Reference data is loaded once and shared across all entities:
-async process(context) {
-const { references } = context;
-// Access pre-loaded lookup tables
-const sectorMap = references\['sector_mappings'\];
-const sector = sectorMap\['AAPL'\];
-}
-Configure in bulltrackers.config.js:
-referenceData: \['sector_mappings', 'ticker_mappings'\]
-# 5\. Business Rules System
-Business rules separate domain logic from computation orchestration. They are completely optional - the framework works without them.
-## 5.1 Using Rules
-Rules are automatically injected into the computation context:
-async process(context) {
-const { data, rules, entityId } = context;
-// Extract data using rules
-const portfolioData = rules.portfolio.extractPortfolioData(
-data\['portfolio_snapshots'\]
-);
-const positions = rules.portfolio.extractPositions(portfolioData);
-// Calculate metrics using rules
-const totalValue = rules.portfolio.calculateTotalValue(positions);
-const exposure = rules.portfolio.getExposure(portfolioData);
-this.setResult(entityId, { totalValue, exposure });
-}
-## 5.2 Available Rule Modules (BullTrackers)
-The current implementation includes finance-specific rules:
-| **Module** | **Purpose** | **Example Functions** |
-| --- | --- | --- |
-| rules.portfolio | Portfolio data extraction and analysis | extractPositions, calculateTotalValue, groupBySector |
-| rules.metrics | Financial metrics calculations | calculateSharpeRatio, calculateMaxDrawdown, calculateVaR |
-| rules.rankings | PI rankings data | getRiskScore, getAUM, calculateQualityScore |
-| rules.trades | Trade history analysis | extractTrades, calculateTradeStats, filterProfitable |
-| rules.social | Social engagement metrics | extractPosts, calculateEngagement, getAverageRating |
-| rules.instruments | Asset and price data | getPrice, calculateSentimentScore, buildTickerMaps |
-## 5.3 Creating Custom Rules
-To add new business rules:
-- Create a new file in rules/ directory
-// rules/mymodule.js
-function calculateSomething(data) {
-// Your logic
-return result;
-}
-module.exports = { calculateSomething };
-- Register in rules/index.js
-const mymodule = require('./mymodule');
-module.exports = {
-portfolio,
-metrics,
-mymodule // Add here
-};
-- Use in computations
-const result = rules.mymodule.calculateSomething(data);
-## 5.4 Rule Versioning
-Rules are automatically version-tracked:
-- Each rule function is hashed based on its source code
-- Computations using a rule include its hash in their version
-- Changing a rule triggers re-runs of dependent computations
-- Only rules actually used (detected via code analysis) affect versioning
-This ensures computation results stay fresh when business logic evolves.
-# 6\. Testing & Debugging
-## 6.1 Using the Admin Test Endpoint
-The system includes a protected admin endpoint for testing:
-curl -X POST <https://REGION-PROJECT.cloudfunctions.net/compute-admin-test> \\
-\-H "Authorization: Bearer \$(gcloud auth print-identity-token)" \\
-\-H "Content-Type: application/json" \\
-\-d '{"action": "run", "computation": "MyComputation"}'
-### Available Actions
-| **Action** | **Purpose** | **Parameters** |
-| --- | --- | --- |
-| status | List all computations | None |
-| analyze | Check what would run for a date | date |
-| run | Execute a computation | computation, date, force, dryRun |
-| run_limited | Run on sample entities only | computation, limit |
-| test_worker | Test worker logic locally | computation, entityIds |
-## 6.2 Development Workflow
-- Create your computation file
-- Deploy to Cloud Functions
-- Test with admin endpoint
-{
-"action": "run_limited",
-"computation": "MyComputation",
-"limit": 5,
-"dryRun": true
-}
-- Verify results in BigQuery
-- Iterate as needed
-## 6.3 Testing Strategies
-### Test Mode
-Mark computations as test to always re-run on today:
-isTest: true
-Perfect for development - runs every time without manual force flags.
-### Dry Run
-Execute logic without saving results:
-{
-"action": "run",
-"computation": "MyComputation",
-"dryRun": true
-}
-### Entity Filtering
-Test on specific entities:
-{
-"action": "run",
-"computation": "MyComputation",
-"entityIds": \["user-123", "user-456"\]
-}
-## 6.4 Debugging Tips
-- Use this.log() for logging (respects computation name prefix)
-this.log('INFO', 'Processing entity: ' + entityId);
-- Check BigQuery for stored results
-- Verify dependencies completed successfully
-- Use analyze action to understand scheduling
-- Review Cloud Function logs for errors
-# 7\. Best Practices & Common Patterns
-## 7.1 Computation Design
-### DO: Keep Computations Simple
-Computations should be thin orchestration layers:
-// GOOD: Simple orchestration
-async process(context) {
-const { data, rules, entityId } = context;
-const portfolio = rules.portfolio.extractPortfolioData(data);
-const positions = rules.portfolio.extractPositions(portfolio);
-const summary = rules.portfolio.summarize(positions);
-this.setResult(entityId, summary);
-}
-### DON'T: Implement Complex Logic
-// BAD: Complex logic in computation
-async process(context) {
-const { data } = context;
-// 100 lines of complex calculation logic here...
-// This should be in business rules instead!
-}
-## 7.2 Dependency Management
-### DO: Declare All Dependencies
-Always list computations you depend on:
-dependencies: \['UserPortfolioSummary', 'AssetPriceAnalysis'\]
-### DON'T: Create Circular Dependencies
-The framework will detect and reject these at startup:
-// A depends on B
-// B depends on C
-// C depends on A <- CIRCULAR! Will fail
-## 7.3 Data Requirements
-### DO: Use Lookback Wisely
-Only request the data you need:
-requires: {
-'portfolio_snapshots': {
-lookback: 30 // Only if you need 30 days
-}
-}
-Note: The system has a MAX_LOOKBACK_DAYS limit (currently 30) to prevent cost spirals.
-### DO: Use Field Selection
-Reduce data transfer by selecting only needed columns:
-requires: {
-'portfolio_snapshots': {
-fields: \['user_id', 'portfolio_data', 'date'\]
-}
-}
-## 7.4 Error Handling
-### DO: Handle Missing Data Gracefully
-async process(context) {
-const { data, entityId } = context;
-const portfolio = data\['portfolio_snapshots'\];
-if (!portfolio) {
-this.log('WARN', 'No portfolio data for ' + entityId);
-return; // Skip this entity
-}
-// Process...
-}
-### DO: Use Mandatory for Critical Data
-requires: {
-'portfolio_snapshots': {
-mandatory: true // Fail if missing
-}
-}
-## 7.5 Performance
-### DO: Use Per-Entity for Scalability
-For user/entity-specific calculations, always use per-entity mode:
-type: 'per-entity'
-Benefits:
-- Automatic parallelization
-- Memory-efficient streaming
-- Worker pool support
-- Checkpointing and resume
-### DON'T: Load Everything Into Memory
-The framework handles batching - don't try to load all entities yourself.
-## 7.6 Common Patterns
-### Pattern: Aggregation
-// Create per-entity computation first
-class UserMetrics extends Computation {
-static getConfig() {
-return { name: 'UserMetrics', type: 'per-entity', ... };
-}
-}
-// Then create global aggregation
-class SystemSummary extends Computation {
-static getConfig() {
-return {
-name: 'SystemSummary',
-type: 'global',
-dependencies: \['UserMetrics'\]
-};
-}
-async process(context) {
-const userMetrics = context.getDependency('UserMetrics');
-// Aggregate all user results
-}
-}
-### Pattern: Time Series
-class RunningBalance extends Computation {
-static getConfig() {
-return {
-name: 'RunningBalance',
-type: 'per-entity',
-isHistorical: true,
-requires: { 'transactions': { lookback: 1 } }
-};
-}
-async process(context) {
-const { previousResult, data, entityId } = context;
-const prevBalance = previousResult?.balance || 0;
-const todayChange = calculateChange(data);
-this.setResult(entityId, {
-balance: prevBalance + todayChange
-});
-}
-}
-# Appendix A: Quick Reference
-## Computation Template
-const { Computation } = require('../framework');
-class MyComputation extends Computation {
-static getConfig() {
-return {
-name: 'MyComputation',
-type: 'per-entity',
-category: 'analytics',
-requires: {
-'table_name': {
-lookback: 0,
-mandatory: true
-}
-},
-dependencies: \[\],
-schedule: 'daily'
-};
-}
-async process(context) {
-const { data, entityId, rules } = context;
-// Your logic here
-const result = {};
-this.setResult(entityId, result);
-}
-}
-module.exports = MyComputation;
-## Common Context Properties
-context.data // Fetched table data
-context.entityId // Current entity (per-entity only)
-context.date // Target date (YYYY-MM-DD)
-context.rules // Business rules modules
-context.references // Global reference data
-context.getDependency(name, id) // Get dependency result
-context.previousResult // Yesterday (isHistorical only)
-## Useful Helper Methods
-this.setResult(entityId, result) // Save result
-this.setGlobalResult(result) // Save (global only)
-this.log(level, message) // Log message
-this.get(obj, 'path.to.prop', default) // Safe access
-# Appendix B: Framework Architecture
-The system is built on a layered architecture that separates concerns:
-## Layer 1: Core Framework
-- Computation base class
-- Manifest builder (DAG construction)
-- Orchestrator (execution engine)
-- Rules registry (business logic injection)
-## Layer 2: Data Layer
-- Schema registry (dynamic schema discovery)
-- Query builder (SQL generation)
-- Data fetcher (BigQuery interface)
-- State repository (result caching)
-## Layer 3: Storage Layer
-- Storage manager (BigQuery & Firestore)
-- Checkpointer (resume capability)
-- Lineage tracker (data provenance)
-## Layer 4: Execution Layer
-- Task runner (middleware chain)
-- Remote task runner (worker pool)
-- Profiler middleware (performance tracking)
-- Cost tracker middleware (billing)
-## Layer 5: Scheduling Layer
-- Scheduler (Cloud Tasks integration)
-- Watchdog (zombie detection)
-- Dispatcher (request routing)
-All layers are domain-agnostic. Business logic lives entirely in the rules modules and computation implementations.
-# End of Developer Manual
-_For additional support, refer to the source code documentation or contact the development team._