bulltrackers-module 1.0.172 → 1.0.173

package/README.MD

@@ -1,179 +1,3 @@
-[ MAIN README ]
-
-Project Overview
-
-Welcome to the Bulltrackers Project
-
-This project is a wide ranging ingestion pipeline for the analysis of an infinitely scalable set of retail portfolios, specifically focusing on the analysis of public eToro accounts. The project covers the pipeline of ingestion of eToro metrics across user portfolios, social data, asset pricing and trading data, builds in-depth computations based on this data, and then iteratively produces further computations based on the results of prior computations. This is all then provided through a customised dynamically generated type-safe frontend integration.
-
-The objective is simple; Transform the possibilities of the everyday Joe, give them the same level of access as hedge funds get, without the millions of dollars in upfront costs. For too long have hedge funds held the upper hand, with pipelines connected directly to brokerages, where every trade you make is immediately sent to a dozen firms trading against you, feeding their algorithms to make decisions.
-
-This project turns the tables, gives you the upper hand, and hands you the data they pay millions for.
-
-What if you could build a system to monitor exactly what the best - and worst - are doing? Copy their trades, or inverse them. 
-This is just one single possibility that this project offers, one of an infinite range.
-If you can think of it, you can build it.
-
-
-[ Architecture ]
-
-This project consists of 3 primary backend packages.
-
-Bulltrackers - This is the core package, consisting of all the cloud functions used in the project, using a heavily customised, extremely efficient and robust pipeline for every functionality of the project. It is designed in such a way that costs are minimised to extreme lengths, processing speeds are lightning fast and creating additional cloud functions and then deploying them is as simple as adding a new file, plugging it into the pipeline and setting up their configs.
-
-Calculations - This is the pairing package for the computations module of the Bulltrackers package. It provides a way to simply insert a new computation into its relevant category, or introduce a new category, without any additional code changes. Simply create a computation, list any dependencies, and the computation system will handle the rest as-is. 
-
-web-components - This is the frontend-as-a-backend integration. Rather than building a complex frontend that must be modified for every time we introduce a new computation, this system is able to produce a completely automated type-safe schema for defining new computations directly from the computation system. We simply produce a new computation, give it a schema, and once it has run, the web-components will pull the new schema in. The result is that we do not need to handle the annoyance of typescript fiddling, we can just pass a chart component the data to render, set the position and the page path and job done.
-
-[ Things you MUST understand, before attempting to understand this project ]
-
-1. 
-This is NOT, a project you can simply "npm install", in fact doing so is a very bad idea. You will rapidly find that it won't work, I have not provided the configurations for api routes, core parameters for inner workings, and much of the logic will not make any sense to you, regardless of your experience in developing anything. This is intentional; I will not ever be providing the routes for finding the data that is processed here, it would be irresponsible of me. Furthermore, if you attempt to find the configuration values yourself through whatever means, you are likely to violate your own country laws, scraping can - and has been - proven to be a criminal act in some jurisdictions, you MUST seek permission by both eToro, and depending on your objectives, the targeted user(s). Without this permission, I heavily caution any interaction with this project, and warn you that you will likely risk both civil and criminal charges that can ruin your life.
-
-Do not take this risk.
-
-2.
-This project revolves around a few key facts on the eToro system, how it works and the fundamental inner workings of what is perhaps a rather unique brokerage. Indeed, without much of how eToro works being so, this project would not be possible. Outside of eToro being my primary brokerage, this is a key reason for choosing eToro as the data provider, their choices in architecture and their own website configurations make this project possible. Not easy, but possible.
-
-One of these key facts is understanding what happens when a user makes their account. Unlike normal sites, eToro does not assign randomised IDs to each user upon account creation, they are simple integers ordered from account creation dates. Account ID 1 is the first account, Account ID 35m is the 35 millionth account, and so on. This is a key fact to understand when reviewing the task engine and orchestrators.
-
-Another fact is how eToro processes requests for user data. It is not possible to fetch most of a private accounts' data, there are some exceptions to this but for the most part, when using an API route that allows for user batching - submitting multiple user IDs into one request - if any of the users' passed to that API are private, eToro returns nothing for that user; not null, but literally nothing, the user will not be included in the response. This is a very useful fact, as you will find later, I use this built-in filter to my advantage and it saves a significant sum of wall-time in the processing, it also forms a mathematically powerful positive feedback loop by allowing us to mark any user we find was a private account, and ensuring we never try to process them again.
-
-3.
-It is not possible to discover how much an account is worth. This was possible in early January 2025, through what was a serious oversight in an update eToro performed, which exposed a parameter called LotCount. This was a parameter that showed the number of shares each Public user owned for a given asset, and by extrapolation you could then infer their portfolio values. The developer of this project was solely responsible for uncovering this bug, reporting it and as a result holds considerable favour within the eToro public, but also staff, community. However, what is possible is uncovering whether a user is of Bronze tier or not, whilst all other tier information is private, eToro intentionally exposes whether a given public account is of Bronze tier, meaning <5k USD portfolio value. This metric is used in some computations to highlight the difference - or lack of - between Bronze and Non-Bronze portfolios. It is repeated again, that this is a public metric, I perform no access to user data beyond what is publicly available and other tier information is not accessible. It has been clarified with eToro that exposing whether a user is of Bronze tier or not, is indeed the intended mechanism.
-
-4.
-Due to the above restriction on it not being possible to identify the exact dollar cost of a single position, no computations revolve around USD values, but a fair number of computations revolve around computing the value of positions based on their P&L. The main portfolio API used in the task engine, returns ONLY the top-level positioning for each users' portfolio, for example it will show they own an exposure of 9% invested in NVDA, but not that the position is 9 positions of NVDA at 1% each. We cannot see the individual position breakdown. It is possible to get this through another endpoint, but requires iterating over every single position one at a time to discover those values; This is not worthwhile for normal user task types, but is used for speculator task types, where we know which position is useful to look at. Of note, the main portfolio API does not return details like leverage, entry point or other such values, it simply returns the averages of the whole position, the individual position API used by the speculator task type, returns this more granular data. On the flip side, the history API used for all task types, returns closed trade values over a YTD period, and each days snapshot can be used to compute the individual position changes. This concept is covered more in depth in task engine documentation.
-
-[ Conventions ]
-
-1. 
-The Pipe architecture used in the core bulltrackers package is a MUST. The existence of this pipe architecture is a fundamental feature of the wider system, and allows for Dependency Injections, making every cloud function reusable, testable and standalone. A single cloud function can break, which is inevitable, and the wider system is unaffected. All cloud functions must use the dependency injection and never define their own require statements within their packages, we simply pass down their dependencies.
-
-2.
-As with all professional systems, DRY applies. Wherever feasible, sensible and logical to do so, functionality for cloud functions should be abstracted into helper modules, or - if used across multiple cloud functions - shifted into the core pipe, which is passed to all cloud functions.
-
-3.
-The use of firestore is huge within this project, and as such, it is a primary cost aspect, as well as significant driver of wall-time. It is therefore an absolute rule to never design queries, reads, writes or deletes within firestore in such a way that they are inefficient. Extreme lengths should be taken to dramatically reduce, simplify and redesign systems that inefficiently use firestore. This is obligatory for any cost efficient system.
-
-4.
-Computations in the computation system are automatically forced to use a strict naming convention, removing underscores and replacing with hyphens. This is a necessity to ensure we do not accidentally name new computations in a way that the system doesn't expect and have a string of errors. The pass architecture of the computation system naturally results in a whole array of errors, if a single computation fails and thus their dependencies cannot proceed, this can take a significant time to debug, so resolving common mistakes early on is key.
-
-5.
-Newly added computations must be integrated into an appropriate category, it is fine to introduce a new category name - this is dynamically auto-handled by the computation system and requires no new code changes, but the name must be understandable and clear to what the computations themselves represent. Furthermore, if a computation requires "yesterday" data, then it must be added to a sub-folder name exactly "historical" within the relevant category, otherwise the computation will fail as it will not receive the data it requires. Please review the computation system documentation to understand this concept and why it matters.
-
-6. 
-New computations MUST - I repeat - MUST define a static getDependencies() within the computation class, IF they use a dependency of another computation. By adding this, you allow the computation manifest to produce the exact order the computation must be run within the wider dependency architecture, as it uses Kahns' algorithm for topological sorting of dependencies. This MUST be given in the computation class in a format like :
-
-```js
-static getDependencies() {
-        return ['user_profitability_tracker']; 
-    }
-```
-
-Any computation that does not define a dependency is automatically assigned as pass 1, and if this is incorrect then the computation will fail.
-
-Furthermore to this, we must ensure that a computation does not try to create a dependency structure that results in circularity, it may well be the case that up to 5 passes is not enough to avoid this, in which case a 6th pass would have to be developed. This isn't a significant change and is easily implementable, however as-is, 5 passes is enough.
-
-The manifest is almost entirely dynamic, and it is likely that you will not need to introduce code changes to the manifest to handle any newly introduced computation; There are some exceptions to this where a computation requires some special handling, but it is intended that we will resolve this by listing the conditions for how the manifest handles special computations, within the computation code itself, for the manifest to then read. A future update will cover this in more detail, and the results of it will be listed in the computation documentation, please refer to that for future updates on this aspect, should you need it.
-
-7.
-There are 2 entry points for this system, there is one in the Bulltrackers module, which defines the exported pipes, and there is another you would build yourself to complete the architecture. The 2nd is not included in this project, as it would expose the configurations used in the deployed cloud functions, but the index.js itself includes the following.
-
-```JS
-const { Firestore, FieldValue, FieldPath } = require('@google-cloud/firestore'); 
-const { PubSub }                           = require('@google-cloud/pubsub');
-const { VertexAI }                         = require('@google-cloud/vertexai');
-const { logger }                           = require('sharedsetup')(__filename);
-const { pipe }                             = require('bulltrackers-module');
-const { calculations, utils }              = require('aiden-shared-calculations-unified');
-
-const fs                                   = require('fs');
-const path                                 = require('path');
-const db                                   = new Firestore();
-const pubsub                               = new PubSub();
-
-const vertexAI                             = new VertexAI({project: 'some_project', location: 'europe-west1'});
-const geminiModel                          = vertexAI.getGenerativeModel({ model: 'gemini-2.5-flash-lite',});
-const configDir                            = path.join(__dirname, 'config');
-const configFiles                          = fs.readdirSync(configDir).filter(f => /_(config|manifest)\.js$/.test(f));
-const cfg                                  = Object.fromEntries(configFiles.map(f => [f.replace(/_(config|manifest)\.js$/, ''),require(path.join(configDir, f))]));
-
-const headerManager                        = new pipe.core.IntelligentHeaderManager(db               , logger, cfg.core);
-const proxyManager                         = new pipe.core.IntelligentProxyManager (db               , logger, cfg.core);
-const batchManager                         = new pipe.core.FirestoreBatchManager   (db, headerManager, logger, cfg.taskEngine);
-
-const allDependencies                      = { db, pubsub, logger, FieldValue, FieldPath, headerManager, proxyManager, batchManager, firestoreUtils: pipe.core.firestoreUtils, pubsubUtils: pipe.core.pubsubUtils, geminiModel, calculationUtils: utils };
-const handlers = {
-    discoveryOrchestrator          : ()     => pipe.orchestrator     .runDiscoveryOrchestrator          (cfg.orchestrator             , allDependencies),
-    updateOrchestrator             : ()     => pipe.orchestrator     .runUpdateOrchestrator             (cfg.orchestrator             , allDependencies),
-    speculatorCleanupOrchestrator  : ()     => pipe.maintenance      .runSpeculatorCleanup              (cfg.cleanup                  , allDependencies),
-    fetchEtoroInsights             : ()     => pipe.maintenance      .runFetchInsights                  (cfg.fetchInsights            , allDependencies),
-    updateClosingPrices            : ()     => pipe.maintenance      .runFetchPrices                    (cfg.priceFetcher             , allDependencies),
-    socialOrchestrator             : ()     => pipe.maintenance      .runSocialOrchestrator             (cfg.social                   , allDependencies),
-    priceBackfill                  : ()     => pipe.maintenance      .runBackfillAssetPrices            (cfg.priceBackfill            , allDependencies),
-
-    taskEngineHandler              : (m, c) => pipe.taskEngine       .handleRequest                     (m, c, cfg.taskEngine         , allDependencies),
-    dispatcher                     : (m, c) => pipe.dispatcher       .handleRequest                     (m, c, cfg.dispatcher         , allDependencies),
-    invalidSpeculatorHandler       : (m, c) => pipe.maintenance      .handleInvalidSpeculator           (m, c, cfg.invalidSpeculator  , allDependencies),
-    socialTaskHandler              : (m, c) => pipe.maintenance      .handleSocialTask                  (m, c, cfg.social             , allDependencies),
-    
-    computationSystemPass1         : ()     => pipe.computationSystem.runComputationPass(cfg.computationSystem, allDependencies, cfg.computation),
-    computationSystemPass2         : ()     => pipe.computationSystem.runComputationPass(cfg.computationSystem, allDependencies, cfg.computation),
-    computationSystemPass3         : ()     => pipe.computationSystem.runComputationPass(cfg.computationSystem, allDependencies, cfg.computation),
-    computationSystemPass4         : ()     => pipe.computationSystem.runComputationPass(cfg.computationSystem, allDependencies, cfg.computation),
-    computationSystemPass5         : ()     => pipe.computationSystem.runComputationPass(cfg.computationSystem, allDependencies, cfg.computation),
-    genericApiV2          : pipe.api.createApiApp(cfg.genericApiV2, allDependencies, calculations)
-};
-Object.assign(exports, handlers); 
-if (require.main === module) {const port = process.env.PORT || 8080; 
-    exports.genericApiV2.listen(port, () => console.log(`API listening on port ${port}`));}
-```
-
-This index.js exposes the entry points for each cloud function, which is then linked to their relevant pipelines, which allows GCP, my chosen cloud computing provider, to deploy each function. I personally use a custom deployment system to handle the commands for each pipeline, and this system will be included in a future separate npm package, it also handles passing of the configurations, .env values and all other key aspects.
-
-The key note to take is that both these 2 core index.js entry points are clean and easy to read, they provide clear names for their intentions, and it is easy to follow the pipes to each individual cloud function. The inner workings of each pipeline are obscured behind their own internal modules, and thus making a change to one pipe, does not have a direct effect on any other pipe, with the sole exception of modifications in the core pipeline, which is used across all functions, but rarely needs modifying.
-
-All cloud functions have their own pipe, with the exception that the orchestrators and handlers for both portfolio and social tasks are passed through a single pipe which internally splits off into 2 sub pipes, which themselves handle the orchestration of the tasks and the handling of the tasks as 2 seperate cloud functions, or in the case of the portfolio processing, it actually uses 2 cloud functions for the orchestration, and 1 cloud function for task processing. More on this architecture and the reasoning for choices can be found in the task engine and social processing documentation.
-
-It is OK for a main pipeline, exposed in the outer index.js, to be handling multiple cloud functions so long as their core functionality is similar. 2 cloud functions that exist to process only slightly differing objectives, can be given the same core pipeline, and then be split off into sub pipes at the inner layers. However, a pipeline should not consist of functionality that is dissimilar to other components of that same pipeline. A future development may chang this, to further simplify the outer index.js into solely the core features. This would result in the discovery & update orchestrator being merged into a main pipeline with the task engine and dispatcher, as they all work together to achieve an end objective - the processing of user portfolios. This would then expose a single primary portfolioProcessor pipeline. 
-
-[ Objectives ]
-
-There are some clear, high-level objectives this project, encompassing the likely dozens of npm packages it will require for an end-result, attempts to achieve.
-
-These are :
-
-1. A world-class demonstration in how to build, from the ground-up, a scalable, highly efficient data ingestion pipeline that not only ingests.
-
-2. A comprehensive data computation layer which allows for extremely advanced, highly complex computations to be integrated seamlessly and dynamically, with the ability to build data upon data, and have a limitless range of possible computations, all of which are scalable and processed in not seconds, but milliseconds.
-
-3. A dynamically built, completely automated frontend integration that exposes easy-to-use displays for all the computed data, along with advanced customisation options and a no-code layer for allowing an end-user to produce their own computations on-top of raw data, or pre-computed data.
-
-4. A demonstration in what is possible for the eToro community, and their own developers, particularly of potential interest to the Popular Investors fortunate enough to be handed access to the API layer. Though I must add the caveat that you will likely not be able to build anywhere similar to the scale of this project as you will have rate limits and much less data capability than this project allows, however it should give you some idea of what you can build yourself. Please check out the separate computation module for some inspiration.
-
-5. A personal test for myself, as the sole developer of this project, and completely expecting very few contributions to the wider system, this is a significant undertaking. I am not a professional developer, and so this is quite the learning curve. I believe in the hidden and unused value that data analysis can offer, and eToro offers, arguably unknowingly, a rare opportunity to build something that is of immense value.
-
-6. To produce a series of computations that are of such value that they are provably predicative in their insight, and thus have financial value to offer to a front-end user.
-
-[ Future Updates ]
-
-1. Producing and exposing unit tests for each computation; These will be dynamically produced based on the contents of the computation.
-2. Refining logging and building a custom tool for detecting problems across all the codebase by creating a log sink to read the results of the custom logging.log() function, across the project.
-3. Reviewing existing computations, refining the results and optimising computations. Then developing pass 5 computations to prove the signals of each signal-generating computations are alpha-generative.
-4. General solidification of the codebase, marking aspects as TODO, abstracting remaining magic numbers into configurable variables. Applying Dependency Injection for the few remaining imports.
-5. Building up the frontend integration, mapping data to charts and stress-testing edge cases in schema formats.
-
-For coverage on each cloud function, please refer to their individual documentation, which you can find linked below.
-
-[View Computation System Documentation](./docs/ComputationSystem.MD)
-[View Computation System Documentation](./docs/TaskEngine.MD)
-
-
-
-
-
-
-
-
-
+This is the directory for the cloud function entry points in the Bulltrackers Platform
+
+Please refer to BullTrackers\Backend\documentation for the full documentation

package/functions/computation-system/controllers/computation_controller.js

@@ -1,53 +1,42 @@
 /**
- * @fileoverview Control Layer - Orchestrates Computation Execution
- * UPDATE: Imported and exposed HistoryExtractor in the math context.
- * UPDATE: Implemented strict User Type segregation. 'All' now defaults to 'Normal'.
+ * FIXED: computation_controller.js
+ * V3.2: Supports Streaming Trading History (todayHistory) separate from Yesterday's Portfolio.
  */
 
-const { 
-    DataExtractor, 
-    HistoryExtractor, 
-    MathPrimitives, 
-    Aggregators, 
-    Validators, 
-    SCHEMAS, 
-    SignalPrimitives 
-} = require('../layers/math_primitives');
-
-const { 
-    loadDailyInsights, 
-    loadDailySocialPostInsights, 
-    getPortfolioPartRefs, 
-    getHistoryPartRefs, 
-    streamPortfolioData, 
-    streamHistoryData 
+const { DataExtractor, 
+        HistoryExtractor, 
+        MathPrimitives, 
+        Aggregators, 
+        Validators, 
+        SCHEMAS, 
+        SignalPrimitives, 
+        DistributionAnalytics, 
+        TimeSeries,
+        priceExtractor } 
+= require('../layers/math_primitives');
+
+const { loadDailyInsights, 
+        loadDailySocialPostInsights,  
 } = require('../utils/data_loader');
 
-// ============================================================================
-// DATA LOADER WRAPPER
-// ============================================================================
-
 class DataLoader {
     constructor(config, dependencies) {
         this.config = config;
         this.deps = dependencies;
         this.cache = { mappings: null, insights: new Map(), social: new Map() };
     }
-
     async loadMappings() {
         if (this.cache.mappings) return this.cache.mappings;
         const { calculationUtils } = this.deps;
         this.cache.mappings = await calculationUtils.loadInstrumentMappings();
         return this.cache.mappings;
     }
-
     async loadInsights(dateStr) {
         if (this.cache.insights.has(dateStr)) return this.cache.insights.get(dateStr);
         const insights = await loadDailyInsights(this.config, this.deps, dateStr);
         this.cache.insights.set(dateStr, insights);
         return insights;
     }
-
     async loadSocial(dateStr) {
         if (this.cache.social.has(dateStr)) return this.cache.social.get(dateStr);
         const social = await loadDailySocialPostInsights(this.config, this.deps, dateStr);
@@ -56,42 +45,51 @@ class DataLoader {
     }
 }
 
-// ============================================================================
-// CONTEXT BUILDER
-// ============================================================================
-
 class ContextBuilder {
     static buildPerUserContext(options) {
-        const {
-            todayPortfolio, yesterdayPortfolio, todayHistory, yesterdayHistory,
-            userId, userType, dateStr, metadata, mappings, insights, socialData,
-            computedDependencies, config, deps
+        const { 
+            todayPortfolio, 
+            yesterdayPortfolio, 
+            todayHistory, 
+            yesterdayHistory,
+            userId, 
+            userType, 
+            dateStr, 
+            metadata, 
+            mappings, 
+            insights, 
+            socialData,
+            computedDependencies, 
+            previousComputedDependencies,
+            config, 
+            deps
         } = options;
 
         return {
-            // User Identity & Data
             user: {
                 id: userId,
                 type: userType,
                 portfolio: { today: todayPortfolio, yesterday: yesterdayPortfolio },
                 history: { today: todayHistory, yesterday: yesterdayHistory }
             },
-            // Global Time & Data
             date: { today: dateStr },
             insights: { today: insights?.today, yesterday: insights?.yesterday },
             social: { today: socialData?.today, yesterday: socialData?.yesterday },
-            // Helpers
             mappings: mappings || {},
-            math: {
+            math: { // Introduced a new computation class in the math primitives? Add it here. Then add it to meta context a little further down.
                 extract: DataExtractor,
                 history: HistoryExtractor,
                 compute: MathPrimitives,
                 aggregate: Aggregators,
                 validate: Validators,
                 signals: SignalPrimitives,
-                schemas: SCHEMAS
+                schemas: SCHEMAS,
+                distribution : DistributionAnalytics,
+                TimeSeries: TimeSeries,
+                priceExtractor : priceExtractor
             },
             computed: computedDependencies || {},
+            previousComputed: previousComputedDependencies || {},
             meta: metadata,
             config,
             deps
@@ -100,8 +98,15 @@ class ContextBuilder {
 
     static buildMetaContext(options) {
         const {
-            dateStr, metadata, mappings, insights, socialData,
-            computedDependencies, config, deps
+            dateStr, 
+            metadata, 
+            mappings, 
+            insights, 
+            socialData,
+            computedDependencies, 
+            previousComputedDependencies,
+            config, 
+            deps
         } = options;
 
         return {
@@ -109,16 +114,20 @@ class ContextBuilder {
             insights: { today: insights?.today, yesterday: insights?.yesterday },
             social: { today: socialData?.today, yesterday: socialData?.yesterday },
             mappings: mappings || {},
-            math: {
+            math: { // Introduced a new computation class in the math primitives? Add it here.
                 extract: DataExtractor,
                 history: HistoryExtractor,
                 compute: MathPrimitives,
                 aggregate: Aggregators,
                 validate: Validators,
                 signals: SignalPrimitives,
-                schemas: SCHEMAS
+                schemas: SCHEMAS,
+                distribution: DistributionAnalytics,
+                TimeSeries: TimeSeries,
+                priceExtractor : priceExtractor
             },
             computed: computedDependencies || {},
+            previousComputed: previousComputedDependencies || {},
             meta: metadata,
             config,
             deps
@@ -126,10 +135,6 @@ class ContextBuilder {
     }
 }
 
-// ============================================================================
-// EXECUTOR
-// ============================================================================
-
 class ComputationExecutor {
     constructor(config, dependencies, dataLoader) {
         this.config = config;
@@ -137,66 +142,57 @@ class ComputationExecutor {
         this.loader = dataLoader;
     }
 
-    async executePerUser(calcInstance, metadata, dateStr, portfolioData, historyData, computedDeps) {
+    /**
+     * UPDATED: Accepts yesterdayPortfolioData AND historyData separately.
+     */
+    async executePerUser(calcInstance, metadata, dateStr, portfolioData, yesterdayPortfolioData, historyData, computedDeps, prevDeps) {
         const { logger } = this.deps;
         
-        // --------------------------------------------------------------------
-        // 1. DETERMINE TARGET SCHEMA
-        // --------------------------------------------------------------------
-        // We strictly enforce separation here. 
-        // Unless 'speculator' is explicitly requested, we default to 'normal'.
-        // This effectively deprecates 'all' by treating it as 'normal' for safety.
-        const targetUserType = (metadata.userType === 'speculator')
-            ? SCHEMAS.USER_TYPES.SPECULATOR 
-            : SCHEMAS.USER_TYPES.NORMAL;
-        
+        // Fix for the 'all' userType discrepancy:
+        const targetUserType = metadata.userType; // 'all', 'normal', or 'speculator'
+
         const mappings = await this.loader.loadMappings();
-        const insights = metadata.rootDataDependencies?.includes('insights') 
-            ? { today: await this.loader.loadInsights(dateStr) } : null;
+        const insights = metadata.rootDataDependencies?.includes('insights') ? { today: await this.loader.loadInsights(dateStr) } : null;
 
-        // Loop through user batch
         for (const [userId, todayPortfolio] of Object.entries(portfolioData)) {
-            const yesterdayPortfolio = historyData ? historyData[userId] : null;
+            // 1. Get Yesterday's Portfolio (if available)
+            const yesterdayPortfolio = yesterdayPortfolioData ? yesterdayPortfolioData[userId] : null;
             
-            // ----------------------------------------------------------------
-            // 2. IDENTIFY ACTUAL DATA TYPE
-            // ----------------------------------------------------------------
-            // We inspect the data structure to know what we are holding.
-            const actualUserType = todayPortfolio.PublicPositions 
-                ? SCHEMAS.USER_TYPES.SPECULATOR 
-                : SCHEMAS.USER_TYPES.NORMAL;
-
-            // ----------------------------------------------------------------
-            // 3. STRICT GATEKEEPING
-            // ----------------------------------------------------------------
-            // If the computation asked for 'normal' (or 'all'), but we have a 'speculator', SKIP.
-            // If the computation asked for 'speculator', but we have a 'normal', SKIP.
-            if (targetUserType !== actualUserType) continue;
+            // 2. Get Today's Trading History (if available)
+            const todayHistory = historyData ? historyData[userId] : null;
+
+            const actualUserType = todayPortfolio.PublicPositions ? SCHEMAS.USER_TYPES.SPECULATOR : SCHEMAS.USER_TYPES.NORMAL;
+            
+            // Filtering Logic
+            if (targetUserType !== 'all') {
+                 const mappedTarget = (targetUserType === 'speculator') ? SCHEMAS.USER_TYPES.SPECULATOR : SCHEMAS.USER_TYPES.NORMAL;
+                 if (mappedTarget !== actualUserType) continue;
+            }
 
             const context = ContextBuilder.buildPerUserContext({
                 todayPortfolio, yesterdayPortfolio,
+                todayHistory, // Passed to context
                 userId, userType: actualUserType, dateStr, metadata, mappings, insights,
-                computedDependencies: computedDeps, config: this.config, deps: this.deps
+                computedDependencies: computedDeps, 
+                previousComputedDependencies: prevDeps,
+                config: this.config, deps: this.deps
             });
 
-            try {
-                await calcInstance.process(context);
-            } catch (e) {
-                logger.log('WARN', `Calc ${metadata.name} failed for user ${userId}: ${e.message}`);
-            }
+            try { await calcInstance.process(context); } 
+            catch (e) { logger.log('WARN', `Calc ${metadata.name} failed for user ${userId}: ${e.message}`); }
         }
     }
 
-    async executeOncePerDay(calcInstance, metadata, dateStr, computedDeps) {
+    async executeOncePerDay(calcInstance, metadata, dateStr, computedDeps, prevDeps) {
         const mappings = await this.loader.loadMappings();
-        const insights = metadata.rootDataDependencies?.includes('insights') 
-            ? { today: await this.loader.loadInsights(dateStr) } : null;
-        const social = metadata.rootDataDependencies?.includes('social')
-            ? { today: await this.loader.loadSocial(dateStr) } : null;
+        const insights = metadata.rootDataDependencies?.includes('insights') ? { today: await this.loader.loadInsights(dateStr) } : null;
+        const social = metadata.rootDataDependencies?.includes('social') ? { today: await this.loader.loadSocial(dateStr) } : null;
 
         const context = ContextBuilder.buildMetaContext({
             dateStr, metadata, mappings, insights, socialData: social,
-            computedDependencies: computedDeps, config: this.config, deps: this.deps
+            computedDependencies: computedDeps,
+            previousComputedDependencies: prevDeps,
+            config: this.config, deps: this.deps
         });
 
         return await calcInstance.process(context);

package/functions/computation-system/helpers/computation_manifest_builder.js

@@ -80,7 +80,7 @@ function findCycles(manifestMap, adjacencyList) {
  */
 function getDependencySet(endpoints, adjacencyList) {
     const required = new Set(endpoints);
-    const queue = [...endpoints];
+    const queue    = [...endpoints];
     while (queue.length > 0) { const calcName = queue.shift(); const dependencies = adjacencyList.get(calcName);
     if (dependencies) { for (const dep of dependencies) { if (!required.has(dep)) { required.add(dep); queue.push(dep); } } } }
     return required;
@@ -104,6 +104,7 @@ function buildManifest(productLinesToRun = [], calculations) {
     const reverseAdjacency = new Map(); 
     const inDegree         = new Map();
     let hasFatalError      = false;
+    
     /* ---------------- 1. Load All Calculations ---------------- */
     log.step('Loading and validating all calculation classes…');
     const allCalculationClasses = new Map();
@@ -121,10 +122,12 @@ function buildManifest(productLinesToRun = [], calculations) {
         if (typeof Class.getDependencies !== 'function') { log.fatal(`Calculation "${normalizedName}" is missing the static getDependencies() method. Build FAILED.`); hasFatalError = true;return; }
         // --- RULE 3: Check for static getSchema() ---
         if (typeof Class.getSchema !== 'function') {log.warn(`Calculation "${normalizedName}" is missing the static getSchema() method. (Recommended)`); }
-        const metadata = Class.getMetadata();
+        const metadata     = Class.getMetadata();
         const dependencies = Class.getDependencies().map(normalizeName);
         // --- RULE 4: Check for isHistorical mismatch ---
-        if (metadata.isHistorical === true && !Class.toString().includes('yesterdayPortfolio')) { log.warn(`Calculation "${normalizedName}" is marked 'isHistorical: true' but does not seem to use 'yesterdayPortfolio'.`); }
+        if (metadata.isHistorical === true && !Class.toString().includes('yesterday')) { // UPDATED FOR MATH LAYER, THIS IS A LITTLE BRITTLE, BUT FOR NOW FINE...
+            log.warn(`Calculation "${normalizedName}" is marked 'isHistorical: true' but does not seem to reference 'yesterday' data.`); 
+        }
         const manifestEntry = {
             name: normalizedName,
             class: Class,
@@ -157,6 +160,7 @@ function buildManifest(productLinesToRun = [], calculations) {
 
     if (hasFatalError) { throw new Error('Manifest build failed due to missing static methods in calculations.'); }
     log.success(`Loaded and validated ${manifestMap.size} total calculations.`);
+
     /* ---------------- 2. Validate Dependency Links ---------------- */
     log.divider('Validating Dependency Links');
     const allNames = new Set(manifestMap.keys());
@@ -177,6 +181,7 @@ function buildManifest(productLinesToRun = [], calculations) {
     }
     if (invalidLinks) { throw new Error('Manifest validation failed. Fix missing or self-referencing dependencies.'); }
     log.success('All dependency links are valid.');
+
     /* ---------------- 3. Filter for Product Lines ---------------- */
     log.divider('Filtering by Product Line');
     // 1. Find all "endpoint" calculations (the final signals) in the target product lines.
@@ -189,17 +194,18 @@ function buildManifest(productLinesToRun = [], calculations) {
     log.info(`Identified ${productLineEndpoints.length} endpoint/core calculations.`);
     log.info(`Traced dependencies: ${requiredCalcs.size} total calculations are required.`);
     // 4. Create the final, filtered maps for sorting.
-    const filteredManifestMap = new Map();
-    const filteredInDegree = new Map();
+    const filteredManifestMap      = new Map();
+    const filteredInDegree         = new Map();
     const filteredReverseAdjacency = new Map();
     for (const name of requiredCalcs) { filteredManifestMap.set(name, manifestMap.get(name)); filteredInDegree.set(name, inDegree.get(name));
     const consumers = (reverseAdjacency.get(name) || []).filter(consumer => requiredCalcs.has(consumer)); filteredReverseAdjacency.set(name, consumers); }
     log.success(`Filtered manifest to ${filteredManifestMap.size} calculations.`);
+
     /* ---------------- 4. Topological Sort (Kahn's Algorithm) ---------------- */
     log.divider('Topological Sorting (Kahn\'s Algorithm)');
     const sortedManifest = [];
-    const queue = [];
-    let maxPass = 0;
+    const queue          = [];
+    let maxPass          = 0;
     for (const [name, degree] of filteredInDegree) { if (degree === 0) { queue.push(name); filteredManifestMap.get(name).pass = 1; maxPass = 1; } }
     queue.sort();
     while (queue.length) {
@@ -216,6 +222,7 @@ function buildManifest(productLinesToRun = [], calculations) {
         const cycles = findCycles(filteredManifestMap, adjacency);
         for (const c of cycles) log.error('Cycle: ' + c.join(' → '));
         throw new Error('Circular dependency detected. Manifest build failed.'); }
+
     /* ---------------- 5. Summary ---------------- */
     log.divider('Manifest Summary');
     log.success(`Total calculations in build: ${sortedManifest.length}`);
@@ -251,10 +258,6 @@ async function generateSvgGraph(manifest, filename = 'dependency-tree.svg') {
     dot += '}\n';
     try {
         const svg = await viz.renderString(dot, { format: 'svg' });
-        
-        // --- MODIFIED PATH ---
-        // Writes to the root of the 'bulltrackers-module' package directory, 
-        // which is a safer, more predictable location.
         const out = path.join('/tmp', filename);
         fs.writeFileSync(out, svg);
         log.success(`Dependency tree generated at ${out}`);
@@ -270,21 +273,13 @@ async function generateSvgGraph(manifest, filename = 'dependency-tree.svg') {
  */
 function build(productLinesToRun, calculations) {
     try {
-        // This function MUST NOT fail or generate SVGs.
-        // It has one job: build and return the manifest array.
         const manifest = buildManifest(productLinesToRun, calculations);
         return manifest; // Returns the array
     } catch (error) { 
         log.error(error.message); 
-        // If buildManifest fails, it will throw.
-        // We return null so the caller can handle the failure.
         return null; 
     }
 }
 
-// --- REMOVED ---
-// The entire 'if (require.main === module)' block has been removed
-// to prevent any local 'require' conflicts. This file is now a pure
-// module and must be called by another script.
 
-module.exports = { build, generateSvgGraph }; // <-- Also exporting generateSvgGraph
+module.exports = { build, generateSvgGraph };