bulltrackers-module 1.0.26 → 1.0.28

package/README.MD

@@ -0,0 +1,153 @@
+# BullTrackers Module (`bulltrackers-module`)
+
+**Version:** 1.0.27
+
+## Overview
+
+This package encapsulates the core logic and shared utilities for the BullTrackers cloud function architecture. It promotes code reuse and separation of concerns by providing abstracted handlers and helper functions for various backend services.
+
+## Core Concepts (Based on your Documentation.md)
+
+* **eToro User IDs (CIDs):** Sequential integers assigned upon registration.
+
+eToro assigns a hidden UID to each users' account in order for their backend configurations to identify users without needing to reference a username.
+This is a common fact across most modern sites, but eToro is slightly unique in that their UIDs are not randomised, they are sequential.
+This means that UID 1 is the first account ever made, and UID 1 million is the 1 millionth account made.
+
+* **Block Sampling:** Randomly sampling users within 1M ID blocks to ensure representative data across different registration cohorts.
+
+Given how the UIDs are assigned on eToro, we can group users by their UID into 1m block ranges, and then from this we can sample users randomly to produce data across each Block.
+Now there is a slight caveat to the current implementation, where our current implementation is designed to fill up a set capacity of users, per user type, per block range. 
+This is statistically biased, because there will be more public users in certain blocks than other blocks.
+The future resolution to this, is to weight the block capacity by that blocks' ratio of public to private users, ensuring representative sampling.
+For now, this is not implemented.
+
+* **User Types:** Differentiating between 'normal' users and 'speculators' based on portfolio activity and specific asset holdings.
+
+This system attempts to categorise user portfolios, and the users themselves, by types and (in the future, sub types).
+These are based on the response of the rankings API call, which returns enough detail to figure out whether a user is speculative, normal, inactive or private.
+In the future the plan is to create sub categories that state the users' risk group alongside their main category.
+Private is not a category, it is rather the lack of a category. Users who are private, are implicitely removed from the response of the API, auto-filtering for the system.
+Inactive users are a future implementation of a user type, to allow us to track users who have not logged into the platformm recently but hold positions.
+
+* **Data Flow:** An architecture primarily using Google Cloud Pub/Sub and Cloud Functions.
+
+The flow of data is, in part, event-driven, and in part scheduler driven.
+The system begins by searching for users to fill up the capacity of the block counts for normal and speculator users. This is the discovery process.
+These users are then verified, and if they meet the criteria of a normal or speculator user, then their UID is stored.
+The update task triggers once a day and finds all the users it needs to update the data of, this will include those users the discovery process located and then verified.
+The update task will look at their portfolios, store the contents. 
+
+The submitting of the task type and UIDs is done by the Orchestrators, either the update or discovery orchestrator.
+In discovery mode, the tasks are submitted directly to the task engine through a pub/sub
+In update mode, the tasks are passed to the dispatcher, to then slowly release to the task engine, an intentional choice to ensure the task engine can scale gradually to handle the incoming requests.
+The task engine is event-driven, triggered by pub/sub messages.
+The portfolio data is then stored, and once a day the computation system is run on a cron schedule trigger, fetches all the portfolio data and applies computations to the data, returning results that are stored.
+The API is then able to read these results.
+
+## Module Structure
+
+The module is organized by function, mirroring the cloud function deployment structure.
+
+### `/core`
+
+Contains utilities shared across multiple functions within the module.
+
+* **`/utils`:**
+    * `firestore_utils.js`: Helpers for common Firestore operations (batching, querying blocks, resetting proxies, getting IDs).
+    * `pubsub_utils.js`: Helpers for publishing messages to Pub/Sub topics in batches.
+    * `intelligent_header_manager.js`: Manages a pool of browser headers, selecting them based on past success rates to mimic real user traffic. Stores performance data in Firestore.
+    * `intelligent_proxy_manager.js`: Manages a pool of Google Apps Script proxies, selecting available ones and handling failures/locking.
+
+### `/orchestrator`
+
+Handles the triggering and coordination of data fetching tasks (Discovery and Update).
+
+* **`index.js`:** Exports factory functions (`createDiscoveryOrchestrator`, `createUpdateOrchestrator`).
+* **`/helpers`:**
+    * `discovery_helpers.js`: Logic for checking if discovery is needed, finding candidate users (prioritizing known users, then random generation with exclusions), managing pending lists, and dispatching tasks.
+    * `update_helpers.js`: Logic for identifying users needing updates based on timestamps and dispatching update tasks.
+
+### `/dispatcher`
+
+Acts as an intermediary buffer, receiving tasks from the Orchestrator and slowly releasing them to the Task Engine to prevent overwhelming it.
+
+* **`index.js`:** Exports the factory function `createDispatcherHandler`.
+* **`/helpers/dispatch_helpers.js`:** Contains the core logic for batching and delaying task publishing.
+
+### `/task-engine`
+
+The core processing unit. Receives tasks (discover, verify, update) for different user types (normal, speculator) and executes the corresponding logic using eToro APIs via the proxy manager.
+
+* **`index.js`:** Exports the factory function `createTaskEngineHandler`.
+* **`handler_creator.js`:** Sets up the handler, initializes clients (Firestore, PubSub, managers), and defines the main processing flow.
+* **`/helpers`:**
+    * `discover_helpers.js`: Handles fetching user ranking/activity data, filtering private/inactive users, applying heuristics (especially for speculators), and chaining to the 'verify' task. Also handles logging invalid speculator IDs.
+    * `verify_helpers.js`: Handles fetching portfolio details, checking for specific assets (for speculators) or any positions (for normal users), storing bronze state, updating block counts, and storing verified users.
+    * `update_helpers.js`: Handles fetching the latest portfolio or position data for already verified users, handling private users, and storing the updated data.
+* **`/utils`:**
+    * `firestore_batch_manager.js`: Manages batch writes to Firestore for portfolios (sharded), timestamps, and removing processed speculators from the pending list.
+---
+
+### `/computation-system`
+
+Handles the orchestration and execution of the **unified computation system**. This system runs scheduled calculations (both daily and historical comparisons) using calculation logic imported from the `aiden-shared-calculations-unified` package. It identifies date ranges needing processing or backfilling, loads the necessary portfolio data, runs the calculations, and stores the results.
+
+* **`index.js`:** Exports the factory function `createComputationSystemHandler` and the main orchestration logic `runComputationOrchestrator`.
+* **`handler_creator.js`:** Creates the HTTP-triggered Cloud Function handler that initiates the computation orchestration.
+* **`/helpers/orchestration_helpers.js`:** Contains the core logic for determining dates to process, identifying missing computations, loading data (using `data_loader.js`), initializing calculators from the unified package, processing data across dates, and committing results to Firestore.
+* **`/utils`:**
+    * `data_loader.js`: Functions specifically for loading portfolio data in parts/shards from Firestore for the computation system.
+    * `utils.js`: Helper functions for categorizing calculations (historical vs. daily), managing Firestore batches, handling date ranges, parallel processing, and finding the earliest data dates.
+
+
+### `/generic-api`
+
+Provides a flexible, cached API endpoint (intended for frontend use) to query computed data stored in Firestore.
+
+* **`index.js`:** Exports the factory function `createApiApp` to set up the Express app.
+* **`/helpers/api_helpers.js`:** Contains logic for request validation, date range generation, data fetching from the unified insights collection, and building the calculation map.
+
+### `/invalid-speculator-handler`
+
+Listens to a Pub/Sub topic for messages containing IDs of users identified as private or inactive speculators and stores them in Firestore for future exclusion during discovery.
+
+* **`index.js`:** Exports helpers.
+* **`/helpers/handler_helpers.js`:** Core logic for finding/creating a Firestore document and adding the invalid IDs, managing document size limits.
+
+### `/speculator-cleanup-orchestrator`
+
+A scheduled function to remove stale speculators from the active blocks based on inactivity and clean up old entries from the pending speculator list.
+
+* **`index.js`:** Exports the factory function `createSpeculatorCleanupHandler`.
+* **`/helpers/cleanup_helpers.js`:** Contains the core logic for querying blocks/pending lists based on grace periods and batching deletions/count updates.
+
+### `/fetch-insights`
+
+Scheduled function to fetch daily instrument ownership insights (buy/sell percentages, total owners) from the eToro API.
+
+* **`index.js`:** Exports the factory function `createFetchInsightsHandler`.
+* **`/helpers/handler_helpers.js`:** Core logic for making the API call (via proxy manager) and storing the results in Firestore.
+
+### `/etoro-price-fetcher`
+
+Scheduled function to fetch daily closing prices for instruments from the eToro API.
+
+* **`index.js`:** Exports the factory function `createPriceFetcherHandler`.
+* **`/helpers/handler_helpers.js`:** Core logic for making the API call (via proxy manager) and batch-writing price updates to Firestore.
+
+### `/appscript-api`
+
+Contains the helper logic intended to be deployed as a Google Apps Script web app, acting as the HTTP proxy layer.
+
+* **`index.js`:** Exports the `handlePost` function.
+* **`/helpers/errors.js`:** Utility for creating standardized error responses.
+
+## Setup & Usage
+
+```javascript
+// Example 
+const { Orchestrator } = require('bulltrackers-module');
+const cfg = require('./config/orchestrator_config'); // Load specific config
+
+exports.discoveryOrchestrator = Orchestrator.createDiscoveryOrchestrator(cfg);

package/functions/appscript-api/helpers/errors.js

@@ -0,0 +1,19 @@
+/**
+ * Creates a structured JSON error response.
+ * @param {string} message
+ * @param {number} statusCode
+ * @returns {GoogleAppsScript.Content.TextOutput}
+ */
+function createErrorResponse(message, statusCode) {
+  const error = {
+    error: {
+      message,
+      code: statusCode,
+    },
+  };
+
+  return ContentService.createTextOutput(JSON.stringify(error))
+    .setMimeType(ContentService.MimeType.JSON);
+}
+
+module.exports = { createErrorResponse };

package/functions/appscript-api/index.js

@@ -0,0 +1,57 @@
+/**
+ * @fileoverview Google Apps Script proxy logic (v2) as a module.
+ * Can be imported and used in a doPost or other GAS handlers.
+ */
+
+const { createErrorResponse } = require('./helpers/error');
+
+/**
+ * Handles the incoming POST request object from Apps Script.
+ * @param {GoogleAppsScript.Events.DoPost} e
+ * @returns {GoogleAppsScript.Content.TextOutput}
+ */
+function handlePost(e) {
+  try {
+    // Parse the request details from the POST body
+    const requestDetails = JSON.parse(e.postData.contents);
+    const { url, headers, method, body } = requestDetails;
+
+    if (!url || !headers) {
+      return createErrorResponse("Invalid request: 'url' and 'headers' are required.", 400);
+    }
+
+    // Prepare the external request options
+    const options = {
+      headers: headers,
+      muteHttpExceptions: true,
+      method: method || 'GET',
+    };
+
+    if (body) {
+      options.payload = body;
+      if (!headers['Content-Type']) {
+        headers['Content-Type'] = 'application/json';
+      }
+    }
+
+    // Make the external request using UrlFetchApp
+    const response = UrlFetchApp.fetch(url, options);
+
+    // Package the response to send back
+    const responseData = {
+      statusCode: response.getResponseCode(),
+      headers: response.getHeaders(),
+      body: response.getContentText(),
+    };
+
+    return ContentService.createTextOutput(JSON.stringify(responseData))
+      .setMimeType(ContentService.MimeType.JSON);
+
+  } catch (error) {
+    return createErrorResponse(error.toString(), 500);
+  }
+}
+
+module.exports = {
+  handlePost,
+};

package/index.js

@@ -14,6 +14,7 @@ const InvalidSpeculatorHandler      = require('./functions/invalid-speculator-ha
 const SpeculatorCleanupOrchestrator = require('./functions/speculator-cleanup-orchestrator'); 
 const FetchInsights                 = require('./functions/fetch-insights'); 
 const EtoroPriceFetcher             = require('./functions/etoro-price-fetcher'); 
+const { handlePost }                = require('./functions/appscript-api');
 
 module.exports = {
   core,
@@ -25,5 +26,6 @@ module.exports = {
   InvalidSpeculatorHandler, 
   SpeculatorCleanupOrchestrator, 
   FetchInsights, 
-  EtoroPriceFetcher 
+  EtoroPriceFetcher,
+  handlePost
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bulltrackers-module",
-  "version": "1.0.26",
+  "version": "1.0.28",
   "description": "Helper Functions for Bulltrackers.",
   "main": "index.js",
   "files": [
@@ -14,7 +14,8 @@
     "functions/invalid-speculator-handler/",
     "functions/speculator-cleanup-orchestrator/",
     "functions/fetch-insights/",
-    "functions/etoro-price-fetcher/"
+    "functions/etoro-price-fetcher/",
+    "functions/appscript-api/"
   ],
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"