aiden-shared-calculations-unified 1.0.82 → 1.0.83

package/README.MD

@@ -1,78 +1,155 @@
-# Unified Calculations Package (`aiden-shared-calculations-unified`)
+---
 
-**Version:** 1.0.0
+# Quantum Test Harness for Computation System
 
 ## Overview
 
-This package centralizes all data calculation logic for the BullTrackers project. It provides a standardized structure for calculations run by the unified Computation System and includes shared utility functions. Calculations are dynamically loaded and categorized.
+The Quantum Test Harness is a **dynamic, dependency-aware testing framework** for running, validating, and profiling the entire computation system.
+
+The new architecture is built around a `run-suite.js` orchestrator. It automatically builds a dependency graph of all calculations, ensuring they are tested in the correct order. This "ground-up" approach allows for robust, efficient, and realistic testing that mimics the production environment.
+
+The harness still tests calculations **without modifying their source code**. It:
+
+*   Builds a dependency graph of all calculations.
+*   Dynamically generates mock data for a consistent "universe" of users and tickers.
+*   Executes calculations in the correct order, passing the results of one test as dependencies to the next.
+*   Monitors performance and validates output against each calculation's `getSchema()`.
+*   Generates rich, interactive HTML reports for each calculation run.
+
+---
+
+## Core Concepts
+
+The harness is built on **four main components**:
+
+### 1. Test Suite Orchestrator (`run-suite.js`)
+
+The main CLI entry point for running automated test suites. Responsibilities:
+
+*   Scans all calculation files and builds a computation graph.
+*   Topologically sorts the graph to determine the correct execution order.
+*   Parses CLI commands (`--all`, `--target`, `--product`) to decide which tests to run.
+*   Manages a results cache to efficiently pass outputs as dependencies.
+
+### 2. Test Worker (`test-worker.js`)
+
+Executes the test for a single calculation. Responsibilities:
+
+*   Receives a calculation to test from the orchestrator, along with its pre-computed dependencies.
+*   Generates fresh, temporal mock data for the specific test.
+*   Instantiates the calculation and wraps it in the Quantum Monitor.
+*   Executes `process()` and `getResult()`.
+*   Triggers the report generation.
+
+### 3. Dynamic Data Generation (`data-generator.js`)
+
+Generates **temporal mock data**:
+
+1.  The orchestrator creates a consistent "universe" (tickers and userIds) for the entire test run.
+2.  The test worker generates "today" and optional "yesterday" data snapshots using this universe, ensuring data is consistent and comparable across tests.
+
+### 4. Instrumentation & Validation (`test-harness.js`)
 
-## Package Structure
+Provides:
 
-### `/utils`
+*   **Instrumentation**: Wraps the calculation instance in a Proxy via `createQuantumRecorder`. Logs performance, memory, and errors without changing the calculation logic.
+*   **Validation**:
+    *   **Schema Check**: Uses Ajv to validate `getResult()` against `getSchema()`.
+    *   **Sanity Check**: Flags "valid but empty" results like `0`, `null`, `[]`, or `{}` as warnings.
 
-Shared utility functions used by calculations or the systems consuming them.
+---
+
+## Usage: Automated Test Suite
+
+To run automated, dependency-aware test suites, use the `run-suite.js` orchestrator. This is the recommended method for all testing.
+
+### Prerequisites
+
+Graphviz is required to render data-flow diagrams. Ensure `dot` is available in your PATH.
+
+| OS      | Installation                    |
+| ------- | ------------------------------- |
+| macOS   | `brew install graphviz`         |
+| Windows | `choco install graphviz`        |
+| Linux   | `sudo apt-get install graphviz` |
+
+---
+
+### Commands
 
-* `firestore_utils.js`: Provides a resilient `withRetry` wrapper for Firestore operations using exponential backoff.
-* `sector_mapping_provider.js`: Provides functions (`loadInstrumentMappings`, `getInstrumentSectorMap`) to fetch and cache instrument-to-ticker and instrument-to-sector mappings from Firestore.
+#### 1. Run ALL Computations
 
-### `/calculations`
+Tests every non-legacy computation, ordered by complexity (dependency-free calculations first).
 
-Contains the core calculation logic, organized into subdirectories representing categories.
+```bash
+node run-suite.js --all
+```
 
-* **Calculation Class Structure:** Each `.js` file defines a class responsible for a specific metric. Every class **must** implement:
-    * `constructor()`: Initializes any internal state needed for aggregation.
-    * `process(portfolioData, userId, context)` OR `process(todayPortfolio, yesterdayPortfolio, userId, context)`: Processes a single user's data. The signature depends on whether the calculation requires historical comparison. `context` provides shared data like mappings.
-    * `getResult()`: Returns the final, calculated result for the aggregation period. **Crucially, this method must perform any final averaging (e.g., sum/count) itself.** It should return the final value or object ready for storage, not raw components.
-    * `reset()`: (Optional but recommended) Resets the internal state, often used by the calling system between processing batches or days.
+#### 2. Run a Specific Target
 
-* **Categories (Examples based on your files):**
-    * `/asset_metrics`: Calculations focused on individual assets (e.g., `asset_dollar_metrics`, `asset_position_size`).
-    * `/behavioural`: Calculations analyzing user trading patterns (e.g., `drawdown_response`, `gain_response`, `paper_vs_diamond_hands`, `position_count_pnl`, `smart_money_flow`).
-    * `/pnl`: Profit and Loss related calculations (e.g., `asset_pnl_status`, `average_daily_pnl_all_users`, `average_daily_pnl_per_sector`, `average_daily_pnl_per_stock`, `average_daily_position_pnl`, `pnl_distribution_per_stock`, `profitability_migration`, `profitability_ratio_per_stock`, `profitability_skew_per_stock`, `user_profitability_tracker`).
-    * `/sanity`: Basic checks and counts (e.g., `users_processed`).
-    * `/sectors`: Calculations aggregated by market sector (e.g., `diversification_pnl`, `sector_dollar_metrics`, `sector_rotation`, `total_long_per_sector`, `total_short_per_sector`).
-    * `/sentiment`: Calculations related to market sentiment (e.g., `crowd_conviction_score`).
-    * `/short_and_long_stats`: Specific counts for short and long positions (e.g., `long_position_per_stock`, `sentiment_per_stock`, `short_position_per_stock`, `total_long_figures`, `total_short_figures`).
-    * `/speculators`: Calculations **specifically** for the 'speculator' user type, often involving leverage, stop-loss, or take-profit data (e.g., `distance_to_stop_loss_per_leverage`, `distance_to_tp_per_leverage`, `entry_distance_to_sl_per_leverage`, `entry_distance_to_tp_per_leverage`, `holding_duration_per_asset`, `leverage_per_asset`, `leverage_per_sector`, `risk_appetite_change`, `risk_reward_ratio_per_asset`, `speculator_asset_sentiment`, `speculator_danger_zone`, `stop_loss_distance_by_sector_short_long_breakdown`, `stop_loss_distance_by_ticker_short_long_breakdown`, `stop_loss_per_asset`, `take_profit_per_asset`, `tsl_effectiveness`, `tsl_per_asset`).
+Tests a single computation and all of its dependencies in the correct order.
 
-* **Output Formats:** Calculations should adhere to the standardized output formats defined in `docs/Notes/output_formats.md`.
+```bash
+# Syntax
+node run-suite.js --target [computation-name]
 
-## Usage
+# Example (note: uses the name, not the file path)
+node run-suite.js --target asset-pnl-status
+```
 
-This package is intended to be consumed primarily by the unified Computation System.
+#### 3. Run a Full Product Line
 
-```javascript
-// Example usage within Computation System
-const { calculations, utils } = require('aiden-shared-calculations-unified');
+Tests all computations belonging to a specific product category (defined in `getMetadata`) and all of their dependencies.
 
-const CalculationClass = calculations.pnl.average_daily_pnl_per_stock;
-const calculator = new CalculationClass();
+```bash
+# Syntax
+node run-suite.js --product [product-name]
 
-// ... load data ...
+# Example
+node run-suite.js --product gem
+```
 
-// In a loop for each user:
-calculator.process(portfolioData, userId, context);
+---
 
-// After processing all users:
-const results = await calculator.getResult();
+### Output
 
-// ... store results ...
-````
+Running a test produces:
 
-## Contributing
+*   **Console Output**: Live logs of the orchestration and test progress.
+*   **Test Reports**: Generated for each executed calculation at `test_reports/[CalculationName]/`:
+    *   `timeline.html` – Interactive performance report.
+    *   `dataflow.svg` – Diagram of the calculation's data flow.
+    *   `computation_result.json` – Raw JSON output from `getResult()`.
+    *   `report.json` – Full metrics collected by the harness.
 
-*(Outline the process for adding new calculations)*
+---
 
-1.  **Determine Category:** Decide which subdirectory (`/calculations/<category>`) the new metric belongs to. Create a new category if necessary.
-2.  **Create File:** Create a new `.js` file using kebab-case (e.g., `my-new-metric.js`).
-3.  **Implement Class:** Define the calculation class, ensuring it has `constructor`, `process`, and `getResult` methods adhering to the standards. Remember `getResult` must return the *final* computed value.
-4.  **Add to Manifest:** The main `index.js` uses `require-all`, so the new calculation should be automatically included in the exports upon the next package build/publish, assuming the file naming and structure are correct.
-5.  **Publish:** Bump the package version (`npm version patch` or minor/major as appropriate) and publish (`npm publish --access public`).
-6.  **Update Consumer:** Update the version dependency in the Computation System's `package.json`.
+## Adding a New Calculation
 
-<!-- end list -->
+The process remains the same.
 
+1.  **Create Your File**: `calculations/my_product/my-new-calc.js`
+2.  **Implement the Class**: Must include `constructor()`, `process()`, and `getResult()`.
+3.  **Add `getDependencies()`**: Define the other calculations this one depends on.
+4.  **Add `getMetadata()`**: Defines the contract for the harness.
+5.  **Add `getSchema()`**: Enables automatic output validation.
+6.  **Run the Test**: Use the `run-suite.js` commands.
+
+```bash
+# Test your new calculation and its dependencies
+node run-suite.js --target my-new-calc
 ```
 
 ---
-```
+
+## Troubleshooting
+
+| Issue                                            | Solution                                                                                                                            |
+| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
+| Graphviz rendering failed                        | Ensure Graphviz is installed and `dot` is in your system's PATH.                                                                    |
+| `SchemaValidation` errors in `report.json`       | The calculation's output does not match its `getSchema()`. Check `computation_result.json` to see the actual output and fix the logic. |
+| `SanityCheck` warnings in `report.json`          | The output is valid according to the schema but is empty (`null`, `0`, `[]`, `{}`). This may be expected, but it's worth verifying.    |
+| `Error: Unknown computation dependency: [name]`  | A calculation lists a dependency in `getDependencies()` that doesn't exist or has a typo. Check the name for errors.                  |
+| `Error: Circular dependency detected!`           | Two or more calculations depend on each other, creating an impossible loop. Review the `getDependencies()` lists for the involved calculations. |
+
+---

package/calculations/helix/winner-loser-flow.js

@@ -119,8 +119,10 @@ class WinnerLoserFlow {
             return;
         }
 
+
+
         if (!this.mappings) {
-            this.mappings = context.mappings;
+             this.mappings = context.instrumentToTickerMap;
         }
 
         const yHoldings = this._getHoldingsWithPnl(yesterdayPortfolio); // Map<InstID, {pnl}>
@@ -132,7 +134,7 @@ class WinnerLoserFlow {
         }
 
         for (const instrumentId of allInstrumentIds) {
-            const ticker = this.mappings.instrumentToTicker[instrumentId];
+            const ticker = this.mappings[instrumentId];
             if (!ticker) continue;
             
             // --- MODIFIED ---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "aiden-shared-calculations-unified",
-  "version": "1.0.82",
+  "version": "1.0.83",
   "description": "Shared calculation modules for the BullTrackers Computation System.",
   "main": "index.js",
   "files": [
@@ -27,15 +27,15 @@
   ],
   "dependencies": {
     "@google-cloud/firestore": "^7.11.3",
-    "sharedsetup": "latest",
     "require-all": "^3.0.0",
-    "dotenv": "latest",
+    "sharedsetup": "latest",
     "viz.js": "^2.1.2"
   },
   "devDependencies": {
-    "bulltracker-deployer": "file:../bulltracker-deployer",
     "@google-cloud/firestore": "^7.11.3",
-    "dotenv": "latest",
+    "ajv": "^8.17.1",
+    "bulltracker-deployer": "file:../bulltracker-deployer",
+    "dotenv": "^17.2.3",
     "node-graphviz": "^0.1.1"
   },
   "engines": {