bulltrackers-module 1.0.293 → 1.0.295

package/functions/computation-system/onboarding.md

@@ -1,210 +0,0 @@
-# BullTrackers Computation System: Architecture & Operational Manual
-
-This document provides a comprehensive overview of the BullTrackers Computation System, a distributed, deterministic, and self-optimizing data pipeline. Unlike traditional task schedulers, this system operates on "Build System" principles, treating data calculations as compiled artifacts with strict versioning and dependency guarantees.
-
----
-
-## 1. System Philosophy & Core Concepts
-
-### The "Build System" Paradigm
-We treat the computation pipeline like a large-scale software build system (e.g., Bazel or Make). Every data point is an "artifact" produced by a specific version of code (Code Hash) acting on specific versions of dependencies (Dependency Hashes).
-*   **Determinism**: If the input data and code haven't changed, the output *must* be identical. We verify this to skip unnecessary work.
-*   **Merkle Tree Structure**: The state of the system is a DAG (Directed Acyclic Graph) of hashes. A change in a root node propagates potential invalidation down the tree, but invalidation stops as soon as a node produces the same output as before (Short-Circuiting).
-
-### Source-of-Truth Architecture
-The **Root Data Index** is the absolute source of truth. No computation can start until the underlying raw data (prices, signals) is indexed and verified "Available" for the target date. This prevents partial runs and "garbage-in-garbage-out".
-
-### The Three-Layer Hash Model
-To optimize execution, we track three distinct hashes for every calculation:
-1.  **Code Hash (Static)**: A SHA-256 hash of the cleaned source code (comments and whitespace stripped). This tells us if the logic *might* have changed.
-2.  **SimHash (Behavioral)**: Generated by running the code against a deterministic "Fabricated" context. This tells us if the logic *actually* changed behavior (e.g., a refactor that changes variable names but not logic will have a different Code Hash but the same SimHash).
-3.  **ResultHash (Output)**: A hash of the actual production output from a run. This tells us if the data changed. Used for downstream short-circuiting.
-
----
-
-## 2. Core Components Overview
-
-### Root Data Indexer
-A scheduled crawler that verifies the availability of raw external data (e.g., asset prices, global signals) for a given date. It produces an "Availability Manifest" that the Dispatcher consults before scheduling anything.
-
-### Manifest Builder
-*   **Role**: Topology Discovery.
-*   **Mechanism**: It scans the `calculations/` directory, loads every module, and builds the global Dependency Graph (DAG) in memory.
-*   **Output**: A topological sort of all calculations assigned to "Passes" (Pass 0, Pass 1, etc.).
-
-### The Dispatcher (`WorkflowOrchestrator.js`)
-The "Brain" of the system. It runs largely stateless, analyzing the `StatusRepository` against the `Manifest`.
-*   **Responsibility**: For a given Grid (Date x Calculation), it determines if the state is `RUNNABLE`, `BLOCKED`, `SKIPPED`, or `IMPOSSIBLE`.
-*   **Key Logic**: It implements the "Short-Circuiting" and "Historical Continuity" checks.
-
-### The Build Optimizer
-A pre-flight tool that attempts to avoiding running tasks by proving they are identical to previous versions.
-*   **Mechanism**: If a calculation's Code Hash changes, the Optimizer runs a **Simulation** (using `SimRunner`) to generate a SimHash. If the SimHash matches the registry, the system acts as if the code never changed, skipping the production re-run.
-
-### The Worker (`StandardExecutor` / `MetaExecutor`)
-The execution unit. It is unaware of the broader topology.
-*   **Input**: A target Calculation and Date.
-*   **Action**: Fetches inputs, runs `process()`, validates results, and writes to Firestore.
-*   **Output**: The computed data + the **ResultHash**.
-
----
-
-## 3. The Daily Lifecycle (Chronological Process)
-
-### Phase 1: Indexing
-The system waits for the `SystemEpoch` to advance. The Root Data Indexer checks for "Canary Blocks" (indicators that external data providers have finished for the day). Once confirmed, the date is marked `OPEN`.
-
-### Phase 2: Pre-Flight Optimization
-Before dispatching workers:
-1.  The system identifies all calculations with new **Code Hashes**.
-2.  It runs `SimRunner` for these calculations to generate fresh **SimHashes**.
-3.  If `SimHash(New) == SimHash(Old)`, the system updates the Status Ledger to enable the new Code Hash without flagging it as "Changed".
-
-### Phase 3: Dispatch Analysis
-The Dispatcher iterates through the Topological Passes (0 -> N). For each calculation, it queries `calculateExecutionStatus`:
-*   Are dependencies done?
-*   Did dependencies change their output (`ResultHash`)?
-*   Is historical context available?
-
-### Phase 4: Execution Waves
-Workers are triggered via Pub/Sub or direct method invocation.
-*   **Pass 1**: Primitive conversions (e.g., Price Extractor).
-*   **Pass 2**: Technical Indicators that depend on Pass 1.
-*   **Pass 3**: Aggregations and Complex Metrics.
-
-### Phase 5: Reconciliation
-After all queues drain, the system performs a final sweep. Any tasks marked `FAILED` are retried (up to a limit). Impossible tasks are finalized as `IMPOSSIBLE`.
-
----
-
-## 4. Deep Dive: Hashing & Dependency Logic
-
-### Intrinsic Code Hashing
-Located in `topology/HashManager.js`.
-We generate a unique fingerprint for every calculation file:
-```javascript
-clean = codeString.replace(comments).replace(whitespace);
-hash = sha256(clean);
-```
-This ensures that changes to comments or formatting do *not* trigger re-runs.
-
-### Behavioral Hashing (SimHash)
-Located in `simulation/SimRunner.js`.
-When code changes, we can't be 100% sure it's safe just by looking at the source.
-1.  **The Fabricator**: Generates a deterministic mock `Context` (prices, previous results) based on the input schema.
-2.  **Simulation Run**: The calculation `process()` method is executed against this mock data.
-3.  **The Registry**: The hash of the *output* of this simulation is stored.
-If a refactor results in the exact same Mock Output, the system considers the change "Cosmetic".
-
-### Dependency Short-Circuiting
-Implemented in `WorkflowOrchestrator.js` (`analyzeDateExecution`).
-Even if an upstream calculation re-runs, downstream dependents might not need to.
-*   **Logic**:
-    *   Calc A (Upstream) re-runs. Old Output Hash: `HashX`. New Output Hash: `HashX`.
-    *   Calc B (Downstream) sees that Calc A "changed" (new timestamp), BUT the content hash `HashX` is identical to what Calc B used last time.
-    *   **Result**: Calc B is `SKIPPED`.
-
----
-
-## 5. Decision Logic & Edge Case Scenarios
-
-### Scenario A: Standard Code Change (Logic)
-*   **Trigger**: You change the formula for `RSI`. Code Hash changes. SimHash changes.
-*   **Dispatcher**: Sees `storedHash !== currentHash`.
-*   **Result**: Marks as `RUNNABLE`. Worker runs.
-
-### Scenario B: Cosmetic Code Change (Refactor)
-*   **Trigger**: You rename a variable in `RSI`. Code Hash changes. SimHash remains identical.
-*   **Optimizer**: Updates the centralized Status Ledger: "Version `Desc_v2` is equivalent to `Desc_v1`".
-*   **Dispatcher**: Sees the new hash in the ledger as "Verified".
-*   **Result**: Task is `SKIPPED`.
-
-### Scenario C: Upstream Invalidation (The Cascade)
-*   **Condition**: `PriceExtractor` fixes a bug. `ResultHash` changes from `HashA` to `HashB`.
-*   **Downstream**: `RSI` checks detailed dependency report.
-*   **Check**: `LastRunDeps['PriceExtractor'] (HashA) !== CurrentDeps['PriceExtractor'] (HashB)`.
-*   **Result**: `RSI` is forced to re-run.
-
-### Scenario D: Upstream Stability (The Firewall)
-*   **Condition**: `PriceExtractor` runs an optimization. Output is exact same data. `ResultHash` remains `HashA`.
-*   **Downstream**: `RSI` checks dependency report.
-*   **Check**: `LastRunDeps['PriceExtractor'] (HashA) === CurrentDeps['PriceExtractor'] (HashA)`.
-*   **Result**: `RSI` is `SKIPPED`. This firewall prevents massive re-calculation storms for non-functional upstream changes.
-
-### Scenario E: The "Impossible" State
-*   **Condition**: Core market data is missing for `1990-01-01`.
-*   **Root Indexer**: Marks date as providing `[]` (empty) for critical inputs.
-*   **Dispatcher**: Marks `PriceExtractor` as `IMPOSSIBLE: NO_DATA`.
-*   **Propagation**: Any calculation depending on `PriceExtractor` sees the `IMPOSSIBLE` status and marks *itself* as `IMPOSSIBLE: UPSTREAM`.
-*   **Benefit**: The system doesn't waste cycles retrying calculations that can never succeed.
-
-### Scenario F: Category Migration
-*   **Condition**: You change `getMetadata()` for a calculation, moving it from `signals` to `risk`.
-*   **Dispatcher**: Detects `storedCategory !== newCategory`.
-*   **Worker**:
-    1.  Runs `process()` and writes to the *new* path (`risk/CalculateX`).
-    2.  Detects the `previousCategory` flag.
-    3.  Deletes the data at the *old* path (`signals/CalculateX`) to prevent orphan data.
-
----
-
-## 6. Data Management & Storage
-
-### Input Streaming
-To handle large datasets without OOM (Out Of Memory) errors:
-*   `StandardExecutor` does not load all users/tickers at once.
-*   It utilizes wait-and-stream logic (e.g., batches of 50 ids) to process the `Context`.
-
-### Transparent Auto-Sharding
-Firestore has a 1MB document limit.
-*   **Write Path**: If a calculation result > 900KB, it is split into `DocID`, `DocID_shard1`, `DocID_shard2`.
-*   **Read Path**: The `DependencyFetcher` automatically detects sharding pointers and re-assembles (hydrates) the full object before passing it to `process()`.
-
-### Compression Strategy
-*   Payloads are inspected before write.
-*   If efficient (high entropy text/JSON), Zlib compression is applied.
-*   Metadata is tagged `encoding: 'zlib'` so readers know to inflate.
-
----
-
-## 7. Quality Assurance & Self-Healing
-
-### The Heuristic Validator
-Before saving *any* result, the Executor runs heuristics:
-*   **NaN Check**: Are there `NaN` or `Infinity` values in key fields?
-*   **Flatline Check**: Is the data variance 0.00 across a large timespan?
-*   **Null Density**: Is >50% of the dataset null?
-*   **Circuit Breaker**: If heuristics fail, the task throws an error. It is better to fail and alert than to persist corrupted data that pollutes the cache.
-
-### Zombie Task Recovery
-*   **Lease Mechanism**: When a task starts, it sets a `startedAt` timestamp.
-*   **Detection**: The Dispatcher checks for tasks marked `RUNNING` where `startedAt` > 15 minutes ago.
-*   **Resolution**: These are assumed crashed (OOM/Timeout). They are reset to `PENDING` (or `FAILED` if retry count exceeded).
-
-### Dead Letter Queue (DLQ)
-Tasks that deterministically fail (crash every time) after N retries are moved to a special DLQ status. This prevents the system from getting stuck in an infinite retry loop.
-
----
-
-## 8. Developer Workflows
-
-### How to Add a New Calculation
-1.  Create `calculations/category/MyNewCalc.js`.
-2.  Implement `getMetadata()` to define dependencies.
-3.  Implement `process(context)`.
-4.  Run `npm run build-manifest` to register it in the topology.
-
-### How to Force a Global Re-Run
-*   Change the `SYSTEM_EPOCH` constant in `system_epoch.js`.
-*   This changes the "Global Salt" for all hashes, processing every calculation as "New".
-
-### How to Backfill History
-*   **Standard Dispatcher**: Good for recent history (last 30 days).
-*   **BatchPriceExecutor**: Specialized for massive historical backfills (e.g., 20 years of price data). It bypasses some topology checks for raw speed.
-
-### Local Debugging
-Run the orchestrator in "Dry Run" mode:
-```bash
-node scripts/run_orchestrator.js --date=2024-01-01 --dry-run
-```
-This prints the `Analysis Report` (Runnable/Blocked lists) without actually triggering workers.