bulltrackers-module 1.0.281 → 1.0.283

package/functions/computation-system/onboarding.md

@@ -1,925 +1,210 @@
-# BullTrackers Computation System & Root Data Indexer
-
-## Table of Contents
-1. [System Philosophy](#system-philosophy)
-2. [Root Data Indexer](#root-data-indexer)
-3. [Computation Architecture](#computation-architecture)
-4. [Context & Dependency Injection](#context--dependency-injection)
-5. [Execution Pipeline](#execution-pipeline)
-6. [Smart Hashing & Versioning](#smart-hashing--versioning)
-7. [Data Loading & Streaming](#data-loading--streaming)
-8. [Auto-Sharding System](#auto-sharding-system)
-9. [Quality Assurance](#quality-assurance)
-10. [Operational Modes](#operational-modes)
+# BullTrackers Computation System: Architecture & Operational Manual
 
----
-
-## System Philosophy
-
-The BullTrackers Computation System is a **dependency-aware, distributed calculation engine** that processes massive financial datasets with strict guarantees:
-
-- **Incremental Recomputation**: Only re-runs when code or dependencies change
-- **Historical Continuity**: Ensures chronological execution for time-series calculations
-- **Transparent Sharding**: Handles Firestore's 1MB document limit automatically
-- **Data Availability Gating**: Never runs when source data is missing
-- **Cascading Invalidation**: Upstream changes automatically invalidate downstream results
-
-### Key Design Principles
-
-**1. Source of Truth Paradigm**
-- The Root Data Indexer creates a daily availability manifest
-- Computations are gated by data availability checks
-- Missing data triggers "IMPOSSIBLE" states, not retries
-
-**2. Merkle Tree Dependency Hashing**
-- Every calculation has a hash that includes:
-  - Its own source code
-  - Hashes of math layers it uses
-  - Hashes of calculations it depends on
-- Changes cascade: updating calculation A invalidates all dependents
-
-**3. Stateless Execution**
-- Each worker receives complete context for its task
-- No inter-worker communication
-- Infinitely horizontally scalable
+This document provides a comprehensive overview of the BullTrackers Computation System, a distributed, deterministic, and self-optimizing data pipeline. Unlike traditional task schedulers, this system operates on "Build System" principles, treating data calculations as compiled artifacts with strict versioning and dependency guarantees.
 
 ---
 
-## Root Data Indexer
-
-### Purpose
-
-The Root Data Indexer runs daily to scan all data sources and create a **centralized availability manifest**. This prevents the computation system from attempting to run calculations when source data doesn't exist.
-
-### Architecture
-
-```
-┌──────────────────────────────────────────────────────┐
-│           Root Data Indexer (Daily Scan)             │
-├──────────────────────────────────────────────────────┤
-│  For each date (2023-01-01 → Tomorrow):              │
-│    1. Check Normal User Portfolios (Canary: 19M)     │
-│    2. Check Speculator Portfolios (Canary: 19M)      │
-│    3. Check Normal Trade History (Canary: 19M)       │
-│    4. Check Speculator Trade History (Canary: 19M)   │
-│    5. Check Daily Insights (/{date})                 │
-│    6. Check Social Posts (/{date}/posts)             │
-│    7. Pre-Load Price Shard (shard_0) → Date Map      │
-│                                                        │
-│  Output: /system_root_data_index/{date}              │
-│    {                                                   │
-│      hasPortfolio: true,                              │
-│      hasHistory: false,                               │
-│      hasInsights: true,                               │
-│      hasSocial: true,                                 │
-│      hasPrices: true,                                 │
-│      details: {                                        │
-│        normalPortfolio: true,                         │
-│        speculatorPortfolio: false,                    │
-│        normalHistory: false,                          │
-│        speculatorHistory: false                       │
-│      }                                                 │
-│    }                                                   │
-└──────────────────────────────────────────────────────┘
-```
-
-### Canary Block System
-
-Instead of scanning every user block (which would be prohibitively expensive), the indexer uses **representative blocks**:
-
-- **Block 19M**: Statistically verified to always contain data when the system is healthy
-- **Part 0**: First part of sharded collections
-
-**Logic**: If Block 19M has data for a date, all blocks have data for that date. This is a deliberate architectural assumption that reduces indexing cost by 99%.
-
-### Granular UserType Tracking
-
-The indexer distinguishes between:
-- **Normal Users**: Traditional portfolio tracking (AggregatedPositions)
-- **Speculators**: Advanced traders (PublicPositions with leverage/SL/TP)
-
-This enables calculations to declare: `userType: 'speculator'` and be gated only when speculator data exists.
-
-### Price Data Optimization
-
-Price data is handled differently:
+## 1. System Philosophy & Core Concepts
 
-1. **Pre-Load Once**: The entire `shard_0` document is loaded into memory
-2. **Extract Date Keys**: All dates with price data are extracted into a `Set`
-3. **Fast Lookup**: Each date check becomes O(1) instead of a Firestore read
+### The "Build System" Paradigm
+We treat the computation pipeline like a large-scale software build system (e.g., Bazel or Make). Every data point is an "artifact" produced by a specific version of code (Code Hash) acting on specific versions of dependencies (Dependency Hashes).
+*   **Determinism**: If the input data and code haven't changed, the output *must* be identical. We verify this to skip unnecessary work.
+*   **Merkle Tree Structure**: The state of the system is a DAG (Directed Acyclic Graph) of hashes. A change in a root node propagates potential invalidation down the tree, but invalidation stops as soon as a node produces the same output as before (Short-Circuiting).
 
-This reduces price availability checks from **~1000 reads/day** to **1 read total**.
+### Source-of-Truth Architecture
+The **Root Data Index** is the absolute source of truth. No computation can start until the underlying raw data (prices, signals) is indexed and verified "Available" for the target date. This prevents partial runs and "garbage-in-garbage-out".
 
-### Index Schema
-
-```javascript
-{
-  date: "2024-12-07",
-  lastUpdated: Timestamp,
-  
-  // Aggregate Flags (true if ANY subtype exists)
-  hasPortfolio: true,   // normalPortfolio OR speculatorPortfolio
-  hasHistory: false,    // normalHistory OR speculatorHistory
-  hasInsights: true,    // Insights document exists
-  hasSocial: true,      // At least 1 social post exists
-  hasPrices: true,      // Price data exists for this date
-  
-  // Granular Breakdown
-  details: {
-    normalPortfolio: true,
-    speculatorPortfolio: false,
-    normalHistory: false,
-    speculatorHistory: false
-  }
-}
-```
-
-### Availability Check Logic (Computation System)
-
-```javascript
-// AvailabilityChecker.js - checkRootDependencies()
-
-if (calculation.rootDataDependencies includes 'portfolio') {
-  if (calculation.userType === 'speculator') {
-    if (!rootDataStatus.speculatorPortfolio) → MISSING
-  } else if (calculation.userType === 'normal') {
-    if (!rootDataStatus.normalPortfolio) → MISSING
-  } else {
-    // userType: 'all' or 'aggregate'
-    if (!rootDataStatus.hasPortfolio) → MISSING
-  }
-}
-
-// Similar logic for 'history' dependency
-// Global data types (insights, social, price) have no subtypes
-```
-
-**Critical Behavior**: 
-- If data is missing for a **historical date** → Mark calculation as `IMPOSSIBLE` (permanent failure)
-- If data is missing for **today's date** → Mark as `BLOCKED` (retriable, data may arrive later)
+### The Three-Layer Hash Model
+To optimize execution, we track three distinct hashes for every calculation:
+1.  **Code Hash (Static)**: A SHA-256 hash of the cleaned source code (comments and whitespace stripped). This tells us if the logic *might* have changed.
+2.  **SimHash (Behavioral)**: Generated by running the code against a deterministic "Fabricated" context. This tells us if the logic *actually* changed behavior (e.g., a refactor that changes variable names but not logic will have a different Code Hash but the same SimHash).
+3.  **ResultHash (Output)**: A hash of the actual production output from a run. This tells us if the data changed. Used for downstream short-circuiting.
 
 ---
 
-## Computation Architecture
-
-### Calculation Types
-
-#### **Standard (Per-User) Computations**
-
-```javascript
-class UserRiskProfile {
-  static getMetadata() {
-    return {
-      type: 'standard',                    // Runs once per user
-      category: 'risk-management',         // Storage category
-      isHistorical: true,                  // Needs yesterday's data
-      rootDataDependencies: ['portfolio', 'history'],
-      userType: 'speculator'               // Only for speculator users
-    };
-  }
-  
-  static getDependencies() {
-    return ['market-volatility'];          // Needs this calc to run first
-  }
-  
-  async process(context) {
-    const { user, computed, math } = context;
-    // Process individual user
-    this.results[user.id] = { riskScore: /* ... */ };
-  }
-}
-```
-
-**Execution Model**:
-- Portfolio data streams in batches (50 users at a time)
-- Each user processed independently
-- Memory-efficient for millions of users
+## 2. Core Components Overview
 
-#### **Meta (Once-Per-Day) Computations**
-
-```javascript
-class MarketMomentum {
-  static getMetadata() {
-    return {
-      type: 'meta',                        // Runs once per day globally
-      category: 'market-signals',
-      rootDataDependencies: ['price', 'insights']
-    };
-  }
-  
-  async process(context) {
-    const { prices, insights, math } = context;
-    // Process all tickers
-    for (const [instId, data] of Object.entries(prices.history)) {
-      this.results[data.ticker] = { momentum: /* ... */ };
-    }
-  }
-}
-```
-
-**Execution Model**:
-- Loads all price data (or processes in shard batches)
-- Runs once, produces global results
-- Used for market-wide analytics
+### Root Data Indexer
+A scheduled crawler that verifies the availability of raw external data (e.g., asset prices, global signals) for a given date. It produces an "Availability Manifest" that the Dispatcher consults before scheduling anything.
 
 ### Manifest Builder
+*   **Role**: Topology Discovery.
+*   **Mechanism**: It scans the `calculations/` directory, loads every module, and builds the global Dependency Graph (DAG) in memory.
+*   **Output**: A topological sort of all calculations assigned to "Passes" (Pass 0, Pass 1, etc.).
 
-The Manifest Builder automatically:
+### The Dispatcher (`WorkflowOrchestrator.js`)
+The "Brain" of the system. It runs largely stateless, analyzing the `StatusRepository` against the `Manifest`.
+*   **Responsibility**: For a given Grid (Date x Calculation), it determines if the state is `RUNNABLE`, `BLOCKED`, `SKIPPED`, or `IMPOSSIBLE`.
+*   **Key Logic**: It implements the "Short-Circuiting" and "Historical Continuity" checks.
 
-1. **Discovers** all calculation classes in the codebase
-2. **Analyzes** their dependencies
-3. **Sorts** them topologically (builds a DAG)
-4. **Assigns** pass numbers (execution waves)
-5. **Generates** smart hashes for each calculation
+### The Build Optimizer
+A pre-flight tool that attempts to avoiding running tasks by proving they are identical to previous versions.
+*   **Mechanism**: If a calculation's Code Hash changes, the Optimizer runs a **Simulation** (using `SimRunner`) to generate a SimHash. If the SimHash matches the registry, the system acts as if the code never changed, skipping the production re-run.
 
-```
-Pass 1 (No Dependencies):
-  - market-volatility
-  - price-momentum
-  
-Pass 2 (Depends on Pass 1):
-  - user-risk-profile (needs market-volatility)
-  - sentiment-score (needs price-momentum)
-  
-Pass 3 (Depends on Pass 2):
-  - combined-signal (needs sentiment-score + user-risk-profile)
-```
-
-**Circular Dependency Detection**: If `A → B → C → A`, the builder throws a fatal error and refuses to generate a manifest.
+### The Worker (`StandardExecutor` / `MetaExecutor`)
+The execution unit. It is unaware of the broader topology.
+*   **Input**: A target Calculation and Date.
+*   **Action**: Fetches inputs, runs `process()`, validates results, and writes to Firestore.
+*   **Output**: The computed data + the **ResultHash**.
 
 ---
 
-## Context & Dependency Injection
+## 3. The Daily Lifecycle (Chronological Process)
 
-### Context Structure
+### Phase 1: Indexing
+The system waits for the `SystemEpoch` to advance. The Root Data Indexer checks for "Canary Blocks" (indicators that external data providers have finished for the day). Once confirmed, the date is marked `OPEN`.
 
-Every calculation receives exactly what it declares:
+### Phase 2: Pre-Flight Optimization
+Before dispatching workers:
+1.  The system identifies all calculations with new **Code Hashes**.
+2.  It runs `SimRunner` for these calculations to generate fresh **SimHashes**.
+3.  If `SimHash(New) == SimHash(Old)`, the system updates the Status Ledger to enable the new Code Hash without flagging it as "Changed".
 
-```javascript
-{
-  // IDENTITY (Standard only)
-  user: {
-    id: "user_123",
-    type: "speculator",
-    portfolio: { today: {...}, yesterday: {...} },
-    history: { today: {...}, yesterday: {...} }
-  },
-  
-  // TEMPORAL
-  date: { today: "2024-12-07" },
-  
-  // ROOT DATA (if declared)
-  insights: { today: {...}, yesterday: {...} },
-  social: { today: {...}, yesterday: {...} },
-  prices: { history: {...} },
-  
-  // MAPPINGS
-  mappings: {
-    tickerToInstrument: { "AAPL": 123 },
-    instrumentToTicker: { 123: "AAPL" }
-  },
-  
-  // MATH LAYERS (always injected)
-  math: {
-    extract: DataExtractor,
-    compute: MathPrimitives,
-    signals: SignalPrimitives,
-    history: HistoryExtractor,
-    insights: InsightsExtractor,
-    priceExtractor: priceExtractor,
-    // ... 20+ utility classes
-  },
-  
-  // DEPENDENCIES (if declared)
-  computed: {
-    "market-volatility": { "AAPL": { volatility: 0.25 } }
-  },
-  
-  // HISTORICAL DEPENDENCIES (if isHistorical: true)
-  previousComputed: {
-    "market-volatility": { "AAPL": { volatility: 0.23 } }
-  }
-}
-```
-
-### Lazy Loading Optimization
-
-The system **only loads what you declare**:
-
-```javascript
-// Calculation A declares:
-rootDataDependencies: ['portfolio']
-
-// Context A receives:
-{ user: { portfolio: {...} } }  // No insights, social, or prices loaded
-
-
-// Calculation B declares:
-rootDataDependencies: ['portfolio', 'insights']
-
-// Context B receives:
-{ 
-  user: { portfolio: {...} }, 
-  insights: { today: {...} }     // Insights fetched on-demand
-}
-```
-
-This prevents unnecessary Firestore reads and keeps memory usage minimal.
-
----
-
-## Execution Pipeline
-
-### Phase 1: Analysis & Dispatch
-
-```
-┌─────────────────────────────────────────────────────┐
-│         Computation Dispatcher                      │
-│  (Smart Pre-Flight Checker)                         │
-├─────────────────────────────────────────────────────┤
-│  For each date in range:                            │
-│    1. Fetch Root Data Index                         │
-│    2. Fetch Computation Status (stored hashes)      │
-│    3. Fetch Yesterday's Status (historical check)   │
-│    4. Run analyzeDateExecution()                    │
-│                                                       │
-│  Decision Logic per Calculation:                    │
-│    ├─ Root Data Missing?                            │
-│    │   ├─ Historical Date → Mark IMPOSSIBLE         │
-│    │   └─ Today's Date → Mark BLOCKED (retriable)   │
-│    │                                                  │
-│    ├─ Dependency Impossible?                        │
-│    │   └─ Mark IMPOSSIBLE (cascading failure)       │
-│    │                                                  │
-│    ├─ Dependency Missing/Hash Mismatch?             │
-│    │   └─ Mark BLOCKED (wait for dependency)        │
-│    │                                                  │
-│    ├─ Historical Continuity Broken?                 │
-│    │   └─ Mark BLOCKED (wait for yesterday)         │
-│    │                                                  │
-│    ├─ Hash Mismatch?                                │
-│    │   └─ Mark RUNNABLE (re-run needed)             │
-│    │                                                  │
-│    └─ Hash Match?                                   │
-│        └─ Mark SKIPPED (up-to-date)                 │
-│                                                       │
-│  5. Create Audit Ledger (PENDING state)             │
-│  6. Publish RUNNABLE tasks to Pub/Sub               │
-└─────────────────────────────────────────────────────┘
-```
-
-### Phase 2: Worker Execution
-
-```
-┌─────────────────────────────────────────────────────┐
-│         Computation Worker                          │
-│  (Processes Single Task)                            │
-├─────────────────────────────────────────────────────┤
-│  1. Parse Pub/Sub Message                           │
-│     { date, pass, computation, previousCategory }   │
-│                                                       │
-│  2. Load Manifest (cached in memory)                │
-│                                                       │
-│  3. Fetch Dependencies                              │
-│     - Load dependency results (auto-reassemble)     │
-│     - Load previous day's results (if historical)   │
-│                                                       │
-│  4. Execute Calculation                             │
-│     ├─ Standard: Stream users in batches            │
-│     └─ Meta: Load global data / price shards        │
-│                                                       │
-│  5. Validate Results (HeuristicValidator)           │
-│     - NaN Detection                                 │
-│     - Flatline Detection (stuck values)             │
-│     - Null/Empty Analysis                           │
-│     - Dead Object Detection                         │
-│                                                       │
-│  6. Store Results                                   │
-│     ├─ Calculate size                               │
-│     ├─ If > 900KB → Auto-shard                      │
-│     └─ Write to Firestore                           │
-│                                                       │
-│  7. Update Status & Ledgers                         │
-│     ├─ computation_status/{date} → New hash         │
-│     ├─ Audit Ledger → COMPLETED                     │
-│     └─ Run History → SUCCESS/FAILURE record         │
-│                                                       │
-│  8. Category Migration (if detected)                │
-│     └─ Delete old category's data                   │
-└─────────────────────────────────────────────────────┘
-```
+### Phase 3: Dispatch Analysis
+The Dispatcher iterates through the Topological Passes (0 -> N). For each calculation, it queries `calculateExecutionStatus`:
+*   Are dependencies done?
+*   Did dependencies change their output (`ResultHash`)?
+*   Is historical context available?
 
-### Error Handling Stages
+### Phase 4: Execution Waves
+Workers are triggered via Pub/Sub or direct method invocation.
+*   **Pass 1**: Primitive conversions (e.g., Price Extractor).
+*   **Pass 2**: Technical Indicators that depend on Pass 1.
+*   **Pass 3**: Aggregations and Complex Metrics.
 
-The system tracks **where** failures occur:
-
-```javascript
-// Run History Schema
-{
-  status: "FAILURE" | "SUCCESS" | "CRASH",
-  error: {
-    message: "...",
-    stage: "EXECUTION" | "PREPARE_SHARDS" | "COMMIT_BATCH" | 
-           "SHARDING_LIMIT_EXCEEDED" | "QUALITY_CIRCUIT_BREAKER" |
-           "MANIFEST_LOAD" | "SYSTEM_CRASH"
-  }
-}
-```
-
-**Stage-Specific Handling**:
-- `QUALITY_CIRCUIT_BREAKER`: Block deployment, data integrity issue
-- `SHARDING_LIMIT_EXCEEDED`: Firestore hard limit hit, needs redesign
-- `SYSTEM_CRASH`: Infrastructure issue, retriable
-- `EXECUTION`: Logic bug in calculation code
+### Phase 5: Reconciliation
+After all queues drain, the system performs a final sweep. Any tasks marked `FAILED` are retried (up to a limit). Impossible tasks are finalized as `IMPOSSIBLE`.
 
 ---
 
-## Smart Hashing & Versioning
-
-### Hash Composition
+## 4. Deep Dive: Hashing & Dependency Logic
 
+### Intrinsic Code Hashing
+Located in `topology/HashManager.js`.
+We generate a unique fingerprint for every calculation file:
 ```javascript
-// Step 1: Intrinsic Hash (Code + System Epoch)
-const codeHash = SHA256(calculation.toString());
-const intrinsicHash = SHA256(codeHash + "|EPOCH:v1.0-epoch-2");
-
-// Step 2: Layer Hashing (Dynamic Detection)
-let compositeHash = intrinsicHash;
-for (const [layer, exports] of MATH_LAYERS) {
-  for (const [exportName, triggerPatterns] of exports) {
-    if (codeString.includes(exportName)) {
-      compositeHash += layerHashes[layer][exportName];
-    }
-  }
-}
-
-// Step 3: Dependency Hashing (Merkle Tree)
-const depHashes = dependencies.map(dep => dep.hash).join('|');
-const finalHash = SHA256(compositeHash + "|DEPS:" + depHashes);
+clean = codeString.replace(comments).replace(whitespace);
+hash = sha256(clean);
 ```
+This ensures that changes to comments or formatting do *not* trigger re-runs.
 
-### Cascading Invalidation Example
-
-```
-Initial State:
-  PriceVolatility    → hash: abc123
-  UserRisk (uses PV) → hash: def456 (includes abc123)
-  Signal (uses UR)   → hash: ghi789 (includes def456)
-
-Developer Updates PriceVolatility:
-  PriceVolatility    → hash: xyz000 (NEW!)
-  UserRisk           → hash: uvw111 (NEW! Dependency changed)
-  Signal             → hash: rst222 (NEW! Cascade)
-
-Next Dispatch:
-  All 3 calculations marked RUNNABLE (hash mismatch)
-```
-
-### System Epoch
-
-`system_epoch.js`:
-```javascript
-module.exports = "v1.0-epoch-2";
-```
+### Behavioral Hashing (SimHash)
+Located in `simulation/SimRunner.js`.
+When code changes, we can't be 100% sure it's safe just by looking at the source.
+1.  **The Fabricator**: Generates a deterministic mock `Context` (prices, previous results) based on the input schema.
+2.  **Simulation Run**: The calculation `process()` method is executed against this mock data.
+3.  **The Registry**: The hash of the *output* of this simulation is stored.
+If a refactor results in the exact same Mock Output, the system considers the change "Cosmetic".
 
-**Purpose**: Changing this string forces **global re-computation** of all calculations, even if code hasn't changed. Used for:
-- Schema migrations
-- Critical bug fixes requiring historical reprocessing
-- Firestore structure changes
+### Dependency Short-Circuiting
+Implemented in `WorkflowOrchestrator.js` (`analyzeDateExecution`).
+Even if an upstream calculation re-runs, downstream dependents might not need to.
+*   **Logic**:
+    *   Calc A (Upstream) re-runs. Old Output Hash: `HashX`. New Output Hash: `HashX`.
+    *   Calc B (Downstream) sees that Calc A "changed" (new timestamp), BUT the content hash `HashX` is identical to what Calc B used last time.
+    *   **Result**: Calc B is `SKIPPED`.
 
 ---
 
-## Data Loading & Streaming
-
-### Streaming Architecture (Standard Computations)
-
-```javascript
-// Problem: 10M users × 5KB portfolio = 50GB
-// Solution: Stream in chunks
-
-async function* streamPortfolioData(dateStr, refs) {
-  const BATCH_SIZE = 50; // 50 users at a time
-  
-  for (let i = 0; i < refs.length; i += BATCH_SIZE) {
-    const batchRefs = refs.slice(i, i + BATCH_SIZE);
-    const userData = await loadDataByRefs(batchRefs);
-    
-    yield userData; // { user1: {...}, user2: {...}, ... }
-    
-    // Memory cleared after each iteration
-  }
-}
-
-// Usage in Executor
-for await (const userBatch of streamPortfolioData(date, refs)) {
-  for (const [userId, portfolio] of Object.entries(userBatch)) {
-    const context = buildContext({ userId, portfolio, ... });
-    await calculation.process(context);
-  }
-}
-```
-
-### Price Data Batching (Meta Computations)
-
-```javascript
-// Problem: 10,000 tickers × 2 years history = 1GB
-// Solution: Process shards sequentially
-
-for (const shardRef of priceShardRefs) {
-  const shardData = await loadPriceShard(shardRef); // ~100 tickers
-  
-  const context = buildMetaContext({ 
-    prices: { history: shardData } 
-  });
-  
-  await calculation.process(context);
-  
-  // Results accumulate across shards
-  // Memory cleared between shards
-}
-```
-
-### Smart Shard Indexing (Optimization)
-
-For targeted price lookups (e.g., "only calculate momentum for AAPL, GOOGL, MSFT"):
-
-```javascript
-// Without Indexing: Load ALL shards, filter after
-// Cost: 100+ Firestore reads
-
-// With Indexing: Pre-map which shard contains each instrument
-const index = {
-  "123": "shard_0",  // AAPL
-  "456": "shard_2",  // GOOGL
-  "789": "shard_0"   // MSFT
-};
-
-const relevantShards = ["shard_0", "shard_2"];
-// Cost: 2 Firestore reads
-```
-
-**Index Building**: Runs once, cached in `/system_metadata/price_shard_index`.
+## 5. Decision Logic & Edge Case Scenarios
+
+### Scenario A: Standard Code Change (Logic)
+*   **Trigger**: You change the formula for `RSI`. Code Hash changes. SimHash changes.
+*   **Dispatcher**: Sees `storedHash !== currentHash`.
+*   **Result**: Marks as `RUNNABLE`. Worker runs.
+
+### Scenario B: Cosmetic Code Change (Refactor)
+*   **Trigger**: You rename a variable in `RSI`. Code Hash changes. SimHash remains identical.
+*   **Optimizer**: Updates the centralized Status Ledger: "Version `Desc_v2` is equivalent to `Desc_v1`".
+*   **Dispatcher**: Sees the new hash in the ledger as "Verified".
+*   **Result**: Task is `SKIPPED`.
+
+### Scenario C: Upstream Invalidation (The Cascade)
+*   **Condition**: `PriceExtractor` fixes a bug. `ResultHash` changes from `HashA` to `HashB`.
+*   **Downstream**: `RSI` checks detailed dependency report.
+*   **Check**: `LastRunDeps['PriceExtractor'] (HashA) !== CurrentDeps['PriceExtractor'] (HashB)`.
+*   **Result**: `RSI` is forced to re-run.
+
+### Scenario D: Upstream Stability (The Firewall)
+*   **Condition**: `PriceExtractor` runs an optimization. Output is exact same data. `ResultHash` remains `HashA`.
+*   **Downstream**: `RSI` checks dependency report.
+*   **Check**: `LastRunDeps['PriceExtractor'] (HashA) === CurrentDeps['PriceExtractor'] (HashA)`.
+*   **Result**: `RSI` is `SKIPPED`. This firewall prevents massive re-calculation storms for non-functional upstream changes.
+
+### Scenario E: The "Impossible" State
+*   **Condition**: Core market data is missing for `1990-01-01`.
+*   **Root Indexer**: Marks date as providing `[]` (empty) for critical inputs.
+*   **Dispatcher**: Marks `PriceExtractor` as `IMPOSSIBLE: NO_DATA`.
+*   **Propagation**: Any calculation depending on `PriceExtractor` sees the `IMPOSSIBLE` status and marks *itself* as `IMPOSSIBLE: UPSTREAM`.
+*   **Benefit**: The system doesn't waste cycles retrying calculations that can never succeed.
+
+### Scenario F: Category Migration
+*   **Condition**: You change `getMetadata()` for a calculation, moving it from `signals` to `risk`.
+*   **Dispatcher**: Detects `storedCategory !== newCategory`.
+*   **Worker**:
+    1.  Runs `process()` and writes to the *new* path (`risk/CalculateX`).
+    2.  Detects the `previousCategory` flag.
+    3.  Deletes the data at the *old* path (`signals/CalculateX`) to prevent orphan data.
 
 ---
 
-## Auto-Sharding System
-
-### The 1MB Problem
-
-Firestore's hard limit: **1MB per document**. A calculation producing results for 5,000 tickers easily exceeds this.
+## 6. Data Management & Storage
 
-### Transparent Sharding Solution
+### Input Streaming
+To handle large datasets without OOM (Out Of Memory) errors:
+*   `StandardExecutor` does not load all users/tickers at once.
+*   It utilizes wait-and-stream logic (e.g., batches of 50 ids) to process the `Context`.
 
-```javascript
-// ResultCommitter.js - prepareAutoShardedWrites()
-
-const totalSize = calculateFirestoreBytes(result);
-
-if (totalSize < 900KB) {
-  // Write normally
-  /results/{date}/{category}/{calc}
-    → { "AAPL": {...}, "GOOGL": {...}, _completed: true }
-  
-} else {
-  // Auto-shard
-  // Step 1: Split into chunks < 900KB each
-  const chunks = splitIntoChunks(result);
-  
-  // Step 2: Write shards
-  /results/{date}/{category}/{calc}/_shards/shard_0 → chunk 1
-  /results/{date}/{category}/{calc}/_shards/shard_1 → chunk 2
-  /results/{date}/{category}/{calc}/_shards/shard_N → chunk N
-  
-  // Step 3: Write pointer
-  /results/{date}/{category}/{calc}
-    → { _sharded: true, _shardCount: N, _completed: true }
-}
-```
-
-### Transparent Reassembly
-
-```javascript
-// DependencyFetcher.js - fetchExistingResults()
-
-const doc = await docRef.get();
-const data = doc.data();
-
-if (data._sharded === true) {
-  // 1. Fetch all shards
-  const shardsCol = docRef.collection('_shards');
-  const snapshot = await shardsCol.get();
-  
-  // 2. Merge back into single object
-  const assembled = {};
-  snapshot.forEach(shard => {
-    Object.assign(assembled, shard.data());
-  });
-  
-  // 3. Return as if never sharded
-  return assembled;
-}
-
-// Normal path: return as-is
-return data;
-```
+### Transparent Auto-Sharding
+Firestore has a 1MB document limit.
+*   **Write Path**: If a calculation result > 900KB, it is split into `DocID`, `DocID_shard1`, `DocID_shard2`.
+*   **Read Path**: The `DependencyFetcher` automatically detects sharding pointers and re-assembles (hydrates) the full object before passing it to `process()`.
 
-**Developer Experience**: You **never** know or care whether data is sharded. Read and write as if documents have no size limit.
-
-### Sharding Limits
-
-**Maximum Calculation Size**: ~450MB (500 shards × 900KB)
-
-If a calculation exceeds this, the system throws:
-```
-error: { 
-  stage: "SHARDING_LIMIT_EXCEEDED",
-  message: "Firestore subcollection limit reached"
-}
-```
-
-**Solution**: Refactor calculation to produce less data or split into multiple calculations.
+### Compression Strategy
+*   Payloads are inspected before write.
+*   If efficient (high entropy text/JSON), Zlib compression is applied.
+*   Metadata is tagged `encoding: 'zlib'` so readers know to inflate.
 
 ---
 
-## Quality Assurance
-
-### HeuristicValidator (Grey Box Testing)
-
-Runs statistical analysis on results **before storage**:
-
-```javascript
-// ResultsValidator.js
-
-1. NaN Detection
-   - Scans sample of results for NaN/Infinity
-   - Threshold: 0% (strict, NaN is always a bug)
+## 7. Quality Assurance & Self-Healing
 
-2. Flatline Detection
-   - Checks if >95% of values are identical
-   - Catches stuck loops or broken RNG
+### The Heuristic Validator
+Before saving *any* result, the Executor runs heuristics:
+*   **NaN Check**: Are there `NaN` or `Infinity` values in key fields?
+*   **Flatline Check**: Is the data variance 0.00 across a large timespan?
+*   **Null Density**: Is >50% of the dataset null?
+*   **Circuit Breaker**: If heuristics fail, the task throws an error. It is better to fail and alert than to persist corrupted data that pollutes the cache.
 
-3. Null/Empty Analysis
-   - Threshold: 90% of results are null/0
-   - Indicates data pipeline failure
+### Zombie Task Recovery
+*   **Lease Mechanism**: When a task starts, it sets a `startedAt` timestamp.
+*   **Detection**: The Dispatcher checks for tasks marked `RUNNING` where `startedAt` > 15 minutes ago.
+*   **Resolution**: These are assumed crashed (OOM/Timeout). They are reset to `PENDING` (or `FAILED` if retry count exceeded).
 
-4. Dead Object Detection
-   - Finds objects where all properties are null/0
-   - Example: { profile: [], score: 0, signal: null }
-
-5. Vector Emptiness (Distribution Calcs)
-   - Checks if histogram/profile arrays are empty
-   - Threshold: 90% empty → FAIL
-```
-
-**Circuit Breaker**: If validation fails, the calculation **does not store results** and is marked as `FAILURE` with stage: `QUALITY_CIRCUIT_BREAKER`.
-
-### Validation Overrides
-
-For legitimately sparse datasets:
-
-```javascript
-// validation_overrides.js
-module.exports = {
-  "bankruptcy-detector": { 
-    maxZeroPct: 100  // Rare event, 100% zeros is expected
-  },
-  "earnings-surprise": {
-    maxNullPct: 99   // Only runs on earnings days
-  }
-};
-```
-
-### Build Reporter (Pre-Deployment Analysis)
-
-```bash
-npm run build-reporter
-```
-
-Generates a **simulation report** without running calculations:
-
-```
-Build Report: v1.2.5_2024-12-07
-================================
-
-Summary:
-  - 1,245 Re-Runs (hash mismatch)
-  - 23 New Calculations
-  - 0 Impossible
-  - 45 Blocked (waiting for data)
-
-Detailed Breakdown:
-
-2024-12-01:
-  Will Re-Run:
-    - user-risk-profile (Hash: abc123 → xyz789)
-    - sentiment-score (Hash: def456 → uvw012)
-  
-  Blocked:
-    - social-sentiment (Missing Root Data: social)
-
-2024-12-02:
-  Will Run:
-    - new-momentum-signal (New calculation)
-```
-
-**Use Case**: Review before deploying to production. If 10,000 re-runs are detected, investigate whether code change was intentional.
+### Dead Letter Queue (DLQ)
+Tasks that deterministically fail (crash every time) after N retries are moved to a special DLQ status. This prevents the system from getting stuck in an infinite retry loop.
 
 ---
 
-## Operational Modes
+## 8. Developer Workflows
 
-### Mode 1: Local Orchestrator (Development)
+### How to Add a New Calculation
+1.  Create `calculations/category/MyNewCalc.js`.
+2.  Implement `getMetadata()` to define dependencies.
+3.  Implement `process(context)`.
+4.  Run `npm run build-manifest` to register it in the topology.
 
-```bash
-# Run all calculations for Pass 1 sequentially
-COMPUTATION_PASS_TO_RUN=1 npm run computation-orchestrator
-```
-
-**Behavior**:
-- Single-process execution
-- Loads manifest
-- Iterates through all dates
-- Runs calculations in order
-- Good for: Debugging, local testing
+### How to Force a Global Re-Run
+*   Change the `SYSTEM_EPOCH` constant in `system_epoch.js`.
+*   This changes the "Global Salt" for all hashes, processing every calculation as "New".
 
-### Mode 2: Dispatcher + Workers (Production)
+### How to Backfill History
+*   **Standard Dispatcher**: Good for recent history (last 30 days).
+*   **BatchPriceExecutor**: Specialized for massive historical backfills (e.g., 20 years of price data). It bypasses some topology checks for raw speed.
 
+### Local Debugging
+Run the orchestrator in "Dry Run" mode:
 ```bash
-# Step 1: Dispatch tasks to Pub/Sub
-COMPUTATION_PASS_TO_RUN=1 npm run computation-dispatcher
-
-# Step 2: Cloud Function workers consume tasks
-# (Auto-scaled by GCP, 0 to 1000+ workers)
+node scripts/run_orchestrator.js --date=2024-01-01 --dry-run
 ```
-
-**Behavior**:
-- Dispatcher analyzes all dates
-- Publishes ~10,000 messages to Pub/Sub
-- Workers process in parallel
-- Each worker handles 1 date
-- Auto-retries on failure (Pub/Sub built-in)
-
-**Scaling**: 1,000 dates × 3 calcs = 3,000 tasks. With 100 workers, completes in ~5 minutes.
-
-### Mode 3: Batch Price Executor (Optimization)
-
-```bash
-# For price-dependent calcs, bulk-process historical data
-npm run batch-price-executor --dates=2024-12-01,2024-12-02 --calcs=momentum-signal
-```
-
-**Behavior**:
-- Loads price shards once
-- Processes multiple dates in a single pass
-- Bypasses Pub/Sub overhead
-- **10x faster** for historical backfills
-
-**Use Case**: After deploying a new price-dependent calculation, backfill 2 years of history in 1 hour instead of 10.
-
----
-
-## Advanced Topics
-
-### Historical Continuity Enforcement
-
-For calculations that depend on their own previous results:
-
-```javascript
-// Example: cumulative-pnl needs yesterday's cumulative-pnl
-
-static getMetadata() {
-  return { isHistorical: true };
-}
-
-// Dispatcher Logic:
-if (calculation.isHistorical) {
-  const yesterday = date - 1 day;
-  const yesterdayStatus = await fetchComputationStatus(yesterday);
-  
-  if (!yesterdayStatus[calcName] || 
-      yesterdayStatus[calcName].hash !== currentHash) {
-    // Yesterday is missing or has wrong hash
-    report.blocked.push({ 
-      reason: "Waiting for historical continuity" 
-    });
-  }
-}
-```
-
-**Result**: Historical calculations run in **strict chronological order**, never skipping days.
-
-### Category Migration System
-
-If a calculation's category changes:
-
-```javascript
-// Before: category: 'signals'
-// After:  category: 'risk-management'
-
-// System detects change:
-manifest.previousCategory = 'signals';
-
-// Worker executes:
-1. Runs calculation normally
-2. Stores in new category: /results/{date}/risk-management/{calc}
-3. Deletes old category: /results/{date}/signals/{calc}
-```
-
-**Automation**: Zero manual data migration needed.
-
-### Audit Ledger vs Run History
-
-**Audit Ledger** (`computation_audit_ledger/{date}/passes/{pass}/tasks/{calc}`):
-- Created **before** dispatch
-- Status: PENDING → COMPLETED
-- Purpose: Track which tasks were dispatched
-
-**Run History** (`computation_run_history/{date}/runs/{runId}`):
-- Created **after** execution attempt
-- Status: SUCCESS | FAILURE | CRASH
-- Purpose: Debug failures, track performance
-
-**Why Both?**: Audit Ledger answers "What should run?", Run History answers "What actually happened?".
-
----
-
-## Summary: The Complete Flow
-
-### For a Standard Calculation
-
-```
-1. Root Data Indexer (Daily)
-   └─ Scans all data sources
-   └─ Creates availability manifest
-
-2. Dispatcher (Per-Pass)
-   ├─ Loads manifest
-   ├─ For each date:
-   │   ├─ Checks root data availability
-   │   ├─ Checks dependency status
-   │   ├─ Checks historical continuity
-   │   └─ Decides: RUNNABLE | BLOCKED | IMPOSSIBLE
-   ├─ Creates Audit Ledger (PENDING)
-   └─ Publishes RUNNABLE tasks to Pub/Sub
-
-3. Worker (Per-Task)
-   ├─ Receives {date, pass, computation}
-   ├─ Loads manifest (cached)
-   ├─ Fetches dependencies (auto-reassembles shards)
-   ├─ Streams portfolio data in batches
-   ├─ For each user:
-   │   ├─ Builds context (dependency injection)
-   │   └─ Calls calculation.process(context)
-   ├─ Validates results (HeuristicValidator)
-   ├─ Auto-shards if > 900KB
-   ├─ Commits to Firestore
-   ├─ Updates status hash
-   ├─ Updates Audit Ledger → COMPLETED
-   └─ Records Run History → SUCCESS
-
-4. Next Pass
-   └─ Depends on results from this pass
-```
-
-### For a Meta Calculation
-
-Same flow, except:
-- **Step 3**: Loads global data instead of streaming users
-- **Context**: No user object, prices/insights instead
-- **Result**: One document with all tickers' data
-
----
-
-## Key Takeaways
-
-1. **Data Availability Gates Everything**: Computations never run when source data is missing
-2. **Smart Hashing Enables Incremental Updates**: Only changed calculations re-run
-3. **Sharding is Invisible**: Read/write as if documents have no size limit
-4. **Streaming Handles Scale**: Process millions of users without OOM
-5. **Quality Checks Prevent Bad Data**: Results validated before storage
-6. **Historical Continuity is Enforced**: Time-series calculations run in order
-7. **Distributed Execution Scales Infinitely**: 1 worker or 1,000 workers, same code
-
----
-
-## Operational Checklist
-
-**Daily (Automated)**:
-- ✅ Root Data Indexer runs at 2 AM UTC
-- ✅ Computation Dispatchers run for each pass (3 AM, 4 AM, 5 AM)
-- ✅ Workers auto-scale based on Pub/Sub queue depth
-
-**After Code Changes**:
-1. Run Build Reporter to preview impact
-2. Review re-run count (expected vs actual)
-3. Deploy to staging, run single date
-4. Validate results in Firestore
-5. Deploy to production
-6. Monitor Run History for failures
-
-**Debugging a Failure**:
-1. Check Run History for error stage
-2. If `QUALITY_CIRCUIT_BREAKER`: Data integrity issue, review validator logs
-3. If `EXECUTION`: Logic bug, reproduce locally with Orchestrator mode
-4. If `SYSTEM_CRASH`: Infrastructure issue, check Cloud Function logs
-5. Fix bug, redeploy, re-trigger specific pass
+This prints the `Analysis Report` (Runnable/Blocked lists) without actually triggering workers.