bulltrackers-module 1.0.591 → 1.0.593

package/functions/generic-api/user-api/FIRESTORE_PATH_MIGRATION_REFERENCE.md

@@ -1,710 +0,0 @@
-# Firestore Path Migration Reference
-
-## Complete Legacy → New Path Mapping
-
-This document provides a comprehensive reference for all Firestore path migrations from legacy formats to the new user-centric structure. All paths include example values for dynamic segments.
-
----
-
-## 1. User Notifications
-
-### Legacy Paths
-```
-user_notifications/{userCid}/notifications/{notificationId}
-user_notifications/{userCid}/counters/{date}
-user_notifications/{firebaseUid}/notifications/{notificationId}  (test alerts)
-```
-
-**Example Legacy Paths:**
-- `user_notifications/29312236/notifications/test_alert_1767063681211_29312236_1_xx0ey9x`
-- `user_notifications/29312236/counters/2025-12-28`
-- `user_notifications/GYXUIoWNnNSxJf3o5CeZm1fljlp1/notifications/test_alert_1767063681211_GYXUIoWNnNSxJf3o5CeZm1fljlp1_1_xx0ey9x`
-
-### New Paths
-```
-SignedInUsers/{cid}/notifications/{notificationId}
-SignedInUsers/{cid}/notifications/counters/{date}  (if needed)
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/notifications/test_alert_1767063681211_29312236_1_xx0ey9x`
-- `SignedInUsers/29312236/notifications/counters/2025-12-28`
-
-**Notes:**
-- Always use CID, not Firebase UID
-- Use `getCidFromFirebaseUid()` helper when converting from Firebase UID
-- Migration: Auto-migrate on read
-
----
-
-## 2. User Alerts
-
-### Legacy Paths
-```
-user_alerts/{userCid}/alerts/{alertId}
-user_alerts/{userCid}/counters/{date}
-```
-
-**Example Legacy Paths:**
-- `user_alerts/29312236/alerts/alert_1767063681211_31075566`
-- `user_alerts/29312236/counters/2025-12-28`
-
-### New Paths
-```
-SignedInUsers/{cid}/alerts/{alertId}
-SignedInUsers/{cid}/alerts/counters/{date}  (if needed)
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/alerts/alert_1767063681211_31075566`
-- `SignedInUsers/29312236/alerts/counters/2025-12-28`
-
-**Notes:**
-- Alert counters track unread/total counts per day
-- Migration: Auto-migrate on read
-
----
-
-## 3. Watchlists (Private)
-
-### Legacy Paths
-```
-watchlists/{userCid}/lists/{watchlistId}
-user_watchlists/{userCid}/lists/{watchlistId}
-```
-
-**Example Legacy Paths:**
-- `watchlists/29312236/lists/watchlist_1766797382685_091cca66`
-- `user_watchlists/29312236/lists/watchlist_1766797382685_091cca66`
-
-### New Paths
-```
-SignedInUsers/{cid}/watchlists/{watchlistId}
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/watchlists/watchlist_1766797382685_091cca66`
-
-**Notes:**
-- Private watchlists only
-- Public watchlists remain in `public_watchlists` collection (no migration needed)
-- Migration: Auto-migrate on read
-
----
-
-## 4. Watchlist Subscriptions
-
-### Legacy Paths
-```
-watchlist_subscriptions/{userCid}/alerts/{piCid}
-```
-
-**Example Legacy Paths:**
-- `watchlist_subscriptions/29312236/alerts/31075566`
-
-### New Paths
-```
-SignedInUsers/{cid}/watchlists/{watchlistId}/subscriptions/{piCid}  (per-watchlist)
-SignedInUsers/{cid}/subscriptions/{piCid}  (global subscriptions)
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/watchlists/watchlist_1766797382685_091cca66/subscriptions/31075566`
-- `SignedInUsers/29312236/subscriptions/31075566`
-
-**Notes:**
-- Per-watchlist subscriptions: tied to a specific watchlist
-- Global subscriptions: not tied to a watchlist
-- Migration: Auto-migrate on read
-
----
-
-## 5. User Verification
-
-### Legacy Paths
-```
-user_verifications/{username}
-```
-
-**Example Legacy Paths:**
-- `user_verifications/daniel81991`
-- `user_verifications/marau2021`
-
-### New Paths
-```
-SignedInUsers/{cid}/verification/data
-```
-
-**Example New Paths:**
-- `SignedInUsers/12336860/verification/data`
-- `SignedInUsers/29312236/verification/data`
-
-**Notes:**
-- Legacy path uses username (before CID is known)
-- New path uses CID (after verification completes)
-- Migration: After `finalizeVerification` gets `realCID` from bio check API, migrate to CID-based path
-- Firebase UID: Store `firebaseUid` in verification document once Firebase account is created
-- Using subcollection with single document named "data" to comply with Firestore even-segment rule (4 segments)
-
-**Schema Example:**
-```javascript
-{
-  username: "daniel81991",
-  cid: 12336860,
-  firebaseUid: "1C2VKLjEqcZKi3NFAFfPsx5Dins1",  // Added after account creation
-  otp: "38K0E1",
-  status: "VERIFIED",
-  createdAt: Timestamp,
-  expiresAt: Timestamp,
-  verifiedAt: Timestamp,
-  isOptOut: false
-}
-```
-
----
-
-## 6. User Sync Requests
-
-### Legacy Paths
-```
-user_sync_requests/{userCid}/requests/{requestId}
-user_sync_requests/{userCid}/global/latest
-```
-
-**Example Legacy Paths:**
-- `user_sync_requests/29312236/requests/sync_1766941189840_801ad9f3`
-- `user_sync_requests/29312236/global/latest`
-
-### New Paths
-```
-SignedInUsers/{cid}/syncRequests/{requestId}
-SignedInUsers/{cid}/syncStatus/latest
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/syncRequests/sync_1766941189840_801ad9f3`
-- `SignedInUsers/29312236/syncStatus/latest`
-
-**Notes:**
-- Sync requests track on-demand data sync history
-- Global status tracks latest request metadata
-- Migration: Auto-migrate on read
-
-**Schema Example (Request):**
-```javascript
-{
-  requestId: "sync_1766941189840_801ad9f3",
-  targetUserCid: 29312236,
-  requestedBy: 29312236,
-  effectiveRequestedBy: 29312236,
-  username: "Marau2021",
-  userType: "SIGNED_IN_USER",
-  status: "completed",
-  createdAt: Timestamp,
-  dispatchedAt: Timestamp,
-  completedAt: Timestamp,
-  updatedAt: Timestamp
-}
-```
-
----
-
-## 7. Portfolio Data (Latest Snapshot)
-
-### Legacy Paths
-```
-{signedInUsersCollection}/{blockId}/snapshots/{date}/parts/part_{N}
-signed_in_users/{blockId}/snapshots/{date}/parts/part_{N}
-```
-
-**Example Legacy Paths:**
-- `signed_in_users/19M/snapshots/2025-12-26/parts/part_0`
-- `signed_in_users/19M/snapshots/2025-12-26/parts/part_1`
-
-### Root Data Paths (Task Engine Format - No Migration)
-```
-SignedInUserPortfolioData/{date}/{cid}/{cid}
-```
-
-**Example Root Data Paths:**
-- `SignedInUserPortfolioData/2025-12-26/29312236/29312236`
-
-### New Paths (Latest Snapshot - Fallback)
-```
-SignedInUsers/{cid}/portfolio/latest
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/portfolio/latest`
-
-**Notes:**
-- Root data format: `{collection}/{date}/{cid}/{cid}` (document ID is CID)
-- Latest snapshot is stored per-user for fallback when computation doesn't exist
-- Task engine writes to root data format
-- Migration: Auto-migrate on read
-
-**Schema Example (Latest Snapshot):**
-```javascript
-{
-  AggregatedPositions: [...],
-  AggregatedMirrors: [...],
-  fetchedAt: Timestamp,
-  date: "2025-12-26",
-  cid: 29312236
-}
-```
-
----
-
-## 8. Trade History (Latest Snapshot)
-
-### Legacy Paths
-```
-{signedInHistoryCollection}/{blockId}/snapshots/{date}/parts/part_{N}
-```
-
-**Example Legacy Paths:**
-- `signed_in_user_history/19M/snapshots/2025-12-26/parts/part_0`
-
-### Root Data Paths (Task Engine Format - No Migration)
-```
-SignedInUserTradeHistoryData/{date}/{cid}/{cid}
-```
-
-**Example Root Data Paths:**
-- `SignedInUserTradeHistoryData/2025-12-26/29312236/29312236`
-
-### New Paths (Latest Snapshot - Fallback)
-```
-SignedInUsers/{cid}/tradeHistory/latest
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/tradeHistory/latest`
-
-**Notes:**
-- Same structure as portfolio data
-- Migration: Auto-migrate on read
-
----
-
-## 9. Social Posts (Latest Snapshot)
-
-### Legacy Paths
-```
-signed_in_users_social/{cid}/posts/{postId}
-```
-
-**Example Legacy Paths:**
-- `signed_in_users_social/29312236/posts/00c0f500-f199-11ee-8080-80000769abad`
-
-### Root Data Paths (Task Engine Format - No Migration)
-```
-SignedInUserSocialPostData/{date}/{cid}/{cid}
-```
-
-**Example Root Data Paths:**
-- `SignedInUserSocialPostData/2025-12-26/29312236/29312236`
-
-### New Paths (Latest Snapshot - Fallback)
-```
-SignedInUsers/{cid}/posts/{postId}
-```
-
-**Example New Paths:**
-- `SignedInUsers/29312236/posts/00c0f500-f199-11ee-8080-80000769abad`
-
-**Notes:**
-- Task engine can write all posts here as well as root data location
-- Migration: Auto-migrate on read
-
----
-
-## 10. Popular Investor Fetch Requests
-
-### Legacy Paths
-```
-pi_fetch_requests/{piCid}/requests/{requestId}
-pi_fetch_requests/{piCid}/user_requests/{userCid}
-pi_fetch_requests/{piCid}/global/latest
-```
-
-**Example Legacy Paths:**
-- `pi_fetch_requests/31075566/requests/req_1766890969236_4c887876`
-- `pi_fetch_requests/31075566/user_requests/647267`
-- `pi_fetch_requests/31075566/global/latest`
-
-### New Paths
-```
-PopularInvestors/{piCid}/fetchRequests/{requestId}
-PopularInvestors/{piCid}/userFetchRequests/{userCid}
-PopularInvestors/{piCid}/fetchStatus/latest
-```
-
-**Example New Paths:**
-- `PopularInvestors/31075566/fetchRequests/req_1766890969236_4c887876`
-- `PopularInvestors/31075566/userFetchRequests/647267`
-- `PopularInvestors/31075566/fetchStatus/latest`
-
-**Notes:**
-- PI-centric data (not user-centric)
-- Rate limiting tracked per user, per PI
-- Migration: Auto-migrate on read
-
-**Schema Example (Request):**
-```javascript
-{
-  requestId: "req_1766890969236_4c887876",
-  actualUserCid: 29312236,
-  createdAt: Timestamp,
-  isImpersonating: true,
-  piCid: 31075566,
-  piUsername: "Gr33nTrader",
-  status: "queued",
-  taskMessageId: "17612318165520680",
-  updatedAt: Timestamp,
-  userCid: 647267
-}
-```
-
----
-
-## 11. Popular Investor Reviews
-
-### Legacy Paths
-```
-pi_reviews/{piCid}_{userCid}
-```
-
-**Example Legacy Paths:**
-- `pi_reviews/29312236_647267`
-
-### New Paths
-```
-PopularInvestors/{piCid}/reviews/{reviewId}
-```
-
-**Example New Paths:**
-- `PopularInvestors/647267/reviews/29312236_647267`
-
-**Notes:**
-- Review ID format: `{userCid}_{piCid}`
-- Migration: Auto-migrate on read
-
-**Schema Example:**
-```javascript
-{
-  comment: "Test Review",
-  createdAt: Timestamp,
-  isAnonymous: false,
-  piCid: 647267,
-  rating: 5,
-  reviewerUsername: "Marau2021",
-  updatedAt: Timestamp,
-  userCid: 29312236
-}
-```
-
----
-
-## 12. Popular Investor Social Posts
-
-### Legacy Paths
-```
-pi_social_posts/{piCid}/posts/{postId}
-```
-
-**Example Legacy Paths:**
-- `pi_social_posts/10050875/posts/00c0f500-f199-11ee-8080-80000769abad`
-
-### New Paths
-```
-PopularInvestors/{piCid}/posts/{postId}
-```
-
-**Example New Paths:**
-- `PopularInvestors/10050875/posts/00c0f500-f199-11ee-8080-80000769abad`
-
-**Notes:**
-- PI-centric data
-- Migration: Auto-migrate on read
-
----
-
-## 13. Profile Views
-
-### Legacy Paths
-```
-profile_views/{piCid}_{date}
-profile_views/individual_views/views/{piCid}_{viewerCid}_{timestamp}
-```
-
-**Example Legacy Paths:**
-- `profile_views/31075566_2025-12-27`
-- `profile_views/individual_views/views/31075566_29312236_1766868534621`
-
-### New Paths
-```
-PopularInvestors/{piCid}/profileViews/{date}
-PopularInvestors/{piCid}/views/{viewId}
-```
-
-**Example New Paths:**
-- `PopularInvestors/31075566/profileViews/2025-12-27`
-- `PopularInvestors/31075566/views/31075566_29312236_1766868534621`
-
-**Notes:**
-- Profile views tracked by date (aggregated)
-- Individual views tracked per view event
-- Migration: Auto-migrate on read
-
-**Schema Example (Aggregated):**
-```javascript
-{
-  date: "2025-12-27",
-  lastUpdated: Timestamp,
-  piCid: 31075566,
-  totalViews: 2,
-  uniqueViewers: ["29312236"]
-}
-```
-
-**Schema Example (Individual):**
-```javascript
-{
-  date: "2025-12-27",
-  piCid: 31075566,
-  viewedAt: Timestamp,
-  viewerCid: 29312236,
-  viewerType: "signed_in"
-}
-```
-
----
-
-## 14. Public Watchlists (No Migration - Already Correct)
-
-### Current Paths (Unchanged)
-```
-public_watchlists/{watchlistId}
-public_watchlists/{watchlistId}/versions/{version}
-```
-
-**Example Paths:**
-- `public_watchlists/watchlist_1766966023323_17e982f0`
-- `public_watchlists/watchlist_1766966023323_17e982f0/versions/1`
-
-**Notes:**
-- Public watchlists are system-wide, not user-centric
-- No migration needed
-
-**Schema Example (Version):**
-```javascript
-{
-  copyCount: 0,
-  createdAt: Timestamp,
-  createdBy: 29312236,
-  description: "",
-  dynamicConfig: {
-    alertConfig: {...},
-    computationName: "RiskScoreIncrease",
-    parameters: {...}
-  },
-  isImmutable: true,
-  name: "testing",
-  snapshotAt: Timestamp,
-  type: "dynamic",
-  version: 1,
-  watchlistId: "watchlist_1766966023323_17e982f0"
-}
-```
-
----
-
-## 15. System Collections (No Migration Needed)
-
-### Firebase Auth Mapping
-```
-signedInUsers/{firebaseUid}
-```
-
-**Example Paths:**
-- `signedInUsers/1C2VKLjEqcZKi3NFAFfPsx5Dins1`
-
-**Notes:**
-- **ONLY for Firebase Auth UID → eToro CID mapping**
-- All user data should use CID-based paths
-- No migration needed
-
-**Schema Example:**
-```javascript
-{
-  accountSetupComplete: true,
-  createdAt: Timestamp,
-  displayName: "Daniel Allison",
-  email: "daniel.allison@y7mail.com",
-  etoroCID: 12336860,
-  etoroUsername: "daniel81991",
-  photoURL: "https://...",
-  setupCompletedAt: Timestamp,
-  verifiedAt: Timestamp
-}
-```
-
-### User GCID Mappings
-```
-user_gcid_mappings/{cid}
-```
-
-**Example Paths:**
-- `user_gcid_mappings/10050875`
-
-**Notes:**
-- Maps eToro CID to GCID (social ID)
-- Used for social post context where eToro uses GCID instead of CID
-- No migration needed
-
-**Schema Example:**
-```javascript
-{
-  gcid: 10337657,
-  updatedAt: Timestamp,
-  username: "Rafnus"
-}
-```
-
----
-
-## 16. Root Data Collections (No Migration - Task Engine Format)
-
-### Signed-In User Root Data
-```
-SignedInUserPortfolioData/{date}/{cid}/{cid}
-SignedInUserTradeHistoryData/{date}/{cid}/{cid}
-SignedInUserSocialPostData/{date}/{cid}/{cid}
-```
-
-**Example Paths:**
-- `SignedInUserPortfolioData/2025-12-26/29312236/29312236`
-- `SignedInUserTradeHistoryData/2025-12-26/29312236/29312236`
-- `SignedInUserSocialPostData/2025-12-26/29312236/29312236`
-
-**Notes:**
-- Document ID is the CID (task engine format)
-- Populated by task engine, price fetcher, insights fetcher
-- No migration needed - this is the correct format
-
-### Popular Investor Root Data
-```
-PopularInvestorPortfolioData/{date}/{cid}
-PopularInvestorTradeHistoryData/{date}/{cid}
-PopularInvestorSocialPostData/{date}/{cid}
-```
-
-**Example Paths:**
-- `PopularInvestorPortfolioData/2025-12-26/10050875`
-- `PopularInvestorTradeHistoryData/2025-12-26/10050875`
-- `PopularInvestorSocialPostData/2025-12-26/10050875`
-
-**Notes:**
-- Document ID is the CID
-- Populated by task engine
-- No migration needed
-
-### Instrument Social Data
-```
-InstrumentFeedSocialPostData/{date}
-```
-
-**Example Paths:**
-- `InstrumentFeedSocialPostData/2025-12-26`
-
-**Notes:**
-- Instrument-specific social posts
-- No migration needed
-
----
-
-## 17. Computation Results (No Migration Needed)
-
-### Computation Results
-```
-unified_insights/{date}/results/{category}/computations/{computationName}
-unified_insights/{date}/results/{category}/computations/{computationName}/_shards/{shardId}
-```
-
-**Example Paths:**
-- `unified_insights/2025-12-26/results/popular-investor/computations/PopularInvestorProfileMetrics`
-- `unified_insights/2025-12-26/results/popular-investor/computations/PopularInvestorProfileMetrics/_shards/shard_0`
-
-**Notes:**
-- May be sharded for large datasets
-- No migration needed
-
----
-
-## 18. Rankings (No Migration Needed)
-
-### Popular Investor Rankings
-```
-popular_investor_rankings/{date}
-```
-
-**Example Paths:**
-- `popular_investor_rankings/2025-12-26`
-
-**Notes:**
-- Daily rankings from eToro API
-- No migration needed
-
----
-
-## Migration Strategy Summary
-
-### Auto-Migration on Read
-1. **Try new path first** - Attempt to read from new CID-based path
-2. **Fallback to legacy** - If not found, try legacy path
-3. **Auto-migrate** - If found in legacy path, automatically migrate to new path
-4. **Return data** - Return data from new path (or legacy if migration failed)
-
-### Dual-Write (During Transition)
-- Write to both new and legacy paths during transition period
-- Ensures consistency and facilitates migration
-
-### Path Resolution
-- Use `readWithMigration()` helper from `core/path_resolution_helpers.js`
-- Use `writeWithMigration()` helper for dual-write during transition
-- All paths defined in `collection_registry.js` with legacy mappings
-
----
-
-## Key Principles
-
-1. **Always use CID** - All user-centric data uses `{cid}`, not `{firebaseUid}`
-2. **Root data format** - Task engine writes to `{collection}/{date}/{cid}/{cid}` (document ID is CID)
-3. **Firebase UID mapping** - `signedInUsers/{firebaseUid}` is ONLY for UID→CID lookup
-4. **Even segments** - All paths must have even number of segments (collection/document/collection/document)
-5. **Auto-migration** - Migrate on read, no batch migration needed initially
-
----
-
-## Path Segment Validation
-
-All user-centric paths follow this structure:
-- **Collection** (1)
-- **Document** (2) - User CID
-- **Collection** (3) - Data type (notifications, alerts, watchlists, etc.)
-- **Document** (4) - Specific item ID
-
-Example: `SignedInUsers/29312236/notifications/notification_123` (4 segments ✅)
-
----
-
-## Notes
-
-- **Legacy paths** are still supported during migration period
-- **New paths** are the target format for all new data
-- **Root data** collections remain in task engine format (no migration)
-- **Public collections** (watchlists, rankings) remain unchanged
-- **System collections** (mappings, configs) remain unchanged
-