@arela/uploader 1.0.2 → 1.0.4

package/docs/MIGRATION_UPLOADER_TO_FILE_STATS.md

@@ -0,0 +1,1020 @@
+# Migration Plan: Uploader Table to file_stats_* Tables
+
+## Overview
+
+This document outlines the strategy to migrate existing data from the legacy `uploader` table to the new `file_stats_<company>_<server>_<path>` table structure without re-uploading files or losing existing metadata.
+
+## Goals
+
+1. ✅ Preserve all uploaded file references and metadata
+2. ✅ Create appropriate `file_stats_*` tables based on existing data
+3. ✅ Maintain upload history and detection results
+4. ✅ Enable continued use of new CLI commands (identify, propagate, push)
+5. ✅ Zero downtime migration where possible
+6. ✅ Rollback capability if issues arise
+
+## Prerequisites
+
+### 1. Understand Current Schema
+
+**Uploader Table (Legacy)**:
+```sql
+-- Assumed schema based on implementation references
+public.uploader (
+  id UUID PRIMARY KEY,
+  original_path TEXT UNIQUE,
+  relative_path TEXT,
+  file_name VARCHAR,
+  file_extension VARCHAR,
+  directory_path TEXT,
+  size_bytes BIGINT,
+  modified_at TIMESTAMP,
+  
+  -- Detection fields
+  detected_type VARCHAR,
+  detected_pedimento VARCHAR,
+  detected_pedimento_year INTEGER,
+  rfc VARCHAR,
+  arela_path TEXT,
+  detection_attempted_at TIMESTAMP,
+  detection_error TEXT,
+  
+  -- Upload fields
+  uploaded_at TIMESTAMP,
+  storage_id UUID,
+  upload_path TEXT,
+  
+  -- Status fields
+  status VARCHAR,
+  processing_status VARCHAR,
+  
+  -- Timestamps
+  created_at TIMESTAMP,
+  updated_at TIMESTAMP
+)
+```
+
+### 2. New Schema (file_stats_*)
+
+**File Stats Table (New)**:
+```sql
+cli.file_stats_<company>_<server>_<path> (
+  id UUID PRIMARY KEY,
+  file_name VARCHAR(500),
+  file_extension VARCHAR(50),
+  directory_path TEXT,
+  relative_path TEXT,
+  absolute_path TEXT UNIQUE,
+  size_bytes BIGINT,
+  modified_at TIMESTAMP,
+  scan_timestamp TIMESTAMP,
+  
+  -- Detection fields
+  detected_type VARCHAR(100),
+  detected_pedimento VARCHAR(50),
+  detected_pedimento_year INTEGER,
+  rfc VARCHAR(20),
+  arela_path TEXT,
+  detection_attempted_at TIMESTAMP,
+  detection_error TEXT,
+  detection_attempts INTEGER DEFAULT 0,
+  max_detection_attempts INTEGER DEFAULT 3,
+  is_not_pedimento BOOLEAN DEFAULT FALSE,
+  
+  -- Propagation fields
+  propagation_attempted_at TIMESTAMP,
+  propagation_attempts INTEGER DEFAULT 0,
+  max_propagation_attempts INTEGER DEFAULT 3,
+  propagation_error TEXT,
+  propagated_from_id UUID,
+  needs_propagation BOOLEAN DEFAULT FALSE,
+  
+  -- Upload fields
+  upload_attempted_at TIMESTAMP,
+  upload_attempts INTEGER DEFAULT 0,
+  max_upload_attempts INTEGER DEFAULT 3,
+  upload_error TEXT,
+  uploaded_at TIMESTAMP,
+  uploaded_to_storage_id UUID,
+  upload_path TEXT,
+  
+  -- Timestamps
+  created_at TIMESTAMP DEFAULT NOW()
+)
+```
+
+## Migration Strategy
+
+### Phase 1: Analysis and Planning
+
+#### 1.1 Analyze Existing Data Distribution
+
+Create analysis queries to understand data structure:
+
+```sql
+-- Identify distinct company/server/path combinations
+WITH path_analysis AS (
+  SELECT 
+    -- Extract company identifier (if exists in path)
+    -- Extract server identifier (if exists in path)
+    -- Extract base path pattern
+    SPLIT_PART(original_path, '/', 1) as path_segment_1,
+    SPLIT_PART(original_path, '/', 2) as path_segment_2,
+    SPLIT_PART(original_path, '/', 3) as path_segment_3,
+    COUNT(*) as file_count,
+    SUM(size_bytes) as total_size_bytes,
+    COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded_count,
+    COUNT(*) FILTER (WHERE detected_type IS NOT NULL) as detected_count
+  FROM public.uploader
+  GROUP BY 1, 2, 3
+  ORDER BY file_count DESC
+)
+SELECT * FROM path_analysis
+LIMIT 100;
+
+-- Statistics by status
+SELECT 
+  status,
+  processing_status,
+  COUNT(*) as count,
+  COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+  COUNT(*) FILTER (WHERE detected_type = 'pedimento_simplificado') as pedimentos,
+  pg_size_pretty(SUM(size_bytes)) as total_size
+FROM public.uploader
+GROUP BY status, processing_status
+ORDER BY count DESC;
+
+-- Uploaded files by RFC and year
+SELECT 
+  rfc,
+  detected_pedimento_year,
+  COUNT(*) as files,
+  COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+  pg_size_pretty(SUM(size_bytes)) as total_size
+FROM public.uploader
+WHERE rfc IS NOT NULL
+GROUP BY rfc, detected_pedimento_year
+ORDER BY files DESC
+LIMIT 50;
+```
+
+**Output**: Save results to `migration_analysis.csv` for planning.
+
+#### 1.2 Define Instance Mapping
+
+Create a mapping file `instance_mapping.json`:
+
+```json
+{
+  "instances": [
+    {
+      "company_slug": "agencia_palco",
+      "server_id": "nas01",
+      "base_path": "/mnt/storage/documentos",
+      "base_path_label": "documentos",
+      "path_pattern": "/mnt/storage/documentos/%",
+      "table_name": "file_stats_agencia_palco_nas01_documentos",
+      "estimated_records": 1500000
+    },
+    {
+      "company_slug": "cliente_xyz",
+      "server_id": "server01",
+      "base_path": "/data/archives",
+      "base_path_label": "archives",
+      "path_pattern": "/data/archives/%",
+      "table_name": "file_stats_cliente_xyz_server01_archives",
+      "estimated_records": 850000
+    }
+  ]
+}
+```
+
+**Action**: Review with stakeholders to confirm instance definitions.
+
+#### 1.3 Estimate Migration Time
+
+```sql
+-- Calculate migration metrics
+SELECT 
+  COUNT(*) as total_records,
+  pg_size_pretty(pg_total_relation_size('public.uploader')) as table_size,
+  COUNT(*) / NULLIF((EXTRACT(EPOCH FROM (MAX(created_at) - MIN(created_at))) / 86400), 0) as records_per_day
+FROM public.uploader;
+```
+
+**Estimates**:
+- **Records per batch**: 10,000
+- **Expected throughput**: ~5,000 records/second
+- **For 5M records**: ~17 minutes
+- **Buffer time**: 2x = ~35 minutes per instance
+
+### Phase 2: Backend Implementation
+
+#### 2.1 Create Migration Service
+
+**File**: `arela-api/src/uploader/services/uploader-migration.service.ts`
+
+```typescript
+import { Injectable, Logger } from '@nestjs/common';
+import { InjectRepository } from '@nestjs/typeorm';
+import { Repository } from 'typeorm';
+import { FileStatsTableManagerService } from './file-stats-table-manager.service';
+import { CliRegistry } from '../entities/cli-registry.entity';
+
+interface MigrationInstance {
+  companySlug: string;
+  serverId: string;
+  basePath: string;
+  basePathLabel: string;
+  pathPattern: string;
+  tableName: string;
+}
+
+interface MigrationProgress {
+  tableName: string;
+  totalRecords: number;
+  migratedRecords: number;
+  errors: number;
+  startTime: Date;
+  estimatedCompletion?: Date;
+}
+
+@Injectable()
+export class UploaderMigrationService {
+  private readonly logger = new Logger(UploaderMigrationService.name);
+
+  constructor(
+    @InjectRepository(CliRegistry)
+    private cliRegistryRepo: Repository<CliRegistry>,
+    private fileStatsManager: FileStatsTableManagerService,
+  ) {}
+
+  /**
+   * Migrate data from uploader table to file_stats_* tables
+   */
+  async migrateInstance(instance: MigrationInstance): Promise<MigrationProgress> {
+    const progress: MigrationProgress = {
+      tableName: instance.tableName,
+      totalRecords: 0,
+      migratedRecords: 0,
+      errors: 0,
+      startTime: new Date(),
+    };
+
+    try {
+      // 1. Count records to migrate
+      const countResult = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count
+        FROM public.uploader
+        WHERE original_path LIKE $1
+      `, [instance.pathPattern]);
+      
+      progress.totalRecords = parseInt(countResult[0].count);
+      this.logger.log(`Starting migration for ${instance.tableName}: ${progress.totalRecords} records`);
+
+      // 2. Register CLI instance (creates table)
+      await this.fileStatsManager.registerAndCreateTable({
+        companySlug: instance.companySlug,
+        serverId: instance.serverId,
+        basePath: instance.basePath,
+        basePathLabel: instance.basePathLabel,
+        sources: [],
+      });
+
+      this.logger.log(`Created table: ${instance.tableName}`);
+
+      // 3. Migrate data in batches
+      const batchSize = 10000;
+      let offset = 0;
+
+      while (offset < progress.totalRecords) {
+        const batch = await this.migrateBatch(
+          instance.pathPattern,
+          instance.tableName,
+          instance.basePath,
+          offset,
+          batchSize,
+        );
+
+        progress.migratedRecords += batch.inserted;
+        progress.errors += batch.errors;
+
+        // Log progress every 10 batches
+        if (offset % (batchSize * 10) === 0) {
+          const pct = ((progress.migratedRecords / progress.totalRecords) * 100).toFixed(1);
+          this.logger.log(`Migration progress: ${pct}% (${progress.migratedRecords}/${progress.totalRecords})`);
+        }
+
+        offset += batchSize;
+      }
+
+      // 4. Update registry with migration info
+      await this.cliRegistryRepo.update(
+        { tableName: instance.tableName },
+        {
+          lastScanAt: new Date(),
+          totalFiles: progress.migratedRecords,
+        },
+      );
+
+      progress.estimatedCompletion = new Date();
+      this.logger.log(`Migration completed for ${instance.tableName}: ${progress.migratedRecords} records`);
+
+      return progress;
+    } catch (error) {
+      this.logger.error(`Migration failed for ${instance.tableName}:`, error);
+      throw error;
+    }
+  }
+
+  /**
+   * Migrate a batch of records
+   */
+  private async migrateBatch(
+    pathPattern: string,
+    tableName: string,
+    basePath: string,
+    offset: number,
+    limit: number,
+  ): Promise<{ inserted: number; errors: number }> {
+    try {
+      // Fetch batch from uploader table
+      const records = await this.fileStatsManager.query(`
+        SELECT 
+          id,
+          original_path,
+          relative_path,
+          file_name,
+          file_extension,
+          directory_path,
+          size_bytes,
+          modified_at,
+          
+          detected_type,
+          detected_pedimento,
+          detected_pedimento_year,
+          rfc,
+          arela_path,
+          detection_attempted_at,
+          detection_error,
+          
+          uploaded_at,
+          storage_id as uploaded_to_storage_id,
+          upload_path,
+          
+          created_at
+        FROM public.uploader
+        WHERE original_path LIKE $1
+        ORDER BY created_at
+        OFFSET $2 LIMIT $3
+      `, [pathPattern, offset, limit]);
+
+      if (records.length === 0) {
+        return { inserted: 0, errors: 0 };
+      }
+
+      // Transform records for new schema
+      const transformedRecords = records.map(record => ({
+        id: record.id,
+        file_name: record.file_name,
+        file_extension: record.file_extension,
+        directory_path: record.directory_path,
+        relative_path: record.relative_path,
+        absolute_path: record.original_path,
+        size_bytes: record.size_bytes,
+        modified_at: record.modified_at,
+        scan_timestamp: record.created_at,
+        
+        // Detection fields
+        detected_type: record.detected_type,
+        detected_pedimento: record.detected_pedimento,
+        detected_pedimento_year: record.detected_pedimento_year,
+        rfc: record.rfc,
+        arela_path: record.arela_path,
+        detection_attempted_at: record.detection_attempted_at,
+        detection_error: record.detection_error,
+        detection_attempts: record.detection_attempted_at ? 1 : 0,
+        max_detection_attempts: 3,
+        is_not_pedimento: record.detection_error?.includes('NOT_PEDIMENTO') || false,
+        
+        // Propagation fields (initialize)
+        propagation_attempts: 0,
+        max_propagation_attempts: 3,
+        needs_propagation: false,
+        
+        // Upload fields
+        uploaded_at: record.uploaded_at,
+        uploaded_to_storage_id: record.uploaded_to_storage_id,
+        upload_path: record.upload_path,
+        upload_attempts: record.uploaded_at ? 1 : 0,
+        max_upload_attempts: 3,
+        
+        created_at: record.created_at,
+      }));
+
+      // Insert into new table using ON CONFLICT
+      const insertQuery = `
+        INSERT INTO cli.${tableName} (
+          id, file_name, file_extension, directory_path, relative_path, 
+          absolute_path, size_bytes, modified_at, scan_timestamp,
+          detected_type, detected_pedimento, detected_pedimento_year, rfc, arela_path,
+          detection_attempted_at, detection_error, detection_attempts, 
+          max_detection_attempts, is_not_pedimento,
+          propagation_attempts, max_propagation_attempts, needs_propagation,
+          uploaded_at, uploaded_to_storage_id, upload_path, upload_attempts, 
+          max_upload_attempts, created_at
+        )
+        SELECT * FROM UNNEST(
+          $1::uuid[], $2::varchar[], $3::varchar[], $4::text[], $5::text[],
+          $6::text[], $7::bigint[], $8::timestamp[], $9::timestamp[],
+          $10::varchar[], $11::varchar[], $12::integer[], $13::varchar[], $14::text[],
+          $15::timestamp[], $16::text[], $17::integer[],
+          $18::integer[], $19::boolean[],
+          $20::integer[], $21::integer[], $22::boolean[],
+          $23::timestamp[], $24::uuid[], $25::text[], $26::integer[],
+          $27::integer[], $28::timestamp[]
+        )
+        ON CONFLICT (absolute_path) DO UPDATE SET
+          detected_type = EXCLUDED.detected_type,
+          detected_pedimento = EXCLUDED.detected_pedimento,
+          detected_pedimento_year = EXCLUDED.detected_pedimento_year,
+          rfc = EXCLUDED.rfc,
+          arela_path = EXCLUDED.arela_path,
+          uploaded_at = EXCLUDED.uploaded_at,
+          uploaded_to_storage_id = EXCLUDED.uploaded_to_storage_id,
+          upload_path = EXCLUDED.upload_path
+      `;
+
+      const params = [
+        transformedRecords.map(r => r.id),
+        transformedRecords.map(r => r.file_name),
+        transformedRecords.map(r => r.file_extension),
+        transformedRecords.map(r => r.directory_path),
+        transformedRecords.map(r => r.relative_path),
+        transformedRecords.map(r => r.absolute_path),
+        transformedRecords.map(r => r.size_bytes),
+        transformedRecords.map(r => r.modified_at),
+        transformedRecords.map(r => r.scan_timestamp),
+        transformedRecords.map(r => r.detected_type),
+        transformedRecords.map(r => r.detected_pedimento),
+        transformedRecords.map(r => r.detected_pedimento_year),
+        transformedRecords.map(r => r.rfc),
+        transformedRecords.map(r => r.arela_path),
+        transformedRecords.map(r => r.detection_attempted_at),
+        transformedRecords.map(r => r.detection_error),
+        transformedRecords.map(r => r.detection_attempts),
+        transformedRecords.map(r => r.max_detection_attempts),
+        transformedRecords.map(r => r.is_not_pedimento),
+        transformedRecords.map(r => r.propagation_attempts),
+        transformedRecords.map(r => r.max_propagation_attempts),
+        transformedRecords.map(r => r.needs_propagation),
+        transformedRecords.map(r => r.uploaded_at),
+        transformedRecords.map(r => r.uploaded_to_storage_id),
+        transformedRecords.map(r => r.upload_path),
+        transformedRecords.map(r => r.upload_attempts),
+        transformedRecords.map(r => r.max_upload_attempts),
+        transformedRecords.map(r => r.created_at),
+      ];
+
+      await this.fileStatsManager.query(insertQuery, params);
+
+      return { inserted: records.length, errors: 0 };
+    } catch (error) {
+      this.logger.error(`Batch migration error:`, error);
+      return { inserted: 0, errors: limit };
+    }
+  }
+
+  /**
+   * Validate migration results
+   */
+  async validateMigration(instance: MigrationInstance): Promise<{
+    valid: boolean;
+    issues: string[];
+  }> {
+    const issues: string[] = [];
+
+    try {
+      // 1. Count comparison
+      const sourceCount = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM public.uploader
+        WHERE original_path LIKE $1
+      `, [instance.pathPattern]);
+
+      const targetCount = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM cli.${instance.tableName}
+      `);
+
+      const sourceCnt = parseInt(sourceCount[0].count);
+      const targetCnt = parseInt(targetCount[0].count);
+
+      if (sourceCnt !== targetCnt) {
+        issues.push(`Record count mismatch: source=${sourceCnt}, target=${targetCnt}`);
+      }
+
+      // 2. Uploaded files comparison
+      const sourceUploaded = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM public.uploader
+        WHERE original_path LIKE $1 AND uploaded_at IS NOT NULL
+      `, [instance.pathPattern]);
+
+      const targetUploaded = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM cli.${instance.tableName}
+        WHERE uploaded_at IS NOT NULL
+      `);
+
+      const sourceUploadedCnt = parseInt(sourceUploaded[0].count);
+      const targetUploadedCnt = parseInt(targetUploaded[0].count);
+
+      if (sourceUploadedCnt !== targetUploadedCnt) {
+        issues.push(`Uploaded count mismatch: source=${sourceUploadedCnt}, target=${targetUploadedCnt}`);
+      }
+
+      // 3. Detected pedimentos comparison
+      const sourceDetected = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM public.uploader
+        WHERE original_path LIKE $1 AND detected_type = 'pedimento_simplificado'
+      `, [instance.pathPattern]);
+
+      const targetDetected = await this.fileStatsManager.query(`
+        SELECT COUNT(*) as count FROM cli.${instance.tableName}
+        WHERE detected_type = 'pedimento_simplificado'
+      `);
+
+      const sourceDetectedCnt = parseInt(sourceDetected[0].count);
+      const targetDetectedCnt = parseInt(targetDetected[0].count);
+
+      if (sourceDetectedCnt !== targetDetectedCnt) {
+        issues.push(`Detected count mismatch: source=${sourceDetectedCnt}, target=${targetDetectedCnt}`);
+      }
+
+      // 4. Sample data comparison
+      const sampleCheck = await this.fileStatsManager.query(`
+        SELECT 
+          u.id,
+          u.original_path,
+          u.detected_type as src_type,
+          u.uploaded_at as src_uploaded,
+          f.detected_type as tgt_type,
+          f.uploaded_at as tgt_uploaded
+        FROM public.uploader u
+        LEFT JOIN cli.${instance.tableName} f ON f.id = u.id
+        WHERE u.original_path LIKE $1
+          AND (
+            u.detected_type IS DISTINCT FROM f.detected_type
+            OR u.uploaded_at IS DISTINCT FROM f.uploaded_at
+          )
+        LIMIT 10
+      `, [instance.pathPattern]);
+
+      if (sampleCheck.length > 0) {
+        issues.push(`Found ${sampleCheck.length} records with data mismatches`);
+      }
+
+      return {
+        valid: issues.length === 0,
+        issues,
+      };
+    } catch (error) {
+      issues.push(`Validation error: ${error.message}`);
+      return { valid: false, issues };
+    }
+  }
+
+  /**
+   * Rollback migration for an instance
+   */
+  async rollbackMigration(tableName: string): Promise<void> {
+    this.logger.warn(`Rolling back migration for ${tableName}`);
+
+    try {
+      // 1. Drop the table
+      await this.fileStatsManager.query(`
+        DROP TABLE IF EXISTS cli.${tableName}
+      `);
+
+      // 2. Remove registry entry
+      await this.cliRegistryRepo.delete({ tableName });
+
+      this.logger.log(`Rollback completed for ${tableName}`);
+    } catch (error) {
+      this.logger.error(`Rollback failed:`, error);
+      throw error;
+    }
+  }
+}
+```
+
+#### 2.2 Create Migration Controller Endpoints
+
+**File**: `arela-api/src/uploader/controllers/uploader-migration.controller.ts`
+
+```typescript
+import { Controller, Post, Get, Delete, Body, Query, UseGuards } from '@nestjs/common';
+import { ApiKeyGuard } from '../../common/guards/api-key.guard';
+import { UploaderMigrationService } from '../services/uploader-migration.service';
+
+@Controller('api/uploader/migration')
+@UseGuards(ApiKeyGuard)
+export class UploaderMigrationController {
+  constructor(private migrationService: UploaderMigrationService) {}
+
+  @Post('migrate-instance')
+  async migrateInstance(@Body() instance: any) {
+    return await this.migrationService.migrateInstance(instance);
+  }
+
+  @Post('validate')
+  async validateMigration(@Body() instance: any) {
+    return await this.migrationService.validateMigration(instance);
+  }
+
+  @Delete('rollback')
+  async rollbackMigration(@Query('tableName') tableName: string) {
+    await this.migrationService.rollbackMigration(tableName);
+    return { message: 'Rollback completed' };
+  }
+}
+```
+
+### Phase 3: CLI Implementation
+
+#### 3.1 Create Migration Command
+
+**File**: `arela-uploader/src/commands/MigrateCommand.js`
+
+```javascript
+import fs from 'fs';
+import path from 'path';
+import chalk from 'chalk';
+import cliProgress from 'cli-progress';
+import { appConfig } from '../config/config.js';
+import { ScanApiService } from '../services/ScanApiService.js';
+
+export class MigrateCommand {
+  constructor() {
+    this.apiService = new ScanApiService();
+  }
+
+  async execute(options = {}) {
+    console.log(chalk.blue('\n🔄 Starting migration from uploader table to file_stats_* tables\n'));
+
+    try {
+      // 1. Load instance mapping
+      const mappingPath = options.mappingFile || './instance_mapping.json';
+      const mapping = JSON.parse(fs.readFileSync(mappingPath, 'utf8'));
+
+      console.log(chalk.cyan(`📋 Found ${mapping.instances.length} instances to migrate\n`));
+
+      // 2. Confirm migration
+      if (!options.confirm) {
+        console.log(chalk.yellow('⚠️  This will create new tables and migrate data.'));
+        console.log(chalk.yellow('   Run with --confirm to proceed.\n'));
+        
+        mapping.instances.forEach((inst, idx) => {
+          console.log(`   ${idx + 1}. ${inst.table_name} (~${inst.estimated_records.toLocaleString()} records)`);
+        });
+        
+        return;
+      }
+
+      // 3. Migrate each instance
+      const results = [];
+      
+      for (const instance of mapping.instances) {
+        console.log(chalk.cyan(`\n📦 Migrating: ${instance.company_slug}/${instance.server_id}/${instance.base_path_label}`));
+        console.log(chalk.gray(`   Table: ${instance.table_name}`));
+        console.log(chalk.gray(`   Pattern: ${instance.path_pattern}`));
+
+        const progressBar = new cliProgress.SingleBar({
+          format: `   📄 {bar} {percentage}% | {value}/{total} records | {duration_formatted}`,
+          barCompleteChar: '\u2588',
+          barIncompleteChar: '\u2591',
+        });
+
+        try {
+          // Start migration via API
+          const startTime = Date.now();
+          const response = await this.apiService.migrateInstance(instance);
+
+          // Show progress (simplified - in real implementation, poll progress endpoint)
+          progressBar.start(response.totalRecords, 0);
+          progressBar.update(response.migratedRecords);
+          progressBar.stop();
+
+          const duration = ((Date.now() - startTime) / 1000).toFixed(1);
+          const throughput = Math.round(response.migratedRecords / duration);
+
+          console.log(chalk.green(`   ✓ Migrated ${response.migratedRecords} records in ${duration}s (${throughput} records/sec)`));
+
+          if (response.errors > 0) {
+            console.log(chalk.yellow(`   ⚠️  ${response.errors} errors encountered`));
+          }
+
+          results.push({
+            instance: instance.table_name,
+            success: true,
+            migrated: response.migratedRecords,
+            errors: response.errors,
+            duration,
+          });
+
+          // 4. Validate migration
+          if (options.validate) {
+            console.log(chalk.cyan('   🔍 Validating migration...'));
+            const validation = await this.apiService.validateMigration(instance);
+
+            if (validation.valid) {
+              console.log(chalk.green('   ✓ Validation passed'));
+            } else {
+              console.log(chalk.red('   ✗ Validation failed:'));
+              validation.issues.forEach(issue => {
+                console.log(chalk.red(`     - ${issue}`));
+              });
+            }
+          }
+        } catch (error) {
+          progressBar.stop();
+          console.log(chalk.red(`   ✗ Migration failed: ${error.message}`));
+          
+          results.push({
+            instance: instance.table_name,
+            success: false,
+            error: error.message,
+          });
+
+          if (!options.continueOnError) {
+            throw error;
+          }
+        }
+      }
+
+      // 5. Summary
+      console.log(chalk.blue('\n📊 Migration Summary\n'));
+      
+      const successful = results.filter(r => r.success).length;
+      const failed = results.filter(r => !r.success).length;
+      const totalMigrated = results.reduce((sum, r) => sum + (r.migrated || 0), 0);
+
+      console.log(`   Total instances: ${results.length}`);
+      console.log(chalk.green(`   ✓ Successful: ${successful}`));
+      if (failed > 0) {
+        console.log(chalk.red(`   ✗ Failed: ${failed}`));
+      }
+      console.log(`   Total records migrated: ${totalMigrated.toLocaleString()}`);
+
+      // Save results
+      const resultPath = `migration_results_${Date.now()}.json`;
+      fs.writeFileSync(resultPath, JSON.stringify(results, null, 2));
+      console.log(chalk.gray(`\n   Results saved to: ${resultPath}`));
+
+      console.log(chalk.green('\n✅ Migration complete!\n'));
+    } catch (error) {
+      console.error(chalk.red(`\n❌ Migration failed: ${error.message}\n`));
+      throw error;
+    }
+  }
+}
+```
+
+#### 3.2 Register Migration Command
+
+**File**: `arela-uploader/src/index.js`
+
+```javascript
+// Add to setupCommands() method
+this.program
+  .command('migrate')
+  .description('Migrate data from uploader table to file_stats_* tables')
+  .option('--mapping-file <path>', 'Path to instance mapping JSON file', './instance_mapping.json')
+  .option('--confirm', 'Confirm migration execution')
+  .option('--validate', 'Validate migration results')
+  .option('--continue-on-error', 'Continue migration even if an instance fails')
+  .action(async (options) => {
+    const command = new MigrateCommand();
+    await command.execute(options);
+  });
+```
+
+### Phase 4: Execution Plan
+
+#### 4.1 Pre-Migration Checklist
+
+- [ ] Backup `uploader` table
+  ```sql
+  CREATE TABLE public.uploader_backup AS SELECT * FROM public.uploader;
+  ```
+- [ ] Document current record counts and statistics
+- [ ] Create `instance_mapping.json` based on analysis
+- [ ] Test migration on staging environment
+- [ ] Schedule maintenance window (if needed)
+- [ ] Notify users of migration
+
+#### 4.2 Migration Steps
+
+**Step 1: Dry Run** (validate without execution)
+```bash
+cd arela-uploader
+node src/index.js migrate --mapping-file instance_mapping.json
+```
+
+**Step 2: Execute Migration**
+```bash
+node src/index.js migrate --mapping-file instance_mapping.json --confirm --validate
+```
+
+**Step 3: Verify Results**
+```sql
+-- Check record counts
+SELECT 
+  tablename, 
+  schemaname,
+  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
+FROM pg_tables 
+WHERE schemaname = 'cli' AND tablename LIKE 'file_stats_%';
+
+-- Check uploaded files
+SELECT 
+  tablename,
+  (SELECT COUNT(*) FROM cli.${tablename} WHERE uploaded_at IS NOT NULL) as uploaded_count
+FROM pg_tables 
+WHERE schemaname = 'cli' AND tablename LIKE 'file_stats_%';
+```
+
+**Step 4: Test New Commands**
+```bash
+# Test identify command
+arela identify --show-stats
+
+# Test propagate command  
+arela propagate --show-stats
+
+# Test push command (dry run)
+arela push --show-stats
+```
+
+#### 4.3 Post-Migration Tasks
+
+1. **Update Documentation**: Update all docs to reference new table structure
+2. **Archive Legacy Data**: Keep `uploader` table for reference, mark as deprecated
+3. **Monitor Performance**: Track query performance on new tables
+4. **Optimize Indexes**: Add additional indexes based on query patterns
+5. **Update Grafana Dashboards**: Point metrics to new tables
+
+### Phase 5: Rollback Procedure
+
+If migration fails or issues are discovered:
+
+**Step 1: Stop New Operations**
+```bash
+# Prevent new scans/uploads
+# Notify users to stop CLI operations
+```
+
+**Step 2: Rollback Specific Instance**
+```bash
+curl -X DELETE \
+  -H "x-api-key: $TOKEN" \
+  "http://localhost:3010/api/uploader/migration/rollback?tableName=file_stats_company_server_path"
+```
+
+**Step 3: Rollback All Instances**
+```sql
+-- Drop all file_stats tables
+DO $$
+DECLARE
+    r RECORD;
+BEGIN
+    FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = 'cli' AND tablename LIKE 'file_stats_%')
+    LOOP
+        EXECUTE 'DROP TABLE IF EXISTS cli.' || r.tablename;
+    END LOOP;
+END $$;
+
+-- Clear registry
+DELETE FROM cli.cli_registry WHERE table_name LIKE 'file_stats_%';
+```
+
+**Step 4: Restore Legacy Operations**
+```bash
+# Re-enable legacy upload command
+# Verify uploader table is intact
+```
+
+### Phase 6: Data Cleanup
+
+After successful migration and validation period (e.g., 30 days):
+
+```sql
+-- Optional: Archive uploader table to separate schema
+CREATE SCHEMA IF NOT EXISTS archive;
+ALTER TABLE public.uploader SET SCHEMA archive;
+
+-- Optional: Drop uploader table (after extensive validation)
+-- DROP TABLE public.uploader;
+```
+
+## Migration Checklist
+
+### Pre-Migration
+- [ ] Analyze existing data distribution
+- [ ] Create instance mapping file
+- [ ] Backup uploader table
+- [ ] Deploy migration service to backend
+- [ ] Deploy migration command to CLI
+- [ ] Test on staging environment
+- [ ] Document rollback procedures
+
+### During Migration
+- [ ] Execute dry run
+- [ ] Review dry run results
+- [ ] Execute actual migration with validation
+- [ ] Monitor progress and logs
+- [ ] Validate record counts match
+- [ ] Test sample queries on new tables
+
+### Post-Migration
+- [ ] Verify all uploaded files preserved
+- [ ] Test new CLI commands (identify, propagate, push)
+- [ ] Update documentation
+- [ ] Monitor performance for 48 hours
+- [ ] Archive or drop legacy uploader table (after validation period)
+
+## Monitoring Queries
+
+```sql
+-- Progress monitoring during migration
+SELECT 
+  table_name,
+  last_scan_at,
+  total_files,
+  status
+FROM cli.cli_registry
+WHERE table_name LIKE 'file_stats_%'
+ORDER BY last_scan_at DESC;
+
+-- Compare source vs target counts
+SELECT 
+  'uploader' as source,
+  COUNT(*) as records,
+  COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+  COUNT(*) FILTER (WHERE detected_type = 'pedimento_simplificado') as pedimentos
+FROM public.uploader
+UNION ALL
+SELECT 
+  'file_stats_*' as source,
+  SUM(c.records),
+  SUM(c.uploaded),
+  SUM(c.pedimentos)
+FROM (
+  SELECT 
+    COUNT(*) as records,
+    COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+    COUNT(*) FILTER (WHERE detected_type = 'pedimento_simplificado') as pedimentos
+  FROM cli.file_stats_agencia_palco_nas01_documentos
+  -- Add UNION ALL for each file_stats table
+) c;
+
+-- Check for orphaned records
+SELECT u.id, u.original_path
+FROM public.uploader u
+LEFT JOIN cli.file_stats_* f ON f.id = u.id
+WHERE f.id IS NULL
+LIMIT 100;
+```
+
+## Risk Assessment
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|-----------|--------|------------|
+| Data loss during migration | Low | High | Full backup before migration, validate counts |
+| Long migration time | Medium | Medium | Batch processing, run during off-peak hours |
+| Table name conflicts | Low | Medium | Validate instance mapping before execution |
+| Query performance degradation | Low | Medium | Create optimized indexes, monitor queries |
+| Incomplete migration | Low | High | Validation step after each instance |
+| Rollback needed | Low | High | Documented rollback procedures, keep uploader table |
+
+## Success Criteria
+
+- ✅ All records from `uploader` table successfully migrated to appropriate `file_stats_*` tables
+- ✅ Record counts match between source and target (±0.1% tolerance for concurrent operations)
+- ✅ All uploaded file references preserved (uploaded_at, storage_id, upload_path)
+- ✅ All detected pedimento metadata preserved (detected_type, rfc, arela_path)
+- ✅ New CLI commands work correctly (identify, propagate, push)
+- ✅ Query performance meets or exceeds legacy performance
+- ✅ Zero data loss
+- ✅ Rollback procedures tested and documented
+
+## Timeline Estimate
+
+**For 5M records across 3 instances:**
+
+- Phase 1 (Analysis): 2-4 hours
+- Phase 2 (Backend Implementation): 1-2 days
+- Phase 3 (CLI Implementation): 1 day
+- Phase 4 (Testing on Staging): 1 day
+- Phase 5 (Production Migration): 2-3 hours
+- Phase 6 (Validation Period): 7-30 days
+- Phase 7 (Cleanup): 1 hour
+
+**Total: 4-5 days development + validation period**