@arela/uploader 1.0.2 → 1.0.3

package/docs/MULTI_LEVEL_DIRECTORY_SCANNING.md

@@ -0,0 +1,494 @@
+# Multi-Level Directory Scanning Implementation
+
+## Overview
+
+The arela CLI now supports creating multiple database tables based on directory structure depth. This allows for better organization when scanning large file systems with multiple subdirectories.
+
+## Configuration
+
+### Environment Variable
+
+```bash
+# Directory depth level for creating separate tables (default: 0)
+# 0 = single table for entire base path (original behavior)
+# 1 = one table per first-level subdirectory
+# 2 = one table per second-level subdirectory, etc.
+SCAN_DIRECTORY_LEVEL=1
+```
+
+Add this to your `.env` file.
+
+## Example
+
+Given this directory structure:
+
+```
+/data
+├── adios
+│   ├── otro
+│   │   └── otro.txt
+│   └── uno
+│       └── prueba.txt
+└── hola
+    ├── otro
+    │   └── otro.txt
+    └── uno
+        └── prueba.txt
+```
+
+### Level 0 (Default)
+
+```bash
+SCAN_DIRECTORY_LEVEL=0
+```
+
+Creates a single table:
+- `file_stats_palco_local_sample`
+
+### Level 1
+
+```bash
+SCAN_DIRECTORY_LEVEL=1
+```
+
+Creates two tables:
+- `file_stats_palco_local_sample_adios`
+- `file_stats_palco_local_sample_hola`
+
+### Level 2
+
+```bash
+SCAN_DIRECTORY_LEVEL=2
+```
+
+Creates four tables:
+- `file_stats_palco_local_sample_adios_otro`
+- `file_stats_palco_local_sample_adios_uno`
+- `file_stats_palco_local_sample_hola_otro`
+- `file_stats_palco_local_sample_hola_uno`
+
+## How It Works
+
+### Backend Changes
+
+1. **New Endpoint**: `GET /api/uploader/scan/instance-tables`
+   - Fetches all tables for a specific company/server/base combination
+   - Supports filtering by pattern matching
+
+2. **FileStatsTableManagerService**:
+   - Added `getInstanceTables()` method
+   - Returns all tables matching base pattern
+
+### CLI Changes
+
+#### 1. ScanCommand
+
+- Discovers directories at specified level
+- Registers multiple instances (one per directory)
+- Creates separate tables for each directory
+- Streams files into appropriate tables
+
+**Output Example**:
+
+```bash
+🔍 Discovering directories...
+📁 Found 2 directories to scan
+
+📝 Registering scan instances...
+  ✓ adios: file_stats_palco_local_sample_adios (new)
+  ✓ hola: file_stats_palco_local_sample_hola (new)
+
+🚀 Starting file scan...
+
+📂 Scanning: adios
+   📄 |████████████████████| 100% | 50/50 files | 25 files/sec
+
+📂 Scanning: hola
+   📄 |████████████████████| 100% | 45/45 files | 23 files/sec
+
+✅ Scan completed successfully!
+
+📊 Scan Statistics:
+   Directories scanned: 2
+   Files scanned: 95
+   Files inserted: 95
+   Files skipped: 0 (excluded patterns)
+   Total size: 2.5 MB
+   Duration: 4.1s
+   Throughput: 23 files/sec
+
+📋 Tables created:
+   - file_stats_palco_local_sample_adios
+   - file_stats_palco_local_sample_hola
+```
+
+#### 2. IdentifyCommand
+
+- Automatically fetches all tables for the instance
+- Processes each table sequentially
+- Shows per-table and total statistics
+
+**Output Example**:
+
+```bash
+🔍 Starting arela identify command
+📊 Fetching instance tables...
+📋 Found 2 tables to process
+   - file_stats_palco_local_sample_adios
+   - file_stats_palco_local_sample_hola
+
+🔍 Processing table: file_stats_palco_local_sample_adios
+   Total PDFs: 20
+   Detected: 0
+   Pending: 20
+   🚀 Processing 20 pending PDFs...
+   📄 |████████████████████| 100% | 20/20 files | 15 files/sec
+
+🔍 Processing table: file_stats_palco_local_sample_hola
+   Total PDFs: 18
+   Detected: 0
+   Pending: 18
+   🚀 Processing 18 pending PDFs...
+   📄 |████████████████████| 100% | 18/18 files | 14 files/sec
+
+✅ Identification Complete!
+
+📊 Total Results:
+   Tables Processed: 2
+   Files Processed: 38
+   Pedimentos Detected: 35
+   Errors: 3
+   Duration: 2.7s
+   Speed: 14 files/sec
+```
+
+#### 3. PropagateCommand
+
+- Fetches all tables for the instance
+- Processes each table's directories
+- Shows per-table and total statistics
+
+**Output Example**:
+
+```bash
+🔄 Starting arela propagate command
+
+🎯 API Target: default
+📦 Batch Size: 50
+
+📋 Found 2 tables to process:
+   - file_stats_palco_local_sample_adios
+   - file_stats_palco_local_sample_hola
+
+🔄 Processing table: file_stats_palco_local_sample_adios
+
+   Total Files: 50
+   With arela_path: 20
+   Pedimento Sources: 18
+   🚀 Found 30 files ready for propagation.
+
+   📄 Propagating |████████████████████| 100% | 18/18 directories | 42 files/sec | 30 files updated
+
+   📊 Results:
+      Pedimentos Processed: 18
+      Directories Processed: 18
+      Files Updated: 30
+      Errors: 0
+
+🔄 Processing table: file_stats_palco_local_sample_hola
+
+   Total Files: 45
+   With arela_path: 18
+   Pedimento Sources: 17
+   🚀 Found 27 files ready for propagation.
+
+   📄 Propagating |████████████████████| 100% | 17/17 directories | 39 files/sec | 27 files updated
+
+   📊 Results:
+      Pedimentos Processed: 17
+      Directories Processed: 17
+      Files Updated: 27
+      Errors: 0
+
+✅ Propagation Complete!
+
+📊 Total Results:
+   Tables Processed: 2
+   Pedimentos Processed: 35
+   Files Updated: 57
+   Files Failed: 0
+   Directories Processed: 35
+   Duration: 1.5s
+   Speed: 38 files/sec
+```
+
+#### 4. PushCommand
+
+- Fetches all tables for the instance
+- Processes files from each table
+- Shows per-table and total statistics
+
+**Output Example**:
+
+```bash
+🚀 Starting arela push command
+
+🎯 Scan API Target: default
+🎯 Upload API Target: default → http://localhost:3010
+📦 Fetch Batch Size: 100
+📤 Upload Batch Size: 10
+
+📊 Fetching instance tables...
+📋 Found 2 tables to process:
+   - file_stats_palco_local_sample_adios
+   - file_stats_palco_local_sample_hola
+
+🚀 Processing table: file_stats_palco_local_sample_adios
+
+  Table Status:
+     Total with arela_path: 50
+     Pending: 50
+     Uploaded: 0
+
+  🚀 Uploading 50 pending files...
+
+  📤 Uploading |████████████████████| 100% | 50/50 files | 8 files/sec | ✓ 48 ✗ 2
+
+  📊 Table Results:
+     Files Processed: 50
+     Uploaded: 48
+     Errors: 2
+
+🚀 Processing table: file_stats_palco_local_sample_hola
+
+  Table Status:
+     Total with arela_path: 45
+     Pending: 45
+     Uploaded: 0
+
+  🚀 Uploading 45 pending files...
+
+  📤 Uploading |████████████████████| 100% | 45/45 files | 7 files/sec | ✓ 44 ✗ 1
+
+  📊 Table Results:
+     Files Processed: 45
+     Uploaded: 44
+     Errors: 1
+
+✅ Push Complete!
+
+📊 Total Results:
+   Tables Processed: 2
+   Files Processed: 95
+   Uploaded: 92
+   Errors: 3
+   Duration: 12.1s
+   Speed: 8 files/sec
+```
+
+## Use Cases
+
+### 1. Large File Systems
+
+Split large file systems into manageable tables:
+
+```bash
+# Instead of one huge table with millions of files
+SCAN_DIRECTORY_LEVEL=0  # ❌ file_stats_company_nas01_data (5M files)
+
+# Create tables per department/year/project
+SCAN_DIRECTORY_LEVEL=1  # ✅ file_stats_company_nas01_data_accounting
+                        # ✅ file_stats_company_nas01_data_sales
+                        # ✅ file_stats_company_nas01_data_legal
+```
+
+### 2. Year-Based Organization
+
+Organize by year for time-based processing:
+
+```bash
+UPLOAD_BASE_PATH=/archive
+UPLOAD_SOURCES=2023|2024|2025
+SCAN_DIRECTORY_LEVEL=1
+
+# Creates:
+# file_stats_company_nas01_archive_2023
+# file_stats_company_nas01_archive_2024
+# file_stats_company_nas01_archive_2025
+```
+
+### 3. Client-Based Organization
+
+Separate tables per client for multi-tenant setups:
+
+```bash
+UPLOAD_BASE_PATH=/clients
+SCAN_DIRECTORY_LEVEL=1
+
+# Creates:
+# file_stats_agency_nas01_clients_acme
+# file_stats_agency_nas01_clients_globex
+# file_stats_agency_nas01_clients_initech
+```
+
+## Performance Considerations
+
+### Benefits
+
+1. **Smaller Tables**: Easier to query and maintain
+2. **Parallel Processing**: Can process multiple tables concurrently (future enhancement)
+3. **Targeted Operations**: Process only relevant tables (e.g., specific year or client)
+4. **Better Indexes**: Smaller tables = more efficient indexes
+
+### Trade-offs
+
+1. **More API Calls**: Sequential processing of multiple tables
+2. **Increased Complexity**: More tables to manage and monitor
+3. **Slower Initial Scan**: Directory discovery adds overhead
+
+## Best Practices
+
+### 1. Choose Appropriate Level
+
+- **Level 0**: Small datasets (< 100K files)
+- **Level 1**: Medium datasets (100K-1M files) or logical separation
+- **Level 2+**: Very large datasets or deep hierarchies
+
+### 2. Consistent Configuration
+
+Always use the same configuration across commands:
+
+```bash
+# Set once in .env
+ARELA_COMPANY_SLUG=my_company
+ARELA_SERVER_ID=nas01
+UPLOAD_BASE_PATH=/data
+SCAN_DIRECTORY_LEVEL=1
+
+# Then run commands without changing these values
+arela scan
+arela identify
+arela propagate
+arela push
+```
+
+### 3. Monitor Table Sizes
+
+Check table sizes periodically:
+
+```sql
+SELECT 
+  tablename, 
+  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
+FROM pg_tables
+WHERE schemaname = 'cli'
+ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
+```
+
+## Troubleshooting
+
+### Problem: No tables found
+
+```bash
+❌ No tables found for this instance. Run "arela scan" first.
+```
+
+**Solution**: Run `arela scan` with the same configuration.
+
+### Problem: Table name collision
+
+If you change `SCAN_DIRECTORY_LEVEL` after initial scan, you might get conflicts.
+
+**Solution**: Use different `ARELA_BASE_PATH_LABEL` or deactivate old tables:
+
+```bash
+# Deactivate old table
+curl -X PATCH http://localhost:3010/api/uploader/scan/deactivate \
+  -H "x-api-key: $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"tableName":"file_stats_old_table"}'
+
+# Then run scan with new level
+arela scan
+```
+
+### Problem: Duplicate files across tables
+
+Files shouldn't appear in multiple tables if directory level is set correctly.
+
+**Check**: Verify directory structure matches your level setting.
+
+## Migration Guide
+
+### From Level 0 to Level 1
+
+1. **Backup existing data** (optional):
+   ```sql
+   CREATE TABLE cli.file_stats_backup AS 
+   SELECT * FROM cli.file_stats_old_table;
+   ```
+
+2. **Deactivate old table**:
+   ```bash
+   curl -X PATCH http://localhost:3010/api/uploader/scan/deactivate \
+     -H "x-api-key: $TOKEN" \
+     -d '{"tableName":"file_stats_old_table"}'
+   ```
+
+3. **Update configuration**:
+   ```bash
+   SCAN_DIRECTORY_LEVEL=1
+   ```
+
+4. **Run full pipeline**:
+   ```bash
+   arela scan
+   arela identify
+   arela propagate
+   arela push
+   ```
+
+## API Reference
+
+### Get Instance Tables
+
+```
+GET /api/uploader/scan/instance-tables?companySlug=X&serverId=Y&basePathLabel=Z
+```
+
+**Response**:
+```json
+[
+  {
+    "id": "uuid",
+    "companySlug": "my_company",
+    "serverId": "nas01",
+    "basePathLabel": "data_adios",
+    "tableName": "file_stats_my_company_nas01_data_adios",
+    "basePathFull": "/data/adios",
+    "lastScanAt": "2025-01-18T10:30:00Z",
+    "totalFiles": 50,
+    "totalSizeBytes": 1048576,
+    "status": "ACTIVE"
+  },
+  {
+    "id": "uuid",
+    "companySlug": "my_company",
+    "serverId": "nas01",
+    "basePathLabel": "data_hola",
+    "tableName": "file_stats_my_company_nas01_data_hola",
+    "basePathFull": "/data/hola",
+    "lastScanAt": "2025-01-18T10:31:00Z",
+    "totalFiles": 45,
+    "totalSizeBytes": 987654,
+    "status": "ACTIVE"
+  }
+]
+```
+
+## Future Enhancements
+
+1. **Parallel Processing**: Process multiple tables concurrently
+2. **Table Merging**: Combine tables when directory level changes
+3. **Selective Processing**: Process only specific tables via CLI flag
+4. **Table Statistics Dashboard**: View all tables and their stats in one place