@arela/uploader 1.0.2 → 1.0.4

package/docs/ARELA_PROPAGATE_QUICKREF.md

@@ -0,0 +1,272 @@
+# Arela Propagate Quick Reference
+
+## Command
+
+```bash
+arela propagate [options]
+```
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--api <target>` | `default` | API target: default\|agencia\|cliente |
+| `-b, --batch-size <size>` | `50` | Pedimentos per batch |
+| `--show-stats` | `false` | Show detailed statistics |
+
+## Prerequisites
+
+1. **Run `arela scan` first** - Propagate requires scanned files
+2. **Run `arela identify` first** - Need detected pedimentos
+3. **Same configuration** - Use same env vars as scan/identify commands
+
+## Required Environment Variables
+
+```bash
+ARELA_COMPANY_SLUG=your_company
+ARELA_SERVER_ID=server01
+UPLOAD_BASE_PATH=/path/to/files
+UPLOAD_SOURCES=2023|2024|2025
+```
+
+## Examples
+
+```bash
+# Basic propagation
+arela propagate
+
+# Use specific API
+arela propagate --api agencia
+
+# Faster for large datasets
+arela propagate --batch-size 100
+
+# With detailed stats
+arela propagate --show-stats
+```
+
+## What It Does
+
+1. Fetches pedimentos with `arela_path` from `file_stats_*` table
+2. For each pedimento, finds files in the same directory
+3. Propagates `arela_path`, `rfc`, and `year` to related files
+4. Tracks attempts and errors for monitoring
+5. Updates results in database in batches
+
+## Output Example
+
+```
+🔄 Starting arela propagate command
+
+📊 Table: file_stats_acme_corp_nas01_data
+🎯 API Target: default
+📦 Batch Size: 50
+
+📈 Propagation Status:
+   Total Files: 5000
+   With arela_path: 850
+   Pedimento Sources: 850
+   Needs Propagation: 2000
+   Pending: 2000
+   Errors: 0
+
+🏷️  Marking files needing propagation...
+✓ Marked 2000 files
+
+🚀 Processing propagation...
+
+📄 Propagating |████████████████████| 100% | 850/850 directories | 145 files/sec | 2000 files updated
+
+📊 Results:
+   Pedimentos Processed: 850
+   Directories Processed: 850
+   Files Updated: 2000
+   Errors: 0
+   Duration: 13.8s
+   Speed: 145 files/sec
+
+✅ Propagation Complete!
+
+📈 Final Status:
+   Total Files: 5000
+   With arela_path: 2850
+   Needs Propagation: 2000
+   Pending: 0
+   Errors: 0
+```
+
+## Backend Endpoints Used
+
+```
+POST  /api/uploader/scan/mark-propagation?tableName=X
+GET   /api/uploader/scan/pedimento-sources?tableName=X&offset=0&limit=50
+GET   /api/uploader/scan/files-by-directory?tableName=X&directoryPath=Y
+PATCH /api/uploader/scan/batch-update-propagation?tableName=X
+GET   /api/uploader/scan/propagation-stats?tableName=X
+```
+
+## Database Fields Updated
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `arela_path` | TEXT | Upload path (RFC/Year/...) |
+| `rfc` | VARCHAR | Tax ID from pedimento |
+| `detected_pedimento_year` | INTEGER | Year from pedimento |
+| `propagated_from_id` | UUID | Reference to source pedimento |
+| `propagation_attempted_at` | TIMESTAMP | When propagation ran |
+| `propagation_attempts` | INTEGER | Number of attempts |
+| `propagation_error` | TEXT | Error if propagation failed |
+| `needs_propagation` | BOOLEAN | Flag for efficient querying |
+
+## Performance Tips
+
+- **Large datasets**: Increase batch size to 100-200
+- **Slow propagation**: Check for many small directories vs few large ones
+- **API latency**: Use `--api` flag to select closer API
+- **Memory usage**: Reduce batch size if high memory usage
+
+## Optimization Features
+
+### 1. **Directory-Based Indexing**
+- Uses exact `directory_path` matching (no regex)
+- Leverages indexes for fast lookups
+- Processes one directory at a time
+
+### 2. **Attempt Tracking**
+- Tracks `propagation_attempts` per file
+- Respects `max_propagation_attempts` (default: 3)
+- Skips files that reached max attempts
+
+### 3. **Preparation Phase**
+- Marks files needing propagation before processing
+- Enables efficient batch queries
+- Reduces unnecessary processing
+
+### 4. **Batch Processing**
+- Fetches pedimentos in configurable batches
+- Updates multiple files per API call
+- Real-time progress with throughput metrics
+
+## Troubleshooting
+
+| Error | Solution |
+|-------|----------|
+| "Scan configuration errors" | Set ARELA_COMPANY_SLUG and ARELA_SERVER_ID |
+| "Table not found" | Run `arela scan` first |
+| "No files need propagation" | Run `arela identify` first |
+| "Failed to fetch pedimento sources" | Check API connectivity |
+
+## Comparison: Legacy vs Optimized
+
+| Feature | Legacy (detect --propagate-arela-path) | New (propagate) |
+|---------|----------------------------------------|-----------------|
+| Table | Global `uploader` | Dynamic `file_stats_*` |
+| API | Supabase direct | Configured API |
+| Query Strategy | Single UPDATE with regex | Directory-based exact match |
+| Progress | Batch count | Real-time throughput |
+| Indexes | Regex-based | Directory + arela_path |
+| Attempt Tracking | No | Yes (max 3 attempts) |
+| Error Handling | Basic | Categorized with tracking |
+
+## Next Steps
+
+After propagation:
+
+```bash
+# Upload files by RFC
+arela push
+```
+
+## Monitoring Queries
+
+```sql
+-- Check propagation progress
+SELECT 
+  COUNT(*) FILTER (WHERE arela_path IS NOT NULL) as with_arela_path,
+  COUNT(*) FILTER (WHERE needs_propagation = TRUE) as needs_propagation,
+  COUNT(*) FILTER (
+    WHERE arela_path IS NULL 
+      AND needs_propagation = TRUE 
+      AND propagation_attempts < max_propagation_attempts
+  ) as pending
+FROM cli.file_stats_<company>_<server>_<path>;
+
+-- View propagation by directory
+SELECT 
+  directory_path,
+  COUNT(*) as total_files,
+  COUNT(*) FILTER (WHERE arela_path IS NOT NULL) as propagated,
+  MAX(propagation_attempted_at) as last_attempt
+FROM cli.file_stats_<company>_<server>_<path>
+WHERE needs_propagation = TRUE
+GROUP BY directory_path
+ORDER BY total_files DESC
+LIMIT 20;
+
+-- Check propagation errors
+SELECT 
+  directory_path,
+  file_name,
+  propagation_error,
+  propagation_attempts
+FROM cli.file_stats_<company>_<server>_<path>
+WHERE propagation_error IS NOT NULL
+ORDER BY propagation_attempts DESC
+LIMIT 20;
+
+-- Find directories with pedimentos but unpropagated files
+SELECT 
+  f.directory_path,
+  COUNT(*) as unpropagated_files,
+  MAX(p.arela_path) as pedimento_arela_path
+FROM cli.file_stats_<company>_<server>_<path> f
+INNER JOIN cli.file_stats_<company>_<server>_<path> p
+  ON f.directory_path = p.directory_path
+  AND p.detected_type = 'pedimento_simplificado'
+  AND p.arela_path IS NOT NULL
+WHERE f.arela_path IS NULL
+  AND (f.detected_type IS NULL OR f.detected_type != 'pedimento_simplificado')
+GROUP BY f.directory_path
+ORDER BY unpropagated_files DESC
+LIMIT 20;
+```
+
+## Reset Propagation
+
+```sql
+-- Reset all propagation for retry
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET 
+  propagation_attempts = 0,
+  propagation_error = NULL,
+  propagation_attempted_at = NULL,
+  needs_propagation = FALSE
+WHERE arela_path IS NULL;
+
+-- Reset specific directory
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET 
+  propagation_attempts = 0,
+  propagation_error = NULL,
+  propagation_attempted_at = NULL
+WHERE directory_path LIKE '2024/3019796/%'
+  AND arela_path IS NULL;
+
+-- Increase max attempts for stubborn files
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET max_propagation_attempts = 5
+WHERE propagation_attempts >= max_propagation_attempts
+  AND arela_path IS NULL;
+```
+
+## Files Involved
+
+### CLI
+- `src/commands/PropagateCommand.js` - Main command
+- `src/services/ScanApiService.js` - API communication
+- `src/config/config.js` - Configuration (reused)
+
+### Backend
+- `src/uploader/services/file-stats-table-manager.service.ts` - Table operations
+- `src/uploader/services/uploader.service.ts` - Business logic
+- `src/uploader/controllers/uploader.controller.ts` - REST endpoints