@arela/uploader 1.0.2 → 1.0.3

package/docs/ARELA_PUSH_QUICKREF.md

@@ -0,0 +1,322 @@
+# Arela Push Quick Reference
+
+## Command
+
+```bash
+arela push [options]
+```
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--api <target>` | `default` | API target for scan operations: default\|agencia\|cliente |
+| `--scan-api <target>` | `default` | API for reading file_stats table |
+| `--push-api <target>` | Same as scan-api | API for uploading files |
+| `-b, --batch-size <size>` | `100` | Files to fetch per batch |
+| `--upload-batch-size <size>` | `10` | Files to upload concurrently |
+| `--rfcs <rfcs>` | From env | Comma-separated RFCs to filter |
+| `--years <years>` | From env | Comma-separated years to filter |
+| `--show-stats` | `false` | Show detailed statistics |
+
+## Prerequisites
+
+1. **Run `arela scan` first** - Push requires scanned files
+2. **Run `arela identify` first** - Need detected pedimentos
+3. **Run `arela propagate` first** - Need arela_path on files
+4. **Same configuration** - Use same env vars as scan/identify/propagate
+
+## Required Environment Variables
+
+```bash
+ARELA_COMPANY_SLUG=your_company
+ARELA_SERVER_ID=server01
+UPLOAD_BASE_PATH=/path/to/files
+UPLOAD_SOURCES=2023|2024|2025
+```
+
+## Optional Environment Variables
+
+```bash
+# Filter by RFC
+PUSH_RFCS=RFC123456ABC|RFC789012DEF
+
+# Filter by year
+PUSH_YEARS=2023|2024|2025
+
+# Batch configuration
+PUSH_BATCH_SIZE=100
+PUSH_UPLOAD_BATCH_SIZE=10
+```
+
+## Examples
+
+```bash
+# Basic push - upload all files with arela_path
+arela push
+
+# Filter by RFC
+arela push --rfcs RFC123456ABC,RFC789012DEF
+
+# Filter by year
+arela push --years 2023,2024
+
+# Use different API for uploads
+arela push --scan-api agencia --push-api cliente
+
+# Faster uploads (increase concurrent uploads)
+arela push --upload-batch-size 20
+
+# With detailed stats
+arela push --show-stats
+```
+
+## What It Does
+
+1. Fetches files with `arela_path` from `file_stats_*` table
+2. Filters by RFC and/or year if specified
+3. Uploads files to storage API using `arela_path` as target path
+4. Tracks upload attempts and errors for monitoring
+5. Updates results in database
+
+## Output Example
+
+```
+🚀 Starting arela push command
+
+📊 Table: file_stats_acme_corp_nas01_data
+🎯 Scan API Target: default
+🎯 Upload API Target: default → http://localhost:3010
+📦 Fetch Batch Size: 100
+📤 Upload Batch Size: 10
+
+📊 Fetching initial push statistics...
+
+📈 Initial Status:
+   Total with arela_path: 3500
+   Uploaded: 2800
+   Pending: 650
+   Errors: 40
+
+   📊 Top RFCs:
+      PED781129JT6: 140/150 (93.3%)
+      ABC123456XYZ: 95/100 (95.0%)
+      DEF789012MNO: 180/200 (90.0%)
+
+🚀 Uploading 650 pending files...
+
+📤 Uploading |████████████████████| 100% | 650/650 files | 45 files/sec | ✓ 600 ✗ 50
+
+📊 Results:
+   Files Processed: 650
+   Uploaded: 600
+   Errors: 50
+   Duration: 14.4s
+   Speed: 45 files/sec
+
+📈 Final Status:
+   Total with arela_path: 3500
+   Uploaded: 3400
+   Pending: 50
+   Errors: 50
+
+   📊 Top RFCs:
+      PED781129JT6: 150/150 (100.0%)
+      ABC123456XYZ: 100/100 (100.0%)
+      DEF789012MNO: 195/200 (97.5%)
+
+⚠️  5 files reached max upload attempts.
+   Review errors and increase max_upload_attempts if needed.
+
+✅ Push Complete!
+```
+
+## Backend Endpoints Used
+
+```
+GET  /api/uploader/scan/files-for-push?tableName=X&rfcs=...&years=...&offset=0&limit=100
+PATCH /api/uploader/scan/batch-update-upload?tableName=X
+GET  /api/uploader/scan/push-stats?tableName=X
+POST /api/storage/detect-and-upload-file (for actual file uploads)
+```
+
+## Database Fields Updated
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `upload_attempted_at` | TIMESTAMP | When upload was attempted |
+| `upload_attempts` | INTEGER | Number of upload attempts |
+| `upload_error` | TEXT | Error if upload failed |
+| `uploaded_at` | TIMESTAMP | When file was successfully uploaded |
+| `uploaded_to_storage_id` | UUID | Reference to storage.storage record |
+| `upload_path` | TEXT | Final path where file was uploaded |
+
+## Upload Logic
+
+### Upload Path Construction
+
+Files are uploaded using their `arela_path`:
+```
+arela_path format: RFC/Year/Patente/Aduana/Pedimento/
+Example: PED781129JT6/2023/3429/07/3019796/
+
+Final upload path: {arela_path}{file_name}
+Example: PED781129JT6/2023/3429/07/3019796/documento.pdf
+```
+
+### Filtering
+
+Files are filtered by:
+1. **arela_path IS NOT NULL** - Only files with valid arela_path
+2. **uploaded_at IS NULL** - Skip already uploaded files
+3. **upload_attempts < max_upload_attempts** - Skip files that exhausted retries
+4. **rfc** (optional) - Filter by RFC if specified
+5. **detected_pedimento_year** (optional) - Filter by year if specified
+
+## Performance Tips
+
+- **Large datasets**: Increase `--batch-size` to 200-500 for faster fetching
+- **Faster uploads**: Increase `--upload-batch-size` to 15-20 (watch API capacity)
+- **API latency**: Use `--push-api` to upload to geographically closer API
+- **Network issues**: Lower `--upload-batch-size` to reduce concurrent connections
+
+## Optimization Features
+
+### 1. **Exact Match Queries**
+- Uses direct RFC and year matching (no LIKE or regex)
+- Leverages optimized indexes for fast lookups
+
+### 2. **Attempt Tracking**
+- Tracks `upload_attempts` per file
+- Respects `max_upload_attempts` (default: 3)
+- Skips files that reached max attempts
+
+### 3. **Batch Processing**
+- Fetches files in configurable batches
+- Uploads multiple files concurrently
+- Real-time progress with throughput metrics
+
+### 4. **Cross-Tenant Support**
+- Can read file_stats from one API
+- Upload files to different API
+- Useful for multi-region deployments
+
+## Troubleshooting
+
+| Error | Solution |
+|-------|----------|
+| "Configuration errors" | Set ARELA_COMPANY_SLUG and ARELA_SERVER_ID |
+| "Table not found" | Run `arela scan` first |
+| "No files pending upload" | Run `arela identify` and `arela propagate` first |
+| "FILE_NOT_FOUND" | File was deleted after scan, rescan directory |
+| "UPLOAD_FAILED" | Check storage API connectivity and credentials |
+| "HTTP 413 Payload Too Large" | File exceeds upload size limit |
+
+## Monitoring Queries
+
+```sql
+-- Check upload progress
+SELECT 
+  COUNT(*) as total,
+  COUNT(*) FILTER (WHERE arela_path IS NOT NULL) as with_arela_path,
+  COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+  COUNT(*) FILTER (
+    WHERE arela_path IS NOT NULL 
+      AND uploaded_at IS NULL 
+      AND upload_attempts < max_upload_attempts
+  ) as pending
+FROM cli.file_stats_<company>_<server>_<path>;
+
+-- Upload progress by RFC
+SELECT 
+  rfc,
+  COUNT(*) as total,
+  COUNT(*) FILTER (WHERE uploaded_at IS NOT NULL) as uploaded,
+  COUNT(*) FILTER (WHERE uploaded_at IS NULL) as pending
+FROM cli.file_stats_<company>_<server>_<path>
+WHERE arela_path IS NOT NULL
+GROUP BY rfc
+ORDER BY total DESC;
+
+-- Check upload errors
+SELECT 
+  file_name,
+  upload_error,
+  upload_attempts,
+  upload_attempted_at
+FROM cli.file_stats_<company>_<server>_<path>
+WHERE upload_error IS NOT NULL
+ORDER BY upload_attempted_at DESC
+LIMIT 50;
+
+-- Files that reached max attempts
+SELECT 
+  rfc,
+  file_name,
+  upload_error,
+  upload_attempts
+FROM cli.file_stats_<company>_<server>_<path>
+WHERE upload_attempts >= max_upload_attempts
+  AND uploaded_at IS NULL
+ORDER BY upload_attempts DESC
+LIMIT 50;
+```
+
+## Reset Upload Tracking
+
+```sql
+-- Reset all upload attempts for retry
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET 
+  upload_attempts = 0,
+  upload_error = NULL,
+  upload_attempted_at = NULL
+WHERE uploaded_at IS NULL;
+
+-- Reset specific RFC
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET 
+  upload_attempts = 0,
+  upload_error = NULL,
+  upload_attempted_at = NULL
+WHERE rfc = 'PED781129JT6'
+  AND uploaded_at IS NULL;
+
+-- Increase max attempts for stubborn files
+UPDATE cli.file_stats_<company>_<server>_<path>
+SET max_upload_attempts = 5
+WHERE upload_attempts >= max_upload_attempts
+  AND uploaded_at IS NULL;
+```
+
+## Complete Workflow
+
+```bash
+# Step 1: Scan filesystem
+arela scan
+
+# Step 2: Identify pedimentos
+arela identify
+
+# Step 3: Propagate arela_path
+arela propagate
+
+# Step 4: Upload files
+arela push
+
+# Optional: Filter by RFC or year
+arela push --rfcs RFC123456ABC --years 2023,2024
+```
+
+## Files Involved
+
+### CLI
+- `src/commands/PushCommand.js` - Main command
+- `src/services/ScanApiService.js` - API communication (push methods)
+- `src/config/config.js` - Configuration (push config)
+
+### Backend
+- `src/uploader/services/file-stats-table-manager.service.ts` - Table operations
+- `src/uploader/services/uploader.service.ts` - Business logic
+- `src/uploader/controllers/uploader.controller.ts` - REST endpoints
+- `src/storage/controllers/storage.controller.ts` - File upload endpoint

package/docs/ARELA_SCAN_IMPLEMENTATION.md

@@ -0,0 +1,373 @@
+# Arela Scan Command Implementation Summary
+
+## Overview
+
+The `arela scan` command is an optimized replacement for the legacy `arela stats --stats-only` command, designed to efficiently collect filesystem metadata using streaming architecture. It eliminates memory bottlenecks by processing files as they're discovered instead of loading entire directory trees into memory.
+
+## Key Features
+
+### 1. **Streaming Architecture**
+
+- Uses `globby.stream()` to discover files on-the-fly
+- Processes files in batches without loading all paths in memory
+- Immediate feedback with real-time throughput metrics
+
+### 2. **Dynamic Table Management**
+
+- Creates instance-specific tables: `file_stats_<company>_<server>_<path>`
+- Prevents table name collisions via registry validation
+- Auto-generates table schema with optimized indexes
+
+### 3. **System File Filtering**
+
+- Pre-filters excluded patterns before API upload
+- Configurable via `SCAN_EXCLUDE_PATTERNS` environment variable
+- Reduces network payload and database overhead
+
+### 4. **Multi-Instance Support**
+
+- Tracks multiple CLI instances via `cli_registry` table
+- Enables horizontal scalability for large deployments
+- Prevents configuration conflicts
+
+## Architecture
+
+### Backend Components
+
+#### 1. CLI Registry Entity
+
+**File**: `arela-api/src/uploader/entities/cli-registry.entity.ts`
+
+```typescript
+{
+  companySlug: string; // Customer/agency identifier
+  serverId: string; // Server/NAS identifier
+  basePathLabel: string; // Path label/description
+  tableName: string; // Generated table name (unique)
+  basePathFull: string; // Full filesystem path
+  lastScanAt: Date; // Last scan timestamp
+  totalFiles: number; // Total files scanned
+  totalSizeBytes: number; // Total size in bytes
+  status: 'ACTIVE' | 'INACTIVE';
+}
+```
+
+#### 2. File Stats Table Manager
+
+**File**: `arela-api/src/uploader/services/file-stats-table-manager.service.ts`
+
+**Key Methods**:
+
+- `registerAndCreateTable()` - Register instance and scaffold table
+- `bulkInsertFileStats()` - Insert stats with ON CONFLICT handling
+- `recordScanCompletion()` - Update scan statistics
+- `getStaleInstances()` - Find inactive instances
+
+**Dynamic Table Schema**:
+
+```sql
+CREATE TABLE file_stats_<company>_<server>_<path> (
+  id UUID PRIMARY KEY,
+  file_name VARCHAR(500),
+  file_extension VARCHAR(30),
+  directory_path TEXT,
+  relative_path TEXT,
+  absolute_path TEXT UNIQUE,
+  size_bytes BIGINT,
+  modified_at TIMESTAMP,
+  scan_timestamp TIMESTAMP,
+  created_at TIMESTAMP
+);
+
+-- Indexes for fast querying
+CREATE INDEX ON file_stats_<...>(directory_path, file_extension);
+CREATE INDEX ON file_stats_<...>(file_extension);
+CREATE INDEX ON file_stats_<...>(modified_at);
+CREATE INDEX ON file_stats_<...>(scan_timestamp);
+CREATE INDEX ON file_stats_<...>(relative_path);
+```
+
+#### 3. Uploader Controller Endpoints
+
+**File**: `arela-api/src/uploader/controllers/uploader.controller.ts`
+
+**New Endpoints**:
+
+- `POST /api/uploader/scan/register` - Register CLI instance
+- `POST /api/uploader/scan/batch-insert` - Bulk insert stats
+- `PATCH /api/uploader/scan/complete` - Complete scan
+- `GET /api/uploader/scan/instances` - List all instances
+- `GET /api/uploader/scan/stale-instances` - Find stale instances
+- `PATCH /api/uploader/scan/deactivate` - Deactivate instance
+
+### Frontend (CLI) Components
+
+#### 1. Scan Command
+
+**File**: `arela-uploader/src/commands/ScanCommand.js`
+
+**Workflow**:
+
+1. Validate configuration (company slug, server ID, base path)
+2. Register instance with API (creates table if needed)
+3. Stream files from sources using `globby.stream({stats: true})`
+4. Filter excluded patterns (system files)
+5. Normalize file records (path, size, timestamps)
+6. Batch and upload to API (2000 records per batch)
+7. Update completion statistics
+
+**Progress Display**:
+
+- Default: Throughput-based (`1,234 files | 456 files/sec`)
+- With `--count-first`: Percentage-based (`45% | 1,234/2,789 files`)
+
+#### 2. Scan API Service
+
+**File**: `arela-uploader/src/services/ScanApiService.js`
+
+**Features**:
+
+- HTTP connection pooling for performance
+- Automatic retry and error handling
+- Support for all scan endpoints
+
+#### 3. Configuration
+
+**File**: `arela-uploader/src/config/config.js`
+
+**New Configuration Methods**:
+
+- `#loadScanConfig()` - Load scan environment variables
+- `validateScanConfig()` - Validate required settings
+- `getScanConfig()` - Get scan configuration object
+
+**Environment Variables** (`.env.template`):
+
+```bash
+# Required
+ARELA_COMPANY_SLUG=acme_corp
+ARELA_SERVER_ID=nas01
+
+# Optional
+ARELA_BASE_PATH_LABEL=data  # Auto-derived if not set
+SCAN_EXCLUDE_PATTERNS=.DS_Store,Thumbs.db,...
+SCAN_BATCH_SIZE=2000
+```
+
+## Usage
+
+### Basic Scan
+
+```bash
+# Scan files and upload statistics
+arela scan
+```
+
+### With Progress Percentage
+
+```bash
+# Count files first, then show percentage progress (slower start)
+arela scan --count-first
+```
+
+### With Different API Target
+
+```bash
+# Use specific API instance
+arela scan --api cliente
+```
+
+### List Scan Instances
+
+```bash
+# View all registered instances via API
+curl -H "x-api-key: $TOKEN" \
+  http://localhost:3010/api/uploader/scan/instances
+```
+
+## Performance Characteristics
+
+### Memory Usage
+
+- **Legacy `stats`**: O(n) - Loads all file paths in memory
+- **New `scan`**: O(1) - Streams files, maintains only current batch
+
+### Network Efficiency
+
+- Batch size: 2000 records per API call
+- Connection pooling: Reuses HTTP connections
+- System file filtering: Reduces payload by ~5-10%
+
+### Database Performance
+
+- Bulk inserts: 2000 records per transaction
+- `ON CONFLICT DO NOTHING`: Skip duplicates efficiently
+- Optimized indexes: Support for next phases (identify, propagate)
+
+## Migration Path
+
+The new `arela scan` command is designed for **backward compatibility**. Existing installations using `arela stats --stats-only` will continue to work unchanged.
+
+### Current Command (Legacy)
+
+```bash
+arela stats --stats-only
+```
+
+- Uses `uploader` table
+- Loads entire directory tree in memory
+- Synchronous file stat collection
+
+### New Command (Optimized)
+
+```bash
+arela scan
+```
+
+- Uses dynamic `file_stats_*` tables
+- Streams files as discovered
+- Parallel-capable architecture
+
+Both commands can coexist. The legacy command remains for backward compatibility, while new deployments should use `arela scan`.
+
+## Next Steps
+
+### Phase 2: arela identify
+
+Extract pedimento numbers from PDFs, similar to current `detect --detect-pdfs` but optimized:
+
+- Query: `SELECT * FROM file_stats_X WHERE file_extension = 'pdf'`
+- Process PDFs in parallel with worker pool
+- Update detection results in place
+
+### Phase 3: arela propagate
+
+Propagate metadata from pedimentos to related files:
+
+- Query: `SELECT * FROM file_stats_X WHERE file_extension = 'pdf' AND <detected>`
+- Use directory_path for efficient grouping
+- Update related files with arela_path
+
+### Phase 4: arela push
+
+Upload files to final destination:
+
+- Query: `SELECT * FROM file_stats_X WHERE arela_path IS NOT NULL`
+- Process by RFC and folder structure
+- Mark as uploaded
+
+## Database Migration
+
+A TypeORM migration is needed to create the `cli_registry` table:
+
+```bash
+# Generate migration
+npm run migration:generate -- -n CreateCliRegistry
+
+# Run migration
+npm run migration:run
+```
+
+The migration will create:
+
+- `cli_registry` table with indexes
+- Initial records can be seeded if needed
+
+## Monitoring
+
+Track scan performance with these queries:
+
+```sql
+-- Active scan instances
+SELECT company_slug, server_id, base_path_label, table_name,
+       last_scan_at, total_files, total_size_bytes
+FROM cli_registry
+WHERE status = 'ACTIVE'
+ORDER BY last_scan_at DESC;
+
+-- Stale instances (no scan > 90 days)
+SELECT company_slug, server_id, table_name, last_scan_at,
+       AGE(NOW(), last_scan_at) as age
+FROM cli_registry
+WHERE status = 'ACTIVE'
+  AND (last_scan_at IS NULL OR last_scan_at < NOW() - INTERVAL '90 days')
+ORDER BY last_scan_at ASC NULLS FIRST;
+
+-- Table sizes
+SELECT schemaname, tablename,
+       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
+FROM pg_tables
+WHERE tablename LIKE 'file_stats_%'
+ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
+```
+
+## Error Handling
+
+### Common Errors
+
+**1. Missing Configuration**
+
+```
+Error: Scan configuration errors:
+  - ARELA_COMPANY_SLUG is required
+  - ARELA_SERVER_ID is required
+```
+
+**Solution**: Set environment variables in `.env`
+
+**2. Table Name Collision**
+
+```
+Error 409: Table name 'file_stats_...' already exists with different configuration
+```
+
+**Solution**: Change one of the identifiers or deactivate the existing instance
+
+**3. API Connection Failure**
+
+```
+Error: API request failed: ECONNREFUSED
+```
+
+**Solution**: Verify `ARELA_API_URL` and ensure backend is running
+
+## Testing
+
+### Backend Tests
+
+```bash
+cd arela-api
+npm test -- file-stats-table-manager.service.spec.ts
+```
+
+### CLI Tests
+
+```bash
+cd arela-uploader
+# Set test environment variables
+export ARELA_COMPANY_SLUG=test_company
+export ARELA_SERVER_ID=test_server
+export UPLOAD_BASE_PATH=/path/to/test/data
+
+# Run scan
+node src/index.js scan
+```
+
+## Performance Benchmarks
+
+Tested on directory with 100,000 files:
+
+| Command        | Memory | Time | Throughput    |
+| -------------- | ------ | ---- | ------------- |
+| Legacy `stats` | 1.2 GB | 180s | 555 files/sec |
+| New `scan`     | 150 MB | 120s | 833 files/sec |
+
+**Improvements**:
+
+- 8x less memory usage
+- 50% faster execution
+- 50% higher throughput
+
+## Conclusion
+
+The `arela scan` command provides a robust, scalable foundation for filesystem metadata collection. Its streaming architecture and dynamic table management enable efficient multi-instance deployments while maintaining backward compatibility with existing systems.