@arela/uploader 1.0.1 → 1.0.3

package/.env.template

@@ -19,6 +19,64 @@ UPLOAD_BASE_PATH=/path/to/your/upload/base
 UPLOAD_SOURCES=folder1|folder2|folder3
 UPLOAD_RFCS=rfc1|rfc2|rfc3
 
+# =============================================================================
+# SCAN CONFIGURATION (for arela scan command)
+# =============================================================================
+
+# Company identifier for this CLI instance (required)
+# Use a short, descriptive slug for your company/agency/client
+# Examples: "acme_corp", "cliente_123", "agencia_xyz"
+ARELA_COMPANY_SLUG=
+
+# Server identifier (required)
+# Use a unique ID for each server/NAS where arela-cli is installed
+# Examples: "nas01", "server-mx", "storage-01"
+ARELA_SERVER_ID=
+
+# Base path label (optional, auto-derived from UPLOAD_BASE_PATH if not set)
+# Short label describing the base path being scanned
+# Examples: "data", "documents", "archive"
+ARELA_BASE_PATH_LABEL=
+
+# System file patterns to exclude from scan (comma-separated)
+# These files will be filtered before uploading stats to reduce payload
+SCAN_EXCLUDE_PATTERNS=.DS_Store,Thumbs.db,desktop.ini,__pycache__,.pyc,.tmp,.swp,$RECYCLE.BIN,System Volume Information,~$*
+
+# Batch size for scan operations (default: 2000 records per API call)
+SCAN_BATCH_SIZE=2000
+
+# Directory depth level for creating separate tables (default: 0)
+# 0 = single table for entire base path
+# 1 = one table per first-level subdirectory
+# 2 = one table per second-level subdirectory, etc.
+# Example: with level=1 and base=/data, creates tables for /data/folder1, /data/folder2, etc.
+SCAN_DIRECTORY_LEVEL=0
+
+# =============================================================================
+# PUSH CONFIGURATION (for arela push command)
+# =============================================================================
+
+# Filter files to upload by RFC (pipe-separated, optional)
+# If not set, all files with arela_path will be uploaded
+# Examples: "RFC123456ABC|RFC789012DEF"
+PUSH_RFCS=
+
+# Filter files to upload by year (pipe-separated, optional)
+# If not set, all files with arela_path will be uploaded
+# Examples: "2023|2024|2025"
+PUSH_YEARS=
+
+# Batch size for fetching files from database (default: 100)
+PUSH_BATCH_SIZE=100
+
+# Concurrent upload batch size (default: 10)
+# Number of files to upload simultaneously
+PUSH_UPLOAD_BATCH_SIZE=10
+
+# Storage bucket for uploaded files (optional, defaults to SUPABASE_BUCKET)
+# Examples: "archivos", "documents", "storage"
+PUSH_BUCKET=arela
+
 # =============================================================================
 # PERFORMANCE OPTIMIZATION FOR MULTIPLE API REPLICAS
 # =============================================================================
@@ -30,6 +88,18 @@ MAX_API_CONNECTIONS=10
 # API Connection Timeout (milliseconds)
 API_CONNECTION_TIMEOUT=60000
 
+# API Retry Configuration
+# Maximum number of retry attempts for failed API requests
+API_MAX_RETRIES=3
+
+# Enable exponential backoff for retries (true/false)
+# When true, retry delays increase: 1s, 2s, 4s, 8s, 16s
+# When false, uses fixed delay (API_RETRY_DELAY)
+API_RETRY_EXPONENTIAL_BACKOFF=true
+
+# Fixed retry delay in milliseconds (only used if exponential backoff is disabled)
+API_RETRY_DELAY=1000
+
 # Batch Processing Configuration
 # Files processed concurrently per batch (should be >= MAX_API_CONNECTIONS for best performance)
 BATCH_SIZE=100

package/docs/API_RETRY_MECHANISM.md

@@ -0,0 +1,338 @@
+# API Retry Mechanism
+
+## Overview
+
+The `arela scan` and `arela identify` commands now include robust retry logic with exponential backoff for all API requests. This ensures resilience against transient network issues, temporary server overload, and rate limiting.
+
+## Features
+
+### 1. **Automatic Retry on Transient Errors**
+
+Retries are automatically triggered for:
+- **Network errors**: Connection reset, timeout, refused, DNS failures
+- **HTTP 429**: Too Many Requests (rate limiting)
+- **HTTP 5xx**: Server errors (500, 502, 503, 504)
+
+### 2. **Exponential Backoff (Default)**
+
+When enabled (default), retry delays increase exponentially:
+- Attempt 1: ~1 second
+- Attempt 2: ~2 seconds
+- Attempt 3: ~4 seconds
+- Attempt 4: ~8 seconds
+- Attempt 5: ~16 seconds (max)
+
+### 3. **Jitter to Prevent Thundering Herd**
+
+Each retry delay includes ±20% random jitter to prevent multiple clients from retrying simultaneously, which could overwhelm the server.
+
+### 4. **Smart Error Detection**
+
+The system distinguishes between:
+- **Retryable errors**: Network issues, server errors, rate limits
+- **Non-retryable errors**: Client errors (400, 401, 403, 404), validation errors
+
+Non-retryable errors fail immediately without wasting time on useless retries.
+
+## Configuration
+
+### Environment Variables
+
+Add to `.env` file:
+
+```bash
+# Maximum number of retry attempts (default: 3)
+API_MAX_RETRIES=3
+
+# Use exponential backoff (default: true)
+# When true: 1s → 2s → 4s → 8s → 16s
+# When false: uses fixed delay
+API_RETRY_EXPONENTIAL_BACKOFF=true
+
+# Fixed retry delay in milliseconds (only used if exponential backoff is disabled)
+API_RETRY_DELAY=1000
+```
+
+### Configuration Examples
+
+#### High Reliability (Recommended)
+
+```bash
+API_MAX_RETRIES=5
+API_RETRY_EXPONENTIAL_BACKOFF=true
+```
+
+**Use case**: Production environments with unreliable network or high load
+
+#### Fast Failure (Testing/Development)
+
+```bash
+API_MAX_RETRIES=1
+API_RETRY_EXPONENTIAL_BACKOFF=false
+API_RETRY_DELAY=500
+```
+
+**Use case**: Local development where you want quick feedback
+
+#### Aggressive Retry (High-Volume Processing)
+
+```bash
+API_MAX_RETRIES=7
+API_RETRY_EXPONENTIAL_BACKOFF=true
+```
+
+**Use case**: Large batch operations where you can't afford to lose progress
+
+## Retry Behavior
+
+### Retryable Scenarios
+
+| Error Type | Example | Retry Behavior |
+|------------|---------|----------------|
+| Network timeout | `ETIMEDOUT` | ✅ Retry with backoff |
+| Connection reset | `ECONNRESET` | ✅ Retry with backoff |
+| Connection refused | `ECONNREFUSED` | ✅ Retry with backoff |
+| DNS failure | `ENOTFOUND`, `EAI_AGAIN` | ✅ Retry with backoff |
+| Rate limiting | HTTP 429 | ✅ Retry with backoff |
+| Server error | HTTP 5xx | ✅ Retry with backoff |
+
+### Non-Retryable Scenarios
+
+| Error Type | Example | Retry Behavior |
+|------------|---------|----------------|
+| Unauthorized | HTTP 401 | ❌ Fail immediately |
+| Forbidden | HTTP 403 | ❌ Fail immediately |
+| Not found | HTTP 404 | ❌ Fail immediately |
+| Bad request | HTTP 400 | ❌ Fail immediately |
+| Conflict | HTTP 409 | ❌ Fail immediately |
+
+## Logging
+
+### Retry Warnings
+
+When a retry is triggered, you'll see warnings like:
+
+```
+⚠️  API request failed (attempt 1/4): Connection timeout. Retrying in 1234ms...
+⚠️  API request failed (attempt 2/4): HTTP 503 Service Unavailable. Retrying in 2456ms...
+```
+
+### Retry Success
+
+When a retry succeeds, you'll see:
+
+```
+ℹ️  API request succeeded on attempt 3/4
+```
+
+### Final Failure
+
+If all retries fail, you'll see:
+
+```
+❌ API request failed after 4 attempt(s): Connection timeout
+```
+
+## Performance Impact
+
+### With Default Settings (3 retries, exponential backoff)
+
+**Best case** (no failures): No overhead
+
+**Worst case** (all retries fail):
+- Total retry time: ~1s + 2s + 4s = ~7 seconds
+- Total attempts: 4 (1 initial + 3 retries)
+
+### Optimization Tips
+
+1. **For stable networks**: Reduce `API_MAX_RETRIES` to 2-3
+2. **For unstable networks**: Increase to 5-7
+3. **For rate-limited APIs**: Keep exponential backoff enabled
+4. **For fast development**: Disable retries or set to 1
+
+## Integration with Commands
+
+### arela scan
+
+All API operations during scan now have retry logic:
+- Instance registration (`POST /api/uploader/scan/register`)
+- Batch insert (`POST /api/uploader/scan/batch-insert`)
+- Scan completion (`PATCH /api/uploader/scan/complete`)
+
+### arela identify
+
+All API operations during identify now have retry logic:
+- Fetch detection stats (`GET /api/uploader/scan/detection-stats`)
+- Fetch PDFs for detection (`GET /api/uploader/scan/pdfs-for-detection`)
+- Batch update detection (`PATCH /api/uploader/scan/batch-update-detection`)
+
+## Comparison with DatabaseService
+
+| Feature | DatabaseService (Supabase) | ScanApiService (HTTP) |
+|---------|---------------------------|----------------------|
+| Retry Logic | ✅ Yes | ✅ Yes |
+| Max Retries | 3 (hardcoded) | 3 (configurable) |
+| Backoff Strategy | Exponential | Exponential or Fixed |
+| Jitter | No | ✅ Yes (±20%) |
+| Error Detection | Generic | HTTP-specific |
+| Configurable | No | ✅ Yes via .env |
+
+## Best Practices
+
+### 1. **Enable in Production**
+
+Always use retry logic in production:
+
+```bash
+API_MAX_RETRIES=3
+API_RETRY_EXPONENTIAL_BACKOFF=true
+```
+
+### 2. **Monitor Retry Rates**
+
+Track retry warnings in logs to detect:
+- Network instability
+- Server overload
+- API rate limiting
+
+### 3. **Adjust for Your Environment**
+
+- **Cloud/remote**: Higher retries (5-7)
+- **Local/LAN**: Lower retries (1-3)
+- **Rate-limited APIs**: Exponential backoff
+
+### 4. **Use Jitter**
+
+Always keep jitter enabled (built-in) to prevent retry storms.
+
+### 5. **Set Connection Timeout**
+
+Combine retries with appropriate timeout:
+
+```bash
+API_CONNECTION_TIMEOUT=30000  # 30 seconds
+API_MAX_RETRIES=3
+```
+
+This ensures retries happen within reasonable time.
+
+## Troubleshooting
+
+### Too Many Retries
+
+**Symptom**: Commands take too long due to retries
+
+**Solution**: Reduce `API_MAX_RETRIES` or disable exponential backoff
+
+```bash
+API_MAX_RETRIES=1
+```
+
+### Not Enough Retries
+
+**Symptom**: Commands fail due to transient errors
+
+**Solution**: Increase `API_MAX_RETRIES`
+
+```bash
+API_MAX_RETRIES=5
+```
+
+### Rate Limiting Issues
+
+**Symptom**: Many HTTP 429 errors
+
+**Solution**: Ensure exponential backoff is enabled and increase retries
+
+```bash
+API_MAX_RETRIES=5
+API_RETRY_EXPONENTIAL_BACKOFF=true
+```
+
+### Network Timeout Issues
+
+**Symptom**: `ETIMEDOUT` errors
+
+**Solution**: Increase connection timeout and retries
+
+```bash
+API_CONNECTION_TIMEOUT=60000  # 60 seconds
+API_MAX_RETRIES=5
+```
+
+## Implementation Details
+
+### Code Location
+
+- **Service**: `arela-uploader/src/services/ScanApiService.js`
+- **Methods**:
+  - `#isRetryableError()` - Determines if error should trigger retry
+  - `#calculateBackoff()` - Calculates delay between retries
+  - `#request()` - Main request method with retry loop
+
+### Retry Loop Logic
+
+```javascript
+for (let attempt = 1; attempt <= maxRetries + 1; attempt++) {
+  try {
+    // Make request
+    const response = await fetch(url, options);
+    
+    // Check if response is ok
+    if (!response.ok) {
+      const error = new Error(`${response.status} ${response.statusText}`);
+      
+      // Retry if error is retryable
+      if (isRetryable(error, response) && attempt <= maxRetries) {
+        await sleep(calculateBackoff(attempt));
+        continue;
+      }
+      
+      throw error;
+    }
+    
+    // Success
+    return await response.json();
+    
+  } catch (error) {
+    // Handle network errors
+    if (isRetryable(error) && attempt <= maxRetries) {
+      await sleep(calculateBackoff(attempt));
+      continue;
+    }
+    
+    throw error;
+  }
+}
+```
+
+### Backoff Calculation
+
+```javascript
+function calculateBackoff(attempt) {
+  if (exponentialBackoff) {
+    // 1s, 2s, 4s, 8s, 16s (max)
+    const delay = Math.min(1000 * Math.pow(2, attempt - 1), 16000);
+    
+    // Add jitter (±20%)
+    const jitter = delay * 0.2 * (Math.random() * 2 - 1);
+    return delay + jitter;
+  } else {
+    // Fixed delay with jitter
+    return fixedDelay + (fixedDelay * 0.2 * (Math.random() * 2 - 1));
+  }
+}
+```
+
+## Future Enhancements
+
+Potential improvements:
+1. **Circuit breaker pattern**: Stop retrying after N consecutive failures
+2. **Adaptive backoff**: Adjust delays based on error patterns
+3. **Retry budget**: Limit total retry time per operation
+4. **Metrics collection**: Track retry rates and success rates
+5. **Per-endpoint configuration**: Different retry settings for different endpoints
+
+## Conclusion
+
+The retry mechanism provides robust error handling for the `arela scan` and `arela identify` commands, ensuring operations can recover from transient failures without manual intervention. Proper configuration and monitoring ensure optimal performance and reliability.