@arela/uploader 1.0.24 → 1.1.0

package/docs/API_RETRY_MECHANISM.md

@@ -1,338 +0,0 @@
-# API Retry Mechanism
-
-## Overview
-
-The `arela scan` and `arela identify` commands now include robust retry logic with exponential backoff for all API requests. This ensures resilience against transient network issues, temporary server overload, and rate limiting.
-
-## Features
-
-### 1. **Automatic Retry on Transient Errors**
-
-Retries are automatically triggered for:
-- **Network errors**: Connection reset, timeout, refused, DNS failures
-- **HTTP 429**: Too Many Requests (rate limiting)
-- **HTTP 5xx**: Server errors (500, 502, 503, 504)
-
-### 2. **Exponential Backoff (Default)**
-
-When enabled (default), retry delays increase exponentially:
-- Attempt 1: ~1 second
-- Attempt 2: ~2 seconds
-- Attempt 3: ~4 seconds
-- Attempt 4: ~8 seconds
-- Attempt 5: ~16 seconds (max)
-
-### 3. **Jitter to Prevent Thundering Herd**
-
-Each retry delay includes ±20% random jitter to prevent multiple clients from retrying simultaneously, which could overwhelm the server.
-
-### 4. **Smart Error Detection**
-
-The system distinguishes between:
-- **Retryable errors**: Network issues, server errors, rate limits
-- **Non-retryable errors**: Client errors (400, 401, 403, 404), validation errors
-
-Non-retryable errors fail immediately without wasting time on useless retries.
-
-## Configuration
-
-### Environment Variables
-
-Add to `.env` file:
-
-```bash
-# Maximum number of retry attempts (default: 3)
-API_MAX_RETRIES=3
-
-# Use exponential backoff (default: true)
-# When true: 1s → 2s → 4s → 8s → 16s
-# When false: uses fixed delay
-API_RETRY_EXPONENTIAL_BACKOFF=true
-
-# Fixed retry delay in milliseconds (only used if exponential backoff is disabled)
-API_RETRY_DELAY=1000
-```
-
-### Configuration Examples
-
-#### High Reliability (Recommended)
-
-```bash
-API_MAX_RETRIES=5
-API_RETRY_EXPONENTIAL_BACKOFF=true
-```
-
-**Use case**: Production environments with unreliable network or high load
-
-#### Fast Failure (Testing/Development)
-
-```bash
-API_MAX_RETRIES=1
-API_RETRY_EXPONENTIAL_BACKOFF=false
-API_RETRY_DELAY=500
-```
-
-**Use case**: Local development where you want quick feedback
-
-#### Aggressive Retry (High-Volume Processing)
-
-```bash
-API_MAX_RETRIES=7
-API_RETRY_EXPONENTIAL_BACKOFF=true
-```
-
-**Use case**: Large batch operations where you can't afford to lose progress
-
-## Retry Behavior
-
-### Retryable Scenarios
-
-| Error Type | Example | Retry Behavior |
-|------------|---------|----------------|
-| Network timeout | `ETIMEDOUT` | ✅ Retry with backoff |
-| Connection reset | `ECONNRESET` | ✅ Retry with backoff |
-| Connection refused | `ECONNREFUSED` | ✅ Retry with backoff |
-| DNS failure | `ENOTFOUND`, `EAI_AGAIN` | ✅ Retry with backoff |
-| Rate limiting | HTTP 429 | ✅ Retry with backoff |
-| Server error | HTTP 5xx | ✅ Retry with backoff |
-
-### Non-Retryable Scenarios
-
-| Error Type | Example | Retry Behavior |
-|------------|---------|----------------|
-| Unauthorized | HTTP 401 | ❌ Fail immediately |
-| Forbidden | HTTP 403 | ❌ Fail immediately |
-| Not found | HTTP 404 | ❌ Fail immediately |
-| Bad request | HTTP 400 | ❌ Fail immediately |
-| Conflict | HTTP 409 | ❌ Fail immediately |
-
-## Logging
-
-### Retry Warnings
-
-When a retry is triggered, you'll see warnings like:
-
-```
-⚠️  API request failed (attempt 1/4): Connection timeout. Retrying in 1234ms...
-⚠️  API request failed (attempt 2/4): HTTP 503 Service Unavailable. Retrying in 2456ms...
-```
-
-### Retry Success
-
-When a retry succeeds, you'll see:
-
-```
-ℹ️  API request succeeded on attempt 3/4
-```
-
-### Final Failure
-
-If all retries fail, you'll see:
-
-```
-❌ API request failed after 4 attempt(s): Connection timeout
-```
-
-## Performance Impact
-
-### With Default Settings (3 retries, exponential backoff)
-
-**Best case** (no failures): No overhead
-
-**Worst case** (all retries fail):
-- Total retry time: ~1s + 2s + 4s = ~7 seconds
-- Total attempts: 4 (1 initial + 3 retries)
-
-### Optimization Tips
-
-1. **For stable networks**: Reduce `API_MAX_RETRIES` to 2-3
-2. **For unstable networks**: Increase to 5-7
-3. **For rate-limited APIs**: Keep exponential backoff enabled
-4. **For fast development**: Disable retries or set to 1
-
-## Integration with Commands
-
-### arela scan
-
-All API operations during scan now have retry logic:
-- Instance registration (`POST /api/uploader/scan/register`)
-- Batch insert (`POST /api/uploader/scan/batch-insert`)
-- Scan completion (`PATCH /api/uploader/scan/complete`)
-
-### arela identify
-
-All API operations during identify now have retry logic:
-- Fetch detection stats (`GET /api/uploader/scan/detection-stats`)
-- Fetch PDFs for detection (`GET /api/uploader/scan/pdfs-for-detection`)
-- Batch update detection (`PATCH /api/uploader/scan/batch-update-detection`)
-
-## Comparison with DatabaseService
-
-| Feature | DatabaseService (Supabase) | ScanApiService (HTTP) |
-|---------|---------------------------|----------------------|
-| Retry Logic | ✅ Yes | ✅ Yes |
-| Max Retries | 3 (hardcoded) | 3 (configurable) |
-| Backoff Strategy | Exponential | Exponential or Fixed |
-| Jitter | No | ✅ Yes (±20%) |
-| Error Detection | Generic | HTTP-specific |
-| Configurable | No | ✅ Yes via .env |
-
-## Best Practices
-
-### 1. **Enable in Production**
-
-Always use retry logic in production:
-
-```bash
-API_MAX_RETRIES=3
-API_RETRY_EXPONENTIAL_BACKOFF=true
-```
-
-### 2. **Monitor Retry Rates**
-
-Track retry warnings in logs to detect:
-- Network instability
-- Server overload
-- API rate limiting
-
-### 3. **Adjust for Your Environment**
-
-- **Cloud/remote**: Higher retries (5-7)
-- **Local/LAN**: Lower retries (1-3)
-- **Rate-limited APIs**: Exponential backoff
-
-### 4. **Use Jitter**
-
-Always keep jitter enabled (built-in) to prevent retry storms.
-
-### 5. **Set Connection Timeout**
-
-Combine retries with appropriate timeout:
-
-```bash
-API_CONNECTION_TIMEOUT=30000  # 30 seconds
-API_MAX_RETRIES=3
-```
-
-This ensures retries happen within reasonable time.
-
-## Troubleshooting
-
-### Too Many Retries
-
-**Symptom**: Commands take too long due to retries
-
-**Solution**: Reduce `API_MAX_RETRIES` or disable exponential backoff
-
-```bash
-API_MAX_RETRIES=1
-```
-
-### Not Enough Retries
-
-**Symptom**: Commands fail due to transient errors
-
-**Solution**: Increase `API_MAX_RETRIES`
-
-```bash
-API_MAX_RETRIES=5
-```
-
-### Rate Limiting Issues
-
-**Symptom**: Many HTTP 429 errors
-
-**Solution**: Ensure exponential backoff is enabled and increase retries
-
-```bash
-API_MAX_RETRIES=5
-API_RETRY_EXPONENTIAL_BACKOFF=true
-```
-
-### Network Timeout Issues
-
-**Symptom**: `ETIMEDOUT` errors
-
-**Solution**: Increase connection timeout and retries
-
-```bash
-API_CONNECTION_TIMEOUT=60000  # 60 seconds
-API_MAX_RETRIES=5
-```
-
-## Implementation Details
-
-### Code Location
-
-- **Service**: `arela-uploader/src/services/ScanApiService.js`
-- **Methods**:
-  - `#isRetryableError()` - Determines if error should trigger retry
-  - `#calculateBackoff()` - Calculates delay between retries
-  - `#request()` - Main request method with retry loop
-
-### Retry Loop Logic
-
-```javascript
-for (let attempt = 1; attempt <= maxRetries + 1; attempt++) {
-  try {
-    // Make request
-    const response = await fetch(url, options);
-    
-    // Check if response is ok
-    if (!response.ok) {
-      const error = new Error(`${response.status} ${response.statusText}`);
-      
-      // Retry if error is retryable
-      if (isRetryable(error, response) && attempt <= maxRetries) {
-        await sleep(calculateBackoff(attempt));
-        continue;
-      }
-      
-      throw error;
-    }
-    
-    // Success
-    return await response.json();
-    
-  } catch (error) {
-    // Handle network errors
-    if (isRetryable(error) && attempt <= maxRetries) {
-      await sleep(calculateBackoff(attempt));
-      continue;
-    }
-    
-    throw error;
-  }
-}
-```
-
-### Backoff Calculation
-
-```javascript
-function calculateBackoff(attempt) {
-  if (exponentialBackoff) {
-    // 1s, 2s, 4s, 8s, 16s (max)
-    const delay = Math.min(1000 * Math.pow(2, attempt - 1), 16000);
-    
-    // Add jitter (±20%)
-    const jitter = delay * 0.2 * (Math.random() * 2 - 1);
-    return delay + jitter;
-  } else {
-    // Fixed delay with jitter
-    return fixedDelay + (fixedDelay * 0.2 * (Math.random() * 2 - 1));
-  }
-}
-```
-
-## Future Enhancements
-
-Potential improvements:
-1. **Circuit breaker pattern**: Stop retrying after N consecutive failures
-2. **Adaptive backoff**: Adjust delays based on error patterns
-3. **Retry budget**: Limit total retry time per operation
-4. **Metrics collection**: Track retry rates and success rates
-5. **Per-endpoint configuration**: Different retry settings for different endpoints
-
-## Conclusion
-
-The retry mechanism provides robust error handling for the `arela scan` and `arela identify` commands, ensuring operations can recover from transient failures without manual intervention. Proper configuration and monitoring ensure optimal performance and reliability.