@arela/uploader 0.2.13 → 1.0.0

package/docs/QUICK_REFERENCE_API_DETECTION.md

@@ -0,0 +1,264 @@
+# Quick Reference: Using API Service for Pedimento Detection
+
+## Overview
+
+The `detectPedimentosInDatabase` method now automatically uses API service when available, with seamless fallback to Supabase.
+
+## How It Works
+
+### Automatic Service Selection
+
+```
+1. Check if API service is available
+   ├─ YES → Use API endpoints (faster, more secure)
+   └─ NO  → Fall back to Supabase client (backward compatible)
+
+2. Process detection in chunks
+   ├─ Fetch records (API or Supabase)
+   ├─ Detect pedimentos locally
+   └─ Batch update results (API or Supabase)
+```
+
+### Running Detection
+
+```bash
+# Standard command - uses API if available
+node src/index.js detect --detect-pdfs
+
+# With custom batch size
+node src/index.js detect --detect-pdfs --batch-size 20
+```
+
+### Log Output
+
+**With API (New):**
+```
+✅ Connected to Arela API
+Using API service for PDF detection...
+Phase 2: Starting PDF detection for pedimento-simplificado documents...
+Processing PDF files in chunks of 100 records...
+```
+
+**With Supabase (Fallback):**
+```
+⚠️ API connection failed, falling back to direct Supabase upload
+✅ Connected to Supabase Storage (direct mode)
+Phase 2: Starting PDF detection for pedimento-simplificado documents...
+Processing PDF files in chunks of 100 records...
+```
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# API Mode (Preferred)
+ARELA_API_URL=https://your-api.com
+ARELA_API_TOKEN=your-api-token-here
+
+# Supabase (Fallback)
+SUPABASE_URL=https://your-project.supabase.co
+SUPABASE_KEY=your-supabase-key
+```
+
+### Performance Tuning
+
+```bash
+# Adjust concurrent API connections
+MAX_API_CONNECTIONS=10
+
+# Adjust API timeout
+API_CONNECTION_TIMEOUT=60000
+
+# Adjust processing batch size
+BATCH_SIZE=50
+```
+
+## API Endpoints Used
+
+### 1. Fetch PDF Records
+```http
+GET /api/uploader/pdf-records?offset=0&limit=100&status=fs-stats&file_extension=pdf&is_like_simplificado=true
+```
+
+### 2. Batch Update Results
+```http
+PATCH /api/uploader/batch-update-detection
+Content-Type: application/json
+
+{
+  "updates": [
+    {
+      "id": "uuid",
+      "status": "detected",
+      "document_type": "pedimento-simplificado",
+      "num_pedimento": "23  12  3456  7890123",
+      "arela_path": "RFC/2023/3456/",
+      "year": "2023",
+      "rfc": "RFC123456789"
+    }
+  ]
+}
+```
+
+## Troubleshooting
+
+### API Not Available
+
+**Symptom:** See "falling back to direct Supabase upload" in logs
+
+**Solutions:**
+1. Check `ARELA_API_URL` and `ARELA_API_TOKEN` are set
+2. Verify backend API is running and accessible
+3. Check network connectivity to API endpoint
+4. Review API logs for errors
+
+**Verification:**
+```bash
+# Test API health endpoint
+curl -H "x-api-key: YOUR_TOKEN" https://your-api.com/api/health
+```
+
+### Slow Performance
+
+**Symptom:** Detection taking longer than expected
+
+**Solutions:**
+1. Increase `MAX_API_CONNECTIONS` (default: 10)
+2. Adjust `BATCH_SIZE` (default: 50)
+3. Check backend API performance
+4. Verify database indexes exist
+
+### Update Failures
+
+**Symptom:** Some records not being updated
+
+**Solutions:**
+1. Check API response for specific errors
+2. Verify record IDs exist in database
+3. Review API logs for validation errors
+4. Check network stability
+
+## Benefits Comparison
+
+| Feature | Direct Supabase | API Service |
+|---------|----------------|-------------|
+| **Security** | Client has DB credentials | API key only |
+| **Updates** | Individual queries | Batch operations |
+| **Monitoring** | Client-side only | Centralized logging |
+| **Flexibility** | Client redeployment needed | Backend updates only |
+| **Performance** | Multiple connections | Optimized batching |
+| **Scalability** | Limited by client | Horizontal scaling |
+
+## Code Usage
+
+### In Your Code
+
+The service is used transparently - no code changes needed:
+
+```javascript
+import databaseService from './services/DatabaseService.js';
+
+// Automatically uses API if available
+const result = await databaseService.detectPedimentosInDatabase({
+  batchSize: 10
+});
+
+console.log(`Detected: ${result.detectedCount}`);
+console.log(`Processed: ${result.processedCount}`);
+console.log(`Errors: ${result.errorCount}`);
+```
+
+### Custom Service Selection (Advanced)
+
+```javascript
+import uploadServiceFactory from './services/upload/UploadServiceFactory.js';
+
+// Check what service is active
+const isApiMode = await uploadServiceFactory.isApiModeAvailable();
+console.log(`API Mode: ${isApiMode ? 'Yes' : 'No'}`);
+
+// Force Supabase mode (bypass API)
+const supabaseService = await uploadServiceFactory.getUploadService(true);
+```
+
+## Testing
+
+### Test API Endpoints
+
+```bash
+# 1. Test fetch endpoint
+curl -H "x-api-key: YOUR_TOKEN" \
+  "https://your-api.com/api/uploader/pdf-records?offset=0&limit=10&status=fs-stats&file_extension=pdf&is_like_simplificado=true"
+
+# 2. Test batch update endpoint
+curl -X PATCH \
+  -H "x-api-key: YOUR_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "updates": [{
+      "id": "test-uuid",
+      "status": "detected",
+      "document_type": "pedimento-simplificado"
+    }]
+  }' \
+  https://your-api.com/api/uploader/batch-update-detection
+```
+
+### Integration Test
+
+```bash
+# Run detection with small batch for testing
+node src/index.js detect --detect-pdfs --batch-size 5
+
+# Check logs for:
+# - "Using API service for PDF detection..." (API mode)
+# - Successful chunk processing
+# - No error messages
+```
+
+## Monitoring
+
+### Key Metrics to Track
+
+1. **Service Mode**: API vs Supabase usage percentage
+2. **Response Times**: Average time per batch operation
+3. **Error Rates**: Failed fetches or updates
+4. **Throughput**: Records processed per minute
+5. **API Health**: Uptime and availability
+
+### Log Analysis
+
+```bash
+# Count API vs Supabase usage
+grep "Using API service" logs/*.log | wc -l
+grep "falling back to direct Supabase" logs/*.log | wc -l
+
+# Check for errors
+grep "ERROR" logs/*.log | grep "detection"
+
+# Performance metrics
+grep "Chunk.*completed" logs/*.log
+```
+
+## Support
+
+For issues or questions:
+
+1. **Check Logs**: Review application logs for detailed error messages
+2. **Verify Config**: Ensure all environment variables are set correctly
+3. **Test Endpoints**: Use curl to test API endpoints directly
+4. **Consult Docs**: See `/docs/API_ENDPOINTS_FOR_DETECTION.md` for API specs
+5. **Fallback**: System automatically falls back to Supabase if API fails
+
+## Next Steps
+
+1. ✅ Client code is ready
+2. ⏳ Implement backend API endpoints (see `/docs/API_ENDPOINTS_FOR_DETECTION.md`)
+3. ⏳ Test with production data
+4. ⏳ Monitor and optimize performance
+5. ⏳ Phase out direct Supabase access
+
+---
+
+**Current Status**: Client refactored and ready. Backend endpoints need implementation.

package/docs/REFACTORING_SUMMARY_DETECT_PEDIMENTOS.md

@@ -0,0 +1,200 @@
+# Refactoring Summary: detectPedimentosInDatabase API Migration
+
+## Overview
+
+Successfully refactored the `detectPedimentosInDatabase` method to use API service instead of direct Supabase client access. This provides better abstraction, security, and maintainability while maintaining backward compatibility.
+
+## Changes Made
+
+### 1. ApiUploadService.js - New API Methods
+
+Added two new methods to communicate with backend API endpoints:
+
+#### `fetchPdfRecordsForDetection(options)`
+- **Purpose**: Fetch PDF records ready for pedimento detection
+- **Endpoint**: `GET /api/uploader/pdf-records`
+- **Parameters**:
+  - `offset`: Pagination offset (default: 0)
+  - `limit`: Number of records to fetch (default: 100)
+- **Query Filters**:
+  - `status=fs-stats`
+  - `file_extension=pdf`
+  - `is_like_simplificado=true`
+- **Returns**: `{ data: Array|null, error: Error|null }`
+
+#### `batchUpdateDetectionResults(updates)`
+- **Purpose**: Batch update detection results for multiple records
+- **Endpoint**: `PATCH /api/uploader/batch-update-detection`
+- **Parameters**:
+  - `updates`: Array of update objects with `{ id, status, document_type, num_pedimento, arela_path, year, rfc, message }`
+- **Returns**: `{ success: boolean, updated: number, errors: Array }`
+
+### 2. DatabaseService.js - Refactored Detection Logic
+
+Updated `detectPedimentosInDatabase` method with the following improvements:
+
+#### Dual-Mode Operation
+- **Primary**: Uses API service when available
+- **Fallback**: Uses direct Supabase client if API is unavailable
+- **Detection**: Automatically checks which service is available at runtime
+
+#### Key Changes
+1. **Service Detection**:
+   ```javascript
+   const uploadService = await uploadServiceFactory.getUploadService();
+   if (uploadService.getServiceName() === 'Arela API') {
+     apiService = uploadService;
+     useApi = true;
+   }
+   ```
+
+2. **Fetch Records** (API or Supabase):
+   ```javascript
+   if (useApi && apiService) {
+     const { data, error } = await apiService.fetchPdfRecordsForDetection({
+       offset,
+       limit: queryBatchSize,
+     });
+   } else {
+     // Fallback to Supabase
+   }
+   ```
+
+3. **Batch Updates** (API or Supabase):
+   - Collects all updates in `batchUpdates` array
+   - Sends as single batch request to API
+   - Falls back to individual Supabase updates if API unavailable
+
+#### Benefits
+- **Reduced Database Load**: Single batch update instead of multiple individual queries
+- **Better Error Handling**: Centralized error management in API layer
+- **Security**: No direct database credentials in client
+- **Maintainability**: Easier to modify query logic in backend
+- **Monitoring**: Better tracking of database operations
+
+### 3. Documentation
+
+Created comprehensive API documentation at `/docs/API_ENDPOINTS_FOR_DETECTION.md` including:
+- Endpoint specifications
+- Request/response formats
+- Security considerations
+- Performance recommendations
+- Testing checklist
+- Migration path
+
+## Backward Compatibility
+
+The refactored code maintains full backward compatibility:
+
+1. **Automatic Fallback**: If API service is not available, automatically falls back to Supabase
+2. **No Breaking Changes**: Same method signature and return format
+3. **Logging**: Clear indication of which service is being used
+4. **Error Handling**: Robust error handling for both modes
+
+## Required Backend Work
+
+To fully enable API mode, implement these endpoints in the backend:
+
+### 1. GET /api/uploader/pdf-records
+Query parameters: `offset`, `limit`, `status`, `file_extension`, `is_like_simplificado`
+
+### 2. PATCH /api/uploader/batch-update-detection
+Request body: `{ updates: [...] }`
+
+See `/docs/API_ENDPOINTS_FOR_DETECTION.md` for complete specifications.
+
+## Testing Recommendations
+
+### Unit Tests
+- [ ] Test API service methods with mock responses
+- [ ] Test fallback to Supabase when API unavailable
+- [ ] Test batch update logic
+
+### Integration Tests
+- [ ] Test with real API endpoints
+- [ ] Test pagination with large datasets
+- [ ] Test error scenarios (network failures, invalid data)
+- [ ] Test performance with 1000+ records
+
+### Manual Testing
+```bash
+# Test detection with API mode
+node src/index.js detect --detect-pdfs
+
+# Monitor logs to verify API usage
+# Should see: "Using API service for PDF detection..."
+```
+
+## Performance Improvements
+
+Expected improvements with API mode:
+1. **Batch Updates**: ~50% reduction in database connections
+2. **Network Efficiency**: Single batch request vs multiple individual requests
+3. **Database Load**: Reduced query overhead
+4. **Scalability**: Better support for concurrent operations
+
+## Migration Path
+
+### Phase 1: Current State ✅
+- Client code refactored
+- API methods implemented
+- Documentation created
+- Backward compatibility maintained
+
+### Phase 2: Backend Implementation (Next)
+- Implement GET /api/uploader/pdf-records
+- Implement PATCH /api/uploader/batch-update-detection
+- Add authentication/authorization
+- Add rate limiting
+
+### Phase 3: Testing & Deployment
+- Test endpoints with client
+- Performance benchmarking
+- Gradual rollout
+- Monitor logs
+
+### Phase 4: Optimization
+- Fine-tune batch sizes
+- Add caching if needed
+- Optimize database queries
+- Consider async processing for large batches
+
+## Code Files Modified
+
+1. `/src/services/upload/ApiUploadService.js`
+   - Added `fetchPdfRecordsForDetection()`
+   - Added `batchUpdateDetectionResults()`
+
+2. `/src/services/DatabaseService.js`
+   - Refactored `detectPedimentosInDatabase()`
+   - Added API service detection and fallback logic
+   - Improved batch update handling
+
+3. `/docs/API_ENDPOINTS_FOR_DETECTION.md` (new)
+   - Complete API endpoint documentation
+
+## Next Steps
+
+1. **Backend Team**: Implement the two required API endpoints
+2. **Testing**: Set up test environment with backend endpoints
+3. **Configuration**: Ensure `ARELA_API_URL` and `ARELA_API_TOKEN` are configured
+4. **Monitoring**: Add metrics to track API vs Supabase usage
+5. **Documentation**: Update user guide with new capabilities
+
+## Questions or Issues?
+
+If you encounter any issues or have questions:
+1. Check logs for service selection messages
+2. Verify API configuration in environment variables
+3. Ensure backend endpoints are deployed and accessible
+4. Review error messages for specific failure points
+
+## Benefits Summary
+
+✅ **Abstraction**: Database logic centralized in backend  
+✅ **Security**: No direct database credentials in client  
+✅ **Maintainability**: Easier to modify without client updates  
+✅ **Performance**: Batch operations reduce overhead  
+✅ **Monitoring**: Better tracking and observability  
+✅ **Backward Compatible**: Seamless fallback to Supabase  
+✅ **Future-Ready**: Prepared for microservices architecture

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arela/uploader",
-  "version": "0.2.13",
+  "version": "1.0.0",
   "description": "CLI to upload files/directories to Arela",
   "bin": {
     "arela": "./src/index.js"
@@ -29,6 +29,7 @@
   "homepage": "https://github.com/inspiraCode/arela-uploader#readme",
   "dependencies": {
     "@supabase/supabase-js": "2.49.4",
+    "chokidar": "^4.0.3",
     "cli-progress": "3.12.0",
     "commander": "13.1.0",
     "dotenv": "16.5.0",
@@ -43,4 +44,4 @@
     "@trivago/prettier-plugin-sort-imports": "5.2.2",
     "prettier": "3.5.3"
   }
-}
+}

package/scripts/cleanup-ds-store.js

@@ -0,0 +1,109 @@
+#!/usr/bin/env node
+
+/**
+ * cleanup-ds-store.js
+ * Remove .DS_Store and other system file records from the uploader table
+ */
+
+import { fileURLToPath } from 'url';
+import path from 'path';
+import logger from './src/services/LoggingService.js';
+import { databaseService } from './src/services/DatabaseService.js';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+
+console.log('\n╔════════════════════════════════════════════════════════════════╗');
+console.log('║         Cleaning System Files from Uploader Table             ║');
+console.log('╚════════════════════════════════════════════════════════════════╝\n');
+
+async function cleanupSystemFiles() {
+  try {
+    const supabase = await databaseService.getSupabaseClient();
+
+    // System file patterns to remove
+    const systemFiles = [
+      '.DS_Store',
+      'Thumbs.db',
+      'desktop.ini',
+      '.directory',
+    ];
+
+    let totalDeleted = 0;
+
+    for (const fileName of systemFiles) {
+      console.log(`\n🔍 Searching for ${fileName}...`);
+      
+      // Find records with this filename
+      const { data: records, error: selectError } = await supabase
+        .from('uploader')
+        .select('*')
+        .eq('filename', fileName);
+
+      if (selectError) {
+        console.error(`❌ Error searching for ${fileName}: ${selectError.message}`);
+        continue;
+      }
+
+      if (!records || records.length === 0) {
+        console.log(`   ✓ No records found for ${fileName}`);
+        continue;
+      }
+
+      console.log(`   Found ${records.length} record(s) for ${fileName}`);
+      
+      // Get the IDs to delete
+      const idsToDelete = records.map(r => r.id);
+      
+      // Delete records
+      const { error: deleteError } = await supabase
+        .from('uploader')
+        .delete()
+        .in('id', idsToDelete);
+
+      if (deleteError) {
+        console.error(`   ❌ Error deleting: ${deleteError.message}`);
+        continue;
+      }
+
+      console.log(`   ✅ Deleted ${records.length} record(s)`);
+      totalDeleted += records.length;
+    }
+
+    // Also delete any records with original_path containing .DS_Store
+    console.log(`\n🔍 Searching for records with .DS_Store in path...`);
+    const { data: pathRecords, error: pathError } = await supabase
+      .from('uploader')
+      .select('*')
+      .ilike('original_path', '%.DS_Store%');
+
+    if (!pathError && pathRecords && pathRecords.length > 0) {
+      console.log(`   Found ${pathRecords.length} record(s)`);
+      const idsToDelete = pathRecords.map(r => r.id);
+      
+      const { error: deleteError } = await supabase
+        .from('uploader')
+        .delete()
+        .in('id', idsToDelete);
+
+      if (!deleteError) {
+        console.log(`   ✅ Deleted ${pathRecords.length} record(s)`);
+        totalDeleted += pathRecords.length;
+      } else {
+        console.error(`   ❌ Error deleting: ${deleteError.message}`);
+      }
+    }
+
+    console.log('\n╔════════════════════════════════════════════════════════════════╗');
+    console.log(`║  ✅ Cleanup Complete: Deleted ${totalDeleted} system file record(s)                  ║`);
+    console.log('╚════════════════════════════════════════════════════════════════╝\n');
+
+    process.exit(0);
+  } catch (error) {
+    console.error('\n❌ ERROR');
+    console.error(error.message);
+    process.exit(1);
+  }
+}
+
+cleanupSystemFiles();

package/scripts/cleanup-system-files.js

@@ -0,0 +1,69 @@
+#!/usr/bin/env node
+
+/**
+ * cleanup-system-files.js
+ * Remove system files (.DS_Store, etc) from the uploader table
+ */
+
+import { fileURLToPath } from 'url';
+import path from 'path';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+
+console.log('\n╔════════════════════════════════════════════════════════════════╗');
+console.log('║           Cleaning System Files from Database                  ║');
+console.log('╚════════════════════════════════════════════════════════════════╝\n');
+
+try {
+  const databaseService = (await import('./src/services/DatabaseService.js')).default;
+  const supabase = await databaseService.getSupabaseClient();
+
+  // System file patterns to remove
+  const systemFilePatterns = ['.DS_Store', '__pycache__', '.pyc', '.swp', '.swo', 'Thumbs.db', 'desktop.ini'];
+
+  console.log('🔍 Searching for system files...\n');
+
+  let totalRemoved = 0;
+
+  for (const pattern of systemFilePatterns) {
+    const { data: records, error } = await supabase
+      .from('uploader')
+      .select('id, filename')
+      .ilike('filename', `%${pattern}%`);
+
+    if (!error && records && records.length > 0) {
+      console.log(`📝 Found ${records.length} record(s) with "${pattern}"`);
+      records.forEach(r => {
+        console.log(`   - ID: ${r.id}, Filename: ${r.filename}`);
+      });
+
+      // Delete them
+      const ids = records.map(r => r.id);
+      const { error: deleteError } = await supabase
+        .from('uploader')
+        .delete()
+        .in('id', ids);
+
+      if (!deleteError) {
+        console.log(`   ✅ Deleted ${records.length} record(s)\n`);
+        totalRemoved += records.length;
+      } else {
+        console.log(`   ❌ Error deleting records: ${deleteError.message}\n`);
+      }
+    }
+  }
+
+  console.log('╔════════════════════════════════════════════════════════════════╗');
+  console.log(`║                    ✅ CLEANUP COMPLETED                        ║`);
+  console.log(`║                                                                ║`);
+  console.log(`║  Total system files removed: ${totalRemoved}`);
+  console.log('╚════════════════════════════════════════════════════════════════╝\n');
+
+  process.exit(0);
+} catch (error) {
+  console.error('\n❌ CLEANUP FAILED');
+  console.error(error.message);
+  console.error(error.stack);
+  process.exit(1);
+}