@arela/uploader 1.0.24 → 1.1.1

package/docs/AUTO_PROCESSING_PIPELINE.md

@@ -0,0 +1,258 @@
+# Automatic 4-Step Processing Pipeline
+
+## Overview
+
+The automatic processing pipeline enables Watch Mode to automatically execute a 4-step workflow whenever a new file is detected in monitored directories. This streamlines the file upload process without manual intervention.
+
+## The 4-Step Pipeline
+
+When auto-processing is enabled and a new file is detected, the pipeline automatically executes:
+
+### Step 1: Stats Collection (`stats --stats-only`)
+- Collects file statistics and metadata
+- Uploads file information to the database (uploads table)
+- Records file properties: size, type, modification date, path
+
+### Step 2: PDF/Pedimento Detection (`detect --detect-pdfs`)
+- Analyzes files to detect "Pedimento Simplificado" documents
+- Identifies which files are simplified customs documents
+- Updates database records with detection results
+- **Important**: Files in the same directory will only be uploaded if a Pedimento Simplificado is detected
+
+### Step 3: Arela Path Propagation (`detect --propagate-arela-path`)
+- Propagates `arela_path` from detected Pedimento records to related files
+- Ensures all documents in the same directory are correctly linked
+- Creates the hierarchical relationship between main and supporting documents
+
+### Step 4: RFC-Based Upload (`upload --upload-by-rfc --folder-structure`)
+- Uploads files based on RFC values from `UPLOAD_RFCS` configuration
+- Uses the configured folder structure for bucket organization
+- Each watched directory can have its own folder structure
+
+## Configuration
+
+### 1. Environment Setup (`.env`)
+
+Configure watched directories with associated folder structures:
+
+```env
+# JSON format: {"directory_path": "folder_structure"}
+WATCH_DIRECTORY_CONFIGS={"../../Documents/2022":"estructura-2022","../../Documents/2023":"estructura-2023"}
+```
+
+Each directory can have a unique folder structure:
+- `../../Documents/2022` → uploads to bucket structure `estructura-2022`
+- `../../Documents/2023` → uploads to bucket structure `estructura-2023`
+
+### 2. CLI Options
+
+Enable auto-processing when running watch mode:
+
+```bash
+# Basic usage with auto-processing enabled
+arela watch --auto-processing
+
+# With custom directories
+arela watch -d "../../Documents/2022,../../Documents/2023" --auto-processing
+
+# With other options
+arela watch --auto-processing -b 10 --debounce 1000 -s batch
+```
+
+### 3. Processing Options
+
+When auto-processing is enabled, you can configure:
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--batch-size` / `-b` | 10 | Files to process per batch in stats collection |
+| `--debounce` | 1000ms | Wait time before processing after file event |
+| `--auto-processing` | - | Enable the 4-step pipeline |
+
+## Workflow Example
+
+### Scenario: New file detected in monitored directory
+
+```
+📄 New file: /Documents/2023/AKS151005E46/invoice.pdf
+    ↓
+⚡ File Event Detected (add)
+    ↓
+┌─────────────────────────────────────────┐
+│ Step 1: Stats Collection                │
+│ → Collects file info                    │
+│ → Updates uploads table                 │
+└─────────────────────────────────────────┘
+    ↓
+┌─────────────────────────────────────────┐
+│ Step 2: PDF Detection                   │
+│ → Searches for Pedimento Simplificado   │
+│ → Updates detection status              │
+└─────────────────────────────────────────┘
+    ↓
+┌─────────────────────────────────────────┐
+│ Step 3: Arela Path Propagation          │
+│ → Links related documents               │
+│ → Propagates metadata                   │
+└─────────────────────────────────────────┘
+    ↓
+┌─────────────────────────────────────────┐
+│ Step 4: RFC Upload                      │
+│ → Identifies RFC from directory/file    │
+│ → Uploads with folder structure         │
+│ → Creates bucket organization           │
+└─────────────────────────────────────────┘
+    ↓
+✅ Processing Complete
+```
+
+## Important Constraints
+
+### Pedimento Detection Requirement
+- If a Pedimento Simplificado is NOT detected in Step 2, documents will not be uploaded
+- Ensure your documents include the proper Pedimento Simplificado format
+- Check database records for detection status:
+  ```bash
+  arela query --ready-files
+  ```
+
+### RFC Configuration
+- Ensure `UPLOAD_RFCS` environment variable contains required RFCs
+- Files from unregistered RFCs will not be processed
+- Example configuration:
+  ```env
+  UPLOAD_RFCS=AKS151005E46|IMS030409FZ0|RDG1107154L7
+  ```
+
+## Monitoring and Debugging
+
+### Check Auto-Processing Status
+
+The watch command logs auto-processing events:
+
+```
+[AutoPipeline] Triggering 4-step processing pipeline for: /path/to/file.pdf
+[AutoPipeline] Step 1/4: Stats collection...
+[AutoPipeline] ✅ Stats collection completed
+[AutoPipeline] Step 2/4: PDF detection...
+[AutoPipeline] ✅ PDF detection completed
+...
+[AutoPipeline] ✅ Pipeline completed successfully (ID: pipeline-xxx-yyy)
+```
+
+### Query Ready Files
+
+Check which files are prepared for upload:
+
+```bash
+arela query --ready-files
+```
+
+### Manual Commands (if needed)
+
+Run individual steps manually:
+
+```bash
+# Stats only
+arela stats --stats-only
+
+# PDF Detection
+arela detect --detect-pdfs
+
+# Path Propagation
+arela detect --propagate-arela-path
+
+# RFC Upload
+arela upload --upload-by-rfc --folder-structure estructura-2023
+```
+
+## Performance Considerations
+
+- **Debounce Time**: Default 1000ms prevents redundant processing
+- **Batch Size**: Affects concurrent processing in Step 1
+- **Pipeline Concurrency**: Only one pipeline runs at a time (prevents conflicts)
+- **Auto-Processing Overhead**: Each new file triggers all 4 steps (approximately 10-30 seconds depending on file count)
+
+## Disabling Auto-Processing
+
+To use watch mode without auto-processing:
+
+```bash
+# Watch mode without auto-processing
+arela watch
+
+# Files detected but not automatically processed
+# Use manual commands instead:
+arela stats --stats-only
+arela detect --detect-pdfs
+arela detect --propagate-arela-path
+arela upload --upload-by-rfc
+```
+
+## Troubleshooting
+
+### Pipeline Not Triggering
+1. Check if `--auto-processing` flag is set
+2. Verify `WATCH_DIRECTORY_CONFIGS` is properly configured in `.env`
+3. Check logs for configuration errors: `--verbose` flag
+
+### Files Not Uploading (Step 4 Failing)
+1. Verify Pedimento was detected: `arela query --ready-files`
+2. Check `UPLOAD_RFCS` configuration
+3. Verify folder structure is valid
+
+### Performance Issues
+- Increase debounce time: `--debounce 2000`
+- Reduce batch size: `-b 5`
+- Check system resources during processing
+
+## Files Modified
+
+- **`.env`**: New `WATCH_DIRECTORY_CONFIGS` format
+- **`src/config/config.js`**: Parser for JSON directory configuration
+- **`src/services/AutoProcessingService.js`**: New service for 4-step pipeline
+- **`src/services/WatchService.js`**: Integration of auto-processing
+- **`src/commands/WatchCommand.js`**: Support for directory configurations
+- **`src/index.js`**: New `--auto-processing` CLI option
+- **`src/utils/WatchEventHandler.js`**: Pipeline invocation support
+
+## API Reference
+
+### WatchService Methods
+
+```javascript
+// Enable automatic processing
+watchService.enableAutoProcessing({ batchSize: 10 });
+
+// Disable automatic processing
+watchService.disableAutoProcessing();
+
+// Check if enabled
+const enabled = watchService.isAutoProcessingEnabled();
+
+// Get stats including pipeline count
+const stats = watchService.getStats();
+// Output: { pipelinesTriggered: 5, ... }
+```
+
+### AutoProcessingService Methods
+
+```javascript
+// Execute the 4-step pipeline
+const result = await autoProcessingService.executeProcessingPipeline({
+  filePath: '/path/to/file.pdf',
+  watchDir: '/watched/directory',
+  folderStructure: 'estructura-2023',
+  batchSize: 10
+});
+
+// Returns:
+// {
+//   pipelineId: 'pipeline-xxx-yyy',
+//   summary: {
+//     success: true,
+//     message: '✅ All 4 steps completed successfully!',
+//     details: { ... }
+//   }
+// }
+```