@arela/uploader 0.2.13 → 1.0.0

package/.env.template

@@ -63,6 +63,72 @@ MAX_CONCURRENT_SOURCES=2
 # MAX_CONCURRENT_SOURCES=1
 # BATCH_DELAY=100
 
+# =============================================================================
+# WATCH MODE CONFIGURATION
+# =============================================================================
+
+# Habilitar watch mode (true/false)
+WATCH_ENABLED=false
+
+# Configuración de directorios a observar (formato JSON)
+# Cada directorio puede tener su propia folderStructure para organizar en el bucket
+# Formato: {"ruta/directorio1":"estructura-1","ruta/directorio2":"estructura-2"}
+# 
+# IMPORTANTE: El folderStructure se usa en el comando:
+# upload --upload-by-rfc --folder-structure <estructura>
+# 
+# Ejemplo:
+WATCH_DIRECTORY_CONFIGS={"../../Documents/2022":"estructura-2022","../../Documents/2023":"estructura-2023"}
+
+# DEPRECATED: Configuración antigua (se mantiene para compatibilidad hacia atrás)
+# WATCH_DIRECTORIES=/ruta/carpeta1,/ruta/carpeta2
+
+# Estrategia de upload (opciones: individual|batch|full-structure)
+# - individual: Sube solo el archivo modificado más reciente
+# - batch: Sube un lote de N archivos recientes
+# - full-structure: Sube la estructura completa de carpetas
+WATCH_STRATEGY=batch
+
+# Debouncing en milisegundos (esperar entre eventos antes de procesar)
+WATCH_DEBOUNCE_MS=1000
+
+# Tamaño de batch para strategy batch
+WATCH_BATCH_SIZE=10
+
+# Usar polling en lugar de eventos nativos del filesystem
+# Útil para sistemas de archivos remotos o NFS
+WATCH_USE_POLLING=false
+
+# Interval de polling en milisegundos (solo si WATCH_USE_POLLING=true)
+WATCH_POLL_INTERVAL=100
+
+# Umbral de estabilidad en ms (esperar a que el archivo deje de cambiar)
+WATCH_STABILITY_THRESHOLD=300
+
+# Patrones a ignorar (separados por coma, se usan como regex)
+WATCH_IGNORE_PATTERNS=*.tmp,*.bak,*.swp
+
+# Detección automática de tipos de documento
+WATCH_AUTO_DETECT=false
+
+# Organización automática de archivos
+WATCH_AUTO_ORGANIZE=false
+
+# =============================================================================
+# WATCH MODE - AUTOMATIC PROCESSING PIPELINE
+# =============================================================================
+# 
+# El pipeline automático ejecuta la siguiente secuencia cuando se detecta un archivo nuevo:
+# 1. Stats Collection   → stats --stats-only (recopila información del archivo)
+# 2. PDF Detection      → detect --detect-pdfs (identifica pedimentos simplificados)
+# 3. Path Propagation   → detect --propagate-arela-path (propaga a documentos relacionados)
+# 4. RFC Upload         → upload --upload-by-rfc --folder-structure (sube con estructura)
+#
+# El pipeline se habilita automáticamente en watch mode y usa la folderStructure
+# definida para cada WATCH_DIRECTORY_CONFIGS
+#
+# Para deshabilitar en CLI, usa: arela watch --no-auto-processing
+
 # =============================================================================
 # LOGGING AND MONITORING
 # =============================================================================

package/README.md

@@ -2,6 +2,14 @@
 
 CLI tool to upload files and directories to Arela API or Supabase Storage with automatic file processing, detection, and organization.
 
+## ✨ What's New in v0.4.0
+
+- 🏢 **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
+- 🔀 **Cross-Tenant Mode**: Read from one API, write to another with `--source-api` and `--target-api`
+- ⚙️ **Dynamic Client Config**: Switch clients by updating `.env` - no code changes needed!
+- 👁️ **Enhanced Watch Mode**: Full cross-tenant support in automatic processing pipeline
+- ⚡ **Optimized Connections**: HTTP Agent with connection pooling for high performance
+
 ## 🚀 OPTIMIZED 4-PHASE WORKFLOW
 
 **New in v0.2.0**: The tool now supports an optimized 4-phase workflow designed for maximum performance when processing large file collections:
@@ -84,6 +92,75 @@ arela --upload-by-rfc        # Phase 4: Upload by RFC
 - 📋 **Upload files by specific RFC values**
 - 🔍 **Propagate arela_path from pedimento documents to related files**
 - ⚡ **4-Phase optimized workflow for maximum performance**
+- 👁️ **Watch Mode** - Monitor directories for changes and upload automatically
+  - Multiple watch strategies (batch, individual, full-structure)
+  - **Multi-tenant and cross-tenant support** ⭐ NEW
+  - Debounce and polling support
+  - Auto-processing pipeline
+  - Dry-run mode for testing
+  - Pattern-based file ignoring
+
+## 🏢 Multi-Tenant API Support
+
+Connect to different API instances: **default**, **agencia**, or **cliente**.
+
+```bash
+# Upload to client API
+arela upload --api cliente --upload-by-rfc
+
+# Collect stats on agencia API
+arela stats --api agencia
+
+# Watch mode with specific API target
+arela watch --api cliente
+```
+
+### Cross-Tenant Mode
+
+Process files from one tenant and upload to another:
+
+```bash
+# Read data from agencia, upload files to client
+arela watch --source-api agencia --target-api cliente
+
+# Same for upload command
+arela upload --source-api agencia --target-api cliente --upload-by-rfc
+```
+
+**How Cross-Tenant Works:**
+| Phase | Description | API Used |
+|-------|-------------|----------|
+| Phase 1 | Stats Collection | `--source-api` |
+| Phase 2 | PDF Detection | `--source-api` |
+| Phase 3 | Path Propagation | `--source-api` |
+| Phase 4 | File Upload | `--target-api` |
+
+### Available API Targets
+
+Only 3 API targets are available: `default`, `agencia`, `cliente`
+
+Configure in your `.env` file:
+
+```env
+# Default API (--api default or no flag)
+ARELA_API_URL=http://localhost:3010
+ARELA_API_TOKEN=your_token
+
+# Agencia API (--api agencia)
+ARELA_API_AGENCIA_URL=http://localhost:4012
+ARELA_API_AGENCIA_TOKEN=your_agencia_token
+
+# Cliente API (--api cliente)
+# Configure the URL/Token for the specific client you need
+ARELA_API_CLIENTE_URL=http://localhost:4014
+ARELA_API_CLIENTE_TOKEN=your_cliente_token
+
+# Examples for different clients:
+# Cliente AUM9207011CA: ARELA_API_CLIENTE_URL=http://localhost:4014
+# Cliente KTJ931117P55: ARELA_API_CLIENTE_URL=http://localhost:4013
+```
+
+> 💡 **Tip**: To switch between clients, just update `ARELA_API_CLIENTE_URL` and `ARELA_API_CLIENTE_TOKEN` in your `.env` file. No code changes needed!
 
 ## Installation
 
@@ -97,78 +174,136 @@ npm install -g @arela/uploader
 
 ```bash
 # Run all phases automatically (most efficient)
-arela --run-all-phases --batch-size 20
+arela upload --run-all-phases --batch-size 20
 
 # Or run phases individually for fine-grained control
-arela --stats-only                    # Phase 1: Filesystem stats only
-arela --detect-pdfs --batch-size 10   # Phase 2: PDF detection
-arela --propagate-arela-path          # Phase 3: Path propagation
-arela --upload-by-rfc --batch-size 5  # Phase 4: RFC-based upload
+arela stats                           # Phase 1: Filesystem stats only
+arela detect                          # Phase 2: PDF detection
+arela detect --propagate-arela-path   # Phase 3: Path propagation
+arela upload --upload-by-rfc          # Phase 4: RFC-based upload
 ```
 
-### Traditional Single-Phase Upload (Legacy)
+### Available Commands
 
-#### Basic Upload with Auto-Processing (API Mode)
+#### 1. **upload** - Upload files to Arela
 ```bash
-arela --batch-size 10 -c 5
-```
+# Basic upload with auto-processing (API Mode)
+arela upload --batch-size 10
 
-### Upload with Auto-Detection of Year/Pedimento
-```bash
-arela --auto-detect-structure --batch-size 10 -c 5
-```
+# Upload with auto-detection of year/pedimento from file paths
+arela upload --auto-detect-structure --batch-size 10
 
-### Upload with Custom Folder Structure
-```bash
-arela --folder-structure "2024/4023260" --batch-size 10 -c 5
+# Upload with custom folder structure
+arela upload --folder-structure "2024/4023260" --batch-size 10
+
+# Upload to Supabase directly (skip API)
+arela upload --force-supabase --prefix "my-folder"
+
+# Upload files by specific RFC values
+arela upload --upload-by-rfc --batch-size 5
+
+# Upload RFC files with custom folder prefix
+arela upload --upload-by-rfc --folder-structure "palco" --batch-size 5
+
+# Upload RFC files with nested folder structure
+arela upload --upload-by-rfc --folder-structure "2024/Q1/processed" --batch-size 15
+
+# Upload with performance statistics
+arela upload --batch-size 10 --show-stats
+
+# Upload with client path tracking
+arela upload --client-path "/client/documents" --batch-size 10
 ```
 
-### Upload with Directory Structure Preservation
+#### 2. **stats** - Collect file statistics without uploading
 ```bash
-arela --batch-size 10 -c 5 --preserve-structure
+# Collect filesystem statistics only (Phase 1)
+arela stats --batch-size 10
+
+# Stats with custom folder organization
+arela stats --folder-structure "2023/3019796" --batch-size 10
+
+# Stats with client path tracking
+arela stats --client-path "/client/documents" --batch-size 10
 ```
 
-### Upload to Supabase Directly (Skip API)
+#### 3. **detect** - Run document detection and path propagation
 ```bash
-arela --force-supabase -p "my-folder"
+# Run PDF detection on existing database records (Phase 2)
+arela detect --batch-size 10
+
+# Propagate arela_path from pedimento records to related files (Phase 3)
+arela detect --propagate-arela-path
 ```
 
-### Upload Files by Specific RFC Values
+#### 4. **watch** - Monitor directories and upload automatically ⭐ NEW
 ```bash
-# Upload all files associated with specific RFCs
-arela --upload-by-rfc --batch-size 5
+# Watch directories for changes with automatic upload
+arela watch --directories "/path/to/watch1,/path/to/watch2"
 
-# Upload RFC files with custom folder prefix
-arela --upload-by-rfc --folder-structure "palco" --batch-size 5
+# Watch with specific API target (single tenant)
+arela watch --api cliente
 
-# Upload RFC files with nested folder structure
-arela --upload-by-rfc --folder-structure "2024/client1/pedimentos" --batch-size 5
-```
+# Watch with cross-tenant mode (read from agencia, upload to client)
+arela watch --source-api agencia --target-api cliente
 
-### Propagate Arela Path from Pedimentos to Related Files
-```bash
-# Copy arela_path from pedimento_simplificado records to related files
-arela --propagate-arela-path
+# Watch with custom upload strategy (default: batch)
+arela watch --directories "/path/to/watch" --strategy individual
+arela watch --directories "/path/to/watch" --strategy full-structure
+
+# Watch with custom debounce delay (default: 1000ms)
+arela watch --directories "/path/to/watch" --debounce 2000
+
+# Watch with automatic 4-step pipeline
+arela watch --directories "/path/to/watch" --auto-processing --batch-size 10
+
+# Watch with polling instead of native file system events
+arela watch --directories "/path/to/watch" --poll 5000
+
+# Watch with pattern ignoring
+arela watch --directories "/path/to/watch" --ignore "node_modules,*.log,*.tmp"
+
+# Watch in dry-run mode (simulate without uploading)
+arela watch --directories "/path/to/watch" --dry-run
+
+# Watch with verbose logging
+arela watch --directories "/path/to/watch" --verbose
 ```
 
-### Stats-Only Mode (No Upload)
+**Watch Strategies:**
+- `batch` **(default)**: Groups files and uploads periodically
+- `individual`: Uploads each file immediately as it changes
+- `full-structure`: Preserves directory structure during upload
+
+**Multi-Tenant Options:**
+- `--api <target>`: Use a single API for all operations
+- `--source-api <target>`: API for reading/processing (phases 1-3)
+- `--target-api <target>`: API for uploading (phase 4)
+
+#### 5. **query** - Query database for file status
 ```bash
-# Only process file stats and insert to database, don't upload
-arela --stats-only --folder-structure "2023/3019796"
+# Show files ready for upload
+arela query --ready-files
 ```
 
-### Upload with Performance Statistics
+#### 6. **config** - Show current configuration
 ```bash
-arela --batch-size 10 -c 5 --show-stats
+# Display all configuration settings
+arela config
 ```
 
-### Upload with Client Path Tracking
+### Legacy Syntax (Still Supported)
+
+The old flag-based syntax is still supported for backward compatibility:
+
 ```bash
-arela --client-path "/client/documents" --batch-size 10 -c 5
+# These are equivalent to the commands above
+arela --stats-only                    # Same as: arela stats
+arela --detect-pdfs                   # Same as: arela detect
+arela --propagate-arela-path          # Same as: arela detect --propagate-arela-path
+arela --upload-by-rfc                 # Same as: arela upload --upload-by-rfc
 ```
 
-### Options
-
 #### Phase Control
 - `--stats-only`: **Phase 1** - Only collect filesystem stats (no file reading)
 - `--detect-pdfs`: **Phase 2** - Process PDF files for pedimento-simplificado detection
@@ -176,36 +311,71 @@ arela --client-path "/client/documents" --batch-size 10 -c 5
 - `--upload-by-rfc`: **Phase 4** - Upload files based on RFC values from UPLOAD_RFCS
 - `--run-all-phases`: **All Phases** - Run complete optimized workflow
 
-#### Performance & Configuration
-- `-c, --concurrency <number>`: Files per batch for processing (default: 10)
-- `--batch-size <number>`: API batch size (default: 10)
-- `--show-stats`: Show detailed processing statistics
+#### Global Options (all commands)
+- `-v, --verbose`: Enable verbose logging
+- `--clear-log`: Clear the log file before starting
+- `-h, --help`: Display help information
+- `--version`: Display version number
 
-#### Upload Configuration
-- `-p, --prefix <prefix>`: Prefix path in bucket (default: "")
-- `-b, --bucket <bucket>`: Bucket name override
-- `--force-supabase`: Force direct Supabase upload (skip API)
-- `--no-auto-detect`: Disable automatic file detection (API mode only)
-- `--no-auto-organize`: Disable automatic file organization (API mode only)
-- `--preserve-structure`: **Preserve original directory structure when using auto-organize**
-- `--folder-structure <structure>`: **Custom folder structure** (e.g., "2024/4023260" or "cliente1/pedimentos")
-- `--auto-detect-structure`: **Automatically detect year/pedimento from file paths**
+#### Upload Command Options
+- `-b, --batch-size <size>`: API batch size (default: 10)
+- `--folder-structure <structure>`: Custom folder structure (e.g., "2024/4023260")
 - `--client-path <path>`: Client path for metadata tracking
+- `--auto-detect-structure`: Automatically detect year/pedimento from file paths
+- `--auto-detect`: Enable automatic document type detection
+- `--auto-organize`: Enable automatic file organization
+- `--force-supabase`: Force direct Supabase upload (skip API)
+- `--skip-processed`: Skip files already processed
+- `--show-stats`: Show performance statistics
+- `--upload-by-rfc`: Upload files based on RFC values from UPLOAD_RFCS
+- `--run-all-phases`: Run all processing phases sequentially
 
-#### Legacy Options
-- `--no-detect`: Disable document type detection in stats-only mode
-- `-v, --version`: Display version number
-- `-h, --help`: Display help information
+#### Stats Command Options
+- `-b, --batch-size <size>`: Batch size for processing (default: 10)
+- `--client-path <path>`: Client path for metadata tracking
+- `--show-stats`: Show performance statistics
+
+#### Detect Command Options
+- `-b, --batch-size <size>`: Batch size for PDF detection (default: 10)
+- `--propagate-arela-path`: Propagate arela_path from pedimento records to related files
+
+#### Watch Command Options
+- `-d, --directories <paths>`: **Comma-separated directories to watch** (required)
+- `-s, --strategy <strategy>`: Upload strategy (default: batch)
+  - `batch`: Groups files and uploads periodically
+  - `individual`: Uploads each file immediately
+  - `full-structure`: Preserves directory structure
+- `--api <target>`: Use a single API target for all operations
+- `--source-api <target>`: API for reading/processing (phases 1-3)
+- `--target-api <target>`: API for uploading (phase 4)
+- `--debounce <ms>`: Debounce delay in milliseconds (default: 1000)
+- `-b, --batch-size <size>`: Batch size for uploads (default: 10)
+- `--poll <ms>`: Use polling instead of native file system events (interval in ms)
+- `--ignore <patterns>`: Comma-separated patterns to ignore
+- `--auto-detect`: Enable automatic document type detection
+- `--auto-organize`: Enable automatic file organization
+- `--auto-processing`: Enable automatic 4-step pipeline (stats, detect, propagate, upload)
+- `--dry-run`: Simulate changes without uploading
+- `--verbose`: Enable verbose logging
 
 ## Environment Variables
 
 Create a `.env` file in your project root:
 
 ```env
-# For API Mode (recommended)
+# Default API (--api default or no flag)
 ARELA_API_URL=http://localhost:3010
 ARELA_API_TOKEN=your_api_token
 
+# Agencia API (--api agencia)
+ARELA_API_AGENCIA_URL=http://localhost:4012
+ARELA_API_AGENCIA_TOKEN=your_agencia_token
+
+# Cliente API (--api cliente)
+# Configure for the specific client you need
+ARELA_API_CLIENTE_URL=http://localhost:4014
+ARELA_API_CLIENTE_TOKEN=your_cliente_token
+
 # For Direct Supabase Mode (fallback)
 SUPABASE_URL=your_supabase_url
 SUPABASE_KEY=your_supabase_anon_key
@@ -218,18 +388,26 @@ UPLOAD_SOURCES=folder1|folder2|file.pdf
 # RFC-based Upload Configuration
 # Pipe-separated list of RFCs to upload files for
 UPLOAD_RFCS=MMJ0810145N1|ABC1234567XY|DEF9876543ZZ
+
+# Watch Mode Configuration (JSON format)
+WATCH_DIRECTORY_CONFIGS={"../../Documents/2022":"palco","../../Documents/2023":"palco"}
 ```
 
 **Environment Variable Details:**
 
-- `ARELA_API_URL`: Base URL for the Arela API service
-- `ARELA_API_TOKEN`: Authentication token for API access
+- `ARELA_API_URL`: Base URL for default API service
+- `ARELA_API_AGENCIA_URL`: URL for agencia API
+- `ARELA_API_CLIENTE_URL`: URL for client API (configure per client)
+- `ARELA_API_TOKEN`: Authentication token for default API
+- `ARELA_API_AGENCIA_TOKEN`: Token for agencia API
+- `ARELA_API_CLIENTE_TOKEN`: Token for client API
 - `SUPABASE_URL`: Your Supabase project URL
 - `SUPABASE_KEY`: Supabase anonymous key for direct uploads
 - `SUPABASE_BUCKET`: Target bucket name in Supabase Storage
 - `UPLOAD_BASE_PATH`: Root directory containing files to upload
 - `UPLOAD_SOURCES`: Pipe-separated list of folders/files to process
 - `UPLOAD_RFCS`: Pipe-separated list of RFC values for targeted uploads
+- `WATCH_DIRECTORY_CONFIGS`: JSON mapping directories to folder structures
 
 ## RFC-Based File Upload
 
@@ -434,7 +612,25 @@ The tool maintains comprehensive logs both locally and remotely:
 
 ## Version History
 
-**v2.0.0** - Latest Release
+**v0.4.0** - Current Release 🆕
+- ✨ **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
+- ✨ **Cross-Tenant Mode**: Read from one API, upload to another
+- ✨ **Dynamic Client Config**: Change client by updating `.env` (no code changes)
+- ✨ New `--api` flag for single API target
+- ✨ New `--source-api` flag for source API (phases 1-3)
+- ✨ New `--target-api` flag for target API (phase 4)
+- ✨ `WATCH_DIRECTORY_CONFIGS` environment variable for watch mode
+- 🔧 Enhanced pipeline routing for cross-tenant operations
+- 📝 Simplified documentation for multi-tenant configuration
+
+**v0.3.0** - Watch Mode Release
+- ✨ Added watch command with chokidar integration
+- ✨ Automatic 4-step pipeline (stats → detect → propagate → upload)
+- ✨ Multiple upload strategies (batch, individual, full-structure)
+- ✨ Configurable debounce and polling options
+- 🔧 Signal handling for graceful shutdown
+
+**v0.2.0** - Pipeline Automation
 - ✨ Added smart year/pedimento auto-detection from file paths
 - ✨ Custom folder structure support with `--folder-structure` option
 - ✨ Client path tracking with `--client-path` option
@@ -446,6 +642,11 @@ The tool maintains comprehensive logs both locally and remotely:
 - 📝 Comprehensive logging with SANITIZED status
 - 🔧 Memory optimization for large file processing
 
+**v0.1.0** - Initial Release
+- 📦 Basic upload functionality
+- 🔌 API and Supabase direct mode support
+- 📂 RFC-based file upload
+
 ## Troubleshooting
 
 **Connection Issues:**