@arela/uploader 1.0.11 → 1.0.13

package/README.md

@@ -1,673 +1,340 @@
 # arela-uploader
 
-CLI tool to upload files and directories to Arela API or Supabase Storage with automatic file processing, detection, and organization.
+CLI tool to scan, detect, and upload files to the Arela API with multi-tenant support, automatic document detection, and directory watching.
 
-## ✨ What's New in v0.4.0
-
-- 🏢 **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
-- 🔀 **Cross-Tenant Mode**: Read from one API, write to another with `--source-api` and `--target-api`
-- ⚙️ **Dynamic Client Config**: Switch clients by updating `.env` - no code changes needed!
-- 👁️ **Enhanced Watch Mode**: Full cross-tenant support in automatic processing pipeline
-- ⚡ **Optimized Connections**: HTTP Agent with connection pooling for high performance
-
-## 🚀 OPTIMIZED 4-PHASE WORKFLOW
-
-**New in v0.2.0**: The tool now supports an optimized 4-phase workflow designed for maximum performance when processing large file collections:
-
-### Phase 1: Filesystem Stats Collection 📊
-```bash
-arela --stats-only
-```
-- ⚡ **ULTRA FAST**: Only reads filesystem metadata (no file content)
-- 📈 **Bulk database operations**: Processes 1000+ files per batch
-- 🔄 **Upsert optimization**: Handles duplicates efficiently
-- 💾 **Minimal memory usage**: No file content loading
+## Installation
 
-### Phase 2: PDF Detection 🔍
 ```bash
-arela --detect-pdfs
+npm install -g @arela/uploader
 ```
-- 🎯 **Targeted processing**: Only processes PDF files from database
-- � **Pedimento-simplificado detection**: Extracts RFC, pedimento numbers, and metadata
-- 🔄 **Batched processing**: Handles large datasets efficiently
-- 📊 **Progress tracking**: Real-time detection statistics
 
-### Phase 3: Path Propagation �📁
-```bash
-arela --propagate-arela-path
-```
-- 🎯 **Smart path copying**: Propagates arela_path from pedimento documents to related files
-- 📦 **Batch updates**: Processes files in groups for optimal database performance
-- 🔗 **Relationship mapping**: Links supporting documents to their pedimento
+## Quick Start
 
-### Phase 4: RFC-based Upload 🚀
-```bash
-arela --upload-by-rfc
-```
-- 🎯 **Targeted uploads**: Only uploads files for specified RFCs
-- 📋 **Supporting documents**: Includes all related files, not just pedimentos
-- 🏗️ **Structure preservation**: Maintains proper folder hierarchy
+The recommended workflow runs 4 phases in sequence:
 
-### Combined Workflow 🎯
 ```bash
-# Run all 4 phases in sequence (recommended)
-arela --run-all-phases
-
-# Or run phases individually for more control
-arela --stats-only           # Phase 1: Collect filesystem stats
-arela --detect-pdfs          # Phase 2: Detect pedimento documents  
-arela --propagate-arela-path # Phase 3: Propagate paths to related files
-arela --upload-by-rfc        # Phase 4: Upload by RFC
+arela scan                # 1. Discover files and register metadata
+arela identify            # 2. Detect document types from PDF content
+arela propagate           # 3. Propagate paths to related files
+arela push                # 4. Upload files to storage
 ```
 
-### Performance Benefits
-
-**Before optimization** (single phase with detection):
-- 🐌 Read every file for detection
-- 💾 High memory usage
-- 🔄 Slow database operations
-- ❌ Process unsupported files
-
-**After optimization** (4-phase approach):
-- ⚡ **10x faster**: Phase 1 only reads filesystem metadata
-- 📊 **Bulk operations**: Database inserts up to 1000 records per batch
-- 🎯 **Targeted processing**: Phase 2 only processes PDFs needing detection
-- 💾 **Memory efficient**: No unnecessary file content loading
-- 🔄 **Optimized I/O**: Separates filesystem, database, and network operations
-
-## Features
-
-- 📁 Upload entire directories or individual files
-- 🤖 **Automatic file detection and organization** (API mode)
-- 🗂️ **Smart year/pedimento auto-detection from file paths**
-- 🏗️ **Custom folder structure support**
-- 🔄 Automatic file renaming to handle problematic characters
-- 📝 Comprehensive logging (local and remote)
-- ⚡ Retry mechanism for failed uploads
-- 🎯 Skip duplicate files automatically
-- 📊 Progress bars and detailed summaries
-- 📂 **Preserve directory structure with auto-organization**
-- 🚀 **Batch processing with configurable concurrency**
-- 🔧 **Performance optimizations with caching**
-- 📋 **Upload files by specific RFC values**
-- 🔍 **Propagate arela_path from pedimento documents to related files**
-- ⚡ **4-Phase optimized workflow for maximum performance**
-- 👁️ **Watch Mode** - Monitor directories for changes and upload automatically
-  - Multiple watch strategies (batch, individual, full-structure)
-  - **Multi-tenant and cross-tenant support** ⭐ NEW
-  - Debounce and polling support
-  - Auto-processing pipeline
-  - Dry-run mode for testing
-  - Pattern-based file ignoring
-
-## 🏢 Multi-Tenant API Support
-
-Connect to different API instances: **default**, **agencia**, or **cliente**.
+## Commands
 
-```bash
-# Upload to client API
-arela upload --api cliente --upload-by-rfc
+### `arela scan`
 
-# Collect stats on agencia API
-arela stats --api agencia
+Scan the filesystem and register file metadata via the API. Supports streaming discovery and multi-level directory partitioning.
 
-# Watch mode with specific API target
-arela watch --api cliente
+```bash
+arela scan                     # Basic scan
+arela scan --api agencia       # Scan using agencia API
+arela scan --count-first       # Count files first for progress %
+arela scan --no-stream         # Synchronous discovery (no streaming)
 ```
 
-### Cross-Tenant Mode
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target (`default`, `agencia`, `cliente`) | `default` |
+| `--count-first` | Count files before scanning for progress tracking | — |
+| `--no-stream` | Use synchronous file discovery | — |
 
-Process files from one tenant and upload to another:
+### `arela identify`
 
-```bash
-# Read data from agencia, upload files to client
-arela watch --source-api agencia --target-api cliente
+Detect document types (e.g. pedimento simplificado) from PDF content using pattern matchers. Runs against existing database records.
 
-# Same for upload command
-arela upload --source-api agencia --target-api cliente --upload-by-rfc
+```bash
+arela identify                        # Identify documents
+arela identify --batch-size 200       # Larger batches
+arela identify --show-stats           # Show performance stats
 ```
 
-**How Cross-Tenant Works:**
-| Phase | Description | API Used |
-|-------|-------------|----------|
-| Phase 1 | Stats Collection | `--source-api` |
-| Phase 2 | PDF Detection | `--source-api` |
-| Phase 3 | Path Propagation | `--source-api` |
-| Phase 4 | File Upload | `--target-api` |
-
-### Available API Targets
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target | `default` |
+| `-b, --batch-size <size>` | Files per batch | `100` |
+| `--show-stats` | Show performance statistics | — |
 
-Only 3 API targets are available: `default`, `agencia`, `cliente`
+### `arela propagate`
 
-Configure in your `.env` file:
+Propagate `arela_path` from identified pedimento records to related files in the same directory.
 
-```env
-# Default API (--api default or no flag)
-ARELA_API_URL=http://localhost:3010
-ARELA_API_TOKEN=your_token
-
-# Agencia API (--api agencia)
-ARELA_API_AGENCIA_URL=http://localhost:4012
-ARELA_API_AGENCIA_TOKEN=your_agencia_token
-
-# Cliente API (--api cliente)
-# Configure the URL/Token for the specific client you need
-ARELA_API_CLIENTE_URL=http://localhost:4014
-ARELA_API_CLIENTE_TOKEN=your_cliente_token
-
-# Examples for different clients:
-# Cliente AUM9207011CA: ARELA_API_CLIENTE_URL=http://localhost:4014
-# Cliente KTJ931117P55: ARELA_API_CLIENTE_URL=http://localhost:4013
+```bash
+arela propagate                       # Propagate paths
+arela propagate --batch-size 100      # Process 100 pedimentos per batch
+arela propagate --show-stats          # Show statistics
 ```
 
-> 💡 **Tip**: To switch between clients, just update `ARELA_API_CLIENTE_URL` and `ARELA_API_CLIENTE_TOKEN` in your `.env` file. No code changes needed!
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target | `default` |
+| `-b, --batch-size <size>` | Pedimentos per batch | `50` |
+| `--show-stats` | Show performance statistics | — |
 
-## Installation
+### `arela push`
+
+Upload files to Arela storage, filtered by RFC and/or year. Supports cross-tenant mode.
 
 ```bash
-npm install -g @arela/uploader
+arela push                                          # Upload all files with arela_path
+arela push --rfcs RFC1,RFC2 --years 2023,2024       # Filter by RFC and year
+arela push --source-api agencia --target-api cliente # Cross-tenant upload
+arela push --folder-structure "prefix/path"          # Add storage path prefix
+arela push --no-auto-organize                        # Disable auto-organization
 ```
 
-## Usage
-
-### 🚀 Optimized 4-Phase Workflow (Recommended)
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target | `default` |
+| `--scan-api <target>` | API for reading scan data | `default` |
+| `--push-api <target>` | API for uploading files | — |
+| `--source-api <target>` | Source API (cross-tenant) | — |
+| `--target-api <target>` | Target API (cross-tenant) | — |
+| `-b, --batch-size <size>` | Files to fetch per batch | `100` |
+| `--upload-batch-size <size>` | Concurrent uploads | `10` |
+| `--rfcs <rfcs>` | Comma-separated RFCs (overrides `PUSH_RFCS`) | — |
+| `--years <years>` | Comma-separated years (overrides `PUSH_YEARS`) | — |
+| `--folder-structure <path>` | Storage path prefix | — |
+| `--no-auto-organize` | Disable automatic file organization | — |
+| `--show-stats` | Show performance statistics | — |
+
+### `arela upload`
+
+Legacy upload command with multiple modes (stats-only, RFC-based, run-all-phases).
 
 ```bash
-# Run all phases automatically (most efficient)
-arela upload --run-all-phases --batch-size 20
-
-# Or run phases individually for fine-grained control
-arela stats                           # Phase 1: Filesystem stats only
-arela detect                          # Phase 2: PDF detection
-arela detect --propagate-arela-path   # Phase 3: Path propagation
-arela upload --upload-by-rfc          # Phase 4: RFC-based upload
+arela upload --batch-size 10                        # Basic upload
+arela upload --upload-by-rfc                        # Upload by RFC
+arela upload --run-all-phases                       # Run all phases
+arela upload --force-supabase --prefix "folder"     # Direct Supabase upload
+arela upload --folder-structure "2024/pedimentos"   # Custom folder structure
+arela upload --auto-detect-structure                # Auto-detect year/pedimento from paths
 ```
 
-### Available Commands
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target | `default` |
+| `--source-api <target>` | Source API (cross-tenant) | — |
+| `--target-api <target>` | Target API (cross-tenant) | — |
+| `-b, --batch-size <size>` | Files per batch | `10` |
+| `-p, --prefix <prefix>` | Prefix for uploaded files | — |
+| `--folder-structure <structure>` | Custom folder structure | — |
+| `--client-path <path>` | Override client path for metadata | — |
+| `--auto-detect-structure` | Auto-detect folder structure from paths | — |
+| `--auto-detect` | Enable document type detection | — |
+| `--auto-organize` | Enable automatic file organization | — |
+| `--force-supabase` | Force direct Supabase upload (skip API) | — |
+| `--skip-processed` | Skip already-processed files | — |
+| `--show-stats` | Show performance statistics | — |
+| `--upload-by-rfc` | Upload based on RFC values from `UPLOAD_RFCS` | — |
+| `--run-all-phases` | Run all processing phases sequentially | — |
+
+### `arela watch`
+
+Monitor directories for file changes and process them automatically.
 
-#### 1. **upload** - Upload files to Arela
 ```bash
-# Basic upload with auto-processing (API Mode)
-arela upload --batch-size 10
-
-# Upload with auto-detection of year/pedimento from file paths
-arela upload --auto-detect-structure --batch-size 10
-
-# Upload with custom folder structure
-arela upload --folder-structure "2024/4023260" --batch-size 10
-
-# Upload to Supabase directly (skip API)
-arela upload --force-supabase --prefix "my-folder"
-
-# Upload files by specific RFC values
-arela upload --upload-by-rfc --batch-size 5
-
-# Upload RFC files with custom folder prefix
-arela upload --upload-by-rfc --folder-structure "palco" --batch-size 5
-
-# Upload RFC files with nested folder structure
-arela upload --upload-by-rfc --folder-structure "2024/Q1/processed" --batch-size 15
-
-# Upload with performance statistics
-arela upload --batch-size 10 --show-stats
-
-# Upload with client path tracking
-arela upload --client-path "/client/documents" --batch-size 10
+arela watch -d "/path/to/dir1,/path/to/dir2"       # Watch directories
+arela watch --api cliente                           # Watch with specific API
+arela watch --source-api agencia --target-api cliente  # Cross-tenant watch
+arela watch --strategy individual                   # Upload each file immediately
+arela watch --auto-processing --batch-size 10       # Auto pipeline (scan→identify→propagate→push)
+arela watch --dry-run                               # Simulate without uploading
+arela watch --poll 5000                             # Use polling (for NFS/remote FS)
 ```
 
-#### 2. **stats** - Collect file statistics without uploading
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--api <target>` | API target | `default` |
+| `--source-api <target>` | Source API (cross-tenant) | — |
+| `--target-api <target>` | Target API (cross-tenant) | — |
+| `-d, --directories <paths>` | Comma-separated directories to watch | — |
+| `-s, --strategy <strategy>` | `batch`, `individual`, or `full-structure` | `batch` |
+| `--debounce <ms>` | Debounce delay in milliseconds | `1000` |
+| `-b, --batch-size <size>` | Files per batch | `10` |
+| `--poll <ms>` | Use polling with interval in ms | — |
+| `--ignore <patterns>` | Comma-separated patterns to ignore | — |
+| `--auto-detect` | Enable document type detection | — |
+| `--auto-organize` | Enable file organization | — |
+| `--auto-processing` | Enable automatic 4-step pipeline | — |
+| `--dry-run` | Simulate without uploading | — |
+| `--verbose` | Verbose logging | — |
+
+**Watch strategies:**
+- **`batch`** (default) — Groups files and uploads periodically
+- **`individual`** — Uploads each file immediately on change
+- **`full-structure`** — Preserves full directory structure during upload
+
+### `arela stats`
+
+Collect filesystem statistics without uploading (legacy, wraps upload in stats-only mode).
+
 ```bash
-# Collect filesystem statistics only (Phase 1)
 arela stats --batch-size 10
-
-# Stats with custom folder organization
-arela stats --folder-structure "2023/3019796" --batch-size 10
-
-# Stats with client path tracking
-arela stats --client-path "/client/documents" --batch-size 10
 ```
 
-#### 3. **detect** - Run document detection and path propagation
-```bash
-# Run PDF detection on existing database records (Phase 2)
-arela detect --batch-size 10
+### `arela detect`
 
-# Propagate arela_path from pedimento records to related files (Phase 3)
-arela detect --propagate-arela-path
-```
+Legacy document detection command. Prefer `arela identify` and `arela propagate` instead.
 
-#### 4. **watch** - Monitor directories and upload automatically ⭐ NEW
 ```bash
-# Watch directories for changes with automatic upload
-arela watch --directories "/path/to/watch1,/path/to/watch2"
-
-# Watch with specific API target (single tenant)
-arela watch --api cliente
-
-# Watch with cross-tenant mode (read from agencia, upload to client)
-arela watch --source-api agencia --target-api cliente
-
-# Watch with custom upload strategy (default: batch)
-arela watch --directories "/path/to/watch" --strategy individual
-arela watch --directories "/path/to/watch" --strategy full-structure
-
-# Watch with custom debounce delay (default: 1000ms)
-arela watch --directories "/path/to/watch" --debounce 2000
-
-# Watch with automatic 4-step pipeline
-arela watch --directories "/path/to/watch" --auto-processing --batch-size 10
-
-# Watch with polling instead of native file system events
-arela watch --directories "/path/to/watch" --poll 5000
-
-# Watch with pattern ignoring
-arela watch --directories "/path/to/watch" --ignore "node_modules,*.log,*.tmp"
-
-# Watch in dry-run mode (simulate without uploading)
-arela watch --directories "/path/to/watch" --dry-run
-
-# Watch with verbose logging
-arela watch --directories "/path/to/watch" --verbose
+arela detect --batch-size 10                  # PDF detection
+arela detect --propagate-arela-path           # Path propagation
 ```
 
-**Watch Strategies:**
-- `batch` **(default)**: Groups files and uploads periodically
-- `individual`: Uploads each file immediately as it changes
-- `full-structure`: Preserves directory structure during upload
-
-**Multi-Tenant Options:**
-- `--api <target>`: Use a single API for all operations
-- `--source-api <target>`: API for reading/processing (phases 1-3)
-- `--target-api <target>`: API for uploading (phase 4)
+### `arela query`
 
-#### 5. **query** - Query database for file status
-```bash
-# Show files ready for upload
-arela query --ready-files
-```
+Query the database for file status.
 
-#### 6. **config** - Show current configuration
 ```bash
-# Display all configuration settings
-arela config
+arela query --ready-files     # Show files ready for upload
 ```
 
-### Legacy Syntax (Still Supported)
+### `arela config`
 
-The old flag-based syntax is still supported for backward compatibility:
+Display current configuration for all API targets and settings.
 
 ```bash
-# These are equivalent to the commands above
-arela --stats-only                    # Same as: arela stats
-arela --detect-pdfs                   # Same as: arela detect
-arela --propagate-arela-path          # Same as: arela detect --propagate-arela-path
-arela --upload-by-rfc                 # Same as: arela upload --upload-by-rfc
-```
-
-#### Phase Control
-- `--stats-only`: **Phase 1** - Only collect filesystem stats (no file reading)
-- `--detect-pdfs`: **Phase 2** - Process PDF files for pedimento-simplificado detection
-- `--propagate-arela-path`: **Phase 3** - Propagate arela_path from pedimento records to related files
-- `--upload-by-rfc`: **Phase 4** - Upload files based on RFC values from UPLOAD_RFCS
-- `--run-all-phases`: **All Phases** - Run complete optimized workflow
-
-#### Global Options (all commands)
-- `-v, --verbose`: Enable verbose logging
-- `--clear-log`: Clear the log file before starting
-- `-h, --help`: Display help information
-- `--version`: Display version number
-
-#### Upload Command Options
-- `-b, --batch-size <size>`: API batch size (default: 10)
-- `--folder-structure <structure>`: Custom folder structure (e.g., "2024/4023260")
-- `--client-path <path>`: Client path for metadata tracking
-- `--auto-detect-structure`: Automatically detect year/pedimento from file paths
-- `--auto-detect`: Enable automatic document type detection
-- `--auto-organize`: Enable automatic file organization
-- `--force-supabase`: Force direct Supabase upload (skip API)
-- `--skip-processed`: Skip files already processed
-- `--show-stats`: Show performance statistics
-- `--upload-by-rfc`: Upload files based on RFC values from UPLOAD_RFCS
-- `--run-all-phases`: Run all processing phases sequentially
-
-#### Stats Command Options
-- `-b, --batch-size <size>`: Batch size for processing (default: 10)
-- `--client-path <path>`: Client path for metadata tracking
-- `--show-stats`: Show performance statistics
-
-#### Detect Command Options
-- `-b, --batch-size <size>`: Batch size for PDF detection (default: 10)
-- `--propagate-arela-path`: Propagate arela_path from pedimento records to related files
-
-#### Watch Command Options
-- `-d, --directories <paths>`: **Comma-separated directories to watch** (required)
-- `-s, --strategy <strategy>`: Upload strategy (default: batch)
-  - `batch`: Groups files and uploads periodically
-  - `individual`: Uploads each file immediately
-  - `full-structure`: Preserves directory structure
-- `--api <target>`: Use a single API target for all operations
-- `--source-api <target>`: API for reading/processing (phases 1-3)
-- `--target-api <target>`: API for uploading (phase 4)
-- `--debounce <ms>`: Debounce delay in milliseconds (default: 1000)
-- `-b, --batch-size <size>`: Batch size for uploads (default: 10)
-- `--poll <ms>`: Use polling instead of native file system events (interval in ms)
-- `--ignore <patterns>`: Comma-separated patterns to ignore
-- `--auto-detect`: Enable automatic document type detection
-- `--auto-organize`: Enable automatic file organization
-- `--auto-processing`: Enable automatic 4-step pipeline (stats, detect, propagate, upload)
-- `--dry-run`: Simulate changes without uploading
-- `--verbose`: Enable verbose logging
-
-## Environment Variables
-
-Create a `.env` file in your project root:
-
-```env
-# Default API (--api default or no flag)
-ARELA_API_URL=http://localhost:3010
-ARELA_API_TOKEN=your_api_token
-
-# Agencia API (--api agencia)
-ARELA_API_AGENCIA_URL=http://localhost:4012
-ARELA_API_AGENCIA_TOKEN=your_agencia_token
-
-# Cliente API (--api cliente)
-# Configure for the specific client you need
-ARELA_API_CLIENTE_URL=http://localhost:4014
-ARELA_API_CLIENTE_TOKEN=your_cliente_token
-
-# For Direct Supabase Mode (fallback)
-SUPABASE_URL=your_supabase_url
-SUPABASE_KEY=your_supabase_anon_key
-SUPABASE_BUCKET=your_bucket_name
-
-# Required for both modes
-UPLOAD_BASE_PATH=/path/to/your/files
-UPLOAD_SOURCES=folder1|folder2|file.pdf
-
-# RFC-based Upload Configuration
-# Pipe-separated list of RFCs to upload files for
-UPLOAD_RFCS=MMJ0810145N1|ABC1234567XY|DEF9876543ZZ
-
-# Watch Mode Configuration (JSON format)
-WATCH_DIRECTORY_CONFIGS={"../../Documents/2022":"palco","../../Documents/2023":"palco"}
+arela config
 ```
 
-**Environment Variable Details:**
+### Global Options
 
-- `ARELA_API_URL`: Base URL for default API service
-- `ARELA_API_AGENCIA_URL`: URL for agencia API
-- `ARELA_API_CLIENTE_URL`: URL for client API (configure per client)
-- `ARELA_API_TOKEN`: Authentication token for default API
-- `ARELA_API_AGENCIA_TOKEN`: Token for agencia API
-- `ARELA_API_CLIENTE_TOKEN`: Token for client API
-- `SUPABASE_URL`: Your Supabase project URL
-- `SUPABASE_KEY`: Supabase anonymous key for direct uploads
-- `SUPABASE_BUCKET`: Target bucket name in Supabase Storage
-- `UPLOAD_BASE_PATH`: Root directory containing files to upload
-- `UPLOAD_SOURCES`: Pipe-separated list of folders/files to process
-- `UPLOAD_RFCS`: Pipe-separated list of RFC values for targeted uploads
-- `WATCH_DIRECTORY_CONFIGS`: JSON mapping directories to folder structures
+| Flag | Description |
+|------|-------------|
+| `-v, --verbose` | Enable verbose logging |
+| `--clear-log` | Clear log file before starting |
+| `--version` | Display version number |
+| `-h, --help` | Display help |
 
-## RFC-Based File Upload
+## Multi-Tenant API
 
-The `--upload-by-rfc` feature allows you to upload files to the Arela API based on specific RFC values. This is useful when you want to upload only files associated with certain companies or entities.
+Three API targets are available: `default`, `agencia`, and `cliente`. Configure each in your `.env` file.
 
-### How it works:
-
-1. **Configure RFCs**: Set the `UPLOAD_RFCS` environment variable with pipe-separated RFC values
-2. **Query Database**: The tool searches the Supabase database for files matching the specified RFCs
-3. **Include Supporting Documents**: Finds all files sharing the same `arela_path` as the RFC matches (not just the pedimento files)
-4. **Apply Folder Structure**: Optionally applies custom folder prefix using `--folder-structure`
-5. **Group and Upload**: Files are grouped by their final destination path and uploaded with proper structure
-
-### Folder Structure Options:
+```bash
+# Single-tenant
+arela scan --api cliente
+arela push --api agencia
 
-**Default Behavior** (no `--folder-structure`):
-- Uses original `arela_path`: `CAD890407NK7/2023/3429/070/230734293000421/`
+# Cross-tenant (read from one API, upload to another)
+arela push --source-api agencia --target-api cliente
+```
 
-**With Custom Prefix** (`--folder-structure "palco"`):
-- Results in: `palco/CAD890407NK7/2023/3429/070/230734293000421/`
+| Phase | `--source-api` | `--target-api` |
+|-------|-----------------|-----------------|
+| Scan / Identify / Propagate | ✓ | — |
+| Push / Upload | — | ✓ |
 
-**With Nested Prefix** (`--folder-structure "2024/client1/pedimentos"`):
-- Results in: `2024/client1/pedimentos/CAD890407NK7/2023/3429/070/230734293000421/`
+## Environment Variables
 
-### Prerequisites:
+Copy `.env.template` to `.env` and configure:
 
-- Files must have been previously processed (have entries in the `uploader` table)
-- Files must have `rfc` field populated (from document detection)
-- Files must have `arela_path` populated (from pedimento processing)
-- Original files must still exist at their `original_path` locations
+### API Configuration
 
-### Example:
+```env
+ARELA_API_URL=https://your-arela-api.example.com
+ARELA_API_TOKEN=your-api-token
 
-```bash
-# Set RFCs in environment
-export UPLOAD_RFCS="MMJ0810145N1|ABC1234567XY|DEF9876543ZZ"
+ARELA_API_AGENCIA_URL=https://agencia-api.example.com
+ARELA_API_AGENCIA_TOKEN=your-agencia-token
 
-# Upload files for these RFCs (original folder structure)
-arela --upload-by-rfc --batch-size 5 --show-stats
+ARELA_API_CLIENTE_URL=https://cliente-api.example.com
+ARELA_API_CLIENTE_TOKEN=your-cliente-token
+```
 
-# Upload with custom folder prefix
-arela --upload-by-rfc --folder-structure "palco" --batch-size 10
+### Supabase (fallback)
 
-# Upload with nested organization
-arela --upload-by-rfc --folder-structure "2024/Q1/processed" --batch-size 15
+```env
+SUPABASE_URL=https://your-project.supabase.co
+SUPABASE_KEY=your-supabase-key
+SUPABASE_BUCKET=your-bucket-name
 ```
 
-The tool will:
-- Find all database records matching the specified RFCs
-- Include ALL supporting documents that share the same `arela_path`
-- Apply the optional folder structure prefix if specified
-- Group files by their final destination folder structure
-- Upload each group maintaining the correct Arela folder hierarchy
-- Provide detailed progress and summary statistics
-- Handle large datasets with automatic pagination (no 1000-file limit)
-
-## File Processing Modes
-
-### API Mode (Default)
-When `ARELA_API_URL` and `ARELA_API_TOKEN` are configured:
-- ✅ Automatic file detection and classification
-- ✅ Intelligent file organization
-- ✅ **Smart year/pedimento auto-detection from paths**
-- ✅ **Custom folder structure support**
-- ✅ Batch processing with progress tracking
-- ✅ Advanced error handling and retry logic
-- ✅ **Performance optimizations with file sanitization caching**
-
-### Auto-Detection Features
-The tool can automatically detect year and pedimento numbers from file paths using multiple patterns:
-
-**Pattern 1: Direct Structure**
-```
-/path/to/2024/4023260/file.pdf
-/path/to/pedimentos/2024/4023260/file.pdf
-```
+### Upload & File Sources
 
-**Pattern 2: Named Patterns**
-```
-/path/to/docs/año2024/ped4023260/file.pdf
-/path/to/files/year2024/pedimento4023260/file.pdf
+```env
+UPLOAD_BASE_PATH=/path/to/files
+UPLOAD_SOURCES=folder1|folder2|folder3
+UPLOAD_RFCS=RFC1|RFC2|RFC3
+UPLOAD_YEARS=2023|2024
 ```
 
-**Pattern 3: Loose Detection**
-- Year: Any 4-digit number starting with "202" (2020-2029)
-- Pedimento: Any 4-8 consecutive digits in path
+### Scan Configuration
 
-Use `--auto-detect-structure` to enable automatic detection:
-```bash
-arela --auto-detect-structure --batch-size 10
+```env
+ARELA_COMPANY_SLUG=your_company          # Company identifier
+ARELA_SERVER_ID=nas01                     # Server/NAS identifier
+ARELA_BASE_PATH_LABEL=                    # Optional (auto-derived from base path)
+SCAN_EXCLUDE_PATTERNS=.DS_Store,Thumbs.db,desktop.ini
+SCAN_BATCH_SIZE=2000                      # Records per API call
+SCAN_DIRECTORY_LEVEL=0                    # Directory depth for table partitioning
 ```
 
-### Custom Folder Structure
-Specify a custom organization pattern:
-```bash
-# Static structure
-arela --folder-structure "2024/4023260" --batch-size 10
+### Push Configuration
 
-# Client-based structure  
-arela --folder-structure "cliente1/pedimentos" --batch-size 10
+```env
+PUSH_RFCS=                               # Pipe-separated RFC filter
+PUSH_YEARS=                              # Pipe-separated year filter
+PUSH_BATCH_SIZE=100                      # Fetch batch size
+PUSH_UPLOAD_BATCH_SIZE=10                # Concurrent uploads
+PUSH_BUCKET=arela                        # Storage bucket
+PUSH_FOLDER_STRUCTURE=                   # Prefix for storage paths
 ```
 
-### Directory Structure Preservation
-Use `--preserve-structure` to maintain your original folder structure even with auto-organization:
-
-```bash
-# Without --preserve-structure
-# Files organized by API: bucket/filename.pdf
+### Performance Tuning
 
-# With --preserve-structure  
-# Files keep structure: bucket/2024/4023260/filename.pdf
-arela --preserve-structure --batch-size 10
+```env
+MAX_API_CONNECTIONS=10                   # Match your API replica count
+API_CONNECTION_TIMEOUT=60000             # Timeout in ms
+API_MAX_RETRIES=3
+API_RETRY_EXPONENTIAL_BACKOFF=true
+BATCH_SIZE=100
+BATCH_DELAY=0
+MAX_CONCURRENT_SOURCES=2
 ```
 
-### Supabase Direct Mode (Fallback)
-When API is unavailable or `--force-supabase` is used:
-- ✅ Direct upload to Supabase Storage
-- ✅ File sanitization and renaming
-- ✅ Basic progress tracking
-- ✅ **Optimized sanitization with pre-compiled regex patterns**
-- ✅ **Performance caching for file name sanitization**
+### Watch Mode
 
-## Performance Features
+```env
+WATCH_ENABLED=false
+WATCH_DIRECTORY_CONFIGS={"../docs/2023":"prefix-2023","../docs/2024":"prefix-2024"}
+WATCH_STRATEGY=batch                     # batch|individual|full-structure
+WATCH_DEBOUNCE_MS=1000
+WATCH_BATCH_SIZE=10
+WATCH_USE_POLLING=false
+WATCH_POLL_INTERVAL=100
+WATCH_STABILITY_THRESHOLD=300
+WATCH_IGNORE_PATTERNS=*.tmp,*.bak,*.swp
+```
 
-### Database Pagination
-- **No Upload Limits**: Handles datasets larger than 1000 files through automatic pagination
-- **Efficient Querying**: Uses Supabase `.range()` method to fetch data in batches
-- **Memory Optimization**: Processes large datasets without memory overflow
+## Document Detection
 
-### File Processing
-- **Pre-compiled Regex**: Sanitization patterns are compiled once for optimal performance  
-- **Caching System**: File name sanitization results are cached to avoid re-processing
-- **Batch Processing**: Configurable batch sizes for optimal upload throughput
+The tool detects and classifies documents:
 
-### RFC Upload Optimizations
-- **Smart Querying**: Three-step query process to efficiently find related files
-- **Supporting Document Inclusion**: Automatically includes all related documents, not just pedimentos
-- **Path Concatenation**: Efficiently combines custom folder structures with arela_paths
+- **Pedimento Simplificado** — Extracts RFC, pedimento number, patente, aduana, and year from PDF content to compose an `arela_path` (`RFC/Year/Patente/Aduana/Pedimento/`)
+- **Supporting Documents** — XML, TXT, and JSON customs-related documents
 
 ## File Sanitization
 
-The tool automatically handles problematic characters using advanced sanitization:
-
-**Character Replacements:**
-- **Accents**: á→a, é→e, í→i, ó→o, ú→u, ñ→n, ç→c
-- **Korean characters**: 멕→meok, 시→si, 코→ko, 용→yong, others→kr  
-- **Special symbols**: &→and, {}[]~^|"<>?*: →-
-- **Email symbols**: @→(removed), spaces→-
-- **Multiple dashes**: collapsed to single dash
-- **Leading/trailing**: dashes and dots removed
-
-**Performance Features:**
-- Pre-compiled regex patterns for faster processing
-- Sanitization result caching to avoid re-processing
-- Unicode normalization (NFD) for consistent handling
+Filenames are automatically sanitized before upload:
 
-### Examples
-
-| Original | Renamed |
-|----------|---------|
+| Original | Sanitized |
+|----------|-----------|
 | `Facturas Importación.pdf` | `Facturas-Importacion.pdf` |
 | `File{with}brackets.pdf` | `File-with-brackets.pdf` |
 | `Document ^& symbols.pdf` | `Document-and-symbols.pdf` |
-| `CI & PL-20221212(멕시코용).xls` | `CI-and-PL-20221212.xls` |
-| `impresora@nereprint.com_file.xml` | `impresoranereprint.com_file.xml` |
-| `07-3429-3000430 HC.pdf` | `07-3429-3000430-HC.pdf` |
-| `FACTURA IN 3000430.pdf` | `FACTURA-IN-3000430.pdf` |
-
-## Logging and Monitoring
+| `file with spaces.pdf` | `file-with-spaces.pdf` |
 
-The tool maintains comprehensive logs both locally and remotely:
+Handles accented characters, Korean glyphs, special symbols, and email addresses.
 
-**Local Logging (`arela-upload.log`):**
-- Upload status (SUCCESS/ERROR/SKIPPED/SANITIZED)
-- File paths and sanitization changes  
-- Error messages and timestamps
-- Rename operations with before/after names
-- Processing statistics and performance metrics
+## Development
 
-**Log Entry Examples:**
-```
-[2025-09-04T01:17:00.141Z] SUCCESS: /Users/.../file.xml -> 2023/2003180/file.xml
-[2025-09-04T01:17:00.822Z] SANITIZED: file name.pdf → file-name.pdf
-[2025-09-04T01:17:00.856Z] SKIPPED: /Users/.../duplicate.pdf (already exists)
+```bash
+npm test                    # Run tests
+npm run test:watch          # Run tests in watch mode
+npm run test:coverage       # Run tests with coverage
+npm run format              # Format code with Prettier
 ```
 
-**Remote Logging:**
-- Integration with Supabase database for centralized logging
-- Upload tracking and audit trails
-- Error reporting and monitoring
-
-## Performance Features
-
-**Version 2.0.0 introduces several performance optimizations:**
-
-- **Pre-compiled Regex Patterns**: Sanitization patterns are compiled once and reused
-- **Sanitization Caching**: File name sanitization results are cached to avoid reprocessing
-- **Batch Processing**: Configurable batch sizes for optimal API usage
-- **Concurrent Processing**: Adjustable concurrency levels for file processing
-- **Smart Skip Logic**: Efficiently skips already processed files using log analysis
-- **Memory Optimization**: Large file outputs are truncated to prevent memory issues
-
-## Version History
-
-**v0.4.0** - Current Release 🆕
-- ✨ **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
-- ✨ **Cross-Tenant Mode**: Read from one API, upload to another
-- ✨ **Dynamic Client Config**: Change client by updating `.env` (no code changes)
-- ✨ New `--api` flag for single API target
-- ✨ New `--source-api` flag for source API (phases 1-3)
-- ✨ New `--target-api` flag for target API (phase 4)
-- ✨ `WATCH_DIRECTORY_CONFIGS` environment variable for watch mode
-- 🔧 Enhanced pipeline routing for cross-tenant operations
-- 📝 Simplified documentation for multi-tenant configuration
-
-**v0.3.0** - Watch Mode Release
-- ✨ Added watch command with chokidar integration
-- ✨ Automatic 4-step pipeline (stats → detect → propagate → upload)
-- ✨ Multiple upload strategies (batch, individual, full-structure)
-- ✨ Configurable debounce and polling options
-- 🔧 Signal handling for graceful shutdown
-
-**v0.2.0** - Pipeline Automation
-- ✨ Added smart year/pedimento auto-detection from file paths
-- ✨ Custom folder structure support with `--folder-structure` option
-- ✨ Client path tracking with `--client-path` option
-- ✨ Performance optimizations with regex pre-compilation
-- ✨ Sanitization result caching for improved speed
-- ✨ Enhanced file sanitization with Korean character support
-- ✨ Improved email character handling in file names
-- ✨ Better error handling and logging
-- 📝 Comprehensive logging with SANITIZED status
-- 🔧 Memory optimization for large file processing
-
-**v0.1.0** - Initial Release
-- 📦 Basic upload functionality
-- 🔌 API and Supabase direct mode support
-- 📂 RFC-based file upload
-
-## Troubleshooting
-
-**Connection Issues:**
-- Verify `ARELA_API_URL` and `ARELA_API_TOKEN` are correct
-- Check network connectivity to the API endpoint
-- The tool will automatically fallback to Supabase direct mode if API is unavailable
-
-**Performance Issues:**
-- Adjust `--batch-size` for optimal API performance (default: 10)
-- Modify `--concurrency` to control parallel processing (default: 10)  
-- Use `--show-stats` to monitor sanitization cache performance
-
-**File Issues:**
-- Check file permissions in `UPLOAD_BASE_PATH`
-- Verify `UPLOAD_SOURCES` paths exist and are accessible
-- Review `arela-upload.log` for detailed error information
-
-## Contributing
-
-Contributions are welcome! Please feel free to submit a Pull Request.
-
 ## License
 
-ISC License - see LICENSE file for details.
+ISC