@arela/uploader 1.0.2 → 1.0.4

package/docs/CROSS_PLATFORM_PATH_HANDLING.md

@@ -0,0 +1,593 @@
+# Absolute Path Strategy & SCAN_DIRECTORY_LEVEL Implementation
+
+**Updated**: January 19, 2026
+
+## Overview
+
+The system now uses **absolute paths throughout** for maximum clarity and correctness. This ensures that different relative paths like `../sample` and `./sample` create different tables as expected.
+
+## ✅ Simplified Strategy: Use Absolute Paths Everywhere
+
+### Key Principle
+**Always resolve paths to absolute paths first, then sanitize for table names.**
+
+### Why This Works Better
+
+1. **Unambiguous**: `../sample` and `./sample` resolve to different absolute paths
+2. **Simpler**: Same absolute path used for both `base_path_label` and `base_path_full`
+3. **Cross-platform**: Forward slashes in stored paths for consistency
+4. **PostgreSQL-friendly**: Special characters sanitized only in table names
+
+### Path Examples
+
+```bash
+# From /data/project:
+../sample     → /data/sample           → scan_palco_local_data_sample
+./sample      → /data/project/sample   → scan_palco_local_data_project_sample
+
+# Windows:
+C:\Users\Docs → C:/Users/Docs (stored) → scan_palco_local_c_users_docs
+
+# Linux:  
+/home/user    → /home/user (stored)    → scan_palco_local_home_user
+```
+
+## Storage Details
+
+### cli_registry Table
+- `company_slug`: Company identifier
+- `server_id`: Server identifier  
+- `base_path_full`: Absolute path with forward slashes (`C:/Users/Documents/2023`)
+- `table_name`: Sanitized version (`scan_palco_local_c_users_documents_2023`)
+- **Removed**: `base_path_label` (was redundant - same as base_path_full)
+
+### File Records (scan_* tables)
+- `directory_path`: Absolute path with forward slashes
+- `relative_path`: Relative to base with forward slashes  
+- `absolute_path`: Absolute path with forward slashes
+
+**Note**: All paths stored with forward slashes for consistency, but work on all platforms.
+
+## Key Features
+
+### 1. Cross-Platform Path Normalization
+
+All paths are normalized to POSIX format (forward slashes) for consistency:
+
+- **Windows paths**: `C:\Users\Documents` → `/Users/Documents`
+- **Unix paths**: `/home/user/docs` → `/home/user/docs`
+- **Relative paths**: `../parent/folder` → `../parent/folder`
+- **Network drives**: `O:/data/files` → `/data/files`
+
+### 2. Full Path in Table Names
+
+Table names now include the complete normalized path:
+
+```
+Format: scan_<company>_<server>_<normalized_path>
+
+Examples:
+- scan_palco_local_data_2023
+- scan_palco_local_users_documents_projects
+- scan_company_server_c_users_admin_files
+```
+
+### 3. SCAN_DIRECTORY_LEVEL Support
+
+The `SCAN_DIRECTORY_LEVEL` environment variable controls table granularity:
+
+- **Level 0** (default): Single table for entire base path
+- **Level 1**: One table per first-level subdirectory
+- **Level 2**: One table per second-level subdirectory
+- **Level N**: One table per N-th level subdirectory
+
+Example with `SCAN_DIRECTORY_LEVEL=1` and base path `/data`:
+```
+/data/2023     → scan_palco_local_data_2023
+/data/2024     → scan_palco_local_data_2024
+/data/archive  → scan_palco_local_data_archive
+```
+
+## Implementation Details
+
+### CLI Components
+
+#### 1. PathNormalizer Utility (`src/utils/PathNormalizer.js`)
+
+Centralized utility for all path operations:
+
+```javascript
+// Normalize any path to POSIX format
+const normalized = PathNormalizer.normalizePath('C:\\Users\\Documents');
+// Result: /Users/Documents
+
+// Generate table name
+const tableName = PathNormalizer.generateTableName({
+  companySlug: 'palco',
+  serverId: 'local',
+  basePathLabel: '/data/2023'
+});
+// Result: scan_palco_local_data_2023
+
+// Build full path label
+const fullPath = PathNormalizer.buildBasePathLabel('/data', '2023/subfolder');
+// Result: /data/2023/subfolder
+```
+
+#### 2. ScanCommand Updates
+
+**Directory Discovery** ([ScanCommand.js](../src/commands/ScanCommand.js)):
+
+```javascript
+async #discoverDirectories(basePath, level) {
+  // Level 0: Single entry
+  if (level === 0) {
+    return sources.map((source) => {
+      const sourcePath = source === '.' ? basePath : path.resolve(basePath, source);
+      const relativePath = source === '.' ? '' : PathNormalizer.getRelativePath(sourcePath, basePath);
+      return { path: sourcePath, label: relativePath };
+    });
+  }
+  
+  // Level > 0: Discover directories at specified depth
+  const levelDirs = await this.#getDirectoriesAtLevel(basePath, level, '');
+  // ... combine with sources
+}
+```
+
+**Table Registration**:
+
+```javascript
+for (const dir of directories) {
+  // Build full normalized path label
+  const fullPathLabel = PathNormalizer.buildBasePathLabel(
+    basePath,
+    dir.label || '',
+  );
+
+  const registration = await this.scanApiService.registerInstance({
+    companySlug: scanConfig.companySlug,
+    serverId: scanConfig.serverId,
+    basePathLabel: fullPathLabel,  // Full normalized path
+    basePathFull: dir.path,         // Absolute system path
+  });
+}
+```
+
+**File Record Normalization**:
+
+```javascript
+#normalizeFileRecord(filePath, fileStats, basePath, scanTimestamp) {
+  return {
+    fileName: path.basename(filePath),
+    fileExtension: path.extname(filePath).toLowerCase().replace('.', ''),
+    directoryPath: PathNormalizer.normalizePath(path.dirname(filePath)),
+    relativePath: PathNormalizer.getRelativePath(filePath, basePath),
+    absolutePath: PathNormalizer.normalizePath(filePath),
+    sizeBytes: Number(fileStats.size),
+    modifiedAt: fileStats.mtime.toISOString(),
+    scanTimestamp,
+  };
+}
+```
+
+#### 3. Config Updates
+
+Configuration now uses normalized paths ([config.js](../src/config/config.js)):
+
+```javascript
+#loadScanConfig() {
+  // Auto-derive basePathLabel from UPLOAD_BASE_PATH
+  if (!basePathLabel && process.env.UPLOAD_BASE_PATH) {
+    const basePath = process.env.UPLOAD_BASE_PATH;
+    // Normalize for consistent table naming across platforms
+    basePathLabel = PathNormalizer.normalizePath(basePath);
+  }
+
+  // Generate table name using PathNormalizer
+  if (companySlug && serverId && basePathLabel) {
+    tableName = PathNormalizer.generateTableName({
+      companySlug,
+      serverId,
+      basePathLabel,
+    });
+  }
+}
+```
+
+### Backend Components
+
+#### 1. PathNormalizer Utility (`src/uploader/utils/path-normalizer.util.ts`)
+
+TypeScript version matching CLI logic exactly:
+
+```typescript
+export class PathNormalizer {
+  static normalizePath(inputPath: string): string { /* ... */ }
+  static generateTableName(config: {...}): string { /* ... */ }
+  static buildBasePathLabel(basePath: string, relativePath: string): string { /* ... */ }
+  // ... other utilities
+}
+```
+
+#### 2. FileStatsTableManagerService Updates
+
+**Table Name Generation** ([file-stats-table-manager.service.ts](../../arela-api/src/uploader/services/file-stats-table-manager.service.ts)):
+
+```typescript
+private generateTableName(config: ScanInstanceConfig): string {
+  // Delegate to PathNormalizer for consistency with CLI
+  return PathNormalizer.generateTableName(config);
+}
+```
+
+**Instance Table Lookup**:
+
+```typescript
+async getInstanceTables(
+  companySlug: string,
+  serverId: string,
+  basePathLabel: string,
+): Promise<CliRegistry[]> {
+  // Normalize the base path
+  const normalizedBasePath = PathNormalizer.normalizePath(basePathLabel);
+  
+  // Match all tables for this company/server with base_path_full starting with normalized path
+  return this.cliRegistryRepository
+    .createQueryBuilder('registry')
+    .where('registry.table_name LIKE :pattern', {
+      pattern: `${basePattern}%`,
+    })
+    .andWhere('registry.company_slug = :companySlug', { companySlug })
+    .andWhere('registry.server_id = :serverId', { serverId })
+    .andWhere(
+      '(registry.base_path_full = :basePath OR registry.base_path_full LIKE :basePathPrefix)',
+      {
+        basePath: normalizedBasePath,
+        basePathPrefix: `${normalizedBasePath}/%`,
+      },
+    )
+    .orderBy('registry.table_name', 'ASC')
+    .getMany();
+}
+```
+
+## Command Flow
+
+### 1. Scan Command
+
+```bash
+arela scan
+```
+
+**Flow**:
+1. Load configuration (company, server, base path)
+2. Normalize base path using PathNormalizer
+3. Discover directories at specified `SCAN_DIRECTORY_LEVEL`
+4. For each directory:
+   - Build full normalized path label
+   - Register instance (creates/finds table)
+   - Scan files with normalized paths
+   - Upload batch to API
+
+### 2. Identify Command
+
+```bash
+arela identify
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Fetch pending PDFs
+   - Detect locally
+   - Update with normalized paths
+
+### 3. Propagate Command
+
+```bash
+arela propagate
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Mark files needing propagation
+   - Process pedimentos by directory
+   - Propagate arela_path to same-directory files
+
+### 4. Push Command
+
+```bash
+arela push
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Fetch files with arela_path
+   - Upload to storage API
+   - Update upload status
+
+## Configuration Examples
+
+### Single-Level Scan
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+SCAN_DIRECTORY_LEVEL=0
+
+# Result: Single table scan_palco_local_data
+```
+
+### Multi-Level Scan (Year-based)
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+SCAN_DIRECTORY_LEVEL=1
+
+# Directory structure:
+# /data/2023/...
+# /data/2024/...
+# /data/archive/...
+
+# Result: Three tables
+# - scan_palco_local_data_2023
+# - scan_palco_local_data_2024
+# - scan_palco_local_data_archive
+```
+
+### Multi-Level Scan with Sources
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+UPLOAD_SOURCES=documents,images
+SCAN_DIRECTORY_LEVEL=1
+
+# Directory structure:
+# /data/2023/documents/...
+# /data/2023/images/...
+# /data/2024/documents/...
+# /data/2024/images/...
+
+# Result: Four tables
+# - scan_palco_local_data_2023_documents
+# - scan_palco_local_data_2023_images
+# - scan_palco_local_data_2024_documents
+# - scan_palco_local_data_2024_images
+```
+
+## Platform-Specific Handling
+
+### Windows
+
+**Input Paths**:
+```
+C:\Users\Documents\2023
+D:\Archive\Files
+O:\Network\Share
+```
+
+**Normalized Paths**:
+```
+/Users/Documents/2023
+/Archive/Files
+/Network/Share
+```
+
+**Table Names**:
+```
+scan_company_server_users_documents_2023
+scan_company_server_archive_files
+scan_company_server_network_share
+```
+
+### Linux/macOS
+
+**Input Paths**:
+```
+/home/user/documents/2023
+/mnt/storage/archive
+/Volumes/backup/files
+```
+
+**Normalized Paths** (unchanged):
+```
+/home/user/documents/2023
+/mnt/storage/archive
+/Volumes/backup/files
+```
+
+**Table Names**:
+```
+scan_company_server_home_user_documents_2023
+scan_company_server_mnt_storage_archive
+scan_company_server_volumes_backup_files
+```
+
+## Database Schema
+
+### cli_registry Table
+
+Stores metadata for each scan instance:
+
+```sql
+CREATE TABLE cli.cli_registry (
+  id UUID PRIMARY KEY,
+  table_name VARCHAR(63) UNIQUE NOT NULL,
+  company_slug VARCHAR(255) NOT NULL,
+  server_id VARCHAR(255) NOT NULL,
+  base_path_label TEXT NOT NULL,      -- Normalized path (e.g., /data/2023)
+  base_path_full TEXT NOT NULL,        -- Original absolute path (e.g., C:\data\2023)
+  status VARCHAR(50) DEFAULT 'active',
+  created_at TIMESTAMP DEFAULT NOW(),
+  last_scan_at TIMESTAMP,
+  total_files INTEGER DEFAULT 0,
+  total_size_bytes BIGINT DEFAULT 0
+);
+
+-- Index for efficient lookup
+CREATE INDEX idx_cli_registry_lookup 
+ON cli.cli_registry(company_slug, server_id, base_path_label);
+```
+
+### scan_* Tables
+
+Dynamic tables created per instance:
+
+```sql
+CREATE TABLE cli.scan_palco_local_data_2023 (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  file_name VARCHAR(255) NOT NULL,
+  file_extension VARCHAR(50),
+  directory_path TEXT NOT NULL,        -- Normalized directory path
+  relative_path TEXT NOT NULL,         -- Normalized relative path
+  absolute_path TEXT NOT NULL,         -- Normalized absolute path
+  size_bytes BIGINT NOT NULL,
+  modified_at TIMESTAMP NOT NULL,
+  scan_timestamp TIMESTAMP NOT NULL,
+  
+  -- Detection fields
+  detected_type VARCHAR(100),
+  detected_pedimento VARCHAR(50),
+  detected_pedimento_year INTEGER,
+  rfc VARCHAR(20),
+  arela_path TEXT,
+  detection_error TEXT,
+  detection_attempts INTEGER DEFAULT 0,
+  is_not_pedimento BOOLEAN DEFAULT FALSE,
+  
+  -- Propagation fields
+  propagated_from_id UUID,
+  propagation_error TEXT,
+  propagation_attempts INTEGER DEFAULT 0,
+  needs_propagation BOOLEAN DEFAULT FALSE,
+  
+  -- Upload fields
+  uploaded BOOLEAN DEFAULT FALSE,
+  upload_path TEXT,
+  uploaded_to_storage_id TEXT,
+  upload_error TEXT,
+  upload_attempts INTEGER DEFAULT 0
+);
+```
+
+## Benefits
+
+### 1. Cross-Platform Consistency
+
+- Same table names regardless of OS
+- Paths stored consistently in database
+- No special handling needed per platform
+
+### 2. Full Path Visibility
+
+- Table names clearly show what directory they represent
+- Easy to identify and manage specific scans
+- Better organization for multi-directory setups
+
+### 3. Scalability
+
+- Support for any directory structure
+- Configurable granularity via `SCAN_DIRECTORY_LEVEL`
+- Efficient batch processing per directory
+
+### 4. Maintainability
+
+- Centralized path logic in PathNormalizer
+- Consistent behavior between CLI and backend
+- Easy to test and debug
+
+## Migration Guide
+
+### For Existing Installations
+
+If you have existing scan tables with old naming:
+
+1. **New scans will create new tables** with the updated naming scheme
+2. **Old tables remain functional** but won't be automatically migrated
+3. **To migrate**:
+   - Re-run `arela scan` to create new tables
+   - Use `identify`, `propagate`, `push` on new tables
+   - Optionally drop old tables when confident
+
+### Backward Compatibility
+
+The system is designed to be backward compatible:
+
+- `getInstanceTables()` matches by `base_path_full` pattern
+- Old tables can coexist with new tables
+- Commands process all matching tables
+
+## Troubleshooting
+
+### Issue: Tables not found
+
+**Cause**: Base path mismatch between scan and other commands
+
+**Solution**: Ensure `UPLOAD_BASE_PATH` is consistent across commands
+
+### Issue: Duplicate tables
+
+**Cause**: Different path formats creating separate tables
+
+**Solution**: System now prevents this with PathNormalizer
+
+### Issue: Table name too long
+
+**Cause**: Very deep directory structures
+
+**Solution**: System automatically truncates and adds hash for uniqueness
+
+## Testing
+
+### Manual Testing
+
+```bash
+# Test different path formats
+UPLOAD_BASE_PATH="C:\Users\Documents" arela scan  # Windows
+UPLOAD_BASE_PATH="/Users/Documents" arela scan     # Unix
+UPLOAD_BASE_PATH="O:/Data/Files" arela scan        # Network
+
+# All should create the same table name (normalized)
+```
+
+### Unit Testing
+
+Tests are provided in [tests/PathNormalizer.test.js](../tests/PathNormalizer.test.js):
+
+```bash
+npm test -- PathNormalizer.test.js
+```
+
+## Future Enhancements
+
+1. **Table Consolidation**: Utility to merge data from old tables to new naming scheme
+2. **Path Aliases**: Support for custom path labels independent of physical path
+3. **Virtual Directories**: Scan multiple non-contiguous directories into one table
+4. **Path Filters**: Regex-based filtering at path level
+
+## Summary
+
+The cross-platform path handling system provides:
+
+✅ Consistent table naming across Windows, Linux, and macOS  
+✅ Full path information in table names  
+✅ Proper support for `SCAN_DIRECTORY_LEVEL`  
+✅ All commands (scan, identify, propagate, push) working seamlessly  
+✅ Normalized paths in database for consistency  
+✅ Centralized path logic for maintainability  
+
+All path operations are now platform-independent and maintain consistency between CLI and backend implementations.