@arela/uploader 1.0.3 → 1.0.5

package/docs/CROSS_PLATFORM_PATH_HANDLING.md

@@ -0,0 +1,597 @@
+# Absolute Path Strategy & SCAN_DIRECTORY_LEVEL Implementation
+
+**Updated**: January 19, 2026
+
+## Overview
+
+The system now uses **absolute paths throughout** for maximum clarity and correctness. This ensures that different relative paths like `../sample` and `./sample` create different tables as expected.
+
+## ✅ Simplified Strategy: Use Absolute Paths Everywhere
+
+### Key Principle
+**Always resolve paths to absolute paths first, then sanitize for table names.**
+
+### Why This Works Better
+
+1. **Unambiguous**: `../sample` and `./sample` resolve to different absolute paths
+2. **Simpler**: Same absolute path used for both `base_path_label` and `base_path_full`
+3. **Cross-platform**: Forward slashes in stored paths for consistency
+4. **PostgreSQL-friendly**: Special characters sanitized only in table names
+
+### Path Examples
+
+```bash
+# From /data/project:
+../sample     → /data/sample           → scan_palco_local_data_sample
+./sample      → /data/project/sample   → scan_palco_local_data_project_sample
+
+# Windows:
+C:\Users\Docs → C:/Users/Docs (stored) → scan_palco_local_c_users_docs
+
+# Linux:  
+/home/user    → /home/user (stored)    → scan_palco_local_home_user
+```
+
+## Storage Details
+
+### cli_registry Table
+- `company_slug`: Company identifier
+- `server_id`: Server identifier  
+- `base_path_full`: Absolute path (preserves original format for Windows paths like `O:\expediente\archivos`)
+- `table_name`: Sanitized version (`scan_palco_local_c_users_documents_2023`)
+- **Removed**: `base_path_label` (was redundant - same as base_path_full)
+
+### File Records (scan_* tables)
+- `directory_path`: Absolute path with forward slashes
+- `relative_path`: Relative to base with forward slashes  
+- `absolute_path`: Absolute path with forward slashes
+
+**Note**: Paths are stored consistently. Windows absolute paths (with drive letters) are preserved when the CLI runs on the same Windows machine or a different OS (macOS/Linux).
+
+## Key Features
+
+### 1. Cross-Platform Path Detection
+
+The CLI correctly identifies Windows absolute paths even when running on macOS/Linux:
+
+- **Windows paths on any OS**: `O:\expediente\archivos` → recognized as absolute, preserved as-is
+- **Unix paths**: `/home/user/docs` → recognized as absolute, normalized
+- **Relative paths**: `./sample` → resolved against cwd
+
+This allows:
+- Running `arela scan` on Windows server → creates table with Windows path in `base_path_full`
+- Running `arela identify` on macOS (for testing) → correctly uses the same Windows path
+- Both resolve to the same table lookup
+
+### 2. Full Path in Table Names
+
+Table names now include the complete normalized path:
+
+```
+Format: scan_<company>_<server>_<normalized_path>
+
+Examples:
+- scan_palco_local_data_2023
+- scan_palco_local_users_documents_projects
+- scan_company_server_c_users_admin_files
+```
+
+### 3. SCAN_DIRECTORY_LEVEL Support
+
+The `SCAN_DIRECTORY_LEVEL` environment variable controls table granularity:
+
+- **Level 0** (default): Single table for entire base path
+- **Level 1**: One table per first-level subdirectory
+- **Level 2**: One table per second-level subdirectory
+- **Level N**: One table per N-th level subdirectory
+
+Example with `SCAN_DIRECTORY_LEVEL=1` and base path `/data`:
+```
+/data/2023     → scan_palco_local_data_2023
+/data/2024     → scan_palco_local_data_2024
+/data/archive  → scan_palco_local_data_archive
+```
+
+## Implementation Details
+
+### CLI Components
+
+#### 1. PathNormalizer Utility (`src/utils/PathNormalizer.js`)
+
+Centralized utility for all path operations:
+
+```javascript
+// Normalize any path to POSIX format
+const normalized = PathNormalizer.normalizePath('C:\\Users\\Documents');
+// Result: /Users/Documents
+
+// Generate table name
+const tableName = PathNormalizer.generateTableName({
+  companySlug: 'palco',
+  serverId: 'local',
+  basePathLabel: '/data/2023'
+});
+// Result: scan_palco_local_data_2023
+
+// Build full path label
+const fullPath = PathNormalizer.buildBasePathLabel('/data', '2023/subfolder');
+// Result: /data/2023/subfolder
+```
+
+#### 2. ScanCommand Updates
+
+**Directory Discovery** ([ScanCommand.js](../src/commands/ScanCommand.js)):
+
+```javascript
+async #discoverDirectories(basePath, level) {
+  // Level 0: Single entry
+  if (level === 0) {
+    return sources.map((source) => {
+      const sourcePath = source === '.' ? basePath : path.resolve(basePath, source);
+      const relativePath = source === '.' ? '' : PathNormalizer.getRelativePath(sourcePath, basePath);
+      return { path: sourcePath, label: relativePath };
+    });
+  }
+  
+  // Level > 0: Discover directories at specified depth
+  const levelDirs = await this.#getDirectoriesAtLevel(basePath, level, '');
+  // ... combine with sources
+}
+```
+
+**Table Registration**:
+
+```javascript
+for (const dir of directories) {
+  // Build full normalized path label
+  const fullPathLabel = PathNormalizer.buildBasePathLabel(
+    basePath,
+    dir.label || '',
+  );
+
+  const registration = await this.scanApiService.registerInstance({
+    companySlug: scanConfig.companySlug,
+    serverId: scanConfig.serverId,
+    basePathLabel: fullPathLabel,  // Full normalized path
+    basePathFull: dir.path,         // Absolute system path
+  });
+}
+```
+
+**File Record Normalization**:
+
+```javascript
+#normalizeFileRecord(filePath, fileStats, basePath, scanTimestamp) {
+  return {
+    fileName: path.basename(filePath),
+    fileExtension: path.extname(filePath).toLowerCase().replace('.', ''),
+    directoryPath: PathNormalizer.normalizePath(path.dirname(filePath)),
+    relativePath: PathNormalizer.getRelativePath(filePath, basePath),
+    absolutePath: PathNormalizer.normalizePath(filePath),
+    sizeBytes: Number(fileStats.size),
+    modifiedAt: fileStats.mtime.toISOString(),
+    scanTimestamp,
+  };
+}
+```
+
+#### 3. Config Updates
+
+Configuration now uses normalized paths ([config.js](../src/config/config.js)):
+
+```javascript
+#loadScanConfig() {
+  // Auto-derive basePathLabel from UPLOAD_BASE_PATH
+  if (!basePathLabel && process.env.UPLOAD_BASE_PATH) {
+    const basePath = process.env.UPLOAD_BASE_PATH;
+    // Normalize for consistent table naming across platforms
+    basePathLabel = PathNormalizer.normalizePath(basePath);
+  }
+
+  // Generate table name using PathNormalizer
+  if (companySlug && serverId && basePathLabel) {
+    tableName = PathNormalizer.generateTableName({
+      companySlug,
+      serverId,
+      basePathLabel,
+    });
+  }
+}
+```
+
+### Backend Components
+
+#### 1. PathNormalizer Utility (`src/uploader/utils/path-normalizer.util.ts`)
+
+TypeScript version matching CLI logic exactly:
+
+```typescript
+export class PathNormalizer {
+  static normalizePath(inputPath: string): string { /* ... */ }
+  static generateTableName(config: {...}): string { /* ... */ }
+  static buildBasePathLabel(basePath: string, relativePath: string): string { /* ... */ }
+  // ... other utilities
+}
+```
+
+#### 2. FileStatsTableManagerService Updates
+
+**Table Name Generation** ([file-stats-table-manager.service.ts](../../arela-api/src/uploader/services/file-stats-table-manager.service.ts)):
+
+```typescript
+private generateTableName(config: ScanInstanceConfig): string {
+  // Delegate to PathNormalizer for consistency with CLI
+  return PathNormalizer.generateTableName(config);
+}
+```
+
+**Instance Table Lookup**:
+
+```typescript
+async getInstanceTables(
+  companySlug: string,
+  serverId: string,
+  basePathLabel: string,
+): Promise<CliRegistry[]> {
+  // Normalize the base path
+  const normalizedBasePath = PathNormalizer.normalizePath(basePathLabel);
+  
+  // Match all tables for this company/server with base_path_full starting with normalized path
+  return this.cliRegistryRepository
+    .createQueryBuilder('registry')
+    .where('registry.table_name LIKE :pattern', {
+      pattern: `${basePattern}%`,
+    })
+    .andWhere('registry.company_slug = :companySlug', { companySlug })
+    .andWhere('registry.server_id = :serverId', { serverId })
+    .andWhere(
+      '(registry.base_path_full = :basePath OR registry.base_path_full LIKE :basePathPrefix)',
+      {
+        basePath: normalizedBasePath,
+        basePathPrefix: `${normalizedBasePath}/%`,
+      },
+    )
+    .orderBy('registry.table_name', 'ASC')
+    .getMany();
+}
+```
+
+## Command Flow
+
+### 1. Scan Command
+
+```bash
+arela scan
+```
+
+**Flow**:
+1. Load configuration (company, server, base path)
+2. Normalize base path using PathNormalizer
+3. Discover directories at specified `SCAN_DIRECTORY_LEVEL`
+4. For each directory:
+   - Build full normalized path label
+   - Register instance (creates/finds table)
+   - Scan files with normalized paths
+   - Upload batch to API
+
+### 2. Identify Command
+
+```bash
+arela identify
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Fetch pending PDFs
+   - Detect locally
+   - Update with normalized paths
+
+### 3. Propagate Command
+
+```bash
+arela propagate
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Mark files needing propagation
+   - Process pedimentos by directory
+   - Propagate arela_path to same-directory files
+
+### 4. Push Command
+
+```bash
+arela push
+```
+
+**Flow**:
+1. Load configuration
+2. Fetch all tables using `getInstanceTables()`
+3. For each table:
+   - Fetch files with arela_path
+   - Upload to storage API
+   - Update upload status
+
+## Configuration Examples
+
+### Single-Level Scan
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+SCAN_DIRECTORY_LEVEL=0
+
+# Result: Single table scan_palco_local_data
+```
+
+### Multi-Level Scan (Year-based)
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+SCAN_DIRECTORY_LEVEL=1
+
+# Directory structure:
+# /data/2023/...
+# /data/2024/...
+# /data/archive/...
+
+# Result: Three tables
+# - scan_palco_local_data_2023
+# - scan_palco_local_data_2024
+# - scan_palco_local_data_archive
+```
+
+### Multi-Level Scan with Sources
+
+```bash
+# .env
+ARELA_COMPANY_SLUG=palco
+ARELA_SERVER_ID=local
+UPLOAD_BASE_PATH=/data
+UPLOAD_SOURCES=documents,images
+SCAN_DIRECTORY_LEVEL=1
+
+# Directory structure:
+# /data/2023/documents/...
+# /data/2023/images/...
+# /data/2024/documents/...
+# /data/2024/images/...
+
+# Result: Four tables
+# - scan_palco_local_data_2023_documents
+# - scan_palco_local_data_2023_images
+# - scan_palco_local_data_2024_documents
+# - scan_palco_local_data_2024_images
+```
+
+## Platform-Specific Handling
+
+### Windows
+
+**Input Paths**:
+```
+C:\Users\Documents\2023
+D:\Archive\Files
+O:\Network\Share
+```
+
+**Normalized Paths**:
+```
+/Users/Documents/2023
+/Archive/Files
+/Network/Share
+```
+
+**Table Names**:
+```
+scan_company_server_users_documents_2023
+scan_company_server_archive_files
+scan_company_server_network_share
+```
+
+### Linux/macOS
+
+**Input Paths**:
+```
+/home/user/documents/2023
+/mnt/storage/archive
+/Volumes/backup/files
+```
+
+**Normalized Paths** (unchanged):
+```
+/home/user/documents/2023
+/mnt/storage/archive
+/Volumes/backup/files
+```
+
+**Table Names**:
+```
+scan_company_server_home_user_documents_2023
+scan_company_server_mnt_storage_archive
+scan_company_server_volumes_backup_files
+```
+
+## Database Schema
+
+### cli_registry Table
+
+Stores metadata for each scan instance:
+
+```sql
+CREATE TABLE cli.cli_registry (
+  id UUID PRIMARY KEY,
+  table_name VARCHAR(63) UNIQUE NOT NULL,
+  company_slug VARCHAR(255) NOT NULL,
+  server_id VARCHAR(255) NOT NULL,
+  base_path_label TEXT NOT NULL,      -- Normalized path (e.g., /data/2023)
+  base_path_full TEXT NOT NULL,        -- Original absolute path (e.g., C:\data\2023)
+  status VARCHAR(50) DEFAULT 'active',
+  created_at TIMESTAMP DEFAULT NOW(),
+  last_scan_at TIMESTAMP,
+  total_files INTEGER DEFAULT 0,
+  total_size_bytes BIGINT DEFAULT 0
+);
+
+-- Index for efficient lookup
+CREATE INDEX idx_cli_registry_lookup 
+ON cli.cli_registry(company_slug, server_id, base_path_label);
+```
+
+### scan_* Tables
+
+Dynamic tables created per instance:
+
+```sql
+CREATE TABLE cli.scan_palco_local_data_2023 (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  file_name VARCHAR(255) NOT NULL,
+  file_extension VARCHAR(50),
+  directory_path TEXT NOT NULL,        -- Normalized directory path
+  relative_path TEXT NOT NULL,         -- Normalized relative path
+  absolute_path TEXT NOT NULL,         -- Normalized absolute path
+  size_bytes BIGINT NOT NULL,
+  modified_at TIMESTAMP NOT NULL,
+  scan_timestamp TIMESTAMP NOT NULL,
+  
+  -- Detection fields
+  detected_type VARCHAR(100),
+  detected_pedimento VARCHAR(50),
+  detected_pedimento_year INTEGER,
+  rfc VARCHAR(20),
+  arela_path TEXT,
+  detection_error TEXT,
+  detection_attempts INTEGER DEFAULT 0,
+  is_not_pedimento BOOLEAN DEFAULT FALSE,
+  
+  -- Propagation fields
+  propagated_from_id UUID,
+  propagation_error TEXT,
+  propagation_attempts INTEGER DEFAULT 0,
+  needs_propagation BOOLEAN DEFAULT FALSE,
+  
+  -- Upload fields
+  uploaded BOOLEAN DEFAULT FALSE,
+  upload_path TEXT,
+  uploaded_to_storage_id TEXT,
+  upload_error TEXT,
+  upload_attempts INTEGER DEFAULT 0
+);
+```
+
+## Benefits
+
+### 1. Cross-Platform Consistency
+
+- Same table names regardless of OS
+- Paths stored consistently in database
+- No special handling needed per platform
+
+### 2. Full Path Visibility
+
+- Table names clearly show what directory they represent
+- Easy to identify and manage specific scans
+- Better organization for multi-directory setups
+
+### 3. Scalability
+
+- Support for any directory structure
+- Configurable granularity via `SCAN_DIRECTORY_LEVEL`
+- Efficient batch processing per directory
+
+### 4. Maintainability
+
+- Centralized path logic in PathNormalizer
+- Consistent behavior between CLI and backend
+- Easy to test and debug
+
+## Migration Guide
+
+### For Existing Installations
+
+If you have existing scan tables with old naming:
+
+1. **New scans will create new tables** with the updated naming scheme
+2. **Old tables remain functional** but won't be automatically migrated
+3. **To migrate**:
+   - Re-run `arela scan` to create new tables
+   - Use `identify`, `propagate`, `push` on new tables
+   - Optionally drop old tables when confident
+
+### Backward Compatibility
+
+The system is designed to be backward compatible:
+
+- `getInstanceTables()` matches by `base_path_full` pattern
+- Old tables can coexist with new tables
+- Commands process all matching tables
+
+## Troubleshooting
+
+### Issue: Tables not found
+
+**Cause**: Base path mismatch between scan and other commands
+
+**Solution**: Ensure `UPLOAD_BASE_PATH` is consistent across commands
+
+### Issue: Duplicate tables
+
+**Cause**: Different path formats creating separate tables
+
+**Solution**: System now prevents this with PathNormalizer
+
+### Issue: Table name too long
+
+**Cause**: Very deep directory structures
+
+**Solution**: System automatically truncates and adds hash for uniqueness
+
+## Testing
+
+### Manual Testing
+
+```bash
+# Test different path formats
+UPLOAD_BASE_PATH="C:\Users\Documents" arela scan  # Windows
+UPLOAD_BASE_PATH="/Users/Documents" arela scan     # Unix
+UPLOAD_BASE_PATH="O:/Data/Files" arela scan        # Network
+
+# All should create the same table name (normalized)
+```
+
+### Unit Testing
+
+Tests are provided in [tests/PathNormalizer.test.js](../tests/PathNormalizer.test.js):
+
+```bash
+npm test -- PathNormalizer.test.js
+```
+
+## Future Enhancements
+
+1. **Table Consolidation**: Utility to merge data from old tables to new naming scheme
+2. **Path Aliases**: Support for custom path labels independent of physical path
+3. **Virtual Directories**: Scan multiple non-contiguous directories into one table
+4. **Path Filters**: Regex-based filtering at path level
+
+## Summary
+
+The cross-platform path handling system provides:
+
+✅ Consistent table naming across Windows, Linux, and macOS  
+✅ Full path information in table names  
+✅ Proper support for `SCAN_DIRECTORY_LEVEL`  
+✅ All commands (scan, identify, propagate, push) working seamlessly  
+✅ Normalized paths in database for consistency  
+✅ Centralized path logic for maintainability  
+
+All path operations are now platform-independent and maintain consistency between CLI and backend implementations.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arela/uploader",
-  "version": "1.0.3",
+  "version": "1.0.5",
   "description": "CLI to upload files/directories to Arela",
   "bin": {
     "arela": "./src/index.js"
@@ -8,7 +8,9 @@
   "type": "module",
   "scripts": {
     "start": "node ./src/index.js",
-    "test": "echo \"Error: no test specified\" && exit 1",
+    "test": "NODE_OPTIONS=--experimental-vm-modules jest",
+    "test:watch": "NODE_OPTIONS=--experimental-vm-modules jest --watch",
+    "test:coverage": "NODE_OPTIONS=--experimental-vm-modules jest --coverage",
     "format": "prettier --write \"src/**/*.js\""
   },
   "repository": {
@@ -43,7 +45,31 @@
     "pdf-parse": "^2.4.5"
   },
   "devDependencies": {
+    "@jest/globals": "^30.2.0",
     "@trivago/prettier-plugin-sort-imports": "5.2.2",
+    "jest": "^30.2.0",
     "prettier": "3.5.3"
+  },
+  "jest": {
+    "testEnvironment": "node",
+    "transform": {},
+    "testMatch": [
+      "**/tests/**/*.test.js"
+    ],
+    "testPathIgnorePatterns": [
+      "/node_modules/",
+      "/scripts/"
+    ],
+    "collectCoverageFrom": [
+      "src/commands/**/*.js",
+      "!src/commands/**/index.js"
+    ],
+    "coverageDirectory": "coverage",
+    "coverageReporters": [
+      "text",
+      "text-summary",
+      "html",
+      "lcov"
+    ]
   }
 }

package/src/commands/IdentifyCommand.js

@@ -66,7 +66,7 @@ export class IdentifyCommand {
       const tables = await this.scanApiService.getInstanceTables(
         scanConfig.companySlug,
         scanConfig.serverId,
-        scanConfig.basePathLabel,
+        scanConfig.basePathFull,
       );
 
       if (tables.length === 0) {
@@ -421,33 +421,6 @@ export class IdentifyCommand {
     return missing;
   }
 
-  /**
-   * Generate table name from scan config (same logic as scan command)
-   * @private
-   * @param {Object} scanConfig - Scan configuration
-   * @returns {string} Table name
-   */
-  #generateTableName(scanConfig) {
-    const { companySlug, serverId, basePathLabel } = scanConfig;
-
-    // Combine components
-    const rawName = `${companySlug}_${serverId}_${basePathLabel}`;
-
-    // Sanitize: lowercase, replace special chars with underscore
-    let sanitized = rawName
-      .toLowerCase()
-      .replace(/[^a-z0-9_]/g, '_')
-      .replace(/_+/g, '_')
-      .replace(/^_|_$/g, '');
-
-    // Add prefix
-    const tableName = 'scan_' + sanitized;
-
-    // Note: Hash truncation logic should match backend
-    // For simplicity, we rely on backend validation
-    return tableName;
-  }
-
   /**
    * Show detailed performance statistics
    * @private

package/src/commands/PropagateCommand.js

@@ -47,7 +47,7 @@ export class PropagateCommand {
       const tables = await this.scanApiService.getInstanceTables(
         scanConfig.companySlug,
         scanConfig.serverId,
-        scanConfig.basePathLabel,
+        scanConfig.basePathFull,
       );
 
       if (tables.length === 0) {

package/src/commands/PushCommand.js

@@ -76,7 +76,7 @@ export class PushCommand {
       const tables = await this.scanApiService.getInstanceTables(
         scanConfig.companySlug,
         scanConfig.serverId,
-        scanConfig.basePathLabel,
+        scanConfig.basePathFull,
       );
 
       if (tables.length === 0) {