@arela/uploader 1.0.1 → 1.0.3

package/docs/STATS_COMMAND_SEQUENCE_DIAGRAM.md

@@ -0,0 +1,287 @@
+# Stats Command Sequence Diagram
+
+## Command: `node src/index.js stats --stats-only`
+
+This diagram shows the detailed flow of the stats command with the `--stats-only` flag, which collects file statistics and records them to the database without uploading files.
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant CLI as ArelaUploaderCLI<br/>(index.js)
+    participant Commander as Commander.js
+    participant UploadCmd as UploadCommand<br/>(UploadCommand.js)
+    participant Config as appConfig<br/>(config.js)
+    participant FileOps as FileOperations<br/>(FileOperations.js)
+    participant Globby as globby
+    participant DBService as DatabaseService<br/>(DatabaseService.js)
+    participant Factory as UploadServiceFactory
+    participant APIService as ApiUploadService
+    participant ArelaAPI as Arela API<br/>/api/uploader/batch-upsert
+    participant Logger as LoggingService
+    participant ProgressBar as cli-progress
+
+    %% Initial Setup
+    User->>CLI: node src/index.js stats --stats-only
+    activate CLI
+    Note over CLI: Constructor initializes<br/>ErrorHandler, UploadCommand,<br/>WatchCommand
+    
+    CLI->>CLI: #setupProgram()
+    CLI->>CLI: #setupCommands()
+    Note over CLI: Register 'stats' command<br/>with Commander
+    
+    CLI->>CLI: #setupErrorHandling()
+    Note over CLI: Register SIGINT, SIGTERM,<br/>uncaughtException handlers
+    
+    CLI->>CLI: run()
+    CLI->>Logger: setVerbose(false)
+    CLI->>Logger: info("Arela Uploader started")
+    
+    %% Command Parsing
+    CLI->>Commander: parseAsync()
+    activate Commander
+    Commander->>Commander: Parse args: ['stats', '--stats-only']
+    Commander->>Commander: Match 'stats' command
+    
+    %% Stats Command Action
+    Commander->>UploadCmd: execute(options)
+    activate UploadCmd
+    Note over UploadCmd: options = {<br/>  statsOnly: true,<br/>  batchSize: '10',<br/>  api: 'default'<br/>}
+    
+    %% API Configuration
+    alt api !== 'default'
+        UploadCmd->>Config: setApiTarget(options.api)
+    end
+    
+    %% Check Watch Mode
+    UploadCmd->>UploadCmd: Check if watch mode active
+    Note over UploadCmd: Prevent upload during watch
+    
+    %% Validation
+    UploadCmd->>UploadCmd: #validateOptions(options)
+    UploadCmd->>Config: validateConfiguration(false)
+    activate Config
+    Config-->>UploadCmd: Configuration valid
+    deactivate Config
+    
+    %% Initialize Services
+    UploadCmd->>Factory: getUploadService(false)
+    activate Factory
+    Factory->>Config: isApiModeAvailable()
+    Config-->>Factory: true/false
+    Factory->>APIService: new ApiUploadService()
+    activate APIService
+    APIService->>Config: getApiConfig()
+    Config-->>APIService: { baseUrl, token }
+    APIService-->>Factory: apiService instance
+    deactivate APIService
+    Factory-->>UploadCmd: uploadService
+    deactivate Factory
+    
+    UploadCmd->>Config: getUploadSources()
+    Config-->>UploadCmd: sources array (e.g., ['sample/2023'])
+    
+    UploadCmd->>Config: getBasePath()
+    Config-->>UploadCmd: basePath
+    
+    UploadCmd->>Logger: info("Starting upload with Arela API")
+    
+    %% Process Each Source
+    loop For each source in sources
+        Note over UploadCmd: Sequential source processing
+        UploadCmd->>UploadCmd: Resolve sourcePath
+        Note over UploadCmd: sourcePath = path.resolve(basePath, source)
+        
+        UploadCmd->>Logger: info("Processing folder: " + sourcePath)
+        
+        %% File Discovery
+        UploadCmd->>UploadCmd: #discoverFiles(sourcePath)
+        UploadCmd->>FileOps: fileExists(sourcePath)
+        FileOps-->>UploadCmd: true
+        
+        UploadCmd->>FileOps: getFileStats(sourcePath)
+        FileOps-->>UploadCmd: stats (isDirectory: true)
+        
+        UploadCmd->>Globby: globby([sourcePath/**/*], {onlyFiles: true})
+        activate Globby
+        Globby-->>UploadCmd: files array
+        deactivate Globby
+        
+        UploadCmd->>Logger: info("Found X files to process")
+        
+        %% Process Files in Batches
+        UploadCmd->>UploadCmd: #processFilesInBatches(files, options, ...)
+        Note over UploadCmd: batchSize = 10 (from options)
+        
+        UploadCmd->>ProgressBar: new SingleBar()
+        activate ProgressBar
+        UploadCmd->>ProgressBar: start(files.length, 0)
+        
+        loop For each batch of files
+            Note over UploadCmd: batch = files.slice(i, i + batchSize)
+            
+            UploadCmd->>UploadCmd: #processBatch(batch, options, ...)
+            
+            %% Stats Only Mode
+            Note over UploadCmd: options.statsOnly === true
+            
+            %% Create File Objects
+            loop For each file in batch
+                UploadCmd->>FileOps: getFileStats(filePath)
+                FileOps-->>UploadCmd: stats
+                Note over UploadCmd: Create fileObject:<br/>{path, originalName, stats}
+            end
+            
+            %% Insert Stats to Database
+            UploadCmd->>DBService: insertStatsOnlyToUploaderTable(fileObjects, options)
+            activate DBService
+            
+            DBService->>Logger: info("Collecting filesystem stats...")
+            
+            %% Filter System Files
+            loop For each file
+                Note over DBService: Skip system files:<br/>.DS_Store, __pycache__, etc.
+                
+                DBService->>DBService: Create record object
+                Note over DBService: record = {<br/>  name, documentType: null,<br/>  size, filename,<br/>  originalPath, arelaPath: null,<br/>  status: 'fs-stats',<br/>  rfc: null, fileExtension,<br/>  modifiedAt, year: null,<br/>  processingStatus: 'PENDING',<br/>  uploadAttempts: 0<br/>}
+            end
+            
+            DBService->>Logger: info("Processing X file stats in batches...")
+            
+            %% Get Upload Service
+            alt options.apiTarget specified
+                DBService->>Factory: getApiServiceForTarget(apiTarget)
+            else default
+                DBService->>Factory: getUploadService(false)
+            end
+            Factory-->>DBService: uploadService (ApiUploadService)
+            
+            %% Batch Processing (1000 records per API call)
+            loop For each batch of 1000 records
+                DBService->>APIService: batchUpsertStats(batch)
+                activate APIService
+                
+                APIService->>APIService: Prepare request
+                Note over APIService: URL: baseUrl + '/api/uploader/batch-upsert'<br/>Headers: x-api-key, Content-Type<br/>Body: JSON.stringify(records)
+                
+                APIService->>ArelaAPI: POST /api/uploader/batch-upsert
+                activate ArelaAPI
+                Note over ArelaAPI: Process batch upsert<br/>(insert or update by original_path)
+                ArelaAPI-->>APIService: { inserted, updated, total }
+                deactivate ArelaAPI
+                
+                APIService-->>DBService: result
+                deactivate APIService
+                
+                alt Batch % 10 === 0 (every 10th batch)
+                    DBService->>Logger: info("Batch N: X new, Y updates")
+                end
+            end
+            
+            DBService->>Logger: success("Phase 1 Summary: X new records, Y updates")
+            DBService-->>UploadCmd: { totalInserted, totalSkipped, totalProcessed }
+            deactivate DBService
+            
+            %% Update Results
+            Note over UploadCmd: batchResults.successCount = totalInserted<br/>batchResults.skippedCount = totalSkipped
+            
+            UploadCmd->>UploadCmd: #updateResults(results, batchResults)
+            
+            %% Update Progress Bar
+            UploadCmd->>ProgressBar: update(processedCount, {success, errors})
+            
+            %% Batch Delay
+            alt appConfig.performance.batchDelay > 0
+                UploadCmd->>UploadCmd: setTimeout(batchDelay)
+                Note over UploadCmd: Wait between batches
+            end
+        end
+        
+        UploadCmd->>ProgressBar: stop()
+        deactivate ProgressBar
+        
+        %% Source Summary
+        UploadCmd->>UploadCmd: #logSourceSummary(source, result, options)
+        UploadCmd->>User: Console: Summary for source<br/>📊 Stats recorded: X<br/>⏭️ Duplicates: Y<br/>❌ Errors: Z
+        
+        UploadCmd->>UploadCmd: #updateGlobalResults(globalResults, result)
+    end
+    
+    %% Final Summary
+    UploadCmd->>UploadCmd: #logFinalSummary(globalResults, options, uploadService)
+    UploadCmd->>User: Console: STATS COLLECTION COMPLETED<br/>📊 Total stats recorded: X<br/>⏭️ Total duplicates: Y<br/>❌ Total errors: Z<br/>📜 Log file: path
+    
+    %% Additional Phases (if --run-all-phases)
+    alt options.runAllPhases && options.statsOnly
+        UploadCmd->>UploadCmd: #runAdditionalPhases(options)
+        Note over UploadCmd: This would run Phase 2 (PDF Detection)<br/>Not executed with --stats-only alone
+    end
+    
+    UploadCmd-->>Commander: Complete
+    deactivate UploadCmd
+    Commander-->>CLI: Command executed
+    deactivate Commander
+    
+    CLI->>Logger: flush()
+    CLI-->>User: Exit code 0
+    deactivate CLI
+```
+
+## Key Flow Points
+
+### 1. **Initialization Phase**
+- CLI instantiates main components (ErrorHandler, UploadCommand, WatchCommand)
+- Sets up Commander.js with all available commands
+- Registers error handlers for graceful shutdown
+
+### 2. **Command Parsing Phase**
+- Commander.js parses arguments and identifies 'stats' command
+- Options are merged: `{ ...options, statsOnly: true }`
+- API target is configured if specified
+
+### 3. **Validation Phase**
+- Validates configuration (API URL, token)
+- Validates batch size (1-100)
+- Checks watch mode isn't active
+
+### 4. **Service Initialization Phase**
+- UploadServiceFactory determines available service (API vs Supabase)
+- Creates ApiUploadService with HTTP agent configuration
+- Retrieves upload sources and base path from config
+
+### 5. **File Discovery Phase**
+- For each source directory
+- Uses globby to recursively find all files
+- Filters to only files (not directories)
+
+### 6. **Batch Processing Phase**
+- Files are processed in configurable batches (default: 10)
+- For each batch:
+  - Reads file stats (size, mtime)
+  - Filters out system files (.DS_Store, __pycache__, etc.)
+  - Creates record objects with status='fs-stats'
+  - Groups records into API batches (1000 per call)
+
+### 7. **Database Recording Phase**
+- ApiUploadService makes POST request to `/api/uploader/batch-upsert`
+- API performs upsert (insert or update by original_path)
+- Returns counts: inserted, updated, total
+- Progress bar updates after each batch
+
+### 8. **Summary Phase**
+- Logs per-source summary (stats recorded, duplicates, errors)
+- Logs final summary across all sources
+- Displays log file path
+
+### 9. **Cleanup Phase**
+- Logger flushes buffered logs
+- Process exits gracefully with code 0
+
+## Important Notes
+
+- **Stats-only mode**: Does NOT detect document types, does NOT upload files
+- **Record status**: All records get `status='fs-stats'` and `processingStatus='PENDING'`
+- **Upsert behavior**: Records are inserted or updated by `original_path` (prevents duplicates)
+- **Batch sizes**: Files batched at 10 (configurable), API calls batched at 1000 records
+- **System files**: Automatically filtered (.DS_Store, __pycache__, .swp, etc.)
+- **Progress tracking**: Real-time progress bar with success/error counts
+- **HTTP agent**: Connection pooling optimized for multiple API replicas (10 concurrent connections)

package/docs/STATS_COMMAND_SIMPLE.md

@@ -0,0 +1,93 @@
+# Stats Command - Simplified Flow
+
+## Command: `node src/index.js stats --stats-only`
+
+A high-level overview of the stats collection process.
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant CLI as CLI Application
+    participant UploadCmd as Upload Command
+    participant Config as Configuration
+    participant FileSystem as File System
+    participant Database as Database Service
+    participant API as Arela API
+
+    User->>CLI: stats --stats-only
+    activate CLI
+    
+    CLI->>UploadCmd: Execute stats command
+    activate UploadCmd
+    
+    UploadCmd->>Config: Get sources & validate config
+    Config-->>UploadCmd: Sources array & settings
+    
+    loop For each source directory
+        UploadCmd->>FileSystem: Discover all files
+        FileSystem-->>UploadCmd: List of file paths
+        
+        loop Process in batches
+            UploadCmd->>FileSystem: Read file stats (size, mtime)
+            FileSystem-->>UploadCmd: File metadata
+            
+            UploadCmd->>Database: Insert stats records
+            activate Database
+            
+            Database->>Database: Filter system files<br/>Create record objects<br/>(status='fs-stats')
+            
+            Database->>API: POST /api/uploader/batch-upsert
+            activate API
+            API-->>Database: {inserted, updated}
+            deactivate API
+            
+            Database-->>UploadCmd: Success count
+            deactivate Database
+            
+            UploadCmd->>User: Update progress bar
+        end
+        
+        UploadCmd->>User: Source summary
+    end
+    
+    UploadCmd->>User: Final summary<br/>📊 Total stats recorded<br/>⏭️ Duplicates<br/>❌ Errors
+    
+    deactivate UploadCmd
+    deactivate CLI
+```
+
+## Process Overview
+
+### 1. **Command Initialization**
+- CLI parses `stats --stats-only` command
+- Validates configuration (API URL, token)
+- Initializes Upload Command
+
+### 2. **File Discovery**
+- Retrieves source directories from configuration
+- Recursively scans for all files in each source
+- Filters out directories, keeps only files
+
+### 3. **Stats Collection**
+- Processes files in batches (default: 10 files)
+- Reads filesystem metadata (size, modified time, extension)
+- Filters out system files (.DS_Store, __pycache__, etc.)
+
+### 4. **Database Recording**
+- Creates record objects with `status='fs-stats'`
+- Sends batches of 1000 records to API
+- API performs upsert by `original_path`
+- Returns inserted/updated counts
+
+### 5. **Progress & Summary**
+- Real-time progress bar during processing
+- Per-source summary after each directory
+- Final summary with total counts
+
+## Key Characteristics
+
+- **No File Upload**: Files are not uploaded, only metadata is recorded
+- **No Detection**: Document types are not detected (all `null`)
+- **Duplicate Prevention**: Upsert by `original_path` prevents duplicates
+- **Batch Processing**: Efficient handling of large file sets
+- **Status**: All records marked as `fs-stats` and `PENDING`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arela/uploader",
-  "version": "1.0.1",
+  "version": "1.0.3",
   "description": "CLI to upload files/directories to Arela",
   "bin": {
     "arela": "./src/index.js"
@@ -38,7 +38,9 @@
     "globby": "14.1.0",
     "mime-types": "3.0.1",
     "node-fetch": "3.3.2",
-    "office-text-extractor": "3.0.3"
+    "office-text-extractor": "3.0.3",
+    "p-limit": "^7.2.0",
+    "pdf-parse": "^2.4.5"
   },
   "devDependencies": {
     "@trivago/prettier-plugin-sort-imports": "5.2.2",