@tb.p/dd 1.2.0 → 1.3.1

package/ISSUES_FOUND.md

@@ -0,0 +1,186 @@
+# Potential Issues Found in @tb.p/dd Codebase
+
+## 🔴 Critical Issues
+
+### 1. **Missing UNIQUE Constraint on file_path** (Data Integrity)
+**Location:** `database/dbConnection.js:69`
+**Issue:** The `copies` table doesn't have a UNIQUE constraint on `file_path`, allowing duplicate entries for the same file path.
+```sql
+CREATE TABLE IF NOT EXISTS copies (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  file_path TEXT NOT NULL,  -- No UNIQUE constraint!
+  ...
+)
+```
+**Impact:** 
+- During resume operations, if a file path exists but isn't in the `existingFilePaths` Set (due to timing), duplicate entries can be created
+- No database-level protection against duplicate file paths
+- Could lead to incorrect duplicate detection results
+**Fix:** Add `UNIQUE(file_path)` constraint or use `INSERT OR IGNORE` / `INSERT OR REPLACE` logic
+
+### 2. **Missing Database Connection Check** (Runtime Error)
+**Location:** `database/dbConnection.js:134-144`
+**Issue:** `query()` and `queryOne()` methods don't verify the database is connected before executing queries.
+```javascript
+async query(sql, params = []) {
+  return new Promise((resolve, reject) => {
+    this.db.all(sql, params, (err, rows) => {  // No check if db is null!
+```
+**Impact:** If called before `connect()`, will throw cryptic "Cannot read property 'all' of null" errors instead of clear error messages.
+**Fix:** Add connection check: `if (!this.isConnected || !this.db) throw new Error('Database not connected')`
+
+### 3. **Memory Issue with Large Files** (Performance/Scalability)
+**Location:** `utils/fileHasher.js:148`
+**Issue:** BLAKE3 hash calculation reads the entire file into memory using `fs.promises.readFile()`.
+```javascript
+const fileBuffer = await fs.promises.readFile(filePath);
+result.hash = await blake3(fileBuffer);
+```
+**Impact:** For very large files (GB+), this can cause out-of-memory errors.
+**Fix:** Use streaming approach similar to other hash algorithms, or add file size limits.
+
+## 🟡 Medium Priority Issues
+
+### 4. **Missing Composite Database Index** (Performance)
+**Location:** `database/dbConnection.js:100-106`
+**Issue:** Missing composite index on `(file_hash, active)` which is frequently queried together.
+**Current indexes:**
+- `idx_copies_file_hash` (single column)
+- No composite index for `(file_hash, active)`
+
+**Impact:** Queries like `WHERE file_hash IS NOT NULL AND active = 1` may be slower on large databases.
+**Fix:** Add composite index: `CREATE INDEX IF NOT EXISTS idx_copies_hash_active ON copies(file_hash, active);`
+
+### 5. **Path Comparison Issue on Windows** (Cross-platform)
+**Location:** `utils/duplicateMover.js:72-73`
+**Issue:** Path comparison uses `startsWith()` which may fail on Windows with different path separators or case sensitivity.
+```javascript
+const absoluteTarget = path.resolve(target);
+if (fileDir.startsWith(absoluteTarget)) {
+```
+**Impact:** Files may not be correctly associated with their target directories on Windows.
+**Fix:** Use `path.relative()` or normalize paths before comparison.
+
+### 6. **Missing Active Filter in getUnhashedCopies** (Logic Bug)
+**Location:** `database/dbOperations.js:69`
+**Issue:** `getUnhashedCopies()` doesn't filter by `active = 1`, but other parts of the code expect only active files.
+```javascript
+let sql = 'SELECT * FROM copies WHERE file_hash IS NULL ORDER BY file_path';
+```
+**Impact:** Inactive files may be processed unnecessarily during resume operations.
+**Fix:** Add `AND active = 1` filter if that's the intended behavior.
+
+### 7. **Missing Index on (file_size, active)** (Performance)
+**Location:** `database/dbConnection.js:100-106`
+**Issue:** Queries frequently filter by `file_size` and `active` together (e.g., in candidate detection).
+**Impact:** `getFilesBySize()` queries may be slower on large databases.
+**Fix:** Add composite index: `CREATE INDEX IF NOT EXISTS idx_copies_size_active ON copies(file_size, active);`
+
+### 8. **Race Condition in Resume Controller** (Data Integrity)
+**Location:** `controllers/resumeController.js:157-182`
+**Issue:** `getExistingFilePaths()` is loaded once at the start. If a file is added during the loop, the Set becomes stale, potentially allowing duplicate inserts.
+**Impact:** Duplicate file entries could be created, especially if the same file path appears in multiple target directories.
+**Fix:** 
+- Add UNIQUE constraint on `file_path` (see issue #1)
+- Use `INSERT OR IGNORE` in `addCopy()` method
+- Or reload the Set periodically during long operations
+
+## 🟢 Low Priority / Code Quality Issues
+
+### 9. **Inconsistent Error Handling**
+**Location:** Multiple files
+**Issue:** Some database operations return `{success, error}` objects, others throw exceptions.
+**Impact:** Inconsistent error handling patterns make code harder to maintain.
+**Suggestion:** Standardize error handling approach across the codebase.
+
+### 10. **Missing Input Validation**
+**Location:** `processors/optionETL.js:32-34`
+**Issue:** `normalizePaths()` doesn't validate that paths are actually valid before resolving.
+**Impact:** Invalid paths may cause errors later in processing.
+**Suggestion:** Add path validation early in the pipeline.
+
+### 11. **Potential Stack Overflow with Very Large Arrays**
+**Location:** `controllers/newController.js:52-55`
+**Issue:** While there's a comment about avoiding stack overflow, the sorting operation could still be problematic.
+```javascript
+const sortedFiles = Array.from(fileMap.values()).sort((a, b) => a.priority - b.priority);
+```
+**Impact:** With millions of files, this could cause issues.
+**Suggestion:** Consider streaming/chunked processing for very large datasets.
+
+### 12. **Missing Transaction Wrappers**
+**Location:** `database/dbOperations.js`
+**Issue:** Multiple related operations (e.g., setting all inactive, then activating specific files) aren't wrapped in transactions.
+**Impact:** Database could be left in inconsistent state if operation fails mid-way.
+**Suggestion:** Wrap related operations in transactions.
+
+### 13. **File Extension Edge Cases**
+**Location:** `processors/optionETL.js:18-20`
+**Issue:** `normalizeExtensions()` removes leading dots, but doesn't handle empty strings or special cases.
+```javascript
+return ext.replace(/^\.+/, '').toLowerCase();
+```
+**Impact:** Files with no extension or unusual extension formats may not be handled correctly.
+**Suggestion:** Add validation and edge case handling.
+
+### 14. **Missing Database Connection Check**
+**Location:** `database/dbConnection.js:135-144`
+**Issue:** `query()` and `queryOne()` methods don't check if database is connected before executing.
+**Impact:** Could cause cryptic errors if called before `connect()`.
+**Suggestion:** Add connection check or use the existing `getDatabase()` method.
+
+### 15. **Hardcoded Timeout Value**
+**Location:** `utils/fileScanner.js:25`
+**Issue:** 5-minute timeout is hardcoded.
+```javascript
+const timeout = setTimeout(() => {
+  reject(new Error('Directory scan timeout...'));
+}, 300000); // 5 minute timeout
+```
+**Impact:** Not configurable for different use cases.
+**Suggestion:** Make timeout configurable via options.
+
+## ✅ Verified Non-Issues (Corrected)
+
+### **SQL Injection** - NOT AN ISSUE
+**Location:** `database/dbOperations.js:72-88`
+**Status:** Code correctly uses parameterized queries with validation:
+```javascript
+const limitInt = parseInt(limit, 10);
+if (isNaN(limitInt) || limitInt < 0) {
+  throw new Error('Invalid limit value');
+}
+sql += ' LIMIT ?';
+params.push(limitInt);
+```
+The code validates inputs and uses `?` placeholders. This is safe.
+
+### **Double Promise Resolution** - NOT AN ISSUE  
+**Location:** `database/dbConnection.js:41-43`
+**Status:** Code correctly resolves only once after `createTables()` completes:
+```javascript
+this.createTables()
+  .then(() => resolve())  // Only one resolve, after tables are created
+  .catch(reject);
+```
+The promise resolves only after tables are created. This is correct.
+
+## Summary
+
+**Critical:** 3 issues (missing UNIQUE constraint, missing connection check, memory issue)
+**Medium:** 5 issues (performance indexes, path comparison, logic bugs)
+**Low:** 7 issues (code quality, edge cases, maintainability)
+
+**Total:** 15 potential issues identified (2 previously identified issues were false positives)
+
+## Recommended Priority Order for Fixes
+
+1. Add UNIQUE constraint on file_path (data integrity - prevents duplicates)
+2. Add database connection check (better error messages)
+3. Fix memory issue with large files (scalability)
+4. Add missing database indexes (performance)
+5. Fix path comparison for cross-platform (compatibility)
+6. Add active filter to getUnhashedCopies (logic)
+7. Add transaction wrappers (data integrity)
+8. Address remaining code quality issues
+

package/database/dbConnection.js

@@ -31,18 +31,17 @@ class DatabaseConnection {
             reject(new Error(`Failed to connect to database: ${err.message}`));
             return;
           }
-          
+
           this.isConnected = true;
-          resolve();
-        });
 
-        // Enable foreign key constraints
-        this.db.run('PRAGMA foreign_keys = ON');
-        
-        // Create tables if they don't exist
-        this.createTables()
-          .then(() => resolve())
-          .catch(reject);
+          // Enable foreign key constraints
+          this.db.run('PRAGMA foreign_keys = ON');
+
+          // Create tables if they don't exist
+          this.createTables()
+            .then(() => resolve())
+            .catch(reject);
+        });
 
       } catch (error) {
         reject(error);
@@ -67,7 +66,7 @@ class DatabaseConnection {
         CREATE TABLE IF NOT EXISTS copies (
           id INTEGER PRIMARY KEY AUTOINCREMENT,
           dir_group TEXT,
-          file_path TEXT NOT NULL,
+          file_path TEXT NOT NULL UNIQUE,
           file_name TEXT NOT NULL,
           file_extension TEXT,
           file_size INTEGER NOT NULL,
@@ -103,6 +102,8 @@ class DatabaseConnection {
         CREATE INDEX IF NOT EXISTS idx_copies_file_name ON copies(file_name);
         CREATE INDEX IF NOT EXISTS idx_copies_dir_group ON copies(dir_group);
         CREATE INDEX IF NOT EXISTS idx_copies_extension ON copies(file_extension);
+        CREATE INDEX IF NOT EXISTS idx_copies_hash_active ON copies(file_hash, active);
+        CREATE INDEX IF NOT EXISTS idx_copies_size_active ON copies(file_size, active);
       `;
 
       this.db.exec(createIndexesSQL, (err) => {

package/database/dbOperations.js

@@ -51,7 +51,7 @@ class DatabaseOperations {
       } = fileInfo;
       
       const result = await this.db.run(
-        'INSERT INTO copies (dir_group, file_path, file_name, file_extension, file_size, file_hash, active, priority) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
+        'INSERT OR IGNORE INTO copies (dir_group, file_path, file_name, file_extension, file_size, file_hash, active, priority) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
         [dirGroup, path, name, extension, size, hash, active, priority]
       );
       return { success: true, id: result.lastID, changes: result.changes };
@@ -67,15 +67,29 @@ class DatabaseOperations {
     try {
       const { limit, offset } = options;
       let sql = 'SELECT * FROM copies WHERE file_hash IS NULL ORDER BY file_path';
-      
+      const params = [];
+
       if (limit) {
-        sql += ` LIMIT ${limit}`;
+        // Validate and sanitize limit as integer
+        const limitInt = parseInt(limit, 10);
+        if (isNaN(limitInt) || limitInt < 0) {
+          throw new Error('Invalid limit value');
+        }
+        sql += ' LIMIT ?';
+        params.push(limitInt);
+
         if (offset) {
-          sql += ` OFFSET ${offset}`;
+          // Validate and sanitize offset as integer
+          const offsetInt = parseInt(offset, 10);
+          if (isNaN(offsetInt) || offsetInt < 0) {
+            throw new Error('Invalid offset value');
+          }
+          sql += ' OFFSET ?';
+          params.push(offsetInt);
         }
       }
 
-      const results = await this.db.query(sql);
+      const results = await this.db.query(sql, params);
       return results;
     } catch (error) {
       throw new Error(`Failed to get unhashed copies: ${error.message}`);

package/index.js

@@ -15,7 +15,7 @@ const program = new Command();
 program
   .name('@tb.p/dd')
   .description('File Deduplication Tool')
-  .version('1.0.0')
+  .version('1.3.1')
   .option('-t, --targets <string>', 'Pipe-separated paths to directories', process.cwd())
   .option('-e, --extensions <string>', 'Pipe-separated file extensions')
   .option('-g, --get-extensions', 'Get file extensions in target directories')

package/package.json

@@ -1,33 +1,41 @@
-{
-  "name": "@tb.p/dd",
-  "version": "1.2.0",
-  "description": "A comprehensive command-line tool for finding and removing duplicate files using content-based hashing",
-  "type": "module",
-  "main": "index.js",
-  "bin": {
-    "@tb.p/dd": "./index.js"
-  },
-  "scripts": {
-    "start": "node index.js",
-    "test": "echo \"Error: no test specified\" && exit 1"
-  },
-  "keywords": [
-    "duplicate",
-    "files",
-    "deduplication",
-    "hash",
-    "cli",
-    "file-management"
-  ],
-  "author": "@tb.p",
-  "license": "MIT",
-  "dependencies": {
-    "commander": "^11.1.0",
-    "hash-wasm": "^4.12.0",
-    "recursive-readdir": "^2.2.3",
-    "sqlite3": "^5.1.6"
-  },
-  "engines": {
-    "node": ">=14.0.0"
-  }
-}
+{
+  "name": "@tb.p/dd",
+  "version": "1.3.1",
+  "description": "A comprehensive command-line tool for finding and removing duplicate files using content-based hashing",
+  "type": "module",
+  "main": "index.js",
+  "bin": {
+    "@tb.p/dd": "./index.js"
+  },
+  "scripts": {
+    "start": "node index.js",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [
+    "duplicate",
+    "files",
+    "deduplication",
+    "hash",
+    "cli",
+    "file-management"
+  ],
+  "author": "@tb.p",
+  "license": "MIT",
+  "dependencies": {
+    "commander": "^11.1.0",
+    "hash-wasm": "^4.12.0",
+    "recursive-readdir": "^2.2.3",
+    "sqlite3": "^5.1.6"
+  },
+  "engines": {
+    "node": ">=14.0.0"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/jhasselbring/dd.git"
+  },
+  "bugs": {
+    "url": "https://github.com/jhasselbring/dd/issues"
+  },
+  "homepage": "https://github.com/jhasselbring/dd#readme"
+}

package/utils/duplicateMover.js

@@ -68,9 +68,11 @@ async function moveDuplicates(options) {
           // Find which target directory this file belongs to
           let targetDir = null;
           for (const target of options.targets) {
-            // Resolve both paths to absolute for proper comparison
+            // Resolve both paths to absolute and normalize for cross-platform comparison
             const absoluteTarget = path.resolve(target);
-            if (fileDir.startsWith(absoluteTarget)) {
+            const normalizedFileDir = path.normalize(fileDir).toLowerCase();
+            const normalizedTarget = path.normalize(absoluteTarget).toLowerCase();
+            if (normalizedFileDir === normalizedTarget || normalizedFileDir.startsWith(normalizedTarget + path.sep)) {
               targetDir = target;
               break;
             }

package/utils/fileScanner.js

@@ -18,13 +18,23 @@ async function scanDirectory(dirPath, options = {}) {
   }
 
   try {
+    // Ignore function to exclude dotfiles and dot directories
+    const ignoreFunc = (file, stats) => {
+      const basename = path.basename(file);
+      // Skip files and directories that start with a dot
+      if (basename.startsWith('.')) {
+        return true;
+      }
+      return false;
+    };
+
     const filePaths = await new Promise((resolve, reject) => {
       // Add timeout to prevent hanging on very large directories
       const timeout = setTimeout(() => {
         reject(new Error('Directory scan timeout - directory may be too large or contain circular references'));
       }, 300000); // 5 minute timeout
-      
-      recursive(dirPath, (err, files) => {
+
+      recursive(dirPath, [ignoreFunc], (err, files) => {
         clearTimeout(timeout);
         if (err) reject(err);
         else resolve(files);