@kadi.build/file-manager 1.0.0

package/README.md

@@ -0,0 +1,268 @@
+# @kadi.build/file-manager
+
+> Foundational file management package providing local and remote file operations, file watching, compression utilities, and file streaming.
+
+## Features
+
+- **Local File Operations** — Upload, download, copy, move, rename, delete files and folders
+- **Remote File Operations** — SSH/SFTP-based remote operations with bastion/jump host support
+- **File Watching** — Real-time file system monitoring with configurable debouncing and filtering
+- **Compression** — ZIP and TAR.GZ compression/decompression with progress tracking and batch operations
+- **File Streaming** — Memory-efficient streaming with range request support, ETag generation, and download tracking
+
+## Installation
+
+```bash
+npm install @kadi.build/file-manager
+```
+
+## Quick Start
+
+```javascript
+import { createFileManager } from '@kadi.build/file-manager';
+
+// Create a file manager instance
+const fm = await createFileManager();
+
+// Local file operations
+await fm.uploadFile('./source.txt', './destination.txt');
+const info = await fm.getFileInfo('./myfile.txt');
+const files = await fm.listFiles('./my-directory');
+
+// Compression
+await fm.compressFile('./folder', './backup.zip', { format: 'zip' });
+await fm.decompressFile('./backup.zip', './extracted/');
+
+// File watching
+const watcher = await fm.startWatching('./watched-dir');
+fm.on('fileEvent', (event) => {
+  console.log(`${event.type}: ${event.path}`);
+});
+```
+
+## API Reference
+
+### FileManager
+
+The main orchestrator that provides a unified interface to all providers.
+
+```javascript
+import { FileManager, ConfigManager } from '@kadi.build/file-manager';
+
+const config = new ConfigManager();
+await config.load();
+const fm = new FileManager(config);
+```
+
+#### Connection & Validation
+
+| Method | Description |
+|--------|-------------|
+| `testConnection(providerName)` | Test connectivity for a specific provider (`'local'`, `'watch'`, `'compression'`, `'remote'`) |
+| `validateProvider(providerName)` | Validate provider configuration |
+| `getSystemInfo()` | Get system information (platform, memory, disk space) |
+
+#### File Operations
+
+| Method | Description |
+|--------|-------------|
+| `uploadFile(source, dest, providerName?)` | Copy/upload a file |
+| `downloadFile(source, dest, providerName?)` | Copy/download a file |
+| `getFileInfo(filePath, providerName?)` | Get file metadata |
+| `listFiles(directory, providerName?, options?)` | List files in a directory |
+| `deleteFile(filePath, providerName?)` | Delete a file |
+| `renameFile(oldPath, newPath, providerName?)` | Rename a file |
+| `copyFile(source, dest, providerName?)` | Copy a file |
+| `moveFile(source, dest, providerName?)` | Move a file |
+| `searchFiles(directory, pattern, options?)` | Search for files matching a pattern |
+
+#### Folder Operations
+
+| Method | Description |
+|--------|-------------|
+| `createFolder(folderPath, providerName?)` | Create a directory (recursive) |
+| `listFolders(directory, providerName?)` | List subdirectories |
+| `deleteFolder(folderPath, options?)` | Delete a directory |
+| `renameFolder(oldPath, newPath)` | Rename a directory |
+| `copyFolder(source, dest)` | Copy a directory recursively |
+| `moveFolder(source, dest)` | Move a directory |
+| `getFolderInfo(folderPath)` | Get directory metadata |
+
+#### Compression
+
+| Method | Description |
+|--------|-------------|
+| `compressFile(input, output, options?)` | Compress a file or directory |
+| `decompressFile(archive, outputDir, options?)` | Decompress an archive |
+| `compress(input, output, options?)` | Alias for `compressFile` |
+| `decompress(archive, outputDir, options?)` | Alias for `decompressFile` |
+| `compressMultipleFiles(files, outputDir, options?)` | Batch compress multiple files |
+| `decompressMultipleFiles(archives, outputDir, options?)` | Batch decompress multiple archives |
+| `getCompressionStatus()` | Get compression provider status |
+| `getCompressionProvider()` | Get the underlying compression provider |
+
+**Compression Options:**
+```javascript
+{
+  format: 'zip' | 'tar.gz',  // Archive format (default: 'zip')
+  level: 1-9,                  // Compression level (default: 6)
+  includeRoot: true,           // Include root directory in archive
+  overwrite: true              // Overwrite existing files during extraction
+}
+```
+
+#### File Watching
+
+| Method | Description |
+|--------|-------------|
+| `startWatching(path, options?)` | Start watching a path for changes |
+| `stopWatching(watchIdOrPath)` | Stop a specific watcher |
+| `stopAllWatching()` | Stop all active watchers |
+| `listActiveWatchers()` | List all active watchers |
+| `getWatcherInfo(watchId)` | Get details about a watcher |
+| `getWatchingStatus()` | Get overall watching status |
+
+**Watch Options:**
+```javascript
+{
+  recursive: true,         // Watch subdirectories
+  events: ['add', 'change', 'unlink'],  // Event types to listen for
+  ignoreDotfiles: false,   // Ignore dotfiles
+  debounceMs: 100          // Debounce interval in ms
+}
+```
+
+**Events:**
+```javascript
+fm.on('fileEvent', (event) => {
+  // event.type: 'add' | 'change' | 'unlink' | 'addDir' | 'unlinkDir'
+  // event.path: string
+  // event.watchId: string
+});
+```
+
+#### Remote Operations
+
+| Method | Description |
+|--------|-------------|
+| `connectRemote()` | Connect to the remote server |
+| `disconnectRemote()` | Disconnect from the remote server |
+| `syncToRemote(localPath, remotePath)` | Upload files to remote |
+| `syncFromRemote(remotePath, localPath)` | Download files from remote |
+
+**Remote Configuration (via environment variables):**
+```
+KADI_REMOTE_HOST=example.com
+KADI_REMOTE_PORT=22
+KADI_REMOTE_USERNAME=user
+KADI_REMOTE_PASSWORD=pass
+KADI_REMOTE_PRIVATE_KEY_PATH=~/.ssh/id_rsa
+```
+
+### Individual Providers
+
+For advanced use cases, you can import providers directly:
+
+```javascript
+import {
+  LocalProvider,
+  WatchProvider,
+  CompressionProvider,
+  RemoteProvider,
+  FileStreamingUtils,
+  FileStreamer,
+  DownloadTracker,
+  PathUtils
+} from '@kadi.build/file-manager';
+```
+
+### Factory Functions
+
+```javascript
+import { createFileManager, compressFile, decompressFile, watchDirectory } from '@kadi.build/file-manager';
+
+// Quick file manager creation
+const fm = await createFileManager(options);
+
+// One-shot compression
+await compressFile('./input', './output.zip', { format: 'zip' });
+
+// One-shot decompression
+await decompressFile('./archive.zip', './output/');
+
+// Quick directory watching
+const watcher = await watchDirectory('./path', (event) => {
+  console.log(event);
+});
+```
+
+## Configuration
+
+Configuration is loaded from multiple sources (in priority order):
+
+1. Constructor options
+2. Environment variables (prefixed with `KADI_`)
+3. `.env` file
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `KADI_LOCAL_ROOT` | Root directory for local operations | `.` |
+| `KADI_WATCH_DEBOUNCE_MS` | Debounce interval for file watching | `100` |
+| `KADI_WATCH_MAX_WATCHERS` | Maximum concurrent watchers | `10` |
+| `KADI_COMPRESSION_FORMAT` | Default compression format | `zip` |
+| `KADI_COMPRESSION_LEVEL` | Default compression level | `6` |
+| `KADI_REMOTE_HOST` | Remote SSH host | — |
+| `KADI_REMOTE_PORT` | Remote SSH port | `22` |
+| `KADI_REMOTE_USERNAME` | Remote SSH username | — |
+| `KADI_REMOTE_PASSWORD` | Remote SSH password | — |
+| `KADI_REMOTE_PRIVATE_KEY_PATH` | Path to SSH private key | — |
+
+## Testing
+
+```bash
+# Run all tests
+npm test
+
+# Run individual test suites
+npm run test:local        # Local file operations (33 tests)
+npm run test:watch        # File watching (30 tests)
+npm run test:compression  # Compression (42 tests)
+npm run test:streaming    # File streaming (12 tests)
+npm run test:remote       # Remote operations (18 tests)
+```
+
+**Total: 135 tests across 5 suites**
+
+## Project Structure
+
+```
+packages/file-manager/
+├── package.json
+├── README.md
+├── src/
+│   ├── index.js              # Main entry point & exports
+│   ├── FileManager.js        # Main orchestrator
+│   ├── ConfigManager.js      # Configuration management
+│   ├── providers/
+│   │   ├── LocalProvider.js       # Local file operations
+│   │   ├── WatchProvider.js       # File watching (chokidar)
+│   │   ├── CompressionProvider.js # ZIP & TAR.GZ compression
+│   │   └── RemoteProvider.js      # SSH/SFTP remote operations
+│   └── utils/
+│       ├── FileStreamingUtils.js  # Streaming, MIME detection, ETags
+│       └── PathUtils.js           # Path validation & normalization
+└── tests/
+    ├── run-all-tests.js
+    ├── test-utils.js
+    ├── test-helpers/
+    │   └── index.js
+    ├── test-local-operations.js
+    ├── test-file-watching.js
+    ├── test-compression.js
+    ├── test-file-streaming.js
+    └── test-remote-operations.js
+```
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,48 @@
+{
+  "name": "@kadi.build/file-manager",
+  "version": "1.0.0",
+  "description": "Complete local and remote file management with watching and compression",
+  "main": "src/index.js",
+  "type": "module",
+  "exports": {
+    ".": {
+      "import": "./src/index.js"
+    }
+  },
+  "files": [
+    "src/",
+    "README.md"
+  ],
+  "scripts": {
+    "test": "node tests/run-all-tests.js",
+    "test:local": "node tests/test-local-operations.js",
+    "test:watch": "node tests/test-file-watching.js",
+    "test:compression": "node tests/test-compression.js",
+    "test:streaming": "node tests/test-file-streaming.js",
+    "test:remote": "node tests/test-remote-operations.js"
+  },
+  "keywords": [
+    "file-management",
+    "file-operations",
+    "file-watching",
+    "compression",
+    "sftp",
+    "ssh",
+    "remote-files"
+  ],
+  "dependencies": {
+    "chokidar": "^3.5.3",
+    "archiver": "^6.0.0",
+    "unzipper": "^0.10.14",
+    "tar": "^6.2.0",
+    "ssh2": "^1.15.0",
+    "ssh2-sftp-client": "^9.1.0"
+  },
+  "devDependencies": {
+    "fs-extra": "^11.2.0"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "license": "MIT"
+}

package/src/ConfigManager.js

@@ -0,0 +1,301 @@
+/**
+ * ConfigManager for @kadi.build/file-manager
+ *
+ * Handles configuration for file operations only.
+ * Simplified from the monolith ConfigManager - no tunnel/server config.
+ */
+
+import { promises as fs } from 'fs';
+import path from 'path';
+import os from 'os';
+
+class ConfigManager {
+  constructor() {
+    this.config = {};
+    this.defaults = {
+      // Local File System Configuration
+      LOCAL_ROOT: process.cwd(),
+      DEFAULT_UPLOAD_DIRECTORY: './uploads',
+      DEFAULT_DOWNLOAD_DIRECTORY: './downloads',
+      DEFAULT_TEMP_DIRECTORY: './temp',
+
+      // File Operation Configuration
+      MAX_FILE_SIZE: 1073741824, // 1GB in bytes
+      CHUNK_SIZE: 8388608, // 8MB chunks
+      MAX_RETRY_ATTEMPTS: 3,
+      TIMEOUT_MS: 30000,
+
+      // File Watching Configuration
+      WATCH_ENABLED: true,
+      WATCH_RECURSIVE: true,
+      WATCH_IGNORE_DOTFILES: true,
+      WATCH_DEBOUNCE_MS: 100,
+      WATCH_MAX_WATCHERS: 50,
+
+      // Compression Configuration
+      COMPRESSION_ENABLED: true,
+      COMPRESSION_LEVEL: 6,
+      COMPRESSION_FORMAT: 'zip',
+      COMPRESSION_MAX_FILE_SIZE: 1073741824,
+      COMPRESSION_CHUNK_SIZE: 8388608,
+      COMPRESSION_ENABLE_PROGRESS_TRACKING: true,
+      COMPRESSION_ENABLE_CHECKSUM_VERIFICATION: true,
+
+      // Remote Configuration
+      REMOTE_HOST: '',
+      REMOTE_PORT: 22,
+      REMOTE_USERNAME: '',
+      REMOTE_PASSWORD: '',
+      REMOTE_PRIVATE_KEY_PATH: '~/.ssh/id_rsa',
+      REMOTE_PASSPHRASE: '',
+      REMOTE_ROOT: '/home',
+      REMOTE_TIMEOUT: 30000,
+      REMOTE_BASTION_HOST: '',
+      REMOTE_BASTION_PORT: 22,
+
+      // Performance Configuration
+      MAX_CONCURRENT_OPERATIONS: 5,
+      ENABLE_PROGRESS_TRACKING: true,
+      ENABLE_CHECKSUM_VERIFICATION: true,
+
+      // Security Configuration
+      ALLOW_SYMLINKS: false,
+      RESTRICT_TO_BASE_PATH: true,
+      MAX_PATH_LENGTH: 255,
+
+      // Logging
+      LOG_LEVEL: 'info'
+    };
+  }
+
+  async load(options = {}) {
+    // Apply defaults
+    this.config = { ...this.defaults };
+
+    // Load from .env file if it exists
+    try {
+      const envPath = path.join(process.cwd(), '.env');
+      const envContent = await fs.readFile(envPath, 'utf8');
+      this._parseEnvContent(envContent);
+    } catch (error) {
+      // .env file not found, that's fine
+    }
+
+    // Load from environment variables
+    this._loadFromEnvironment();
+
+    // Apply passed options (highest priority)
+    if (options) {
+      for (const [key, value] of Object.entries(options)) {
+        // Support both UPPER_CASE config keys and camelCase options
+        const upperKey = key.replace(/([A-Z])/g, '_$1').toUpperCase().replace(/^_/, '');
+        if (this.defaults.hasOwnProperty(upperKey)) {
+          this.config[upperKey] = value;
+        } else if (this.defaults.hasOwnProperty(key)) {
+          this.config[key] = value;
+        } else {
+          // Store as-is for custom options
+          this.config[key] = value;
+        }
+      }
+    }
+
+    // Ensure required directories exist
+    await this._ensureDirectories();
+
+    return this;
+  }
+
+  _parseEnvContent(content) {
+    const lines = content.split('\n');
+    for (const line of lines) {
+      const trimmed = line.trim();
+      if (!trimmed || trimmed.startsWith('#')) continue;
+
+      const eqIndex = trimmed.indexOf('=');
+      if (eqIndex === -1) continue;
+
+      const key = trimmed.substring(0, eqIndex).trim();
+      let value = trimmed.substring(eqIndex + 1).trim();
+
+      // Remove surrounding quotes
+      if ((value.startsWith('"') && value.endsWith('"')) ||
+          (value.startsWith("'") && value.endsWith("'"))) {
+        value = value.slice(1, -1);
+      }
+
+      // Only apply if it's a known config key
+      if (this.defaults.hasOwnProperty(key)) {
+        this.config[key] = this._parseValue(value, this.defaults[key]);
+      }
+    }
+  }
+
+  _loadFromEnvironment() {
+    for (const key of Object.keys(this.defaults)) {
+      // Check KADI_ prefixed env vars
+      const envKey = `KADI_${key}`;
+      if (process.env[envKey] !== undefined) {
+        this.config[key] = this._parseValue(process.env[envKey], this.defaults[key]);
+      }
+      // Also check non-prefixed env vars
+      if (process.env[key] !== undefined) {
+        this.config[key] = this._parseValue(process.env[key], this.defaults[key]);
+      }
+    }
+  }
+
+  _parseValue(value, defaultValue) {
+    if (typeof defaultValue === 'boolean') {
+      return value === 'true' || value === '1' || value === true;
+    }
+    if (typeof defaultValue === 'number') {
+      const parsed = parseInt(value, 10);
+      return isNaN(parsed) ? defaultValue : parsed;
+    }
+    return value;
+  }
+
+  async _ensureDirectories() {
+    const dirs = [
+      this.config.DEFAULT_UPLOAD_DIRECTORY,
+      this.config.DEFAULT_DOWNLOAD_DIRECTORY,
+      this.config.DEFAULT_TEMP_DIRECTORY
+    ];
+
+    for (const dir of dirs) {
+      try {
+        await fs.mkdir(dir, { recursive: true });
+      } catch (error) {
+        // Directory creation failures are non-fatal
+      }
+    }
+  }
+
+  // ============================================================================
+  // CONFIG GETTERS
+  // ============================================================================
+
+  getLocalConfig() {
+    return {
+      localRoot: this.config.LOCAL_ROOT,
+      uploadDirectory: this.config.DEFAULT_UPLOAD_DIRECTORY,
+      downloadDirectory: this.config.DEFAULT_DOWNLOAD_DIRECTORY,
+      tempDirectory: this.config.DEFAULT_TEMP_DIRECTORY,
+      maxFileSize: this.config.MAX_FILE_SIZE,
+      chunkSize: this.config.CHUNK_SIZE,
+      maxRetryAttempts: this.config.MAX_RETRY_ATTEMPTS,
+      timeoutMs: this.config.TIMEOUT_MS,
+      allowSymlinks: this.config.ALLOW_SYMLINKS,
+      restrictToBasePath: this.config.RESTRICT_TO_BASE_PATH,
+      maxPathLength: this.config.MAX_PATH_LENGTH
+    };
+  }
+
+  getWatchConfig() {
+    return {
+      localRoot: this.config.LOCAL_ROOT,
+      enabled: this.config.WATCH_ENABLED,
+      recursive: this.config.WATCH_RECURSIVE,
+      ignoreDotfiles: this.config.WATCH_IGNORE_DOTFILES,
+      debounceMs: this.config.WATCH_DEBOUNCE_MS,
+      maxWatchers: this.config.WATCH_MAX_WATCHERS
+    };
+  }
+
+  getCompressionConfig() {
+    return {
+      localRoot: this.config.LOCAL_ROOT,
+      enabled: this.config.COMPRESSION_ENABLED,
+      level: this.config.COMPRESSION_LEVEL,
+      format: this.config.COMPRESSION_FORMAT,
+      maxFileSize: this.config.COMPRESSION_MAX_FILE_SIZE,
+      chunkSize: this.config.COMPRESSION_CHUNK_SIZE,
+      enableProgressTracking: this.config.COMPRESSION_ENABLE_PROGRESS_TRACKING,
+      enableChecksumVerification: this.config.COMPRESSION_ENABLE_CHECKSUM_VERIFICATION
+    };
+  }
+
+  getRemoteConfig() {
+    return {
+      host: this.config.REMOTE_HOST,
+      port: this.config.REMOTE_PORT,
+      username: this.config.REMOTE_USERNAME,
+      password: this.config.REMOTE_PASSWORD,
+      privateKeyPath: this.config.REMOTE_PRIVATE_KEY_PATH,
+      passphrase: this.config.REMOTE_PASSPHRASE,
+      remoteRoot: this.config.REMOTE_ROOT,
+      timeout: this.config.REMOTE_TIMEOUT,
+      bastionHost: this.config.REMOTE_BASTION_HOST,
+      bastionPort: this.config.REMOTE_BASTION_PORT
+    };
+  }
+
+  getPerformanceConfig() {
+    return {
+      maxConcurrentOperations: this.config.MAX_CONCURRENT_OPERATIONS,
+      enableProgressTracking: this.config.ENABLE_PROGRESS_TRACKING,
+      enableChecksumVerification: this.config.ENABLE_CHECKSUM_VERIFICATION
+    };
+  }
+
+  hasRemoteConfig() {
+    return !!(this.config.REMOTE_HOST && this.config.REMOTE_USERNAME);
+  }
+
+  validate() {
+    const errors = [];
+    const warnings = [];
+
+    if (this.config.MAX_FILE_SIZE <= 0) {
+      errors.push('MAX_FILE_SIZE must be positive');
+    }
+
+    if (this.config.CHUNK_SIZE <= 0) {
+      errors.push('CHUNK_SIZE must be positive');
+    }
+
+    if (this.config.WATCH_MAX_WATCHERS > 100) {
+      warnings.push('High number of watchers may impact performance');
+    }
+
+    if (this.config.COMPRESSION_LEVEL < 0 || this.config.COMPRESSION_LEVEL > 9) {
+      errors.push('COMPRESSION_LEVEL must be between 0 and 9');
+    }
+
+    if (this.hasRemoteConfig()) {
+      if (!this.config.REMOTE_PASSWORD && !this.config.REMOTE_PRIVATE_KEY_PATH) {
+        warnings.push('Remote configuration missing both password and private key path');
+      }
+    }
+
+    return {
+      isValid: errors.length === 0,
+      errors,
+      warnings
+    };
+  }
+
+  /**
+   * Get a specific config value
+   */
+  get(key) {
+    return this.config[key];
+  }
+
+  /**
+   * Set a specific config value
+   */
+  set(key, value) {
+    this.config[key] = value;
+  }
+
+  /**
+   * Get all config as a plain object
+   */
+  toJSON() {
+    return { ...this.config };
+  }
+}
+
+export { ConfigManager };