rag-lite-ts 2.2.0 → 2.3.1

package/README.md

@@ -11,7 +11,7 @@
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg?style=flat-square)](https://www.typescriptlang.org/)
 [![Node.js](https://img.shields.io/badge/Node.js-18+-green.svg?style=flat-square)](https://nodejs.org/)
 
-[Quick Start](#quick-start) • [Features](#features) • [Documentation](#documentation) • [Examples](#examples) • [MCP Integration](#mcp-server-integration)
+[Quick Start](#quick-start) • [Features](#features) • [Documentation](#documentation) • [Examples](#examples) • [UI](#web-interface-ui) • [MCP Integration](#mcp-server-integration)
 
 </div>
 
@@ -27,7 +27,7 @@ raglite ingest ./docs/
 raglite search "your query here"
 ```
 
-**That's it.** No API keys, no cloud services, no configuration hell.
+**That's it.** No API keys, no cloud services, no configuration hell. Prefer a UI? Run `raglite ui` for a visual interface with drag‑and‑drop ingestion and search.
 
 ### 🎬 See It In Action
 
@@ -101,6 +101,7 @@ See [CHANGELOG.md](CHANGELOG.md) for complete details.
 - [How It Works](#-how-it-works)
 - [Supported Models](#-supported-models)
 - [Documentation](#-documentation)
+- [Web Interface (UI)](#-web-interface-ui)
 - [MCP Server Integration](#-mcp-server-integration)
 - [Development](#-development)
 - [Contributing](#-contributing)
@@ -114,7 +115,7 @@ See [CHANGELOG.md](CHANGELOG.md) for complete details.
 npm install -g rag-lite-ts
 ```
 
-### Basic Usage
+### Basic Usage (CLI)
 
 ```bash
 # Ingest documents
@@ -127,11 +128,32 @@ raglite search "machine learning concepts"
 raglite search "API documentation" --top-k 10 --rerank
 ```
 
+### Web Interface (UI)
+
+Launch the visual web interface for interactive document management:
+
+```bash
+# Start the UI (opens in browser)
+raglite ui
+
+# UI provides:
+# - Drag & drop file upload
+# - Real-time ingestion progress
+# - Visual search interface
+# - Image search with upload
+# - Knowledge base statistics
+```
+
+→ **[Complete UI Guide](docs/ui-guide.md)**
+
 ### Using Different Models
 
 ```bash
-# Use higher quality model (auto-rebuilds if needed)
-raglite ingest ./docs/ --model Xenova/all-mpnet-base-v2 --rebuild-if-needed
+# Use higher quality model
+raglite ingest ./docs/ --model Xenova/all-mpnet-base-v2
+
+# Switch models or refresh your data (DESTRUCTIVE: wipes DB+index and rebuilds from scratch)
+raglite ingest ./docs/ --model Xenova/all-mpnet-base-v2 --force-rebuild
 
 # Search automatically uses the correct model
 raglite search "complex query"
@@ -441,6 +463,27 @@ Now Claude can search your docs directly! Works with any MCP-compatible AI tool.
 - **Model compatibility** - Auto-detection, rebuilds
 - **Error recovery** - Clear messages, helpful hints
 
+</td>
+</tr>
+<tr>
+<td width="50%">
+
+### 🎨 Web Interface (UI)
+- **Visual ingestion** - Drag & drop file upload
+- **Real-time progress** - Live ingestion tracking
+- **Image search** - Upload images to find similar content
+- **Interactive search** - Visual results with previews
+- **Knowledge base stats** - Overview of your data
+
+</td>
+<td width="50%">
+
+### 📚 Documentation
+- **Comprehensive guides** - CLI, API, multimodal tutorials
+- **Quick start guides** - Get running in minutes
+- **Troubleshooting** - Solutions for common issues
+- **Examples** - Real-world use cases
+
 </td>
 </tr>
 </table>
@@ -579,6 +622,7 @@ Choose the right model for your use case:
 
 ### 🚀 Getting Started
 - [CLI Reference](docs/cli-reference.md)
+- [UI Guide](docs/ui-guide.md) 🆕
 - [API Reference](docs/api-reference.md)
 - [Multimodal Tutorial](docs/multimodal-tutorial.md)
 - [Unified Content System](docs/unified-content-system.md)
@@ -610,6 +654,7 @@ Choose the right model for your use case:
 | I want to... | Start here |
 |--------------|------------|
 | 🆕 Try it out | [CLI Reference](docs/cli-reference.md) → `npm i -g rag-lite-ts` |
+| 🎨 Use visual interface | [UI Guide](docs/ui-guide.md) → `raglite ui` |
 | 🖼️ Search images | [Multimodal Tutorial](docs/multimodal-tutorial.md) → `--mode multimodal` |
 | 💻 Build an app | [API Reference](docs/api-reference.md) → `new SearchEngine()` |
 | 🤖 Integrate with AI | [MCP Guide](docs/mcp-server-multimodal-guide.md) → `raglite-mcp` |
@@ -618,6 +663,44 @@ Choose the right model for your use case:
 
 **📖 [Complete Documentation Hub](docs/README.md)**
 
+## 🎨 Web Interface (UI)
+
+RAG-lite TS includes a modern web-based interface for visual document management and search.
+
+### Quick Start
+
+```bash
+# Launch the UI (starts both frontend and backend)
+raglite ui
+
+# Opens in browser at http://localhost:3000
+```
+
+### Features
+
+- **📤 Visual Ingestion**: Drag & drop file upload with real-time progress
+- **🔍 Interactive Search**: Text and image search with visual results
+- **📊 Knowledge Base Stats**: Overview of documents, chunks, and model information
+- **⚙️ Configuration**: Visual interface for all ingestion options
+- **🖼️ Image Search**: Upload images to find similar content
+- **📁 Folder Support**: Upload entire directory structures
+
+### UI vs CLI
+
+| Feature | UI | CLI |
+|---------|----|----|
+| File upload | ✅ Drag & drop | ✅ Command-line |
+| Progress tracking | ✅ Visual bars | ✅ Console output |
+| Image search | ✅ Upload interface | ✅ File path |
+| Configuration | ✅ Visual options | ✅ Command flags |
+| Batch processing | ✅ | ✅ |
+| Scripting | ❌ | ✅ |
+
+**Use UI for:** Interactive exploration, visual feedback, learning the system  
+**Use CLI for:** Automation, scripting, headless environments
+
+→ **[Complete UI Guide](docs/ui-guide.md)**
+
 ## 🔌 MCP Server Integration
 
 **Give your AI agents semantic memory.** RAG-lite TS includes a built-in Model Context Protocol (MCP) server.

package/dist/cjs/cli/indexer.js

@@ -80,7 +80,7 @@ async function validateModeConfiguration(options) {
  */
 export async function runIngest(path, options = {}) {
     try {
-        // Handle --rebuild-if-needed flag immediately to prevent dimension mismatch error
+        // Handle --force-rebuild flag immediately to prevent dimension mismatch errors
         // Validate path exists
         const resolvedPath = resolve(path);
         if (!existsSync(resolvedPath)) {
@@ -159,26 +159,52 @@ export async function runIngest(path, options = {}) {
             factoryOptions.mode = options.mode;
             console.log(`Using processing mode: ${options.mode}`);
         }
-        if (options.rebuildIfNeeded) {
+        if (options.forceRebuild) {
             factoryOptions.forceRebuild = true;
-            console.log('Force rebuild enabled due to rebuildIfNeeded option');
-            // Delete old index file immediately to prevent dimension mismatch errors
-            const indexPath = process.env.RAG_INDEX_FILE || './vector-index.bin';
-            const { existsSync, unlinkSync } = await import('fs');
-            if (existsSync(indexPath)) {
-                try {
-                    unlinkSync(indexPath);
-                    console.log('🗑️ Removed old index file to prevent dimension mismatch');
-                }
-                catch (error) {
-                    console.warn(`⚠️ Could not remove old index file: ${error}`);
-                }
-            }
+            console.log('Force rebuild enabled (--force-rebuild)');
         }
         // Validate mode-specific model and strategy combinations
         await validateModeConfiguration(factoryOptions);
         const dbPath = process.env.RAG_DB_FILE || './db.sqlite';
         const indexPath = process.env.RAG_INDEX_FILE || './vector-index.bin';
+        // --force-rebuild: Always delete DB (and sidecars) and index to guarantee a clean rebuild.
+        if (options.forceRebuild) {
+            try {
+                const { existsSync: fsExistsSync, unlinkSync } = await import('fs');
+                console.log('🗑️ Deleting existing database and index to perform a clean rebuild...');
+                // Remove WAL/SHM if present (common on SQLite with WAL journaling).
+                const sidecars = [`${dbPath}-wal`, `${dbPath}-shm`];
+                for (const p of sidecars) {
+                    if (fsExistsSync(p)) {
+                        try {
+                            unlinkSync(p);
+                        }
+                        catch (e) {
+                            console.warn(`⚠️ Could not remove SQLite sidecar file (${p}):`, e);
+                        }
+                    }
+                }
+                if (fsExistsSync(dbPath)) {
+                    try {
+                        unlinkSync(dbPath);
+                    }
+                    catch (e) {
+                        console.warn(`⚠️ Could not remove database file (${dbPath}):`, e);
+                    }
+                }
+                if (fsExistsSync(indexPath)) {
+                    try {
+                        unlinkSync(indexPath);
+                    }
+                    catch (e) {
+                        console.warn(`⚠️ Could not remove index file (${indexPath}):`, e);
+                    }
+                }
+            }
+            catch (error) {
+                console.warn('⚠️ Could not delete existing database/index for clean rebuild:', error instanceof Error ? error.message : String(error));
+            }
+        }
         // Setup graceful cleanup
         setupCLICleanup(dbPath);
         // Check if database is busy before starting
@@ -218,6 +244,22 @@ export async function runIngest(path, options = {}) {
                 console.log(`Processing rate: ${chunksPerSecond} chunks/second`);
             }
             console.log('\nIngestion completed successfully!');
+            // Run VACUUM to compact the SQLite database after ingestion
+            try {
+                const { openDatabase } = await import('../core/db.js');
+                const vacuumDb = await openDatabase(dbPath);
+                try {
+                    console.log('Running VACUUM to optimize database size...');
+                    await vacuumDb.run('VACUUM');
+                    console.log('VACUUM completed successfully.');
+                }
+                finally {
+                    await vacuumDb.close();
+                }
+            }
+            catch (vacuumError) {
+                console.warn('⚠️  VACUUM operation failed or was skipped:', vacuumError instanceof Error ? vacuumError.message : String(vacuumError));
+            }
             // Display mode-specific information
             const mode = options.mode || 'text';
             if (mode === 'multimodal') {
@@ -397,6 +439,22 @@ export async function runRebuild() {
                 console.log('All embeddings have been regenerated with the current model.');
                 console.log('');
                 console.log('You can now search your documents using: raglite search "your query"');
+                // Run VACUUM to compact the SQLite database after rebuild
+                try {
+                    const { openDatabase } = await import('../core/db.js');
+                    const vacuumDb = await openDatabase(dbPath);
+                    try {
+                        console.log('Running VACUUM to optimize database size after rebuild...');
+                        await vacuumDb.run('VACUUM');
+                        console.log('VACUUM completed successfully.');
+                    }
+                    finally {
+                        await vacuumDb.close();
+                    }
+                }
+                catch (vacuumError) {
+                    console.warn('⚠️  VACUUM operation failed or was skipped:', vacuumError instanceof Error ? vacuumError.message : String(vacuumError));
+                }
             }
             finally {
                 await db.close();

package/dist/cjs/cli/search.js

@@ -132,7 +132,7 @@ export async function runSearch(query, options = {}) {
                     process.exit(EXIT_CODES.MODEL_ERROR);
                 }
             }
-            // Prepare search options
+            // Prepare search options (with generation support)
             const searchOptions = {};
             if (options['top-k'] !== undefined) {
                 searchOptions.top_k = options['top-k'];
@@ -164,9 +164,60 @@ export async function runSearch(query, options = {}) {
             }
             // Track whether reranking will actually be used in this search
             const rerankingUsed = searchOptions.rerank === true;
+            // Handle generation options (experimental, text mode only)
+            const generateResponse = options.generate === true;
+            const generatorModel = options.generator;
+            const maxGenerationTokens = options['max-tokens'];
+            const generationTemperature = options.temperature;
+            const maxChunksForContext = options['max-chunks'];
+            // Generation only supported in text mode
+            if (generateResponse && isImage) {
+                console.warn('⚠️  [EXPERIMENTAL] Generation is only supported for text searches.');
+                console.warn('   Image search results will be returned without generation.');
+                console.warn('');
+            }
+            // Generation requires reranking - enable it automatically
+            let rerankingEnabledForGeneration = false;
+            if (generateResponse && !isImage && !searchOptions.rerank) {
+                searchOptions.rerank = true;
+                rerankingEnabledForGeneration = true;
+                console.log('📋 Reranking automatically enabled (required for generation)');
+            }
+            // Set up generator if generation is requested (text mode only)
+            let generateFn;
+            if (generateResponse && !isImage) {
+                try {
+                    console.log('🤖 [EXPERIMENTAL] Initializing response generator...');
+                    const { createGenerateFunctionFromModel, getDefaultGeneratorModel } = await import('../factories/generator-factory.js');
+                    const { getDefaultMaxChunksForContext } = await import('../core/generator-registry.js');
+                    const modelToUse = generatorModel || getDefaultGeneratorModel();
+                    const defaultChunks = getDefaultMaxChunksForContext(modelToUse) || 3;
+                    console.log(`   Model: ${modelToUse}`);
+                    console.log(`   Max chunks for context: ${maxChunksForContext || defaultChunks} (default: ${defaultChunks})`);
+                    generateFn = await createGenerateFunctionFromModel(modelToUse);
+                    searchEngine.setGenerateFunction(generateFn);
+                    console.log('✅ Generator initialized');
+                    console.log('');
+                }
+                catch (error) {
+                    console.error('❌ [EXPERIMENTAL] Failed to initialize generator:', error instanceof Error ? error.message : 'Unknown error');
+                    console.error('   Continuing without generation...');
+                    console.error('');
+                }
+            }
+            // Set generation options if generator is ready
+            if (generateFn && generateResponse && !isImage) {
+                searchOptions.generateResponse = true;
+                searchOptions.generationOptions = {
+                    maxTokens: maxGenerationTokens,
+                    temperature: generationTemperature,
+                    maxChunksForContext: maxChunksForContext
+                };
+            }
             // Perform search
             const startTime = Date.now();
             let results;
+            let generationResult;
             if (isImage && embedder) {
                 // Image-based search: embed the image and search with the vector
                 console.log('Embedding image...');
@@ -174,8 +225,14 @@ export async function runSearch(query, options = {}) {
                 console.log('Searching with image embedding...');
                 results = await searchEngine.searchWithVector(imageEmbedding.vector, searchOptions);
             }
+            else if (generateResponse && generateFn) {
+                // Text-based search with generation
+                const searchResult = await searchEngine.searchWithGeneration(query, searchOptions);
+                results = searchResult.results;
+                generationResult = searchResult.generation;
+            }
             else {
-                // Text-based search
+                // Standard text-based search
                 results = await searchEngine.search(query, searchOptions);
             }
             const searchTime = Date.now() - startTime;
@@ -216,6 +273,21 @@ export async function runSearch(query, options = {}) {
                     }
                     console.log('');
                 });
+                // Display generated response if available (experimental)
+                if (generationResult) {
+                    console.log('─'.repeat(50));
+                    console.log('🤖 Generated Response [EXPERIMENTAL]');
+                    console.log(`Model: ${generationResult.modelUsed}`);
+                    console.log('─'.repeat(50));
+                    console.log('');
+                    console.log(generationResult.response);
+                    console.log('');
+                    console.log('─'.repeat(50));
+                    console.log(`⏱️  Generation: ${(generationResult.generationTimeMs / 1000).toFixed(1)}s | ` +
+                        `📊 ${generationResult.tokensUsed} tokens | ` +
+                        `📄 ${generationResult.chunksUsedForContext} chunks used` +
+                        (generationResult.truncated ? ' (context truncated)' : ''));
+                }
                 // Show search statistics
                 const stats = await searchEngine.getStats();
                 console.log('─'.repeat(50));
@@ -233,6 +305,9 @@ export async function runSearch(query, options = {}) {
                 else {
                     console.log('Reranking: disabled');
                 }
+                if (generationResult) {
+                    console.log('Generation: enabled [EXPERIMENTAL]');
+                }
             }
         }
         finally {

package/dist/cjs/cli/ui-server.d.ts

@@ -0,0 +1,5 @@
+/**
+ * Launch the UI server
+ */
+export declare function runUI(options?: any): Promise<void>;
+//# sourceMappingURL=ui-server.d.ts.map

package/dist/cjs/cli/ui-server.js

@@ -0,0 +1,152 @@
+import { fileURLToPath } from 'url';
+import { dirname, join } from 'path';
+import { spawn } from 'child_process';
+import fs from 'fs';
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = dirname(__filename);
+/**
+ * Get the project root directory
+ * When built, CLI is at dist/esm/cli/ui-server.js, so go up 3 levels
+ * When running from source, CLI is at src/cli/ui-server.ts, so go up 2 levels
+ */
+function getProjectRoot() {
+    // Try going up 3 levels first (for built version)
+    const builtPath = join(__dirname, '../../..');
+    if (fs.existsSync(join(builtPath, 'package.json'))) {
+        return builtPath;
+    }
+    // Fallback: go up 2 levels (for source execution)
+    return join(__dirname, '../..');
+}
+/**
+ * Launch the UI server
+ */
+export async function runUI(options = {}) {
+    const port = options.port || 3000;
+    const backendPort = options.backendPort || 3001;
+    console.log('🚀 Launching RAG-lite TS UI...');
+    // Resolve UI paths from project root
+    const projectRoot = getProjectRoot();
+    const backendBuiltPath = join(projectRoot, 'ui', 'backend', 'dist', 'index.js');
+    const backendSourcePath = join(projectRoot, 'ui', 'backend', 'src', 'index.ts');
+    const frontendBuiltPath = join(projectRoot, 'ui', 'frontend', 'dist');
+    const frontendSourcePath = join(projectRoot, 'ui', 'frontend');
+    // Check if built files exist
+    const useBuiltBackend = fs.existsSync(backendBuiltPath);
+    const useBuiltFrontend = fs.existsSync(frontendBuiltPath);
+    if (!useBuiltBackend && !fs.existsSync(backendSourcePath)) {
+        console.error(`❌ UI backend not found at: ${backendSourcePath}`);
+        console.error('   Make sure the UI is set up in the ui/ directory.');
+        process.exit(1);
+    }
+    if (!useBuiltFrontend && !fs.existsSync(frontendSourcePath)) {
+        console.error(`❌ UI frontend not found at: ${frontendSourcePath}`);
+        console.error('   Make sure the UI is set up in the ui/ directory.');
+        process.exit(1);
+    }
+    // Pass the working directory where 'raglite ui' was called to the backend
+    const workingDir = process.cwd();
+    // Built mode: single server on port (UI + API). Dev mode: backend on backendPort, frontend on port.
+    const effectiveBackendPort = useBuiltFrontend ? port : backendPort;
+    console.log(`📡 Starting backend on port ${effectiveBackendPort}...`);
+    // Start backend server - use built version if available
+    const backendCommand = useBuiltBackend ? 'node' : 'npx';
+    const backendArgs = useBuiltBackend
+        ? [backendBuiltPath]
+        : ['tsx', backendSourcePath];
+    const backendProcess = spawn(backendCommand, backendArgs, {
+        stdio: 'pipe',
+        env: {
+            ...process.env,
+            PORT: effectiveBackendPort.toString(),
+            RAG_WORKING_DIR: workingDir,
+            UI_FRONTEND_DIST: useBuiltFrontend ? frontendBuiltPath : undefined
+        },
+        shell: true
+    });
+    backendProcess.on('error', (err) => {
+        console.error('❌ Failed to start backend process:', err);
+        process.exit(1);
+    });
+    // Forward backend output with prefix
+    backendProcess.stdout?.on('data', (data) => {
+        process.stdout.write(`[Backend] ${data}`);
+    });
+    backendProcess.stderr?.on('data', (data) => {
+        process.stderr.write(`[Backend] ${data}`);
+    });
+    // Only start frontend dev server if built version doesn't exist
+    let frontendProcess = null;
+    if (!useBuiltFrontend) {
+        console.log(`🎨 Starting frontend dev server on port ${port}...`);
+        frontendProcess = spawn('npm', ['run', 'dev'], {
+            cwd: frontendSourcePath,
+            stdio: 'pipe',
+            env: {
+                ...process.env,
+                VITE_API_URL: `http://localhost:${effectiveBackendPort}`
+            },
+            shell: true
+        });
+        frontendProcess.on('error', (err) => {
+            console.error('❌ Failed to start frontend process:', err);
+            backendProcess.kill();
+            process.exit(1);
+        });
+        // Forward frontend output with prefix
+        frontendProcess.stdout?.on('data', (data) => {
+            process.stdout.write(`[Frontend] ${data}`);
+        });
+        frontendProcess.stderr?.on('data', (data) => {
+            process.stderr.write(`[Frontend] ${data}`);
+        });
+    }
+    else {
+        console.log(`🎨 Using built frontend from ${frontendBuiltPath}`);
+        console.log(`   Frontend will be served by backend on port ${effectiveBackendPort}`);
+    }
+    // Wait a bit for servers to start
+    await new Promise(resolve => setTimeout(resolve, 2000));
+    console.log(`\n✨ UI Access:`);
+    if (useBuiltFrontend) {
+        console.log(`   Frontend & Backend: http://localhost:${port}`);
+    }
+    else {
+        console.log(`   Frontend: http://localhost:${port}`);
+        console.log(`   Backend:  http://localhost:${effectiveBackendPort}`);
+    }
+    console.log(`\n💡 Press Ctrl+C to stop both servers\n`);
+    // Keep the process alive and handle cleanup
+    return new Promise((resolve) => {
+        const cleanup = () => {
+            console.log('\n🛑 Shutting down servers...');
+            backendProcess.kill();
+            if (frontendProcess) {
+                frontendProcess.kill();
+            }
+            resolve();
+        };
+        process.on('SIGINT', cleanup);
+        process.on('SIGTERM', cleanup);
+        // Handle process exits
+        backendProcess.on('exit', (code) => {
+            if (code !== 0 && code !== null) {
+                console.error(`\n❌ Backend process exited with code ${code}`);
+                if (frontendProcess) {
+                    frontendProcess.kill();
+                }
+                resolve();
+            }
+        });
+        if (frontendProcess) {
+            frontendProcess.on('exit', (code) => {
+                if (code !== 0 && code !== null) {
+                    console.error(`\n❌ Frontend process exited with code ${code}`);
+                    backendProcess.kill();
+                    resolve();
+                }
+            });
+        }
+    });
+}
+//# sourceMappingURL=ui-server.js.map