@autodev/codebase 0.0.4 → 0.0.6

package/README.md

@@ -1,231 +1,321 @@
-
-
 # @autodev/codebase
 
 <div align="center">
-  <img src="src/images/image2.png" alt="Image 2" style="display: inline-block; width: 350px; margin: 0 10px;" />
-  <img src="src/images/image3.png" alt="Image 3" style="display: inline-block; width: 200px; margin: 0 10px;" />
+
+[![npm version](https://img.shields.io/npm/v/@autodev/codebase)](https://www.npmjs.com/package/@autodev/codebase)
+[![GitHub stars](https://img.shields.io/github/stars/anrgct/autodev-codebase)](https://github.com/anrgct/autodev-codebase)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
 </div>
 
-<br />
+```sh
+╭─ ~/workspace/autodev-codebase 
+╰─❯ codebase --demo --search="user manage"
+Found 3 results in 2 files for: "user manage"
+
+==================================================
+File: "hello.js"
+==================================================
+< class UserManager > (L7-20)
+class UserManager {
+  constructor() {
+    this.users = [];
+  }
+
+  addUser(user) {
+    this.users.push(user);
+    console.log('User added:', user.name);
+  }
+
+  getUsers() {
+    return this.users;
+  }
+}
 
-A platform-agnostic code analysis library with semantic search capabilities and MCP (Model Context Protocol) server support. This library provides intelligent code indexing, vector-based semantic search, and can be integrated into various development tools and IDEs.
+==================================================
+File: "README.md" | 2 snippets
+==================================================
+< md_h1 Demo Project > md_h2 Usage > md_h3 JavaScript Functions > (L16-20)
+### JavaScript Functions
+
+- greetUser(name) - Greets a user by name
+- UserManager - Class for managing user data
+
+─────
+< md_h1 Demo Project > md_h2 Search Examples > (L27-38)
+## Search Examples
+
+Try searching for:
+- "greet user"
+- "process data"
+- "user management"
+- "batch processing"
+- "YOLO model"
+- "computer vision"
+- "object detection"
+- "model training"
+
+```
+A vector embedding-based code semantic search tool with MCP server and multi-model integration. Can be used as a pure CLI tool. Supports Ollama for fully local embedding and reranking, enabling complete offline operation and privacy protection for your code repository.
 
 ## 🚀 Features
 
-- **Semantic Code Search**: Vector-based code search using embeddings
-- **MCP Server Support**: HTTP-based MCP server for IDE integration
-- **Terminal UI**: Interactive CLI with rich terminal interface
-- **Tree-sitter Parsing**: Advanced code parsing and analysis
-- **Vector Storage**: Qdrant vector database integration
-- **Flexible Embedding**: Support for various embedding models via Ollama
+- **🔍 Semantic Code Search**: Vector-based search using advanced embedding models
+- **🌐 MCP Server**: HTTP-based MCP server with SSE and stdio adapters
+- **💻 Pure CLI Tool**: Standalone command-line interface without GUI dependencies
+- **⚙️ Layered Configuration**: CLI, project, and global config management
+- **🎯 Advanced Path Filtering**: Glob patterns with brace expansion and exclusions
+- **🌲 Tree-sitter Parsing**: Support for 40+ programming languages
+- **💾 Qdrant Integration**: High-performance vector database
+- **🔄 Multiple Providers**: OpenAI, Ollama, Jina, Gemini, Mistral, OpenRouter, Vercel
+- **📊 Real-time Watching**: Automatic index updates
+- **⚡ Batch Processing**: Efficient parallel processing
 
 ## 📦 Installation
 
-### 1. Install and Start Ollama
-
+### 1. Dependencies
 ```bash
-# Install Ollama (macOS)
-brew install ollama
-
-# Start Ollama service
+brew install ollama ripgrep
 ollama serve
-
-# In a new terminal, pull the embedding model
 ollama pull nomic-embed-text
 ```
 
-### 2. Install and Start Qdrant
-
-Start Qdrant using Docker:
+### 2. Qdrant
+```bash
+docker run -d -p 6333:6333 -p 6334:6334 --name qdrant qdrant/qdrant
+```
 
+### 3. Install
 ```bash
-# Start Qdrant container
-docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant
+npm install -g @autodev/codebase
+codebase --set-config embedderProvider=ollama,embedderModelId=nomic-embed-text
 ```
 
-Or download and run Qdrant directly:
+## 🛠️ Quick Start
 
 ```bash
-# Download and run Qdrant
-wget https://github.com/qdrant/qdrant/releases/latest/download/qdrant-x86_64-unknown-linux-gnu.tar.gz
-tar -xzf qdrant-x86_64-unknown-linux-gnu.tar.gz
-./qdrant
+# Demo mode (recommended for first-time)
+# Creates a demo directory in current working directory for testing
+
+# Index & search
+codebase --demo --index
+codebase --demo --search="user greet"
+
+# MCP server
+codebase --demo --serve
 ```
 
-### 3. Verify Services Are Running
+## 📋 Commands
 
+### Indexing & Search
 ```bash
-# Check Ollama
-curl http://localhost:11434/api/tags
+# Index the codebase
+codebase --index --path=/my/project --force
+
+# Search with filters
+codebase --search="error handling" --path-filters="src/**/*.ts"
+
+# Search with custom limit and minimum score
+codebase --search="authentication" --limit=20 --min-score=0.7
+codebase --search="API" -l 30 -s 0.5
+
+# Search in JSON format
+codebase --search="authentication" --json
 
-# Check Qdrant
-curl http://localhost:6333/collections
+# Clear index data
+codebase --clear --path=/my/project
 ```
-### 4. Install Autodev-codebase
 
+### MCP Server
 ```bash
-npm install -g @autodev/codebase
-```
+# HTTP mode (recommended)
+codebase --serve --port=3001 --path=/my/project
 
-Alternatively, you can install it locally:
+# Stdio adapter
+codebase --stdio-adapter --server-url=http://localhost:3001/mcp
 ```
-git clone https://github.com/anrgct/autodev-codebase
-cd autodev-codebase
-npm install
-npm run build
-npm link
+
+### Configuration
+```bash
+# View config
+codebase --get-config
+codebase --get-config embedderProvider --json
+
+# Set config
+codebase --set-config embedderProvider=ollama,embedderModelId=nomic-embed-text
+codebase --set-config --global qdrantUrl=http://localhost:6333
 ```
-## 🛠️ Usage
 
-### Command Line Interface
+### Advanced Features
 
-The CLI provides two main modes:
+#### 🔍 LLM-Powered Search Reranking
+Enable LLM reranking to dramatically improve search relevance:
 
-#### 1. Interactive TUI Mode (Default)
 ```bash
-# Basic usage: index your current folder as the codebase.
-# Be cautious when running this command if you have a large number of files.
-codebase
+# Enable reranking with Ollama (recommended)
+codebase --set-config rerankerEnabled=true,rerankerProvider=ollama,rerankerOllamaModelId=qwen3-vl:4b-instruct
 
+# Or use OpenAI-compatible providers
+codebase --set-config rerankerEnabled=true,rerankerProvider=openai-compatible,rerankerOpenAiCompatibleModelId=deepseek-chat
 
-# With custom options
-codebase --demo # Create a local demo directory and test the indexing service, recommend for setup
-codebase --path=/my/project
-codebase --path=/my/project --log-level=info
+# Search with automatic reranking
+codebase --search="user authentication"  # Results are automatically reranked by LLM
 ```
 
-#### 2. MCP Server Mode (Recommended for IDE Integration)
+**Benefits:**
+- 🎯 **Higher precision**: LLM understands semantic relevance beyond vector similarity
+- 📊 **Smart scoring**: Results are reranked on a 0-10 scale based on query relevance  
+- ⚡ **Batch processing**: Efficiently handles large result sets with configurable batch sizes
+- 🎛️ **Threshold control**: Filter results with `rerankerMinScore` to keep only high-quality matches
+
+#### Path Filtering & Export
 ```bash
-# Start long-running MCP server
-cd /my/project
-codebase mcp-server
+# Path filtering with brace expansion and exclusions
+codebase --search="API" --path-filters="src/**/*.ts,lib/**/*.js"
+codebase --search="utils" --path-filters="{src,test}/**/*.ts"
 
-# With custom configuration
-codebase mcp-server --port=3001 --host=localhost
-codebase mcp-server --path=/workspace --port=3002
+# Export results in JSON format for scripts
+codebase --search="auth" --json
 ```
 
-### IDE Integration (Cursor/Claude)
+## ⚙️ Configuration
+
+### Config Layers (Priority Order)
+1. **CLI Arguments** - Runtime parameters (`--path`, `--config`, `--log-level`, `--force`, etc.)
+2. **Project Config** - `./autodev-config.json` (or custom path via `--config`)
+3. **Global Config** - `~/.autodev-cache/autodev-config.json`
+4. **Built-in Defaults** - Fallback values
+
+**Note:** CLI arguments provide runtime override for paths, logging, and operational behavior. For persistent configuration (embedderProvider, API keys, search parameters), use `--set-config` to save to config files.
 
-Configure your IDE to connect to the MCP server:
+### Common Config Examples
 
+**Ollama:**
 ```json
 {
-  "mcpServers": {
-    "codebase": {
-      "url": "http://localhost:3001/sse"
-    }
-  }
+  "embedderProvider": "ollama",
+  "embedderModelId": "nomic-embed-text",
+  "qdrantUrl": "http://localhost:6333"
 }
 ```
 
-For clients that do not support SSE MCP, you can use the following configuration:
-
+**OpenAI:**
 ```json
 {
-  "mcpServers": {
-    "codebase": {
-      "command": "codebase",
-      "args": [
-        "stdio-adapter",
-        "--server-url=http://localhost:3001/sse"
-      ]
-    }
-  }
+  "embedderProvider": "openai",
+  "embedderModelId": "text-embedding-3-small",
+  "embedderOpenAiApiKey": "sk-your-key",
+  "qdrantUrl": "http://localhost:6333"
 }
 ```
 
-### Library Usage
+**OpenAI-Compatible:**
+```json
+{
+  "embedderProvider": "openai-compatible",
+  "embedderModelId": "text-embedding-3-small",
+  "embedderOpenAiCompatibleApiKey": "sk-your-key",
+  "embedderOpenAiCompatibleBaseUrl": "https://api.openai.com/v1"
+}
+```
 
-#### Node.js Usage
-```typescript
-import { createNodeDependencies } from '@autodev/codebase/adapters/nodejs'
-import { CodeIndexManager } from '@autodev/codebase'
+### Key Configuration Options
 
-const deps = createNodeDependencies({ 
-  workspacePath: '/path/to/project',
-  storageOptions: { /* ... */ },
-  loggerOptions: { /* ... */ },
-  configOptions: { /* ... */ }
-})
+| Category | Options | Description |
+|----------|---------|-------------|
+| **Embedding** | `embedderProvider`, `embedderModelId`, `embedderModelDimension` | Provider and model settings |
+| **API Keys** | `embedderOpenAiApiKey`, `embedderOpenAiCompatibleApiKey` | Authentication |
+| **Vector Store** | `qdrantUrl`, `qdrantApiKey` | Qdrant connection |
+| **Search** | `vectorSearchMinScore`, `vectorSearchMaxResults` | Search behavior |
+| **Reranker** | `rerankerEnabled`, `rerankerProvider` | Result reranking |
 
-const manager = CodeIndexManager.getInstance(deps)
-await manager.initialize()
-await manager.startIndexing()
-```
+**Key CLI Arguments:**
+- `--serve` / `--index` / `--search` - Core operations
+- `--get-config` / `--set-config` - Configuration management  
+- `--path`, `--demo`, `--force` - Common options
+- `--limit` / `-l <number>` - Maximum number of search results (default: from config, max 50)
+- `--min-score` / `-s <number>` - Minimum similarity score for search results (0-1, default: from config)
+- `--help` - Show all available options
 
-## 🔧 CLI Options
+For complete CLI reference, see [CONFIG.md](CONFIG.md).
 
-### Global Options
-- `--path=<path>` - Workspace path (default: current directory)
-- `--demo` - Create demo files in workspace
-- `--ollama-url=<url>` - Ollama API URL (default: http://localhost:11434)
-- `--qdrant-url=<url>` - Qdrant vector DB URL (default: http://localhost:6333)
-- `--model=<model>` - Embedding model (default: nomic-embed-text)
-- `--config=<path>` - Config file path
-- `--storage=<path>` - Storage directory path
-- `--cache=<path>` - Cache directory path
-- `--log-level=<level>` - Log level: error|warn|info|debug (default: error)
-- `--help, -h` - Show help
+**Configuration Commands:**
+```bash
+# View config
+codebase --get-config
+codebase --get-config --json
 
-### MCP Server Options
-- `--port=<port>` - HTTP server port (default: 3001)
-- `--host=<host>` - HTTP server host (default: localhost)
+# Set config (saves to file)
+codebase --set-config embedderProvider=ollama,embedderModelId=nomic-embed-text
+codebase --set-config --global embedderProvider=openai,embedderOpenAiApiKey=sk-xxx
 
-## 🌐 MCP Server Features
+# Use custom config file
+codebase --config=/path/to/config.json --get-config
+codebase --config=/path/to/config.json --set-config embedderProvider=ollama
 
-### Web Interface
-- **Home Page**: `http://localhost:3001` - Server status and configuration
-- **Health Check**: `http://localhost:3001/health` - JSON status endpoint
-- **MCP Endpoint**: `http://localhost:3001/sse` - SSE/HTTP MCP protocol endpoint
+# Runtime override (paths, logging, etc.)
+codebase --index --path=/my/project --log-level=info --force
+```
 
-### Available MCP Tools
-- **`search_codebase`** - Semantic search through your codebase
-  - Parameters: `query` (string), `limit` (number), `filters` (object)
-  - Returns: Formatted search results with file paths, scores, and code blocks
-- **`get_search_stats`** - Get indexing status and statistics
-- **`configure_search`** - Configure search parameters at runtime
+For complete configuration reference, see [CONFIG.md](CONFIG.md).
 
+## 🔌 MCP Integration
 
-### Scripts
+### HTTP Streamable Mode (Recommended)
 ```bash
-# Development mode with demo files
-npm run dev
+codebase --serve --port=3001
+```
 
-# Build for production
-npm run build
+**IDE Config:**
+```json
+{
+  "mcpServers": {
+    "codebase": {
+      "url": "http://localhost:3001/mcp"
+    }
+  }
+}
+```
 
-# Type checking
-npm run type-check
+### Stdio Adapter
+```bash
+# First start the MCP server in one terminal
+codebase --serve --port=3001
 
-# Run TUI demo
-npm run demo-tui
+# Then connect via stdio adapter in another terminal (for IDEs that require stdio)
+codebase --stdio-adapter --server-url=http://localhost:3001/mcp
+```
 
-# Start MCP server demo
-npm run mcp-server
+**IDE Config:**
+```json
+{
+  "mcpServers": {
+    "codebase": {
+      "command": "codebase",
+      "args": ["stdio-adapter", "--server-url=http://localhost:3001/mcp"]
+    }
+  }
+}
 ```
 
-## 💡 Why Use MCP Server Mode?
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request or open an Issue on [GitHub](https://github.com/anrgct/autodev-codebase).
 
-### Problems Solved
-- **❌ Repeated Indexing**: Every IDE connection re-indexes, wasting time and resources
-- **❌ Complex Configuration**: Each project needs different path parameters in IDE
-- **❌ Resource Waste**: Multiple IDE windows start multiple server instances
+## 📄 License
 
-### Benefits
-- **✅ One-time Indexing**: Server runs long-term, index persists
-- **✅ Simplified Configuration**: Universal IDE configuration, no project-specific paths
-- **✅ Resource Efficiency**: One server instance per project
-- **✅ Better Developer Experience**: Start server in project directory intuitively
-- **✅ Backward Compatible**: Still supports traditional per-connection mode
-- **✅ Web Interface**: Status monitoring and configuration help
-- **✅ Dual Mode**: Can run both TUI and MCP server simultaneously
+This project is licensed under the [MIT License](https://opensource.org/licenses/MIT).
 
+## 🙏 Acknowledgments
 
-This is a platform-agnostic library extracted from the roo-code VSCode plugin. 
-## 📚 Examples
+This project is a fork and derivative work based on [Roo Code](https://github.com/RooCodeInc/Roo-Code). We've built upon their excellent foundation to create this specialized codebase analysis tool with enhanced features and MCP server capabilities.
 
-See the `examples/` directory for complete usage examples:
-- `nodejs-usage.ts` - Node.js integration examples
-- `run-demo-tui.tsx` - TUI demo application
+---
+
+<div align="center">
+
+**🌟 If you find this tool helpful, please give us a [star on GitHub](https://github.com/anrgct/autodev-codebase)!**
+
+Made with ❤️ for the developer community
+
+</div>