opencode-codebase-index 0.1.10 → 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kenneth Helweg
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -14,6 +14,7 @@
 
 - 🧠 **Semantic Search**: Finds "user authentication" logic even if the function is named `check_creds`.
 - ⚡ **Blazing Fast Indexing**: Powered by a Rust native module using `tree-sitter` and `usearch`. Incremental updates take milliseconds.
+- 🌿 **Branch-Aware**: Seamlessly handles git branch switches — reuses embeddings, filters stale results.
 - 🔒 **Privacy Focused**: Your vector index is stored locally in your project.
 - 🔌 **Model Agnostic**: Works out-of-the-box with GitHub Copilot, OpenAI, Gemini, or local Ollama models.
 
@@ -31,11 +32,12 @@
    }
    ```
 
-3. **Start Searching**
-   Load OpenCode and ask:
-   > "Find the function that handles credit card validation errors"
+3. **Index your codebase**
+   Run `/index` or ask the agent to index your codebase. This only needs to be done once — subsequent updates are incremental.
 
-   *The plugin will automatically index your codebase on the first run.*
+4. **Start Searching**
+   Ask:
+   > "Find the function that handles credit card validation errors"
 
 ## 🔍 See It In Action
 
@@ -68,6 +70,28 @@ src/api/checkout.ts:89      (Route handler for /pay)
 
 **Rule of thumb**: Semantic search for discovery → grep for precision.
 
+## 📊 Token Usage
+
+In our testing across open-source codebases (axios, express), we observed **up to 90% reduction in token usage** for conceptual queries like *"find the error handling middleware"*.
+
+### Why It Saves Tokens
+
+- **Without plugin**: Agent explores files, reads code, backtracks, explores more
+- **With plugin**: Semantic search returns relevant code immediately → less exploration
+
+### Key Takeaways
+
+1. **Significant savings possible**: Up to 90% reduction in the best cases
+2. **Results vary**: Savings depend on query type, codebase structure, and agent behavior
+3. **Best for discovery**: Conceptual queries benefit most; exact identifier lookups should use grep
+4. **Complements existing tools**: Provides a faster initial signal, doesn't replace grep/explore
+
+### When the Plugin Helps Most
+
+- **Conceptual queries**: "Where is the authentication logic?" (no keywords to grep for)
+- **Unfamiliar codebases**: You don't know what to search for yet
+- **Large codebases**: Semantic search scales better than exhaustive exploration
+
 ## 🛠️ How It Works
 
 ```mermaid
@@ -75,25 +99,72 @@ graph TD
     subgraph Indexing
     A[Source Code] -->|Tree-sitter| B[Semantic Chunks]
     B -->|Embedding Model| C[Vectors]
-    C -->|uSearch| D[(Local Vector Store)]
+    C -->|uSearch| D[(Vector Store)]
+    C -->|SQLite| G[(Embeddings DB)]
+    B -->|BM25| E[(Inverted Index)]
+    B -->|Branch Catalog| G
     end
 
     subgraph Searching
     Q[User Query] -->|Embedding Model| V[Query Vector]
     V -->|Cosine Similarity| D
-    D --> R[Ranked Results]
+    Q -->|BM25| E
+    G -->|Branch Filter| F
+    D --> F[Hybrid Fusion]
+    E --> F
+    F --> R[Ranked Results]
     end
 ```
 
-1. **Parsing**: We use `tree-sitter` to intelligently parse your code into meaningful blocks (functions, classes, interfaces).
-2. **Embedding**: These blocks are converted into vector representations using your configured AI provider.
-3. **Storage**: Vectors are stored in a high-performance local index using `usearch`.
-4. **Search**: Your natural language queries are matched against this index to find the most semantically relevant code.
+1. **Parsing**: We use `tree-sitter` to intelligently parse your code into meaningful blocks (functions, classes, interfaces). JSDoc comments and docstrings are automatically included with their associated code.
+2. **Chunking**: Large blocks are split with overlapping windows to preserve context across chunk boundaries.
+3. **Embedding**: These blocks are converted into vector representations using your configured AI provider.
+4. **Storage**: Embeddings are stored in SQLite (deduplicated by content hash) and vectors in `usearch` with F16 quantization for 50% memory savings. A branch catalog tracks which chunks exist on each branch.
+5. **Hybrid Search**: Combines semantic similarity (vectors) with BM25 keyword matching, filtered by current branch.
 
 **Performance characteristics:**
 - **Incremental indexing**: ~50ms check time — only re-embeds changed files
-- **Smart chunking**: Understands code structure to keep functions whole
+- **Smart chunking**: Understands code structure to keep functions whole, with overlap for context
 - **Native speed**: Core logic written in Rust for maximum performance
+- **Memory efficient**: F16 vector quantization reduces index size by 50%
+- **Branch-aware**: Automatically tracks which chunks exist on each git branch
+
+## 🌿 Branch-Aware Indexing
+
+The plugin automatically detects git branches and optimizes indexing across branch switches.
+
+### How It Works
+
+When you switch branches, code changes but embeddings for unchanged content remain the same. The plugin:
+
+1. **Stores embeddings by content hash**: Embeddings are deduplicated across branches
+2. **Tracks branch membership**: A lightweight catalog tracks which chunks exist on each branch
+3. **Filters search results**: Queries only return results relevant to the current branch
+
+### Benefits
+
+| Scenario | Without Branch Awareness | With Branch Awareness |
+|----------|-------------------------|----------------------|
+| Switch to feature branch | Re-index everything | Instant — reuse existing embeddings |
+| Return to main | Re-index everything | Instant — catalog already exists |
+| Search on branch | May return stale results | Only returns current branch's code |
+
+### Automatic Behavior
+
+- **Branch detection**: Automatically reads from `.git/HEAD`
+- **Re-indexing on switch**: Triggers when you switch branches (via file watcher)
+- **Legacy migration**: Automatically migrates old indexes on first run
+- **Garbage collection**: Health check removes orphaned embeddings and chunks
+
+### Storage Structure
+
+```
+.opencode/index/
+├── codebase.db           # SQLite: embeddings, chunks, branch catalog
+├── vectors.usearch       # Vector index (uSearch)
+├── inverted-index.json   # BM25 keyword index
+└── file-hashes.json      # File change detection
+```
 
 ## 🧰 Tools Available
 
@@ -117,22 +188,17 @@ The plugin exposes these tools to the OpenCode agent:
 ### `index_codebase`
 Manually trigger indexing.
 - **Use for**: Forcing a re-index or checking stats.
-- **Parameters**: `force` (rebuild all), `estimateOnly` (check costs).
+- **Parameters**: `force` (rebuild all), `estimateOnly` (check costs), `verbose` (show skipped files and parse failures).
 
 ### `index_status`
 Checks if the index is ready and healthy.
 
 ### `index_health_check`
-Maintenance tool to remove stale entries from deleted files.
+Maintenance tool to remove stale entries from deleted files and orphaned embeddings/chunks from the database.
 
 ## 🎮 Slash Commands
 
-For easier access, you can add slash commands to your project.
-
-Copy the commands:
-```bash
-cp -r node_modules/opencode-codebase-index/commands/* .opencode/command/
-```
+The plugin automatically registers these slash commands:
 
 | Command | Description |
 | ------- | ----------- |
@@ -151,7 +217,9 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
   "indexing": {
     "autoIndex": false,
     "watchFiles": true,
-    "maxFileSize": 1048576
+    "maxFileSize": 1048576,
+    "maxChunksPerFile": 100,
+    "semanticOnly": false
   },
   "search": {
     "maxResults": 20,
@@ -172,6 +240,10 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 | `autoIndex` | `false` | Automatically index on plugin load |
 | `watchFiles` | `true` | Re-index when files change |
 | `maxFileSize` | `1048576` | Skip files larger than this (bytes). Default: 1MB |
+| `maxChunksPerFile` | `100` | Maximum chunks to index per file (controls token costs for large files) |
+| `semanticOnly` | `false` | When `true`, only index semantic nodes (functions, classes) and skip generic blocks |
+| `retries` | `3` | Number of retry attempts for failed embedding API calls |
+| `retryDelayMs` | `1000` | Delay between retries in milliseconds |
 | **search** | | |
 | `maxResults` | `20` | Maximum results to return |
 | `minScore` | `0.1` | Minimum similarity score (0-1). Lower = more results |
@@ -204,19 +276,16 @@ Be aware of these characteristics:
    npm run build
    ```
 
-2. **Deploy to OpenCode Cache**:
-   ```bash
-   # Deploy script
-   rm -rf ~/.cache/opencode/node_modules/opencode-codebase-index
-   mkdir -p ~/.cache/opencode/node_modules/opencode-codebase-index
-   cp -R dist native commands skill package.json ~/.cache/opencode/node_modules/opencode-codebase-index/
+2. **Register in Test Project** (use `file://` URL in `opencode.json`):
+   ```json
+   {
+     "plugin": [
+       "file:///path/to/opencode-codebase-index"
+     ]
+   }
    ```
-
-3. **Register in Test Project**:
-   ```bash
-   mkdir -p .opencode/plugin
-    echo 'export { default } from "$HOME/.cache/opencode/node_modules/opencode-codebase-index/dist/index.js"' > .opencode/plugin/codebase-index.ts
-    ```
+   
+   This loads directly from your source directory, so changes take effect after rebuilding.
 
 ## 🤝 Contributing
 
@@ -237,12 +306,13 @@ CI will automatically run tests and type checking on your PR.
 │   ├── config/               # Configuration schema
 │   ├── embeddings/           # Provider detection and API calls
 │   ├── indexer/              # Core indexing logic + inverted index
+│   ├── git/                  # Git utilities (branch detection)
 │   ├── tools/                # OpenCode tool definitions
 │   ├── utils/                # File collection, cost estimation
 │   ├── native/               # Rust native module wrapper
-│   └── watcher/              # File change watcher
+│   └── watcher/              # File/git change watcher
 ├── native/
-│   └── src/                  # Rust: tree-sitter, usearch, xxhash
+│   └── src/                  # Rust: tree-sitter, usearch, xxhash, SQLite
 ├── tests/                    # Unit tests (vitest)
 ├── commands/                 # Slash command definitions
 ├── skill/                    # Agent skill guidance
@@ -252,8 +322,10 @@ CI will automatically run tests and type checking on your PR.
 ### Native Module
 
 The Rust native module handles performance-critical operations:
-- **tree-sitter**: Language-aware code parsing
-- **usearch**: High-performance vector similarity search
+- **tree-sitter**: Language-aware code parsing with JSDoc/docstring extraction
+- **usearch**: High-performance vector similarity search with F16 quantization
+- **SQLite**: Persistent storage for embeddings, chunks, and branch catalog
+- **BM25 inverted index**: Fast keyword search for hybrid retrieval
 - **xxhash**: Fast content hashing for change detection
 
 Rebuild with: `npm run build:native` (requires Rust toolchain)