opencode-codebase-index 0.1.8 → 0.1.11

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kenneth Helweg
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -68,6 +68,28 @@ src/api/checkout.ts:89      (Route handler for /pay)
 
 **Rule of thumb**: Semantic search for discovery → grep for precision.
 
+## 📊 Token Usage
+
+In our testing across open-source codebases (axios, express), we observed **up to 90% reduction in token usage** for conceptual queries like *"find the error handling middleware"*.
+
+### Why It Saves Tokens
+
+- **Without plugin**: Agent explores files, reads code, backtracks, explores more
+- **With plugin**: Semantic search returns relevant code immediately → less exploration
+
+### Key Takeaways
+
+1. **Significant savings possible**: Up to 90% reduction in the best cases
+2. **Results vary**: Savings depend on query type, codebase structure, and agent behavior
+3. **Best for discovery**: Conceptual queries benefit most; exact identifier lookups should use grep
+4. **Complements existing tools**: Provides a faster initial signal, doesn't replace grep/explore
+
+### When the Plugin Helps Most
+
+- **Conceptual queries**: "Where is the authentication logic?" (no keywords to grep for)
+- **Unfamiliar codebases**: You don't know what to search for yet
+- **Large codebases**: Semantic search scales better than exhaustive exploration
+
 ## 🛠️ How It Works
 
 ```mermaid
@@ -75,25 +97,31 @@ graph TD
     subgraph Indexing
     A[Source Code] -->|Tree-sitter| B[Semantic Chunks]
     B -->|Embedding Model| C[Vectors]
-    C -->|uSearch| D[(Local Vector Store)]
+    C -->|uSearch| D[(Vector Store)]
+    B -->|BM25| E[(Inverted Index)]
     end
 
     subgraph Searching
     Q[User Query] -->|Embedding Model| V[Query Vector]
     V -->|Cosine Similarity| D
-    D --> R[Ranked Results]
+    Q -->|BM25| E
+    D --> F[Hybrid Fusion]
+    E --> F
+    F --> R[Ranked Results]
     end
 ```
 
-1. **Parsing**: We use `tree-sitter` to intelligently parse your code into meaningful blocks (functions, classes, interfaces).
-2. **Embedding**: These blocks are converted into vector representations using your configured AI provider.
-3. **Storage**: Vectors are stored in a high-performance local index using `usearch`.
-4. **Search**: Your natural language queries are matched against this index to find the most semantically relevant code.
+1. **Parsing**: We use `tree-sitter` to intelligently parse your code into meaningful blocks (functions, classes, interfaces). JSDoc comments and docstrings are automatically included with their associated code.
+2. **Chunking**: Large blocks are split with overlapping windows to preserve context across chunk boundaries.
+3. **Embedding**: These blocks are converted into vector representations using your configured AI provider.
+4. **Storage**: Vectors are stored in a high-performance local index using `usearch` with F16 quantization for 50% memory savings.
+5. **Hybrid Search**: Combines semantic similarity (vectors) with BM25 keyword matching for best results.
 
 **Performance characteristics:**
 - **Incremental indexing**: ~50ms check time — only re-embeds changed files
-- **Smart chunking**: Understands code structure to keep functions whole
+- **Smart chunking**: Understands code structure to keep functions whole, with overlap for context
 - **Native speed**: Core logic written in Rust for maximum performance
+- **Memory efficient**: F16 vector quantization reduces index size by 50%
 
 ## 🧰 Tools Available
 
@@ -117,7 +145,7 @@ The plugin exposes these tools to the OpenCode agent:
 ### `index_codebase`
 Manually trigger indexing.
 - **Use for**: Forcing a re-index or checking stats.
-- **Parameters**: `force` (rebuild all), `estimateOnly` (check costs).
+- **Parameters**: `force` (rebuild all), `estimateOnly` (check costs), `verbose` (show skipped files and parse failures).
 
 ### `index_status`
 Checks if the index is ready and healthy.
@@ -127,12 +155,7 @@ Maintenance tool to remove stale entries from deleted files.
 
 ## 🎮 Slash Commands
 
-For easier access, you can add slash commands to your project.
-
-Copy the commands:
-```bash
-cp -r node_modules/opencode-codebase-index/commands/* .opencode/command/
-```
+The plugin automatically registers these slash commands:
 
 | Command | Description |
 | ------- | ----------- |
@@ -151,7 +174,9 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
   "indexing": {
     "autoIndex": false,
     "watchFiles": true,
-    "maxFileSize": 1048576
+    "maxFileSize": 1048576,
+    "maxChunksPerFile": 100,
+    "semanticOnly": false
   },
   "search": {
     "maxResults": 20,
@@ -172,6 +197,10 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 | `autoIndex` | `false` | Automatically index on plugin load |
 | `watchFiles` | `true` | Re-index when files change |
 | `maxFileSize` | `1048576` | Skip files larger than this (bytes). Default: 1MB |
+| `maxChunksPerFile` | `100` | Maximum chunks to index per file (controls token costs for large files) |
+| `semanticOnly` | `false` | When `true`, only index semantic nodes (functions, classes) and skip generic blocks |
+| `retries` | `3` | Number of retry attempts for failed embedding API calls |
+| `retryDelayMs` | `1000` | Delay between retries in milliseconds |
 | **search** | | |
 | `maxResults` | `20` | Maximum results to return |
 | `minScore` | `0.1` | Minimum similarity score (0-1). Lower = more results |
@@ -204,19 +233,16 @@ Be aware of these characteristics:
    npm run build
    ```
 
-2. **Deploy to OpenCode Cache**:
-   ```bash
-   # Deploy script
-   rm -rf ~/.cache/opencode/node_modules/opencode-codebase-index
-   mkdir -p ~/.cache/opencode/node_modules/opencode-codebase-index
-   cp -R dist native commands skill package.json ~/.cache/opencode/node_modules/opencode-codebase-index/
+2. **Register in Test Project** (use `file://` URL in `opencode.json`):
+   ```json
+   {
+     "plugin": [
+       "file:///path/to/opencode-codebase-index"
+     ]
+   }
    ```
-
-3. **Register in Test Project**:
-   ```bash
-   mkdir -p .opencode/plugin
-    echo 'export { default } from "$HOME/.cache/opencode/node_modules/opencode-codebase-index/dist/index.js"' > .opencode/plugin/codebase-index.ts
-    ```
+   
+   This loads directly from your source directory, so changes take effect after rebuilding.
 
 ## 🤝 Contributing
 
@@ -252,8 +278,9 @@ CI will automatically run tests and type checking on your PR.
 ### Native Module
 
 The Rust native module handles performance-critical operations:
-- **tree-sitter**: Language-aware code parsing
-- **usearch**: High-performance vector similarity search
+- **tree-sitter**: Language-aware code parsing with JSDoc/docstring extraction
+- **usearch**: High-performance vector similarity search with F16 quantization
+- **BM25 inverted index**: Fast keyword search for hybrid retrieval
 - **xxhash**: Fast content hashing for change detection
 
 Rebuild with: `npm run build:native` (requires Rust toolchain)