opencode-codebase-index 0.6.0 → 0.7.0

package/README.md

@@ -17,6 +17,8 @@
 - [🎯 When to Use What](#-when-to-use-what)
 - [🧰 Tools Available](#-tools-available)
 - [🎮 Slash Commands](#-slash-commands)
+- [📚 Knowledge Base](#-knowledge-base)
+- [🔄 Reranking](#-reranking)
 - [⚙️ Configuration](#️-configuration)
 - [🤝 Contributing](#-contributing)
 
@@ -130,12 +132,13 @@ src/api/checkout.ts:89      (Route handler for /pay)
 | Don't know the function name | `codebase_search` | Semantic search finds by meaning |
 | Exploring unfamiliar codebase | `codebase_search` | Discovers related code across files |
 | Just need to find locations | `codebase_peek` | Returns metadata only, saves ~90% tokens |
+| Need the authoritative definition site | `implementation_lookup` | Prioritizes real implementation definitions over docs/tests |
 | Understand code flow | `call_graph` | Find callers/callees of any function |
 | Know exact identifier | `grep` | Faster, finds all occurrences |
 | Need ALL matches | `grep` | Semantic returns top N only |
 | Mixed discovery + precision | `/find` (hybrid) | Best of both worlds |
 
-**Rule of thumb**: `codebase_peek` to find locations → `Read` to examine → `grep` for precision.
+**Rule of thumb**: `codebase_peek` to find locations → `Read` to examine → `grep` for precision. For symbol-definition questions, use `implementation_lookup` first.
 
 ## 📊 Token Usage
 
@@ -186,7 +189,23 @@ graph TD
 
 1. **Parsing**: We use `tree-sitter` to intelligently parse your code into meaningful blocks (functions, classes, interfaces). JSDoc comments and docstrings are automatically included with their associated code.
 
-**Supported Languages**: TypeScript, JavaScript, Python, Rust, Go, Java, C#, Ruby, PHP, Bash, C, C++, JSON, TOML, YAML
+**Supported Languages (Tree-sitter semantic parsing)**: TypeScript, JavaScript, Python, Rust, Go, Java, C#, Ruby, PHP, Bash, C, C++, JSON, TOML, YAML
+
+**Additional Supported Formats (line-based chunking)**: TXT, HTML, HTM, Markdown, Shell scripts
+
+**Default File Patterns**:
+```
+**/*.{ts,tsx,js,jsx,mjs,cjs}    **/*.{py,pyi}
+**/*.{go,rs,java,kt,scala}      **/*.{c,cpp,cc,h,hpp}
+**/*.{rb,php,inc,swift}         **/*.{vue,svelte,astro}
+**/*.{sql,graphql,proto}        **/*.{yaml,yml,toml}
+**/*.{md,mdx}                   **/*.{sh,bash,zsh}
+**/*.{txt,html,htm}
+```
+
+Use `include` to replace defaults, or `additionalInclude` to extend (e.g. `"**/*.pdf"`, `"**/*.csv"`).
+
+**Max File Size**: Default 1MB (1048576 bytes). Configure via `indexing.maxFileSize` (bytes).
 2. **Chunking**: Large blocks are split with overlapping windows to preserve context across chunk boundaries.
 3. **Embedding**: These blocks are converted into vector representations using your configured AI provider.
 4. **Storage**: Embeddings are stored in SQLite (deduplicated by content hash) and vectors in `usearch` with F16 quantization for 50% memory savings. A branch catalog tracks which chunks exist on each branch.
@@ -237,6 +256,14 @@ When you switch branches, code changes but embeddings for unchanged content rema
 └── file-hashes.json      # File change detection
 ```
 
+### File Exclusions
+
+The following files/folders are excluded from indexing by default:
+
+- **Hidden files/folders**: Files starting with `.` (e.g., `.github`, `.vscode`, `.env`)
+- **Build folders**: Folders containing "build" in their name (e.g., `build`, `mingwBuildDebug`, `cmake-build-debug`)
+- **Default excludes**: `node_modules`, `dist`, `vendor`, `__pycache__`, `target`, `coverage`, etc.
+
 ## 🧰 Tools Available
 
 The plugin exposes these tools to the OpenCode agent:
@@ -270,6 +297,12 @@ The plugin exposes these tools to the OpenCode agent:
   ```
 - **Workflow**: `codebase_peek` → find locations → `Read` specific files
 
+### `implementation_lookup`
+**Definition-first lookup.** Jumps to the authoritative definition site for a symbol or natural-language definition query.
+- **Use for**: "Where is X defined?", symbol-definition requests, and cases where you want the implementation site rather than all usages.
+- **Behavior**: Prefers real implementation files over tests, docs, examples, and fixtures.
+- **Fallback**: If nothing authoritative is found, use `codebase_search` for broader discovery.
+
 ### `find_similar`
 Find code similar to a provided snippet.
 - **Use for**: Duplicate detection, refactor prep, pattern mining.
@@ -300,6 +333,20 @@ Query the call graph to find callers or callees of a function/method. Automatica
 - **Parameters**: `name` (function name), `direction` (`callers` or `callees`), `symbolId` (required for `callees`, returned by previous queries).
 - **Example**: Find who calls `validateToken` → `call_graph(name="validateToken", direction="callers")`
 
+### `add_knowledge_base`
+Add a folder as a knowledge base to be indexed alongside project code.
+- **Use for**: Indexing external documentation, API references, example programs.
+- **Parameters**: `path` (folder path, absolute or relative), `reindex` (optional, default `true`).
+- **Example**: `add_knowledge_base(path="/path/to/docs")`
+
+### `list_knowledge_bases`
+List all configured knowledge base folders and their status.
+
+### `remove_knowledge_base`
+Remove a knowledge base folder from the index.
+- **Parameters**: `path` (folder path to remove), `reindex` (optional, default `false`).
+- **Example**: `remove_knowledge_base(path="/path/to/docs")`
+
 ## 🎮 Slash Commands
 
 The plugin automatically registers these slash commands:
@@ -312,38 +359,218 @@ The plugin automatically registers these slash commands:
 | `/index` | **Update Index**. Forces a refresh of the codebase index. |
 | `/status` | **Check Status**. Shows if indexed, chunk count, and provider info. |
 
+## 📚 Knowledge Base
+
+The plugin can index **external documentation** alongside your project code. The indexed codebase includes:
+
+- **Project Source Code** — all code files in the current workspace
+- **API References** — hardware API docs, library documentation
+- **Usage Guides** — tutorials, how-to guides
+- **Example Programs** — code samples, demo projects
+
+### Adding Knowledge Base Folders
+
+Use the built-in tools to add documentation folders:
+
+```
+add_knowledge_base(path="/path/to/api-docs")
+add_knowledge_base(path="/path/to/examples")
+```
+
+The folder will be indexed into the **same database** as your project code. All searches automatically include both sources.
+
+### Managing Knowledge Bases
+
+```
+list_knowledge_bases          # Show configured knowledge bases
+remove_knowledge_base(path="/path/to/api-docs")  # Remove a knowledge base
+```
+
+### Configuration Example
+
+Project-level config (`.opencode/codebase-index.json`):
+```json
+{
+  "knowledgeBases": [
+    "/home/user/docs/esp-idf",
+    "/home/user/docs/arduino"
+  ]
+}
+```
+
+Global-level config (`~/.config/opencode/codebase-index.json`):
+```json
+{
+  "embeddingProvider": "custom",
+  "customProvider": {
+    "baseUrl": "https://api.siliconflow.cn/v1",
+    "model": "BAAI/bge-m3",
+    "dimensions": 1024,
+    "apiKey": "{env:SILICONFLOW_API_KEY}"
+  }
+}
+```
+
+Config merging: Global config is the base, project config overrides. Knowledge bases from both levels are merged.
+
+### Syncing Changes
+
+- **Project code**: Auto-synced via file watcher (real-time)
+- **Knowledge base folders**: Manual sync — run `/index force` after changes
+
+## 🔄 Reranking
+
+The plugin supports **API-based reranking** for improved search result quality. Reranking uses a cross-encoder model to rescore the top search results.
+
+### Enable Reranking
+
+Add to your config (`.opencode/codebase-index.json` or global config):
+
+```json
+{
+  "reranker": {
+    "enabled": true,
+    "baseUrl": "https://api.siliconflow.cn/v1",
+    "model": "BAAI/bge-reranker-v2-m3",
+    "apiKey": "{env:SILICONFLOW_API_KEY}",
+    "topN": 20
+  }
+}
+```
+
+### Reranker Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `enabled` | `false` | Enable reranking |
+| `baseUrl` | - | Rerank API endpoint |
+| `model` | - | Reranking model name |
+| `apiKey` | - | API key (use `{env:VAR}` for security) |
+| `topN` | `20` | Number of top results to rerank |
+| `timeoutMs` | `30000` | Request timeout |
+
+### How It Works
+
+```
+Query → Embedding Search → BM25 Search → Fusion → Reranking → Results
+```
+
+1. **Embedding Search**: Semantic similarity via vector search
+2. **BM25 Search**: Keyword matching via inverted index
+3. **Fusion**: Combine semantic + keyword results (RRF or weighted)
+4. **Reranking**: Cross-encoder rescores top N results via API
+5. **Results**: Final ranked results
+
+### Supported Reranking APIs
+
+Any OpenAI-compatible reranking endpoint. Examples:
+- **SiliconFlow**: `BAAI/bge-reranker-v2-m3`
+- **Cohere**: `rerank-english-v3.0`
+- **Local models**: Any server implementing `/v1/rerank` format
+
 ## ⚙️ Configuration
 
 Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-index.json`:
 
+### Full Configuration Example
+
 ```json
 {
-  "embeddingProvider": "auto",
-  "scope": "project",
+  // === Embedding Provider ===
+  "embeddingProvider": "custom",              // auto | github-copilot | openai | google | ollama | custom
+  "scope": "project",                         // project (per-repo) | global (shared)
+
+  // === Custom Embedding API (when embeddingProvider is "custom") ===
+  "customProvider": {
+    "baseUrl": "https://api.siliconflow.cn/v1",
+    "model": "BAAI/bge-m3",
+    "dimensions": 1024,
+    "apiKey": "{env:SILICONFLOW_API_KEY}",
+    "maxTokens": 8192,                        // Max tokens per input text
+    "timeoutMs": 30000,                       // Request timeout (ms)
+    "concurrency": 3,                         // Max concurrent requests
+    "requestIntervalMs": 1000,                // Min delay between requests (ms)
+    "maxBatchSize": 64                        // Max inputs per /embeddings request
+  },
+
+  // === File Patterns ===
+  "include": [                                // Override default include patterns
+    "**/*.{ts,js,py,go,rs}"
+  ],
+  "exclude": [                                // Override default exclude patterns
+    "**/node_modules/**"
+  ],
+  "additionalInclude": [                      // Extend defaults (not replace)
+    "**/*.{txt,html,htm}",
+    "**/*.pdf"
+  ],
+
+  // === Knowledge Bases ===
+  "knowledgeBases": [                         // External docs to index alongside code
+    "/home/user/docs/esp-idf",
+    "/home/user/docs/arduino"
+  ],
+
+  // === Indexing ===
   "indexing": {
-    "autoIndex": false,
-    "watchFiles": true,
-    "maxFileSize": 1048576,
-    "maxChunksPerFile": 100,
-    "semanticOnly": false,
-    "autoGc": true,
-    "gcIntervalDays": 7,
-    "gcOrphanThreshold": 100,
-    "requireProjectMarker": true
+    "autoIndex": false,                       // Auto-index on plugin load
+    "watchFiles": true,                       // Re-index on file changes
+    "maxFileSize": 1048576,                   // Max file size in bytes (default: 1MB)
+    "maxChunksPerFile": 100,                  // Max chunks per file
+    "semanticOnly": false,                    // Only index functions/classes (skip blocks)
+    "retries": 3,                             // Embedding API retry attempts
+    "retryDelayMs": 1000,                     // Delay between retries (ms)
+    "autoGc": true,                           // Auto garbage collection
+    "gcIntervalDays": 7,                      // GC interval (days)
+    "gcOrphanThreshold": 100,                 // GC trigger threshold
+    "requireProjectMarker": true,             // Require .git/package.json to index
+    "maxDepth": 5,                            // Max directory depth (-1=unlimited, 0=root only)
+    "maxFilesPerDirectory": 100,              // Max files per directory (smallest first)
+    "fallbackToTextOnMaxChunks": true         // Fallback to text chunking on maxChunksPerFile
   },
+
+  // === Search ===
   "search": {
-    "maxResults": 20,
-    "minScore": 0.1,
-    "hybridWeight": 0.5,
-    "fusionStrategy": "rrf",
-    "rrfK": 60,
-    "rerankTopN": 20,
-    "contextLines": 0
+    "maxResults": 20,                         // Max results to return
+    "minScore": 0.1,                          // Min similarity score (0-1)
+    "hybridWeight": 0.5,                      // Keyword (1.0) vs semantic (0.0)
+    "fusionStrategy": "rrf",                  // rrf | weighted
+    "rrfK": 60,                               // RRF smoothing constant
+    "rerankTopN": 20,                         // Deterministic rerank depth
+    "contextLines": 0,                        // Extra lines before/after match
+    "routingHints": true                      // Runtime nudges for local discovery/definition queries
   },
-  "debug": {
+  "reranker": {
     "enabled": false,
-    "logLevel": "info",
-    "metrics": false
+    "provider": "cohere",
+    "model": "rerank-v3.5",
+    "apiKey": "{env:RERANK_API_KEY}",
+    "topN": 15,
+    "timeoutMs": 10000
+  },
+  "debug": {
+    "enabled": false,                         // Enable debug logging
+    "logLevel": "info",                       // error | warn | info | debug
+    "logSearch": true,                        // Log search operations
+    "logEmbedding": true,                     // Log embedding API calls
+    "logCache": true,                         // Log cache hits/misses
+    "logGc": true,                            // Log garbage collection
+    "logBranch": true,                        // Log branch detection
+    "metrics": false                          // Enable metrics collection
+  }
+}
+```
+
+String values in `codebase-index.json` can reference environment variables with `{env:VAR_NAME}` when the placeholder is the entire string value. Variable names must match `[A-Z_][A-Z0-9_]*`. This is useful for secrets such as custom provider API keys so they do not need to be committed to the config file.
+
+```json
+{
+  "embeddingProvider": "custom",
+  "customProvider": {
+    "baseUrl": "{env:EMBED_BASE_URL}",
+    "model": "nomic-embed-text",
+    "dimensions": 768,
+    "apiKey": "{env:EMBED_API_KEY}"
   }
 }
 ```
@@ -354,6 +581,10 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 |--------|---------|-------------|
 | `embeddingProvider` | `"auto"` | Which AI to use: `auto`, `github-copilot`, `openai`, `google`, `ollama`, `custom` |
 | `scope` | `"project"` | `project` = index per repo, `global` = shared index across repos |
+| `include` | (defaults) | Override the default include patterns (replaces defaults) |
+| `exclude` | (defaults) | Override the default exclude patterns (replaces defaults) |
+| `additionalInclude` | `[]` | Additional file patterns to include (extends defaults, e.g. `"**/*.txt"`, `"**/*.html"`) |
+| `knowledgeBases` | `[]` | External directories to index as knowledge bases (absolute or relative paths) |
 | **indexing** | | |
 | `autoIndex` | `false` | Automatically index on plugin load |
 | `watchFiles` | `true` | Re-index when files change |
@@ -366,6 +597,9 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 | `gcIntervalDays` | `7` | Run GC on initialization if last GC was more than N days ago |
 | `gcOrphanThreshold` | `100` | Run GC after indexing if orphan count exceeds this threshold |
 | `requireProjectMarker` | `true` | Require a project marker (`.git`, `package.json`, etc.) to enable file watching and auto-indexing. Prevents accidentally indexing large directories like home. Set to `false` to index any directory. |
+| `maxDepth` | `5` | Max directory traversal depth. `-1` = unlimited, `0` = only files in root dir, `1` = one level of subdirectories, etc. |
+| `maxFilesPerDirectory` | `100` | Max files to index per directory. Always picks the smallest files first. |
+| `fallbackToTextOnMaxChunks` | `true` | When a file exceeds `maxChunksPerFile`, fallback to text-based (line-by-line) chunking instead of skipping the rest of the file. |
 | **search** | | |
 | `maxResults` | `20` | Maximum results to return |
 | `minScore` | `0.1` | Minimum similarity score (0-1). Lower = more results |
@@ -374,6 +608,15 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 | `rrfK` | `60` | RRF smoothing constant. Higher values flatten rank impact, lower values prioritize top-ranked candidates more strongly |
 | `rerankTopN` | `20` | Deterministic rerank depth cap. Applies lightweight name/path/chunk-type rerank to top-N only |
 | `contextLines` | `0` | Extra lines to include before/after each match |
+| `routingHints` | `true` | Inject lightweight runtime hints for local conceptual discovery and definition lookups. Set to `false` to disable plugin-side routing nudges. |
+| **reranker** | | Optional second-stage model reranker for the top candidate pool |
+| `enabled` | `false` | Turn external reranking on/off |
+| `provider` | `"custom"` | Hosted shortcuts: `cohere`, `jina`, or `custom` |
+| `model` | — | Reranker model name required when enabled |
+| `baseUrl` | provider default | Override reranker endpoint base URL. `cohere` → `https://api.cohere.ai/v1`, `jina` → `https://api.jina.ai/v1` |
+| `apiKey` | — | API key for hosted reranker providers |
+| `topN` | `15` | Number of top candidates to send to the external reranker |
+| `timeoutMs` | `10000` | Timeout for external rerank requests |
 | **debug** | | |
 | `enabled` | `false` | Enable debug logging and metrics collection |
 | `logLevel` | `"info"` | Log level: `error`, `warn`, `info`, `debug` |
@@ -386,9 +629,11 @@ Zero-config by default (uses `auto` mode). Customize in `.opencode/codebase-inde
 
 ### Retrieval ranking behavior
 
-- `codebase_search` and `codebase_peek` use the hybrid path: semantic + keyword retrieval → fusion (`fusionStrategy`) → deterministic rerank (`rerankTopN`) → filtering.
+- `codebase_search` and `codebase_peek` use the hybrid path: semantic + keyword retrieval → fusion (`fusionStrategy`) → deterministic rerank (`rerankTopN`) → optional external reranker (`reranker`) → filtering.
+- When `search.routingHints` is enabled (default), the plugin adds tiny per-turn runtime hints for local conceptual discovery and definition queries. Conceptual discovery is nudged toward `codebase_peek` / `codebase_search`, while definition questions are nudged toward `implementation_lookup`. Exact identifier and unrelated operational tasks are left alone.
 - `find_similar` stays semantic-only: semantic retrieval + deterministic rerank only (no keyword retrieval, no RRF).
 - For compatibility rollbacks, set `search.fusionStrategy` to `"weighted"` to use the legacy weighted fusion path.
+- When enabled, the external reranker sees path metadata plus a bounded on-disk code snippet for each candidate so it can distinguish real implementations from docs/tests more reliably.
 - Retrieval benchmark artifacts are separated by role:
   - baseline (versioned): `benchmarks/baselines/retrieval-baseline.json`
   - latest candidate run (generated): `benchmark-results/retrieval-candidate.json`
@@ -557,6 +802,10 @@ ollama pull nomic-embed-text
 }
 ```
 
+The built-in `ollama` provider uses Ollama's native `/api/embeddings` endpoint and is the simplest setup when you want to use `nomic-embed-text`.
+
+If you want to use a different Ollama embedding model through its OpenAI-compatible API, use the `custom` provider instead and set `customProvider.baseUrl` to `http://127.0.0.1:11434/v1` so the plugin calls `.../v1/embeddings`.
+
 ## 📈 Performance
 
 The plugin is built for speed with a Rust native module (`tree-sitter`, `usearch`, SQLite). In practice, indexing and retrieval remain fast enough for interactive use on medium/large repositories.
@@ -604,16 +853,37 @@ Works with any server that implements the OpenAI `/v1/embeddings` API format (ll
 {
   "embeddingProvider": "custom",
   "customProvider": {
-    "baseUrl": "http://localhost:11434/v1",
+    "baseUrl": "{env:EMBED_BASE_URL}",
     "model": "nomic-embed-text",
     "dimensions": 768,
-    "apiKey": "optional-api-key",
+    "apiKey": "{env:EMBED_API_KEY}",
     "maxTokens": 8192,
-    "timeoutMs": 30000
+    "timeoutMs": 30000,
+    "maxBatchSize": 64
+  }
+}
+```
+Required fields: `baseUrl`, `model`, `dimensions` (positive integer). Optional: `apiKey`, `maxTokens`, `timeoutMs` (default: 30000), `maxBatchSize` (or `max_batch_size`) to cap inputs per `/embeddings` request for servers like text-embeddings-inference. `{env:VAR_NAME}` placeholders are resolved before config validation for fields that are actually used and throw if the referenced environment variable is missing or malformed.
+
+**Custom Ollama models via OpenAI-compatible API**
+If you are running Ollama locally and want to use an embedding model other than the built-in `ollama` setup, point the custom provider at Ollama's OpenAI-compatible base URL with the `/v1` suffix:
+
+```json
+{
+  "embeddingProvider": "custom",
+  "customProvider": {
+    "baseUrl": "http://127.0.0.1:11434/v1",
+    "model": "qwen3-embedding:0.6b",
+    "dimensions": 1024,
+    "apiKey": "ollama"
   }
 }
 ```
-Required fields: `baseUrl`, `model`, `dimensions` (positive integer). Optional: `apiKey`, `maxTokens`, `timeoutMs` (default: 30000).
+
+Notes:
+- The plugin appends `/embeddings`, so `baseUrl` should be `http://127.0.0.1:11434/v1`, not just `http://127.0.0.1:11434`.
+- Ollama ignores the API key, but some OpenAI-compatible clients expect one, so a placeholder like `"ollama"` is fine.
+- Make sure `dimensions` matches the actual output size of the model you pulled locally.
 
 ## ⚠️ Tradeoffs
 

package/commands/index.md

@@ -9,12 +9,14 @@ User input: $ARGUMENTS
 Parse the input and set tool arguments:
 - force=true if input contains "force"
 - estimateOnly=true if input contains "estimate" 
-- verbose=true (always, for detailed output)
+- verbose=false (default, for token efficiency)
+- verbose=true if input contains "verbose" (for detailed output)
 
 Examples:
-- `/index` → force=false, estimateOnly=false, verbose=true
-- `/index force` → force=true, estimateOnly=false, verbose=true
-- `/index estimate` → force=false, estimateOnly=true, verbose=true
+- `/index` → force=false, estimateOnly=false, verbose=false
+- `/index force` → force=true, estimateOnly=false, verbose=false
+- `/index estimate` → force=false, estimateOnly=true, verbose=false
+- `/index verbose` → force=false, estimateOnly=false, verbose=true
 
 IMPORTANT: You MUST pass the parsed arguments to `index_codebase`. Do not ignore them.