opencode-rag-plugin 2.10.0 → 2.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ReadMe.md +40 -28
- package/dist/api.js +1 -0
- package/dist/cli/commands/clickhouse.d.ts +61 -0
- package/dist/cli/commands/clickhouse.js +335 -0
- package/dist/cli/commands/eval-tuner.js +1 -1
- package/dist/cli/commands/eval.js +1 -1
- package/dist/cli/commands/index-command.js +2 -1
- package/dist/cli/commands/index.d.ts +1 -0
- package/dist/cli/commands/index.js +1 -0
- package/dist/cli/commands/init-helpers.js +2 -2
- package/dist/cli/commands/init.js +49 -11
- package/dist/cli/commands/quirk.js +6 -6
- package/dist/cli/commands/status.js +70 -6
- package/dist/cli/commands/ui.js +2 -2
- package/dist/cli/index.js +2 -1
- package/dist/cli/types.d.ts +14 -0
- package/dist/core/bootstrap.d.ts +7 -0
- package/dist/core/bootstrap.js +40 -9
- package/dist/core/config.d.ts +128 -5
- package/dist/core/config.js +205 -8
- package/dist/core/interfaces.d.ts +54 -0
- package/dist/core/runtime-overrides.d.ts +9 -0
- package/dist/core/runtime-overrides.js +44 -0
- package/dist/core/state-sync.d.ts +91 -0
- package/dist/core/state-sync.js +273 -0
- package/dist/eval/fast-index.js +2 -2
- package/dist/eval/rerank-eval.js +2 -2
- package/dist/eval/run-branch-compare.js +2 -2
- package/dist/eval/run-token-test.js +2 -2
- package/dist/index.d.ts +5 -2
- package/dist/index.js +2 -1
- package/dist/indexer/pipeline.d.ts +32 -0
- package/dist/indexer/pipeline.js +230 -65
- package/dist/indexer/progress.d.ts +75 -0
- package/dist/indexer/progress.js +114 -0
- package/dist/indexer/stats.d.ts +14 -0
- package/dist/plugin.d.ts +6 -3
- package/dist/plugin.js +39 -20
- package/dist/quirks/quirk-store.d.ts +10 -0
- package/dist/quirks/quirk-store.js +110 -15
- package/dist/quirks/types.d.ts +7 -0
- package/dist/tui.js +134 -113
- package/dist/vectorstore/clickhouse.d.ts +150 -0
- package/dist/vectorstore/clickhouse.js +931 -0
- package/dist/vectorstore/dimension.d.ts +19 -0
- package/dist/vectorstore/dimension.js +36 -0
- package/dist/vectorstore/factory.d.ts +9 -3
- package/dist/vectorstore/factory.js +11 -2
- package/dist/vectorstore/lancedb.d.ts +27 -4
- package/dist/vectorstore/lancedb.js +180 -15
- package/dist/watcher.d.ts +29 -0
- package/dist/watcher.js +76 -2
- package/dist/web/api.d.ts +3 -4
- package/dist/web/api.js +18 -3
- package/dist/web/server.js +28 -2
- package/dist/web/ui/assets/index-B-RH6fko.css +2 -0
- package/dist/web/ui/index.html +2 -2
- package/package.json +2 -1
- package/dist/web/ui/assets/index-CWcp0jNf.css +0 -2
- /package/dist/web/ui/assets/{index-DbXFrn53.js → index-ChGi2bSp.js} +0 -0
package/ReadMe.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# OpenCodeRAG
|
|
2
2
|
|
|
3
|
-
OpenCodeRAG is a **local-first RAG plugin** for semantic code and image search, with a built-in **local decision model** for fast classification. It converts your codebase into vector indices and retrieves relevant code chunks on natural language queries. The primary aim is to save tokens by replacing full-file reads with targeted chunk retrieval and to speed-up tool calls for large codebases. Integrates seamlessly with [OpenCode](https://opencode.ai) and works as a standalone MCP server or CLI tool for other AI harnesses.
|
|
3
|
+
OpenCodeRAG is a **local-first RAG plugin** for semantic code and image search, with a built-in **local decision model** for fast classification and an optional **Shell Guard** that vets shell commands before they run. It converts your codebase into vector indices and retrieves relevant code chunks on natural language queries. The primary aim is to save tokens by replacing full-file reads with targeted chunk retrieval and to speed-up tool calls for large codebases. Integrates seamlessly with [OpenCode](https://opencode.ai) and works as a standalone MCP server or CLI tool for other AI harnesses.
|
|
4
4
|
|
|
5
5
|
You don't need a dedicated GPU to run smaller embedding LLMs, as these models can still run performant on modern CPUs.
|
|
6
6
|
|
|
@@ -8,6 +8,8 @@ You don't need a dedicated GPU to run smaller embedding LLMs, as these models ca
|
|
|
8
8
|
|
|
9
9
|
> ⚠️ **Note:** Don't confuse this with the npm package `opencode-rag` (a discontinued project by a different author).
|
|
10
10
|
|
|
11
|
+
> 🛡️ **Shell Guard (optional):** every `shell`/`bash` command is classified by the local decision model **before it runs**. Flagged commands ask for confirmation with the full verdict (`destructive`/`malicious` probabilities + risk category); clean commands run unchanged. Enable once, machine-wide — see [Shell Guard](#shell-guard-optional).
|
|
12
|
+
|
|
11
13
|
## Quick Start
|
|
12
14
|
|
|
13
15
|
```bash
|
|
@@ -23,11 +25,18 @@ opencode-rag index
|
|
|
23
25
|
|
|
24
26
|
# 4. CLI search test
|
|
25
27
|
opencode-rag query "authentication middleware"
|
|
28
|
+
|
|
29
|
+
# 5. Optional: vet every shell command (once per machine)
|
|
30
|
+
mkdir -p ~/.config/opencode
|
|
31
|
+
echo '{ "shellGuard": { "enabled": true } }' > ~/.config/opencode/opencode-rag.json
|
|
32
|
+
# Requires the decision model: ollama pull tev1:0.8b
|
|
26
33
|
```
|
|
27
34
|
|
|
35
|
+
> **Try it:** with Shell Guard enabled, `rm -rf ./dist` is flagged (`delete_files`, ~0.96 destructive) and prompts for confirmation, while `ls -la src` passes untouched (~0.2).
|
|
36
|
+
|
|
28
37
|
`opencode-rag setup` installs/verifies the machine-level runtime and, when
|
|
29
38
|
run inside a project, initializes the workspace (`.opencode/`, plugin entries,
|
|
30
|
-
skill file,
|
|
39
|
+
skill file, `.opencode/opencode-rag.json`) in one step. When the installed version is
|
|
31
40
|
outdated, it fetches and installs the latest published version from npm.
|
|
32
41
|
`opencode-rag init` remains available for re-syncing an already initialized
|
|
33
42
|
workspace, and `opencode-rag update` upgrades to a newer release.
|
|
@@ -42,6 +51,7 @@ workspace, and `opencode-rag update` upgrades to a newer release.
|
|
|
42
51
|
|
|
43
52
|
| Feature | Description |
|
|
44
53
|
|---|---|
|
|
54
|
+
| **🛡️ Shell guard** | Optional **pre-execution vetting of every `shell`/`bash` command** by the local decision model — flagged commands ask for confirmation (with probabilities + risk category), clean commands run normally, fail-open by default. Machine-wide via the global config. [Details](#shell-guard-optional) |
|
|
45
55
|
| **MCP server** | `opencode-rag mcp` - stdio-based MCP server exposing `search_semantic`, `get_file_skeleton`, `find_usages`, and `describe_image` tools for any MCP-compatible client |
|
|
46
56
|
| **AST chunking** | 26 languages via tree-sitter (TS, JS, Python, Java, Go, Rust, C/C++, C#, Ruby, Kotlin, Swift, Bash, PHP, PowerShell, SQL, JSON, HTML, CSS, XML (including SVG), YAML, TOML, INI, Dockerfile, Markdown, LaTeX, Razor) |
|
|
47
57
|
| **Configurable parsers** | Remap file extensions to any parser via `chunking.parsers` (e.g. `.c` → `cpp`, or index a new `.cu` as C++) — applied to chunking, `get_file_skeleton`, and read labels ([docs](doc/chunking.md#parser-overrides-per-extension-chunkingparsers)) |
|
|
@@ -50,11 +60,11 @@ workspace, and `opencode-rag update` upgrades to a newer release.
|
|
|
50
60
|
| **Hybrid search** | Vector similarity + TF×IDF keyword fusion — the keyword index is repaired from the store whenever it drifts behind it, so the BM25 half never covers only part of the corpus |
|
|
51
61
|
| **Retrieval calibration** | `opencode-rag eval:tune` sweeps the fusion weights against the labelled golden set and reports (or writes) the best configuration for your embedder and corpus |
|
|
52
62
|
| **Decision model** | Local [tev1](https://ollama.com/library/tev1) classification — `make_decision` labels, routes, and scores short text with probabilities (40–150 ms warm); optional pre-ask routing so the agent asks fewer questions |
|
|
53
|
-
| **Shell guard** | Optional pre-execution vetting of every `shell`/`bash` command by the local decision model — flagged commands ask for confirmation, clean ones run normally (fail-open by default) |
|
|
54
63
|
| **OpenCode plugin** | Auto-inject context, read-tool override, TUI settings, Ctrl+Enter to add RAG context, MCP registration on `init` |
|
|
55
64
|
| **Incremental indexing** | File-hash manifest, background watcher, auto-rebuild on corruption |
|
|
65
|
+
| **Central vector store** | Optional ClickHouse backend (`vectorStore.provider: "clickhouse"`) — one shared index for many machines/workspaces with HNSW ANN, epoch rebuilds and state sync; `opencode-rag clickhouse setup` configures it ([docs](doc/clickhouse.md)) |
|
|
56
66
|
| **Privacy-first** | All processing stays local (when using Ollama) |
|
|
57
|
-
| **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `decide`, `ui`, `mcp`, `setup`, `quirk`, `eval:sessions`, `eval:analyze`, `eval:compare`, `eval:gate`, `eval:decide`, `eval:tune` |
|
|
67
|
+
| **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `decide`, `ui`, `mcp`, `setup`, `clickhouse`, `quirk`, `eval:sessions`, `eval:analyze`, `eval:compare`, `eval:gate`, `eval:decide`, `eval:tune` |
|
|
58
68
|
| **Proxy-aware** | Corporate proxy support with raw-socket localhost bypass |
|
|
59
69
|
| **OpenAI / Anthropic / Cohere** | Use alternate embedding providers with API key auto-resolution |
|
|
60
70
|
| **Evaluation** | Session-level token tracking, RAG-on vs RAG-off comparison, tiktoken BPE counting |
|
|
@@ -64,6 +74,31 @@ workspace, and `opencode-rag update` upgrades to a newer release.
|
|
|
64
74
|
| **Quirk memory** | Persistent experiential memory — agents recall/persist gotchas, preferences, decisions across sessions (`add_quirk` / `recall_quirks`, `opencode-rag quirk`); failed tool calls are auto-recorded as quirks |
|
|
65
75
|
| **AGENTS.md directive** | `opencode-rag init` merges a tool-usage directive into `AGENTS.md` via sentinel markers |
|
|
66
76
|
|
|
77
|
+
## Shell Guard (Optional)
|
|
78
|
+
|
|
79
|
+
The same local decision model can vet **every shell command before it runs**. With `shellGuard.enabled: true`, each `shell`/`bash` call is classified for destructive and malicious intent; a command whose `p(destructive)` or `p(malicious)` reaches the threshold is escalated to OpenCode's confirmation prompt with the full verdict — otherwise it executes unchanged.
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"shellGuard": {
|
|
84
|
+
"enabled": true,
|
|
85
|
+
"threshold": 0.5,
|
|
86
|
+
"timeoutMs": 10000,
|
|
87
|
+
"onError": "allow"
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
- **Off by default** — enabling adds one local model call to the critical path of every shell command.
|
|
93
|
+
- **Fast** — a warm verdict costs ~200–230 ms (`rm -rf ./dist` → 0.96 destructive, `delete_files`; `ls -la src` → 0.23, allowed). Repeated identical commands hit the 10-minute cache.
|
|
94
|
+
- **Machine-wide** — put the block in `~/.config/opencode/opencode-rag.json` (the global layer, see [Configuration](doc/configuration.md#config-file-discovery)) and every workspace inherits it; a workspace config can still override or disable it.
|
|
95
|
+
- **Fail-open by default** (`onError: "allow"`): if the classifier is down or times out, the command runs and a warning is logged. Set `onError: "ask"` to require confirmation instead.
|
|
96
|
+
- **Verdict cache** (10 min, 256 entries): repeated identical commands are answered instantly; the key includes the provider/model, so switching models invalidates it, and cached probabilities are re-scored against the current threshold on every call.
|
|
97
|
+
- **Config edits apply without a restart** — the guard re-reads `opencode-rag.json` when it changes.
|
|
98
|
+
- **Backend** defaults to the `decision` section, and any `/v1/systemone` endpoint works — e.g. llama-swap with `"baseUrl": "http://localhost:11437", "model": "Clef-Flash"`.
|
|
99
|
+
|
|
100
|
+
Every verdict is logged under `scope: shell.guard`, so you can tune the threshold against real commands. See [Configuration](doc/configuration.md#shellguard) and [Plugin documentation](doc/plugin.md#shell-guard-command-vetting).
|
|
101
|
+
|
|
67
102
|
## Web UI
|
|
68
103
|
|
|
69
104
|
A browser-based dashboard for exploring the indexed vector database - browse, visualize and inspect chunks and evaluate the OpenCode sessions in terms of retrieved chunks, consumed tokens and more.
|
|
@@ -81,6 +116,7 @@ Launch with `opencode-rag ui`. See [Web UI documentation](doc/webui.md) for deta
|
|
|
81
116
|
| [Configuration](doc/configuration.md) | All options: embedding, indexing, retrieval, description, image description, decision, shell guard, plugin |
|
|
82
117
|
| [Chunking](doc/chunking.md) | Language matrix, adding new chunkers, custom chunkers |
|
|
83
118
|
| [Embedding](doc/embedding.md) | Providers, model recommendations, proxy, dimension probing |
|
|
119
|
+
| [ClickHouse store](doc/clickhouse.md) | Central vector store backend: setup, epochs, ANN, multi-machine sync |
|
|
84
120
|
| [Retrieval](doc/retrieval.md) | Pipeline, hybrid search, score fusion, caching |
|
|
85
121
|
| [Plugin](doc/plugin.md) | OpenCode integration, tools, hooks, TUI, troubleshooting |
|
|
86
122
|
| [CLI Reference](doc/cli.md) | All commands, options, examples |
|
|
@@ -147,30 +183,6 @@ ollama pull tev1:4b # 4.4 GB, more accurate — recommended when scoring
|
|
|
147
183
|
|
|
148
184
|
`opencode-rag decide` exposes the same engine from the CLI, and `init` creates a dedicated `make-decision` agent skill. See [Plugin documentation](doc/plugin.md#make_decision) and [Configuration](doc/configuration.md#decision).
|
|
149
185
|
|
|
150
|
-
## Shell Guard (Optional)
|
|
151
|
-
|
|
152
|
-
The same local decision model can vet **every shell command before it runs**. With `shellGuard.enabled: true`, each `shell`/`bash` call is classified for destructive and malicious intent; a command whose `p(destructive)` or `p(malicious)` reaches the threshold is escalated to OpenCode's confirmation prompt with the full verdict — otherwise it executes unchanged.
|
|
153
|
-
|
|
154
|
-
```json
|
|
155
|
-
{
|
|
156
|
-
"shellGuard": {
|
|
157
|
-
"enabled": true,
|
|
158
|
-
"threshold": 0.5,
|
|
159
|
-
"timeoutMs": 10000,
|
|
160
|
-
"onError": "allow"
|
|
161
|
-
}
|
|
162
|
-
}
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
- **Off by default** — enabling adds one local model call to the critical path of every shell command.
|
|
166
|
-
- **Machine-wide** — put the block in `~/.config/opencode/opencode-rag.json` (the global layer, see [Configuration](doc/configuration.md#config-file-discovery)) and every workspace inherits it; a workspace config can still override or disable it.
|
|
167
|
-
- **Fail-open by default** (`onError: "allow"`): if the classifier is down or times out, the command runs and a warning is logged. Set `onError: "ask"` to require confirmation instead.
|
|
168
|
-
- **Verdict cache** (10 min, 256 entries): repeated identical commands are answered instantly; the key includes the provider/model, so switching models invalidates it, and cached probabilities are re-scored against the current threshold on every call.
|
|
169
|
-
- **Config edits apply without a restart** — the guard re-reads `opencode-rag.json` when it changes.
|
|
170
|
-
- **Backend** defaults to the `decision` section, and any `/v1/systemone` endpoint works — e.g. llama-swap with `"baseUrl": "http://localhost:11437", "model": "Clef-Flash"`.
|
|
171
|
-
|
|
172
|
-
Every verdict is logged under `scope: shell.guard`, so you can tune the threshold against real commands. See [Configuration](doc/configuration.md#shellguard) and [Plugin documentation](doc/plugin.md#shell-guard-command-vetting).
|
|
173
|
-
|
|
174
186
|
## Image Indexing
|
|
175
187
|
|
|
176
188
|
OpenCodeRAG can index image files (PNG, JPEG, WebP, etc.) by sending them to a vision-capable LLM and storing the generated text descriptions as searchable vector chunks. This makes visual assets discoverable via natural language queries (e.g., "login screen screenshot", "architecture diagram").
|
package/dist/api.js
CHANGED
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @fileoverview `clickhouse setup` — interactively configure and initialize the
|
|
3
|
+
* ClickHouse vector store.
|
|
4
|
+
*
|
|
5
|
+
* Prompts for the endpoint, port, database, credentials, namespace, embedding
|
|
6
|
+
* dimension and ANN mode; probes the server (version/auth), creates the
|
|
7
|
+
* database and the project tables, then writes the `vectorStore` block into the
|
|
8
|
+
* workspace or global config file. Existing config keys are preserved.
|
|
9
|
+
*/
|
|
10
|
+
import type { Command } from "commander";
|
|
11
|
+
import { createInterface } from "node:readline/promises";
|
|
12
|
+
/** Options for `opencode-rag clickhouse setup`. */
|
|
13
|
+
export interface ClickHouseSetupOptions {
|
|
14
|
+
url?: string;
|
|
15
|
+
port?: string;
|
|
16
|
+
database?: string;
|
|
17
|
+
username?: string;
|
|
18
|
+
password?: string;
|
|
19
|
+
passwordEnv?: string;
|
|
20
|
+
namespace?: string;
|
|
21
|
+
ann?: string;
|
|
22
|
+
dimension?: string;
|
|
23
|
+
scope?: string;
|
|
24
|
+
storePassword?: boolean;
|
|
25
|
+
skipInit?: boolean;
|
|
26
|
+
yes?: boolean;
|
|
27
|
+
/** Working directory override (tests); defaults to `process.cwd()`. */
|
|
28
|
+
cwd?: string;
|
|
29
|
+
}
|
|
30
|
+
interface SetupInput {
|
|
31
|
+
url: string;
|
|
32
|
+
database: string;
|
|
33
|
+
username: string;
|
|
34
|
+
password: string;
|
|
35
|
+
passwordEnv: string;
|
|
36
|
+
namespace: string;
|
|
37
|
+
ann: "off" | "hnsw";
|
|
38
|
+
dimension: number;
|
|
39
|
+
storeInlinePassword: boolean;
|
|
40
|
+
}
|
|
41
|
+
/**
|
|
42
|
+
* Merge a ClickHouse `vectorStore` block into an existing config object.
|
|
43
|
+
* Exported for tests; preserves every unrelated top-level key and the old
|
|
44
|
+
* cache path when ClickHouse was already active.
|
|
45
|
+
*/
|
|
46
|
+
export declare function mergeClickHouseConfig(existing: Record<string, unknown> | undefined, input: Pick<SetupInput, "url" | "database" | "username" | "password" | "passwordEnv" | "namespace" | "ann" | "storeInlinePassword">): Record<string, unknown>;
|
|
47
|
+
/**
|
|
48
|
+
* Prompt for a password without echoing it.
|
|
49
|
+
*
|
|
50
|
+
* Readline's internal `_writeToOutput` hook is muted while the answer is read —
|
|
51
|
+
* an active readline interface otherwise echoes every keystroke, even when the
|
|
52
|
+
* caller switches the stream to raw mode.
|
|
53
|
+
*/
|
|
54
|
+
export declare function promptHidden(rl: ReturnType<typeof createInterface>, question: string): Promise<string>;
|
|
55
|
+
/** Compose `scheme://host:port` from a user-supplied URL and port. */
|
|
56
|
+
export declare function composeClickHouseUrl(rawUrl: string, port: string): string;
|
|
57
|
+
/** Run the interactive (or flag-driven) ClickHouse setup. */
|
|
58
|
+
export declare function runClickHouseSetup(options?: ClickHouseSetupOptions): Promise<void>;
|
|
59
|
+
/** Register the `clickhouse` command group. */
|
|
60
|
+
export declare function registerClickHouseCommand(program: Command): void;
|
|
61
|
+
export {};
|
|
@@ -0,0 +1,335 @@
|
|
|
1
|
+
import path from "node:path";
|
|
2
|
+
import { copyFileSync, existsSync, mkdirSync } from "node:fs";
|
|
3
|
+
import { createInterface } from "node:readline/promises";
|
|
4
|
+
import { loadWorkspaceConfig, globalConfigPath, findConfigFile, projectConfigPath } from "../../core/config.js";
|
|
5
|
+
import { ClickHouseStore, deriveNamespace, resolveClickHouseOptions, sanitizeNamespace, } from "../../vectorstore/clickhouse.js";
|
|
6
|
+
import { readJsonObject, writeJsonFile } from "../helpers.js";
|
|
7
|
+
import { c } from "../format.js";
|
|
8
|
+
/**
|
|
9
|
+
* Merge a ClickHouse `vectorStore` block into an existing config object.
|
|
10
|
+
* Exported for tests; preserves every unrelated top-level key and the old
|
|
11
|
+
* cache path when ClickHouse was already active.
|
|
12
|
+
*/
|
|
13
|
+
export function mergeClickHouseConfig(existing, input) {
|
|
14
|
+
const base = existing ?? {};
|
|
15
|
+
const current = (base.vectorStore ?? {});
|
|
16
|
+
const currentClickhouse = (current.clickhouse ?? {});
|
|
17
|
+
const cachePath = current.provider === "clickhouse" && typeof current.path === "string"
|
|
18
|
+
? current.path
|
|
19
|
+
: "./.opencode/rag_ch";
|
|
20
|
+
const clickhouse = {
|
|
21
|
+
...currentClickhouse,
|
|
22
|
+
url: input.url,
|
|
23
|
+
database: input.database,
|
|
24
|
+
username: input.username,
|
|
25
|
+
};
|
|
26
|
+
delete clickhouse.password;
|
|
27
|
+
if (input.storeInlinePassword && input.password)
|
|
28
|
+
clickhouse.password = input.password;
|
|
29
|
+
if (input.passwordEnv)
|
|
30
|
+
clickhouse.passwordEnv = input.passwordEnv;
|
|
31
|
+
else
|
|
32
|
+
delete clickhouse.passwordEnv;
|
|
33
|
+
if (input.namespace)
|
|
34
|
+
clickhouse.namespace = sanitizeNamespace(input.namespace);
|
|
35
|
+
else
|
|
36
|
+
delete clickhouse.namespace;
|
|
37
|
+
clickhouse.ann = input.ann;
|
|
38
|
+
if (input.ann === "hnsw") {
|
|
39
|
+
clickhouse.annFetchMultiplier = 4;
|
|
40
|
+
clickhouse.annWithRescoring = true;
|
|
41
|
+
}
|
|
42
|
+
return {
|
|
43
|
+
...base,
|
|
44
|
+
vectorStore: {
|
|
45
|
+
...current,
|
|
46
|
+
path: cachePath,
|
|
47
|
+
provider: "clickhouse",
|
|
48
|
+
clickhouse,
|
|
49
|
+
},
|
|
50
|
+
};
|
|
51
|
+
}
|
|
52
|
+
/** Parse "major.minor.patch.build" into a comparable pair. */
|
|
53
|
+
function parseVersion(version) {
|
|
54
|
+
const parts = version.split(".").map((part) => Number.parseInt(part, 10));
|
|
55
|
+
const major = parts[0];
|
|
56
|
+
const minor = parts[1];
|
|
57
|
+
return {
|
|
58
|
+
major: major !== undefined && Number.isFinite(major) ? major : 0,
|
|
59
|
+
minor: minor !== undefined && Number.isFinite(minor) ? minor : 0,
|
|
60
|
+
};
|
|
61
|
+
}
|
|
62
|
+
/** ClickHouse >= 25.8 has the `vector_similarity` HNSW index. */
|
|
63
|
+
function supportsAnn(version) {
|
|
64
|
+
const { major, minor } = parseVersion(version);
|
|
65
|
+
return major > 25 || (major === 25 && minor >= 8);
|
|
66
|
+
}
|
|
67
|
+
/** Prompt for a line with a default value. */
|
|
68
|
+
async function promptLine(rl, question, fallback) {
|
|
69
|
+
const answer = (await rl.question(`${question}${fallback ? ` [${fallback}]` : ""}: `)).trim();
|
|
70
|
+
return answer.length > 0 ? answer : fallback;
|
|
71
|
+
}
|
|
72
|
+
/**
|
|
73
|
+
* Prompt for a password without echoing it.
|
|
74
|
+
*
|
|
75
|
+
* Readline's internal `_writeToOutput` hook is muted while the answer is read —
|
|
76
|
+
* an active readline interface otherwise echoes every keystroke, even when the
|
|
77
|
+
* caller switches the stream to raw mode.
|
|
78
|
+
*/
|
|
79
|
+
export async function promptHidden(rl, question) {
|
|
80
|
+
const mutable = rl;
|
|
81
|
+
const original = mutable._writeToOutput;
|
|
82
|
+
process.stdout.write(question);
|
|
83
|
+
if (typeof mutable._writeToOutput === "function") {
|
|
84
|
+
mutable._writeToOutput = () => { };
|
|
85
|
+
}
|
|
86
|
+
try {
|
|
87
|
+
return (await rl.question("")).trim();
|
|
88
|
+
}
|
|
89
|
+
finally {
|
|
90
|
+
mutable._writeToOutput = original;
|
|
91
|
+
process.stdout.write("\n");
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
/** Compose `scheme://host:port` from a user-supplied URL and port. */
|
|
95
|
+
export function composeClickHouseUrl(rawUrl, port) {
|
|
96
|
+
const parsed = new URL(rawUrl);
|
|
97
|
+
if (port.trim())
|
|
98
|
+
parsed.port = port.trim();
|
|
99
|
+
return parsed.toString().replace(/\/+$/, "");
|
|
100
|
+
}
|
|
101
|
+
/** Probe the server and return its version, or throw a readable error. */
|
|
102
|
+
async function probeServer(url, username, password) {
|
|
103
|
+
const { createClient } = await import("@clickhouse/client");
|
|
104
|
+
const client = createClient({
|
|
105
|
+
url,
|
|
106
|
+
username,
|
|
107
|
+
password,
|
|
108
|
+
database: "default",
|
|
109
|
+
request_timeout: 10_000,
|
|
110
|
+
});
|
|
111
|
+
try {
|
|
112
|
+
const result = await client.query({ query: "SELECT version() AS v", format: "JSONEachRow" });
|
|
113
|
+
const rows = await result.json();
|
|
114
|
+
return rows[0]?.v ?? "unknown";
|
|
115
|
+
}
|
|
116
|
+
finally {
|
|
117
|
+
await client.close().catch(() => { });
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
/** Create the database and project tables (and the ANN index when requested). */
|
|
121
|
+
async function initializeStore(input, cwd) {
|
|
122
|
+
const cfg = loadWorkspaceConfig(cwd);
|
|
123
|
+
cfg.vectorStore.provider = "clickhouse";
|
|
124
|
+
cfg.vectorStore.clickhouse = {
|
|
125
|
+
...(cfg.vectorStore.clickhouse ?? {}),
|
|
126
|
+
url: input.url,
|
|
127
|
+
database: input.database,
|
|
128
|
+
username: input.username,
|
|
129
|
+
password: input.password,
|
|
130
|
+
passwordEnv: "",
|
|
131
|
+
namespace: input.namespace,
|
|
132
|
+
ann: input.ann,
|
|
133
|
+
annFetchMultiplier: 4,
|
|
134
|
+
annWithRescoring: true,
|
|
135
|
+
};
|
|
136
|
+
const options = resolveClickHouseOptions(cfg, cwd);
|
|
137
|
+
const store = new ClickHouseStore(options, input.dimension);
|
|
138
|
+
try {
|
|
139
|
+
const dimension = await store.getVectorDimension();
|
|
140
|
+
const ann = await store.hasVectorIndex();
|
|
141
|
+
const count = await store.count();
|
|
142
|
+
console.log(`\n ${c.success("Initialized:")} ${input.database}.${options.chunksTable}`);
|
|
143
|
+
console.log(` ${c.label("State table:")} ${input.database}.${options.stateTable}`);
|
|
144
|
+
console.log(` ${c.label("Dimension:")} ${dimension ?? input.dimension}`);
|
|
145
|
+
console.log(` ${c.label("ANN:")} ${ann ? "hnsw (vector_similarity)" : input.ann === "hnsw" ? "missing" : "off (exact scans)"}`);
|
|
146
|
+
console.log(` ${c.label("Chunks:")} ${count}`);
|
|
147
|
+
}
|
|
148
|
+
finally {
|
|
149
|
+
await store.close().catch(() => { });
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
/** Run the interactive (or flag-driven) ClickHouse setup. */
|
|
153
|
+
export async function runClickHouseSetup(options = {}) {
|
|
154
|
+
const cwd = options.cwd ?? process.cwd();
|
|
155
|
+
const interactive = Boolean(process.stdin.isTTY) && !options.yes;
|
|
156
|
+
const rl = interactive ? createInterface({ input: process.stdin, output: process.stdout }) : undefined;
|
|
157
|
+
try {
|
|
158
|
+
const cfg = loadWorkspaceConfig(cwd);
|
|
159
|
+
const defaultDimension = cfg.embedding.vectorDimension && cfg.embedding.vectorDimension > 0
|
|
160
|
+
? String(cfg.embedding.vectorDimension)
|
|
161
|
+
: "384";
|
|
162
|
+
const scopeDefault = options.scope === "global" || options.scope === "workspace" ? options.scope : "workspace";
|
|
163
|
+
const rawUrl = interactive
|
|
164
|
+
? await promptLine(rl, "ClickHouse URL", options.url ?? "http://localhost:8123")
|
|
165
|
+
: options.url ?? "http://localhost:8123";
|
|
166
|
+
const defaultPort = (() => {
|
|
167
|
+
try {
|
|
168
|
+
return new URL(rawUrl).port || "8123";
|
|
169
|
+
}
|
|
170
|
+
catch {
|
|
171
|
+
return "8123";
|
|
172
|
+
}
|
|
173
|
+
})();
|
|
174
|
+
const port = interactive ? await promptLine(rl, "Port", options.port ?? defaultPort) : options.port ?? defaultPort;
|
|
175
|
+
let url;
|
|
176
|
+
try {
|
|
177
|
+
url = composeClickHouseUrl(rawUrl, port);
|
|
178
|
+
}
|
|
179
|
+
catch {
|
|
180
|
+
throw new Error(`Invalid ClickHouse URL: "${rawUrl}"`);
|
|
181
|
+
}
|
|
182
|
+
const database = interactive
|
|
183
|
+
? await promptLine(rl, "Database", options.database ?? cfg.vectorStore.clickhouse?.database ?? "OpenCodeRAG")
|
|
184
|
+
: options.database ?? cfg.vectorStore.clickhouse?.database ?? "OpenCodeRAG";
|
|
185
|
+
const username = interactive
|
|
186
|
+
? await promptLine(rl, "Username", options.username ?? cfg.vectorStore.clickhouse?.username ?? "default")
|
|
187
|
+
: options.username ?? cfg.vectorStore.clickhouse?.username ?? "default";
|
|
188
|
+
let password = options.password ?? "";
|
|
189
|
+
if (!password && interactive)
|
|
190
|
+
password = await promptHidden(rl, "Password: ");
|
|
191
|
+
if (!password) {
|
|
192
|
+
const envName = options.passwordEnv ?? cfg.vectorStore.clickhouse?.passwordEnv ?? "OPENCODERAG_CLICKHOUSE_PASSWORD";
|
|
193
|
+
password = process.env[envName] ?? "";
|
|
194
|
+
if (!password) {
|
|
195
|
+
throw new Error("A ClickHouse password is required — pass --password, set the password env var, or run the command interactively.");
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
// 1. Probe the server before writing anything.
|
|
199
|
+
let version;
|
|
200
|
+
try {
|
|
201
|
+
version = await probeServer(url, username, password);
|
|
202
|
+
}
|
|
203
|
+
catch (err) {
|
|
204
|
+
throw new Error(`Could not connect/authenticate to ${url}: ${err.message}` +
|
|
205
|
+
(process.env.HTTP_PROXY || process.env.HTTPS_PROXY
|
|
206
|
+
? "\nIf an internal hostname is involved, make sure NO_PROXY includes it."
|
|
207
|
+
: ""));
|
|
208
|
+
}
|
|
209
|
+
const annSupported = supportsAnn(version);
|
|
210
|
+
console.log(`\n ${c.success("Connected:")} ClickHouse ${version}${annSupported ? " (ANN available)" : " (no ANN, needs 25.8+)"}\n`);
|
|
211
|
+
const scope = interactive
|
|
212
|
+
? (await promptLine(rl, "Write the config to workspace or global", scopeDefault)).toLowerCase() === "global"
|
|
213
|
+
? "global"
|
|
214
|
+
: "workspace"
|
|
215
|
+
: scopeDefault;
|
|
216
|
+
const derived = deriveNamespace(cwd);
|
|
217
|
+
const namespaceDefault = options.namespace ?? (scope === "global" ? "" : cfg.vectorStore.clickhouse?.namespace ?? derived);
|
|
218
|
+
const namespace = interactive
|
|
219
|
+
? await promptLine(rl, scope === "global" ? "Namespace (blank = per-workspace tables)" : "Namespace", namespaceDefault)
|
|
220
|
+
: namespaceDefault;
|
|
221
|
+
const dimension = interactive
|
|
222
|
+
? Number.parseInt(await promptLine(rl, "Embedding dimension (ANN index is bound to it)", options.dimension ?? defaultDimension), 10)
|
|
223
|
+
: Number.parseInt(options.dimension ?? defaultDimension, 10);
|
|
224
|
+
if (!Number.isFinite(dimension) || dimension < 2 || dimension > 16000) {
|
|
225
|
+
throw new Error(`Invalid embedding dimension: ${options.dimension ?? defaultDimension}`);
|
|
226
|
+
}
|
|
227
|
+
const annDefault = options.ann ?? (annSupported ? "hnsw" : "off");
|
|
228
|
+
let ann;
|
|
229
|
+
if (interactive) {
|
|
230
|
+
const answer = await promptLine(rl, "Enable the HNSW ANN index (hnsw/off)", annDefault);
|
|
231
|
+
ann = answer.toLowerCase() === "off" ? "off" : "hnsw";
|
|
232
|
+
}
|
|
233
|
+
else {
|
|
234
|
+
ann = annDefault === "hnsw" ? "hnsw" : "off";
|
|
235
|
+
}
|
|
236
|
+
if (ann === "hnsw" && !annSupported) {
|
|
237
|
+
throw new Error(`ANN requires ClickHouse >= 25.8 — ${url} runs ${version}. Use --ann off.`);
|
|
238
|
+
}
|
|
239
|
+
const storeEnvDefault = options.passwordEnv ?? cfg.vectorStore.clickhouse?.passwordEnv ?? "OPENCODERAG_CLICKHOUSE_PASSWORD";
|
|
240
|
+
let storeInlinePassword = options.storePassword ?? false;
|
|
241
|
+
let passwordEnv = storeEnvDefault;
|
|
242
|
+
if (interactive) {
|
|
243
|
+
storeInlinePassword = /^y/i.test(await promptLine(rl, "Store the password in the config file (plaintext)", "n"));
|
|
244
|
+
if (!storeInlinePassword) {
|
|
245
|
+
passwordEnv = await promptLine(rl, "Password env var name", storeEnvDefault);
|
|
246
|
+
}
|
|
247
|
+
}
|
|
248
|
+
if (!storeInlinePassword && !passwordEnv) {
|
|
249
|
+
throw new Error("Set a password env var name or use --store-password.");
|
|
250
|
+
}
|
|
251
|
+
// 2. Initialize the database/tables before writing the config.
|
|
252
|
+
const setupInput = {
|
|
253
|
+
url,
|
|
254
|
+
database,
|
|
255
|
+
username,
|
|
256
|
+
password,
|
|
257
|
+
passwordEnv,
|
|
258
|
+
namespace: namespace ? sanitizeNamespace(namespace) : "",
|
|
259
|
+
ann,
|
|
260
|
+
dimension,
|
|
261
|
+
storeInlinePassword,
|
|
262
|
+
};
|
|
263
|
+
if (!options.skipInit) {
|
|
264
|
+
await initializeStore(setupInput, cwd);
|
|
265
|
+
}
|
|
266
|
+
// 3. Write the config (workspace or the machine-global user config).
|
|
267
|
+
const targetPath = scope === "global"
|
|
268
|
+
? globalConfigPath()
|
|
269
|
+
: findOrCreateWorkspaceConfig(cwd);
|
|
270
|
+
const existing = readJsonObject(targetPath);
|
|
271
|
+
const next = mergeClickHouseConfig(existing, setupInput);
|
|
272
|
+
if (interactive) {
|
|
273
|
+
const write = await promptLine(rl, `Write this vectorStore block to ${targetPath}`, "y");
|
|
274
|
+
if (!/^y/i.test(write)) {
|
|
275
|
+
console.log(" (config not written)");
|
|
276
|
+
return;
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
if (existsSync(targetPath)) {
|
|
280
|
+
copyFileSync(targetPath, `${targetPath}.bak`);
|
|
281
|
+
}
|
|
282
|
+
mkdirSync(path.dirname(targetPath), { recursive: true });
|
|
283
|
+
writeJsonFile(targetPath, next);
|
|
284
|
+
console.log(`\n ${c.success("Config written:")} ${targetPath}`);
|
|
285
|
+
console.log(` ${c.label("Backend:")} clickhouse ${url}/${database}`);
|
|
286
|
+
console.log(` ${c.label("Namespace:")} ${setupInput.namespace || "(derived per workspace)"}`);
|
|
287
|
+
console.log(` ${c.label("ANN:")} ${ann}`);
|
|
288
|
+
if (!storeInlinePassword) {
|
|
289
|
+
const setCommand = process.platform === "win32"
|
|
290
|
+
? `[Environment]::SetEnvironmentVariable("${passwordEnv}", "<password>", "User")`
|
|
291
|
+
: `export ${passwordEnv}="<password>" # add to ~/.bashrc or ~/.zshrc`;
|
|
292
|
+
console.log(`\n ${c.warn("Set the password before using it:")}\n ${setCommand}`);
|
|
293
|
+
}
|
|
294
|
+
console.log(`\n Next: ${c.value("opencode-rag index")} (then ${c.value("opencode-rag status")})\n`);
|
|
295
|
+
}
|
|
296
|
+
finally {
|
|
297
|
+
rl?.close();
|
|
298
|
+
}
|
|
299
|
+
}
|
|
300
|
+
/** Use an existing workspace config file, or the canonical `.opencode/opencode-rag.json` for a new one. */
|
|
301
|
+
function findOrCreateWorkspaceConfig(cwd) {
|
|
302
|
+
return findConfigFile(cwd) ?? projectConfigPath(cwd);
|
|
303
|
+
}
|
|
304
|
+
/** Register the `clickhouse` command group. */
|
|
305
|
+
export function registerClickHouseCommand(program) {
|
|
306
|
+
const group = program
|
|
307
|
+
.command("clickhouse")
|
|
308
|
+
.description("Manage the ClickHouse vector store");
|
|
309
|
+
group
|
|
310
|
+
.command("setup")
|
|
311
|
+
.description("Interactively configure and initialize a ClickHouse vector store")
|
|
312
|
+
.option("--url <url>", "ClickHouse endpoint (e.g. http://localhost:8123)")
|
|
313
|
+
.option("--port <port>", "ClickHouse HTTP port (default 8123)")
|
|
314
|
+
.option("--database <name>", "database name (default OpenCodeRAG)")
|
|
315
|
+
.option("--username <name>", "ClickHouse user (default default)")
|
|
316
|
+
.option("--password <password>", "password (prefer the prompt or an env var)")
|
|
317
|
+
.option("--password-env <name>", "env var holding the password")
|
|
318
|
+
.option("--namespace <name>", "project namespace / table suffix (blank = per-workspace)")
|
|
319
|
+
.option("--ann <mode>", "ANN mode: hnsw or off")
|
|
320
|
+
.option("--dimension <n>", "embedding dimension the ANN index is bound to")
|
|
321
|
+
.option("--scope <scope>", "write the config to workspace or global")
|
|
322
|
+
.option("--store-password", "write the password into the config file (plaintext)")
|
|
323
|
+
.option("--skip-init", "only write the config; do not create the database/tables")
|
|
324
|
+
.option("--yes", "non-interactive: use the flags/defaults without prompting")
|
|
325
|
+
.action(async (options) => {
|
|
326
|
+
try {
|
|
327
|
+
await runClickHouseSetup(options);
|
|
328
|
+
}
|
|
329
|
+
catch (err) {
|
|
330
|
+
console.error(`\n${c.error("ClickHouse setup failed:")} ${err.message}\n`);
|
|
331
|
+
process.exitCode = 1;
|
|
332
|
+
}
|
|
333
|
+
});
|
|
334
|
+
}
|
|
335
|
+
//# sourceMappingURL=clickhouse.js.map
|
|
@@ -189,7 +189,7 @@ export function registerEvalTuneCommand(program) {
|
|
|
189
189
|
let written = false;
|
|
190
190
|
if (options.write) {
|
|
191
191
|
if (!ctx.configPath) {
|
|
192
|
-
throw new Error("No config file to write to — pass --config <path> or run this in a workspace with opencode-rag.json.");
|
|
192
|
+
throw new Error("No config file to write to — pass --config <path> or run this in a workspace with .opencode/opencode-rag.json.");
|
|
193
193
|
}
|
|
194
194
|
const tuned = applyTuningToConfig(ctx.config, best);
|
|
195
195
|
const hybridWeights = tuned.retrieval.hybridSearch ?? { enabled: true, keywordWeight: best.keywordWeight };
|
|
@@ -377,7 +377,7 @@ export function registerEvalDecideCommand(program) {
|
|
|
377
377
|
};
|
|
378
378
|
}
|
|
379
379
|
if (!decisionCfg) {
|
|
380
|
-
console.error(c.error("\nNo enabled decision config. Set `decision.enabled` in opencode-rag.json or pass --base-url/--model.\n"));
|
|
380
|
+
console.error(c.error("\nNo enabled decision config. Set `decision.enabled` in .opencode/opencode-rag.json or pass --base-url/--model.\n"));
|
|
381
381
|
await cleanupContext(ctx);
|
|
382
382
|
process.exit(2);
|
|
383
383
|
}
|
|
@@ -59,7 +59,7 @@ export function registerIndexCommand(program) {
|
|
|
59
59
|
const cwd = process.cwd();
|
|
60
60
|
let logFilePath = path.resolve(cwd, ".opencode", "opencode-rag.log");
|
|
61
61
|
const ctx = await resolveCliContext(options, logFilePath);
|
|
62
|
-
const { config, embedder, store, storePath, keywordIndex, descriptionProvider, dimension } = ctx;
|
|
62
|
+
const { config, embedder, store, storePath, keywordIndex, descriptionProvider, dimension, stateSync } = ctx;
|
|
63
63
|
logFilePath = ctx.logFilePath;
|
|
64
64
|
// ── Abort controller for graceful Ctrl+C during the initial pass ──
|
|
65
65
|
const abortController = new AbortController();
|
|
@@ -111,6 +111,7 @@ export function registerIndexCommand(program) {
|
|
|
111
111
|
embedder,
|
|
112
112
|
keywordIndex,
|
|
113
113
|
descriptionProvider,
|
|
114
|
+
stateSync,
|
|
114
115
|
force: !!(options.force && !watchTriggered),
|
|
115
116
|
abortSignal,
|
|
116
117
|
dimension,
|
|
@@ -9,6 +9,7 @@ export { registerListCommand } from "./list.js";
|
|
|
9
9
|
export { registerShowCommand } from "./show.js";
|
|
10
10
|
export { registerDumpCommand } from "./dump.js";
|
|
11
11
|
export { registerInitCommand } from "./init.js";
|
|
12
|
+
export { registerClickHouseCommand } from "./clickhouse.js";
|
|
12
13
|
export { registerUiCommand } from "./ui.js";
|
|
13
14
|
export { registerMcpCommand } from "./mcp.js";
|
|
14
15
|
export { registerSetupCommand } from "./setup.js";
|
|
@@ -9,6 +9,7 @@ export { registerListCommand } from "./list.js";
|
|
|
9
9
|
export { registerShowCommand } from "./show.js";
|
|
10
10
|
export { registerDumpCommand } from "./dump.js";
|
|
11
11
|
export { registerInitCommand } from "./init.js";
|
|
12
|
+
export { registerClickHouseCommand } from "./clickhouse.js";
|
|
12
13
|
export { registerUiCommand } from "./ui.js";
|
|
13
14
|
export { registerMcpCommand } from "./mcp.js";
|
|
14
15
|
export { registerSetupCommand } from "./setup.js";
|
|
@@ -283,7 +283,7 @@ export function generateSkillFile() {
|
|
|
283
283
|
"- `find_usages` is essential before refactoring — it shows every reference",
|
|
284
284
|
"- Pass `systemPrompt` to `describe_image` when you need specific details (e.g. `\"focus on the chart's axes and values\"`)",
|
|
285
285
|
"- If no results appear, the workspace may not be indexed yet — run `opencode-rag index`",
|
|
286
|
-
"- Image descriptions are generated at index time using the configured vision provider; ensure `imageDescription` is configured in
|
|
286
|
+
"- Image descriptions are generated at index time using the configured vision provider; ensure `imageDescription` is configured in `.opencode/opencode-rag.json` (its provider/model options live in the global config) if your project includes images",
|
|
287
287
|
"",
|
|
288
288
|
].join("\n");
|
|
289
289
|
}
|
|
@@ -308,7 +308,7 @@ export function generateDecisionSkillFile() {
|
|
|
308
308
|
"",
|
|
309
309
|
"### Prerequisites",
|
|
310
310
|
"",
|
|
311
|
-
"- `decision.enabled` must be `true` in
|
|
311
|
+
"- `decision.enabled` must be `true` in `.opencode/opencode-rag.json` (off by default).",
|
|
312
312
|
"- Requires Ollama >= 0.35 with a pulled tev1 model (`ollama pull tev1` or `ollama pull tev1:0.8b`).",
|
|
313
313
|
"- Keep `state` short — the effective context is ~2k tokens. Pass the relevant excerpt, not a whole file.",
|
|
314
314
|
"",
|