opencode-rag-plugin 2.5.3 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ReadMe.md +10 -4
- package/dist/cli/commands/eval-tuner.d.ts +15 -0
- package/dist/cli/commands/eval-tuner.js +216 -0
- package/dist/cli/commands/index.d.ts +1 -0
- package/dist/cli/commands/index.js +1 -0
- package/dist/cli/commands/status.js +20 -0
- package/dist/cli/index.js +2 -1
- package/dist/core/config.d.ts +27 -0
- package/dist/core/config.js +20 -0
- package/dist/core/interfaces.d.ts +8 -0
- package/dist/eval/tuner.d.ts +118 -0
- package/dist/eval/tuner.js +158 -0
- package/dist/indexer/keyword-rehydration.d.ts +48 -0
- package/dist/indexer/keyword-rehydration.js +113 -0
- package/dist/indexer/pipeline.js +27 -0
- package/dist/plugin.js +124 -15
- package/dist/quirks/quirk-store.d.ts +10 -0
- package/dist/quirks/quirk-store.js +9 -1
- package/dist/quirks/tool-error-create.d.ts +111 -0
- package/dist/quirks/tool-error-create.js +173 -0
- package/dist/quirks/tool-error-recall.d.ts +77 -2
- package/dist/quirks/tool-error-recall.js +58 -22
- package/dist/retriever/keyword-index.d.ts +10 -0
- package/dist/retriever/keyword-index.js +12 -0
- package/package.json +1 -1
package/ReadMe.md
CHANGED
|
@@ -47,19 +47,20 @@ workspace, and `opencode-rag update` upgrades to a newer release.
|
|
|
47
47
|
| **Configurable parsers** | Remap file extensions to any parser via `chunking.parsers` (e.g. `.c` → `cpp`, or index a new `.cu` as C++) — applied to chunking, `get_file_skeleton`, and read labels ([docs](doc/chunking.md#parser-overrides-per-extension-chunkingparsers)) |
|
|
48
48
|
| **Document support** | Markdown, LaTeX, PDF, DOCX, DOC, Excel |
|
|
49
49
|
| **Image indexing** | Describe images via vision LLM and store descriptions as searchable vector chunks |
|
|
50
|
-
| **Hybrid search** | Vector similarity + TF×IDF keyword fusion |
|
|
50
|
+
| **Hybrid search** | Vector similarity + TF×IDF keyword fusion — the keyword index is repaired from the store whenever it drifts behind it, so the BM25 half never covers only part of the corpus |
|
|
51
|
+
| **Retrieval calibration** | `opencode-rag eval:tune` sweeps the fusion weights against the labelled golden set and reports (or writes) the best configuration for your embedder and corpus |
|
|
51
52
|
| **Decision model** | Local [tev1](https://ollama.com/library/tev1) classification — `make_decision` labels, routes, and scores short text with probabilities (40–150 ms warm); optional pre-ask routing so the agent asks fewer questions |
|
|
52
53
|
| **OpenCode plugin** | Auto-inject context, read-tool override, TUI settings, Ctrl+Enter to add RAG context, MCP registration on `init` |
|
|
53
54
|
| **Incremental indexing** | File-hash manifest, background watcher, auto-rebuild on corruption |
|
|
54
55
|
| **Privacy-first** | All processing stays local (when using Ollama) |
|
|
55
|
-
| **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `decide`, `ui`, `mcp`, `setup`, `quirk`, `eval:sessions`, `eval:analyze`, `eval:compare`, `eval:gate`, `eval:decide` |
|
|
56
|
+
| **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `decide`, `ui`, `mcp`, `setup`, `quirk`, `eval:sessions`, `eval:analyze`, `eval:compare`, `eval:gate`, `eval:decide`, `eval:tune` |
|
|
56
57
|
| **Proxy-aware** | Corporate proxy support with raw-socket localhost bypass |
|
|
57
58
|
| **OpenAI / Anthropic / Cohere** | Use alternate embedding providers with API key auto-resolution |
|
|
58
59
|
| **Evaluation** | Session-level token tracking, RAG-on vs RAG-off comparison, tiktoken BPE counting |
|
|
59
60
|
| **Documentation mode** | `/doc` slash command: agent adds JSDoc/TSDoc to undocumented files, progress tracked per subdirectory |
|
|
60
61
|
| **Wiki mode** | `/wiki` slash command: agent maintains a persistent knowledge wiki at `.opencode/wiki/` (ingest, query, lint, seed) |
|
|
61
62
|
| **Context optimization** | Post-retrieval dedup, per-file chunk limits, adjacent-merge to fit the context window |
|
|
62
|
-
| **Quirk memory** | Persistent experiential memory — agents recall/persist gotchas, preferences, decisions across sessions (`add_quirk` / `recall_quirks`, `opencode-rag quirk`) |
|
|
63
|
+
| **Quirk memory** | Persistent experiential memory — agents recall/persist gotchas, preferences, decisions across sessions (`add_quirk` / `recall_quirks`, `opencode-rag quirk`); failed tool calls are auto-recorded as quirks |
|
|
63
64
|
| **AGENTS.md directive** | `opencode-rag init` merges a tool-usage directive into `AGENTS.md` via sentinel markers |
|
|
64
65
|
|
|
65
66
|
## Web UI
|
|
@@ -210,6 +211,9 @@ OpenCodeRAG gives your agent **persistent, cross-session memory** of non-obvious
|
|
|
210
211
|
"autoInjectMinScore": 0.75,
|
|
211
212
|
"autoInjectTopK": 1,
|
|
212
213
|
"autoInjectLatencyBudgetMs": 2000,
|
|
214
|
+
"injectOnToolError": true,
|
|
215
|
+
"autoCreateOnToolError": true,
|
|
216
|
+
"toolErrorCreateMaxPerSession": 2,
|
|
213
217
|
"decay": { "enabled": false, "halfLifeDays": 30 }
|
|
214
218
|
}
|
|
215
219
|
}
|
|
@@ -240,6 +244,8 @@ opencode-rag quirk test "npm needs --legacy-peer-deps"
|
|
|
240
244
|
|
|
241
245
|
When `memory.autoInject` is `true`, the plugin checks for relevant quirks on every user message using the combined agent-response + user-query as the search query. Quirks are only injected when their raw vector similarity to that query clears `autoInjectMinScore` (default 0.75) - the same threshold for the user-message and system-prompt injections; `recallMinScore` (default 0.72) is the bar for manual `recall_quirks` calls. A latency budget (`autoInjectLatencyBudgetMs`, default 2000ms) prevents slow embedders from blocking message processing. To avoid polluting the context window, each quirk is injected **at most once per session** — once recalled, it is filtered out from all subsequent auto-injections. Every `add_quirk` and every content-changing `update_quirk` is vetted by an immutable trust monitor that rejects destructive patterns (e.g. `rm -rf`, `force push`, `bypass security`). Outdated or fixed quirks should be corrected with `update_quirk` / `delete_quirk` rather than left to contradict newer memory. See [Plugin documentation](doc/plugin.md#9-quirk-memory-experiential-memory) and [CLI Reference: `quirk`](doc/cli.md#quirk).
|
|
242
246
|
|
|
247
|
+
**Failed tool calls become memory.** When a tool call fails, the plugin looks for a stored quirk that covers the failure and injects it into the next request (`memory.injectOnToolError`, default on). If **no** quirk covers it, it writes one — `memory.autoCreateOnToolError` (default on) — using a deterministic template built from the tool name and the error text, so no LLM is involved. The new quirk is tagged `tool-error` and recorded for *future* sessions; it is deliberately excluded from the generic auto-injection above, so a raw error dump never leaks into an unrelated turn. Creation is bounded by a per-session budget (2) and a per-error cooldown (10 min), and the quirk it would write is suppressed as soon as a similar one already exists.
|
|
248
|
+
|
|
243
249
|
## MCP Server (Optional)
|
|
244
250
|
|
|
245
251
|
OpenCodeRAG ships a CLI-based [MCP (Model Context Protocol)](https://spec.modelcontextprotocol.io/) server that exposes semantic code tools to any MCP-compatible client (Claude Desktop, Cursor, etc.).
|
|
@@ -312,7 +318,7 @@ opencode-rag eval:compare <A> <B> # side-by-side comparison
|
|
|
312
318
|
|
|
313
319
|
Token counting uses tiktoken BPE (cl100k_base) for accurate code tokenization. See [Evaluation documentation](doc/evaluation.md) for details.
|
|
314
320
|
|
|
315
|
-
Retrieval quality has its own harness: `opencode-rag eval:gate` runs the labelled golden set and fails when Hit@5 / MRR / nDCG@10 drop below the documented floors, and `opencode-rag eval:decide` measures tev1's accuracy, ECE, and Brier score on a labelled calibration set.
|
|
321
|
+
Retrieval quality has its own harness: `opencode-rag eval:gate` runs the labelled golden set and fails when Hit@5 / MRR / nDCG@10 drop below the documented floors, and `opencode-rag eval:decide` measures tev1's accuracy, ECE, and Brier score on a labelled calibration set. `opencode-rag eval:tune` sweeps the retrieval fusion weights against the same golden set and reports the best configuration for your setup — the shipped `keywordWeight=0.4` is a default, not an optimum (on a code embedder it costs ~8 Hit@5 points), so measure it instead of guessing.
|
|
316
322
|
|
|
317
323
|
---
|
|
318
324
|
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @fileoverview `eval:tune` command — sweeps the retrieval fusion weights
|
|
3
|
+
* against the labelled golden set and reports the best configuration.
|
|
4
|
+
*/
|
|
5
|
+
import type { Command } from "commander";
|
|
6
|
+
/**
|
|
7
|
+
* Register the `eval:tune` command on the given Commander program.
|
|
8
|
+
*
|
|
9
|
+
* Measures the current retrieval configuration against the golden set, sweeps a
|
|
10
|
+
* grid of fusion weights, and reports the winner. Exits `0` when the sweep ran
|
|
11
|
+
* (even when the current configuration already wins), `2` when it could not run.
|
|
12
|
+
*
|
|
13
|
+
* @param program - The Commander `Command` instance to register on.
|
|
14
|
+
*/
|
|
15
|
+
export declare function registerEvalTuneCommand(program: Command): void;
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
import path from "node:path";
|
|
2
|
+
import { c, resolveCliContext, cleanupContext, logCliError } from "../format.js";
|
|
3
|
+
/** Parse a comma-separated list of numbers. */
|
|
4
|
+
function parseNumberList(raw, option) {
|
|
5
|
+
if (raw === undefined)
|
|
6
|
+
return undefined;
|
|
7
|
+
const parts = raw
|
|
8
|
+
.split(",")
|
|
9
|
+
.map((p) => p.trim())
|
|
10
|
+
.filter((p) => p.length > 0);
|
|
11
|
+
if (parts.length === 0) {
|
|
12
|
+
throw new Error(`${option} needs at least one number (e.g. "${option} 0,0.2,0.4")`);
|
|
13
|
+
}
|
|
14
|
+
const values = parts.map((p) => Number(p));
|
|
15
|
+
for (const part of parts) {
|
|
16
|
+
const value = Number(part);
|
|
17
|
+
if (!Number.isFinite(value)) {
|
|
18
|
+
throw new Error(`${option} contains a non-numeric value: "${part}"`);
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
return values;
|
|
22
|
+
}
|
|
23
|
+
/**
|
|
24
|
+
* Register the `eval:tune` command on the given Commander program.
|
|
25
|
+
*
|
|
26
|
+
* Measures the current retrieval configuration against the golden set, sweeps a
|
|
27
|
+
* grid of fusion weights, and reports the winner. Exits `0` when the sweep ran
|
|
28
|
+
* (even when the current configuration already wins), `2` when it could not run.
|
|
29
|
+
*
|
|
30
|
+
* @param program - The Commander `Command` instance to register on.
|
|
31
|
+
*/
|
|
32
|
+
export function registerEvalTuneCommand(program) {
|
|
33
|
+
program
|
|
34
|
+
.command("eval:tune")
|
|
35
|
+
.description("Sweep the retrieval fusion weights against the golden set and report the best configuration")
|
|
36
|
+
.option("-c, --config <path>", "path to config file")
|
|
37
|
+
.option("--labels <path>", "golden-set label file (default: src/eval/rerank-labels.json)")
|
|
38
|
+
.option("--category <name>", "only run labels of this category")
|
|
39
|
+
.option("--limit <n>", "only run the first N labels")
|
|
40
|
+
.option("--metric <name>", "metric the winner is picked by (ndcg10 | hit5 | hit1 | mrr, default ndcg10)")
|
|
41
|
+
.option("--keyword-weights <list>", "comma-separated keyword weights to try (default 0,0.1,...,0.6)")
|
|
42
|
+
.option("--symbol-weights <list>", "comma-separated symbol keyword weights to try (default: same as keyword weights)")
|
|
43
|
+
.option("--min-scores <list>", "comma-separated minScore values to try (default: the configured one)")
|
|
44
|
+
.option("--topks <list>", "comma-separated topK values to try (default: the configured one)")
|
|
45
|
+
.option("--write", "write the winning values into the config file")
|
|
46
|
+
.option("--json", "print a machine-readable result as the last stdout line")
|
|
47
|
+
.action(async (options) => {
|
|
48
|
+
const cwd = process.cwd();
|
|
49
|
+
const logFilePath = path.resolve(cwd, ".opencode", "opencode-rag.log");
|
|
50
|
+
let ctx;
|
|
51
|
+
let exitCode = 0;
|
|
52
|
+
try {
|
|
53
|
+
const { loadGoldenLabels } = await import("../../eval/golden-set.js");
|
|
54
|
+
const { applyTuningToConfig, buildCandidateGrid, runTuner, selectBestTuning, tuningDelta, } = await import("../../eval/tuner.js");
|
|
55
|
+
const { updateConfigValue } = await import("../../core/config.js");
|
|
56
|
+
const { retrieve } = await import("../../retriever/retriever.js");
|
|
57
|
+
const labelsPath = options.labels
|
|
58
|
+
? path.resolve(cwd, options.labels)
|
|
59
|
+
: path.join(cwd, "src", "eval", "rerank-labels.json");
|
|
60
|
+
let labels = loadGoldenLabels(labelsPath);
|
|
61
|
+
if (options.category)
|
|
62
|
+
labels = labels.filter((label) => label.category === options.category);
|
|
63
|
+
if (options.limit !== undefined) {
|
|
64
|
+
const limit = Number(options.limit);
|
|
65
|
+
if (!Number.isInteger(limit) || limit <= 0) {
|
|
66
|
+
throw new Error(`--limit must be a positive integer (got "${options.limit}")`);
|
|
67
|
+
}
|
|
68
|
+
labels = labels.slice(0, limit);
|
|
69
|
+
}
|
|
70
|
+
if (labels.length === 0) {
|
|
71
|
+
throw new Error("No labels selected (check --category/--limit).");
|
|
72
|
+
}
|
|
73
|
+
const metric = (options.metric ?? "ndcg10");
|
|
74
|
+
if (!["ndcg10", "hit5", "hit1", "mrr"].includes(metric)) {
|
|
75
|
+
throw new Error(`--metric must be one of ndcg10, hit5, hit1, mrr (got "${options.metric}")`);
|
|
76
|
+
}
|
|
77
|
+
ctx = await resolveCliContext(options, logFilePath);
|
|
78
|
+
const indexedCount = await ctx.store.count();
|
|
79
|
+
if (indexedCount === 0) {
|
|
80
|
+
console.error(c.error(`\nTuning cannot run: the index is empty. Run 'opencode-rag index' first.\n`));
|
|
81
|
+
await cleanupContext(ctx);
|
|
82
|
+
process.exit(2);
|
|
83
|
+
}
|
|
84
|
+
const hybrid = ctx.config.retrieval.hybridSearch;
|
|
85
|
+
const base = {
|
|
86
|
+
keywordWeight: hybrid?.keywordWeight ?? 0.4,
|
|
87
|
+
symbolKeywordWeight: hybrid?.symbolKeywordWeight,
|
|
88
|
+
minScore: ctx.config.retrieval.minScore,
|
|
89
|
+
topK: ctx.config.retrieval.topK,
|
|
90
|
+
};
|
|
91
|
+
const candidates = buildCandidateGrid(base, {
|
|
92
|
+
keywordWeights: parseNumberList(options.keywordWeights, "--keyword-weights"),
|
|
93
|
+
symbolKeywordWeights: parseNumberList(options.symbolWeights, "--symbol-weights"),
|
|
94
|
+
minScores: parseNumberList(options.minScores, "--min-scores"),
|
|
95
|
+
topKs: parseNumberList(options.topks, "--topks"),
|
|
96
|
+
});
|
|
97
|
+
console.log(`\n${c.heading("Retrieval Tuning")}\n`);
|
|
98
|
+
console.log(` Indexed chunks: ${c.num(indexedCount)}`);
|
|
99
|
+
console.log(` Embedder: ${c.value(`${ctx.config.embedding.provider}/${ctx.config.embedding.model}`)} (dim ${c.num(ctx.dimension)})`);
|
|
100
|
+
console.log(` Labels: ${c.num(labels.length)} from ${c.file(path.relative(cwd, labelsPath).replace(/\\/g, "/"))}`);
|
|
101
|
+
console.log(` Current config: keywordWeight=${c.num(base.keywordWeight)} symbolKeywordWeight=${c.num(base.symbolKeywordWeight ?? "unset")} minScore=${c.num(base.minScore)} topK=${c.num(base.topK)}`);
|
|
102
|
+
console.log(` Candidates: ${c.num(candidates.length)} · winner picked by ${c.value(metric)}\n`);
|
|
103
|
+
const results = await runTuner({
|
|
104
|
+
labels,
|
|
105
|
+
root: cwd,
|
|
106
|
+
candidates,
|
|
107
|
+
retrieve: (label, candidate) => retrieve(label.query, ctx.embedder, ctx.store, {
|
|
108
|
+
topK: candidate.topK,
|
|
109
|
+
minScore: candidate.minScore,
|
|
110
|
+
keywordIndex: ctx.keywordIndex,
|
|
111
|
+
keywordWeight: candidate.keywordWeight,
|
|
112
|
+
symbolKeywordWeight: candidate.symbolKeywordWeight,
|
|
113
|
+
docKeywordDemotion: hybrid?.docKeywordDemotion,
|
|
114
|
+
testKeywordDemotion: hybrid?.testKeywordDemotion,
|
|
115
|
+
hybridEnabled: hybrid?.enabled,
|
|
116
|
+
queryPrefix: ctx.config.embedding.queryPrefix,
|
|
117
|
+
}),
|
|
118
|
+
log: (line) => console.log(line),
|
|
119
|
+
});
|
|
120
|
+
const best = selectBestTuning(results, metric);
|
|
121
|
+
if (!best) {
|
|
122
|
+
throw new Error("The sweep produced no results.");
|
|
123
|
+
}
|
|
124
|
+
const current = results.find((r) => r.keywordWeight === base.keywordWeight &&
|
|
125
|
+
(r.symbolKeywordWeight ?? "unset") === (base.symbolKeywordWeight ?? "unset") &&
|
|
126
|
+
r.minScore === base.minScore &&
|
|
127
|
+
r.topK === base.topK);
|
|
128
|
+
const delta = tuningDelta(best, current);
|
|
129
|
+
console.log(`\n ${c.heading("Best configuration")}`);
|
|
130
|
+
console.log(` keywordWeight=${c.num(best.keywordWeight)} symbolKeywordWeight=${c.num(best.symbolKeywordWeight ?? "unset")} minScore=${c.num(best.minScore)} topK=${c.num(best.topK)}`);
|
|
131
|
+
console.log(` Hit@1 ${(best.summary.hit1 * 100).toFixed(1)}% · Hit@3 ${(best.summary.hit3 * 100).toFixed(1)}% · ` +
|
|
132
|
+
`Hit@5 ${(best.summary.hit5 * 100).toFixed(1)}% · Hit@10 ${(best.summary.hit10 * 100).toFixed(1)}% · ` +
|
|
133
|
+
`MRR ${best.summary.mrr.toFixed(3)} · nDCG@10 ${best.summary.ndcg10.toFixed(3)}`);
|
|
134
|
+
if (current) {
|
|
135
|
+
const sign = (v) => (v >= 0 ? "+" : "");
|
|
136
|
+
console.log(` ${c.dim("vs. current config:")} Hit@5 ${sign(delta.hit5)}${(delta.hit5 * 100).toFixed(1)} pts · ` +
|
|
137
|
+
`MRR ${sign(delta.mrr)}${delta.mrr.toFixed(3)} · nDCG@10 ${sign(delta.ndcg10)}${delta.ndcg10.toFixed(3)}`);
|
|
138
|
+
}
|
|
139
|
+
let written = false;
|
|
140
|
+
if (options.write) {
|
|
141
|
+
if (!ctx.configPath) {
|
|
142
|
+
throw new Error("No config file to write to — pass --config <path> or run this in a workspace with opencode-rag.json.");
|
|
143
|
+
}
|
|
144
|
+
const tuned = applyTuningToConfig(ctx.config, best);
|
|
145
|
+
const hybridWeights = tuned.retrieval.hybridSearch ?? { enabled: true, keywordWeight: best.keywordWeight };
|
|
146
|
+
const updates = [
|
|
147
|
+
[["retrieval", "hybridSearch", "keywordWeight"], hybridWeights.keywordWeight],
|
|
148
|
+
[["retrieval", "hybridSearch", "symbolKeywordWeight"], hybridWeights.symbolKeywordWeight],
|
|
149
|
+
[["retrieval", "minScore"], tuned.retrieval.minScore],
|
|
150
|
+
[["retrieval", "topK"], tuned.retrieval.topK],
|
|
151
|
+
];
|
|
152
|
+
for (const [keyPath, value] of updates) {
|
|
153
|
+
if (value === undefined)
|
|
154
|
+
continue;
|
|
155
|
+
if (!updateConfigValue(ctx.configPath, keyPath, value)) {
|
|
156
|
+
throw new Error(`Could not write ${keyPath.join(".")} to ${ctx.configPath}`);
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
written = true;
|
|
160
|
+
console.log(` ${c.success(`Wrote the winning configuration to ${path.relative(cwd, ctx.configPath)}`)}`);
|
|
161
|
+
}
|
|
162
|
+
else {
|
|
163
|
+
console.log(` ${c.dim("Re-run with --write to persist these values.")}`);
|
|
164
|
+
}
|
|
165
|
+
if (options.json) {
|
|
166
|
+
console.log(JSON.stringify({
|
|
167
|
+
schema: 1,
|
|
168
|
+
status: "ok",
|
|
169
|
+
labels: labels.length,
|
|
170
|
+
labelsPath: path.relative(cwd, labelsPath).replace(/\\/g, "/"),
|
|
171
|
+
metric,
|
|
172
|
+
candidates: results.map((r) => ({
|
|
173
|
+
keywordWeight: r.keywordWeight,
|
|
174
|
+
symbolKeywordWeight: r.symbolKeywordWeight,
|
|
175
|
+
minScore: r.minScore,
|
|
176
|
+
topK: r.topK,
|
|
177
|
+
hit1: r.summary.hit1,
|
|
178
|
+
hit5: r.summary.hit5,
|
|
179
|
+
mrr: r.summary.mrr,
|
|
180
|
+
ndcg10: r.summary.ndcg10,
|
|
181
|
+
durationMs: r.durationMs,
|
|
182
|
+
})),
|
|
183
|
+
best: {
|
|
184
|
+
keywordWeight: best.keywordWeight,
|
|
185
|
+
symbolKeywordWeight: best.symbolKeywordWeight,
|
|
186
|
+
minScore: best.minScore,
|
|
187
|
+
topK: best.topK,
|
|
188
|
+
metrics: {
|
|
189
|
+
hit1: best.summary.hit1,
|
|
190
|
+
hit5: best.summary.hit5,
|
|
191
|
+
mrr: best.summary.mrr,
|
|
192
|
+
ndcg10: best.summary.ndcg10,
|
|
193
|
+
},
|
|
194
|
+
},
|
|
195
|
+
delta,
|
|
196
|
+
written,
|
|
197
|
+
}));
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
catch (err) {
|
|
201
|
+
const message = err.message || String(err);
|
|
202
|
+
console.error(c.error(`\nTuning could not run: ${message}\n`));
|
|
203
|
+
if (message.toLowerCase().includes("fetch") || message.toLowerCase().includes("econnrefused")) {
|
|
204
|
+
console.error(c.warn("Hint: Is your embedding provider running?\n"));
|
|
205
|
+
}
|
|
206
|
+
logCliError(logFilePath, "eval:tune", `Tuning failed: ${message}`, err);
|
|
207
|
+
exitCode = 2;
|
|
208
|
+
}
|
|
209
|
+
finally {
|
|
210
|
+
if (ctx)
|
|
211
|
+
await cleanupContext(ctx);
|
|
212
|
+
}
|
|
213
|
+
process.exit(exitCode);
|
|
214
|
+
});
|
|
215
|
+
}
|
|
216
|
+
//# sourceMappingURL=eval-tuner.js.map
|
|
@@ -14,6 +14,7 @@ export { registerMcpCommand } from "./mcp.js";
|
|
|
14
14
|
export { registerSetupCommand } from "./setup.js";
|
|
15
15
|
export { registerUpdateCommand } from "./update.js";
|
|
16
16
|
export { registerEvalSessionsCommand, registerEvalAnalyzeCommand, registerEvalCompareCommand, registerEvalGateCommand, registerEvalDecideCommand } from "./eval.js";
|
|
17
|
+
export { registerEvalTuneCommand } from "./eval-tuner.js";
|
|
17
18
|
export { registerDescribeImageCommand } from "./describe-image.js";
|
|
18
19
|
export { registerDecideCommand } from "./decide.js";
|
|
19
20
|
export { registerQuirkCommand } from "./quirk.js";
|
|
@@ -14,6 +14,7 @@ export { registerMcpCommand } from "./mcp.js";
|
|
|
14
14
|
export { registerSetupCommand } from "./setup.js";
|
|
15
15
|
export { registerUpdateCommand } from "./update.js";
|
|
16
16
|
export { registerEvalSessionsCommand, registerEvalAnalyzeCommand, registerEvalCompareCommand, registerEvalGateCommand, registerEvalDecideCommand } from "./eval.js";
|
|
17
|
+
export { registerEvalTuneCommand } from "./eval-tuner.js";
|
|
17
18
|
export { registerDescribeImageCommand } from "./describe-image.js";
|
|
18
19
|
export { registerDecideCommand } from "./decide.js";
|
|
19
20
|
export { registerQuirkCommand } from "./quirk.js";
|
|
@@ -9,6 +9,7 @@ import os from "node:os";
|
|
|
9
9
|
import fs from "node:fs";
|
|
10
10
|
import { c, resolveCliContext, logCliError, logCliInfo, formatTimestamp, formatRelativeAge } from "../format.js";
|
|
11
11
|
import { getIndexStatusSummary } from "../../indexer.js";
|
|
12
|
+
import { findKeywordIndexGaps } from "../../indexer/keyword-rehydration.js";
|
|
12
13
|
import { deriveWatcherHealth } from "../../watcher.js";
|
|
13
14
|
import { getPackageMetadata } from "../helpers.js";
|
|
14
15
|
import { checkForUpdate } from "../../core/version-check.js";
|
|
@@ -142,6 +143,25 @@ export function registerStatusCommand(program) {
|
|
|
142
143
|
? keywordIndex?.count() ?? 0
|
|
143
144
|
: 0;
|
|
144
145
|
logCliInfo(logFilePath, "status", `${c.label("Keyword index:")} ${config.retrieval.hybridSearch?.enabled ? c.enabled("enabled") : c.disabled("disabled")} (${c.num(kiCount)} chunks)`);
|
|
146
|
+
// The keyword index and the vector store are two independent stores of
|
|
147
|
+
// the same corpus. A file the store holds but the keyword index does
|
|
148
|
+
// not cover is unreachable through keyword search — report the gap
|
|
149
|
+
// instead of printing two unrelated chunk counts.
|
|
150
|
+
if (config.retrieval.hybridSearch?.enabled && keywordIndex) {
|
|
151
|
+
const gaps = await findKeywordIndexGaps(store, keywordIndex);
|
|
152
|
+
if (gaps.length > 0) {
|
|
153
|
+
logCliInfo(logFilePath, "status", `${c.label("Keyword coverage:")} ${c.warn(`${gaps.length} file(s) in the store are missing from the keyword index`)} ${c.dim("(run 'opencode-rag index' to rehydrate — no re-embedding needed)")}`);
|
|
154
|
+
for (const gap of gaps.slice(0, 5)) {
|
|
155
|
+
logCliInfo(logFilePath, "status", ` ${c.file(path.relative(cwd, gap) || gap)}`);
|
|
156
|
+
}
|
|
157
|
+
if (gaps.length > 5) {
|
|
158
|
+
logCliInfo(logFilePath, "status", ` ${c.dim(`… and ${gaps.length - 5} more`)}`);
|
|
159
|
+
}
|
|
160
|
+
}
|
|
161
|
+
else {
|
|
162
|
+
logCliInfo(logFilePath, "status", `${c.label("Keyword coverage:")} ${c.success("complete")} ${c.dim("(every indexed file is searchable by keyword)")}`);
|
|
163
|
+
}
|
|
164
|
+
}
|
|
145
165
|
if (summary.rebuildRequired) {
|
|
146
166
|
logCliInfo(logFilePath, "status", `${c.label("Rebuild required:")} ${c.warn("yes")} (manifest missing/corrupt)`);
|
|
147
167
|
}
|
package/dist/cli/index.js
CHANGED
|
@@ -15,7 +15,7 @@ import { realpathSync } from "node:fs";
|
|
|
15
15
|
import { basename, dirname } from "node:path";
|
|
16
16
|
import { fileURLToPath } from "node:url";
|
|
17
17
|
import { getPackageMetadata } from "./helpers.js";
|
|
18
|
-
import { registerIndexCommand, registerQueryCommand, registerClearCommand, registerStatusCommand, registerListCommand, registerShowCommand, registerDumpCommand, registerInitCommand, registerUiCommand, registerMcpCommand, registerSetupCommand, registerUpdateCommand, registerEvalSessionsCommand, registerEvalAnalyzeCommand, registerEvalCompareCommand, registerEvalGateCommand, registerEvalDecideCommand, registerDescribeImageCommand, registerDecideCommand, registerQuirkCommand, } from "./commands/index.js";
|
|
18
|
+
import { registerIndexCommand, registerQueryCommand, registerClearCommand, registerStatusCommand, registerListCommand, registerShowCommand, registerDumpCommand, registerInitCommand, registerUiCommand, registerMcpCommand, registerSetupCommand, registerUpdateCommand, registerEvalSessionsCommand, registerEvalAnalyzeCommand, registerEvalCompareCommand, registerEvalGateCommand, registerEvalDecideCommand, registerEvalTuneCommand, registerDescribeImageCommand, registerDecideCommand, registerQuirkCommand, } from "./commands/index.js";
|
|
19
19
|
/**
|
|
20
20
|
* The top-level Commander program instance that defines the `opencode-rag` CLI.
|
|
21
21
|
*
|
|
@@ -56,6 +56,7 @@ registerEvalAnalyzeCommand(program);
|
|
|
56
56
|
registerEvalCompareCommand(program);
|
|
57
57
|
registerEvalGateCommand(program);
|
|
58
58
|
registerEvalDecideCommand(program);
|
|
59
|
+
registerEvalTuneCommand(program);
|
|
59
60
|
registerInitCommand(program);
|
|
60
61
|
registerQuirkCommand(program);
|
|
61
62
|
// ── Auto-run detection ──────────────────────────────────────────
|
package/dist/core/config.d.ts
CHANGED
|
@@ -279,6 +279,33 @@ export interface MemoryConfig {
|
|
|
279
279
|
toolErrorMaxPerSession: number;
|
|
280
280
|
/** Latency budget (ms) for tool-error quirk recall; exceeded -> no injection (default 1500). */
|
|
281
281
|
toolErrorLatencyBudgetMs: number;
|
|
282
|
+
/**
|
|
283
|
+
* Automatically create and persist a quirk when a failed tool call has no
|
|
284
|
+
* matching quirk memory (default true). The new quirk is a deterministic
|
|
285
|
+
* template built from the tool name and the error text — no LLM involved.
|
|
286
|
+
* It is recorded for FUTURE sessions; it is not injected into the turn that
|
|
287
|
+
* produced the failure (see `toolErrorInjectCreated`).
|
|
288
|
+
*/
|
|
289
|
+
autoCreateOnToolError: boolean;
|
|
290
|
+
/** Max tool-error quirks CREATED per session (default 2). */
|
|
291
|
+
toolErrorCreateMaxPerSession: number;
|
|
292
|
+
/** Cooldown per normalized tool-error signature for creation in ms (default 600000 = 10 min). */
|
|
293
|
+
toolErrorCreateCooldownMs: number;
|
|
294
|
+
/**
|
|
295
|
+
* Minimum raw similarity (0-1) for an existing quirk to count as "an
|
|
296
|
+
* equivalent quirk already exists" and therefore suppress creation
|
|
297
|
+
* (default 0.6). Deliberately below `toolErrorMinScore` (0.7): a probable
|
|
298
|
+
* match still blocks a duplicate, it is just not injected on its own.
|
|
299
|
+
*/
|
|
300
|
+
toolErrorCreateMinScore: number;
|
|
301
|
+
/** Latency budget (ms) for the tool-error creation path (embed + store write); default 2000. */
|
|
302
|
+
toolErrorCreateLatencyBudgetMs: number;
|
|
303
|
+
/**
|
|
304
|
+
* Inject a freshly created tool-error quirk into the turn that produced the
|
|
305
|
+
* failure (default false). The agent already saw the raw error, so the
|
|
306
|
+
* created quirk carries no new information for that turn.
|
|
307
|
+
*/
|
|
308
|
+
toolErrorInjectCreated: boolean;
|
|
282
309
|
/** Decay settings for aging quirks. */
|
|
283
310
|
decay: {
|
|
284
311
|
/** Whether confidence decays over time. */
|
package/dist/core/config.js
CHANGED
|
@@ -322,6 +322,12 @@ export const DEFAULT_CONFIG = {
|
|
|
322
322
|
toolErrorCooldownMs: 60_000,
|
|
323
323
|
toolErrorMaxPerSession: 3,
|
|
324
324
|
toolErrorLatencyBudgetMs: 1500,
|
|
325
|
+
autoCreateOnToolError: true,
|
|
326
|
+
toolErrorCreateMaxPerSession: 2,
|
|
327
|
+
toolErrorCreateCooldownMs: 600_000,
|
|
328
|
+
toolErrorCreateMinScore: 0.6,
|
|
329
|
+
toolErrorCreateLatencyBudgetMs: 2000,
|
|
330
|
+
toolErrorInjectCreated: false,
|
|
325
331
|
decay: {
|
|
326
332
|
enabled: false,
|
|
327
333
|
halfLifeDays: 30,
|
|
@@ -467,6 +473,20 @@ export function validateConfig(config) {
|
|
|
467
473
|
if (config.memory?.toolErrorLatencyBudgetMs != null && config.memory.toolErrorLatencyBudgetMs < 0) {
|
|
468
474
|
warnings.push("memory.toolErrorLatencyBudgetMs must be >= 0");
|
|
469
475
|
}
|
|
476
|
+
if (config.memory?.toolErrorCreateMinScore != null) {
|
|
477
|
+
const r = config.memory.toolErrorCreateMinScore;
|
|
478
|
+
if (r < 0 || r > 1)
|
|
479
|
+
warnings.push("memory.toolErrorCreateMinScore must be between 0 and 1");
|
|
480
|
+
}
|
|
481
|
+
if (config.memory?.toolErrorCreateMaxPerSession != null && config.memory.toolErrorCreateMaxPerSession < 0) {
|
|
482
|
+
warnings.push("memory.toolErrorCreateMaxPerSession must be >= 0");
|
|
483
|
+
}
|
|
484
|
+
if (config.memory?.toolErrorCreateCooldownMs != null && config.memory.toolErrorCreateCooldownMs < 0) {
|
|
485
|
+
warnings.push("memory.toolErrorCreateCooldownMs must be >= 0");
|
|
486
|
+
}
|
|
487
|
+
if (config.memory?.toolErrorCreateLatencyBudgetMs != null && config.memory.toolErrorCreateLatencyBudgetMs < 0) {
|
|
488
|
+
warnings.push("memory.toolErrorCreateLatencyBudgetMs must be >= 0");
|
|
489
|
+
}
|
|
470
490
|
if (config.memory?.decay && config.memory.decay.halfLifeDays <= 0) {
|
|
471
491
|
warnings.push("memory.decay.halfLifeDays must be > 0");
|
|
472
492
|
}
|
|
@@ -249,6 +249,14 @@ export interface KeywordIndex {
|
|
|
249
249
|
count(): number;
|
|
250
250
|
/** Whether a chunk with the given id is indexed. Used by quirk reconcile. */
|
|
251
251
|
hasChunk(id: string): boolean;
|
|
252
|
+
/**
|
|
253
|
+
* Normalized file paths that currently have at least one indexed chunk.
|
|
254
|
+
*
|
|
255
|
+
* Used to detect a keyword index that drifted behind the vector store: an
|
|
256
|
+
* incremental index pass only re-adds the files it reprocessed, so a
|
|
257
|
+
* once-partial index stays partial unless it is compared against the store.
|
|
258
|
+
*/
|
|
259
|
+
getIndexedFilePaths(): string[];
|
|
252
260
|
/** Persist the index to disk as JSON. */
|
|
253
261
|
save(filePath?: string): Promise<void>;
|
|
254
262
|
}
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @fileoverview Golden-set-driven retrieval hyperparameter tuner.
|
|
3
|
+
*
|
|
4
|
+
* The retrieval fusion weights shipped in the config (`keywordWeight`,
|
|
5
|
+
* `symbolKeywordWeight`, …) are a single hardcoded default, but the optimum
|
|
6
|
+
* depends on the embedder, the corpus and the query mix — a code embedder
|
|
7
|
+
* ranks identifiers well through the vector half alone, while a general
|
|
8
|
+
* embedder leans on the keyword half. This module sweeps a small grid of
|
|
9
|
+
* candidates against the labelled golden set and reports which one wins.
|
|
10
|
+
*
|
|
11
|
+
* The metric implementation is shared with `eval:gate` (and therefore with
|
|
12
|
+
* the rerank A/B harness) so the tuner can never report different numbers
|
|
13
|
+
* than the gate for the same labels.
|
|
14
|
+
*/
|
|
15
|
+
import type { RagConfig } from "../core/config.js";
|
|
16
|
+
import type { SearchResult } from "../core/interfaces.js";
|
|
17
|
+
import type { GoldenLabel, GoldenOutcome, GoldenSummary } from "./golden-set.js";
|
|
18
|
+
/** One point in the retrieval search space. */
|
|
19
|
+
export interface TuningCandidate {
|
|
20
|
+
/** RRF keyword weight: 0 = pure vector ranking, 1 = pure keyword ranking. */
|
|
21
|
+
keywordWeight: number;
|
|
22
|
+
/** Keyword weight for bare-identifier queries. Defaults to `keywordWeight`. */
|
|
23
|
+
symbolKeywordWeight?: number;
|
|
24
|
+
/** Minimum fused score a result must reach. */
|
|
25
|
+
minScore: number;
|
|
26
|
+
/** Number of results requested from the pipeline. */
|
|
27
|
+
topK: number;
|
|
28
|
+
}
|
|
29
|
+
/** A candidate together with its measured quality. */
|
|
30
|
+
export interface TuningResult extends TuningCandidate {
|
|
31
|
+
/** Aggregate metrics over the whole label set. */
|
|
32
|
+
summary: GoldenSummary;
|
|
33
|
+
/** Per-label outcomes, for drill-down. */
|
|
34
|
+
outcomes: GoldenOutcome[];
|
|
35
|
+
/** Wall-clock duration of the candidate run in milliseconds. */
|
|
36
|
+
durationMs: number;
|
|
37
|
+
}
|
|
38
|
+
/** Options for {@link buildCandidateGrid}. */
|
|
39
|
+
export interface CandidateGridOptions {
|
|
40
|
+
/** Keyword weights to try. Defaults to `[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6]`. */
|
|
41
|
+
keywordWeights?: number[];
|
|
42
|
+
/** Symbol keyword weights to try. Defaults to the keyword weights. */
|
|
43
|
+
symbolKeywordWeights?: number[];
|
|
44
|
+
/** Minimum scores to try. Defaults to `[base.minScore]`. */
|
|
45
|
+
minScores?: number[];
|
|
46
|
+
/** Top-K values to try. Defaults to `[base.topK]`. */
|
|
47
|
+
topKs?: number[];
|
|
48
|
+
/** Hard cap on the number of candidates (default 64). */
|
|
49
|
+
maxCandidates?: number;
|
|
50
|
+
}
|
|
51
|
+
/** Metrics a candidate can be ranked by. */
|
|
52
|
+
export type TuningMetric = "ndcg10" | "hit5" | "hit1" | "mrr";
|
|
53
|
+
/**
|
|
54
|
+
* Build the candidate grid.
|
|
55
|
+
*
|
|
56
|
+
* The base candidate is always included so the sweep can be compared against
|
|
57
|
+
* the configuration that is actually running. Duplicates (same weights, same
|
|
58
|
+
* minScore, same topK) are removed, and the grid is capped so a wide sweep
|
|
59
|
+
* cannot turn into an accidental benchmark.
|
|
60
|
+
*
|
|
61
|
+
* @param base - The candidate describing the current configuration.
|
|
62
|
+
* @param options - Grid overrides.
|
|
63
|
+
* @returns The candidate list, base candidate first.
|
|
64
|
+
*/
|
|
65
|
+
export declare function buildCandidateGrid(base: TuningCandidate, options?: CandidateGridOptions): TuningCandidate[];
|
|
66
|
+
/** Options for {@link runTuner}. */
|
|
67
|
+
export interface RunTunerOptions {
|
|
68
|
+
/** The labelled queries to measure. */
|
|
69
|
+
labels: GoldenLabel[];
|
|
70
|
+
/** Workspace root used to relativize result paths. */
|
|
71
|
+
root: string;
|
|
72
|
+
/**
|
|
73
|
+
* Retrieve closure for one candidate. The caller owns the embedder/store
|
|
74
|
+
* wiring and maps the candidate onto the production retrieval options.
|
|
75
|
+
*/
|
|
76
|
+
retrieve: (label: GoldenLabel, candidate: TuningCandidate) => Promise<SearchResult[]>;
|
|
77
|
+
/** The candidates to evaluate. */
|
|
78
|
+
candidates: TuningCandidate[];
|
|
79
|
+
/** Progress line callback (defaults to console.log). */
|
|
80
|
+
log?: (line: string) => void;
|
|
81
|
+
}
|
|
82
|
+
/**
|
|
83
|
+
* Evaluate every candidate against the golden set.
|
|
84
|
+
*
|
|
85
|
+
* @param opts - Labels, root, retrieve closure, candidates, logger.
|
|
86
|
+
* @returns One result per candidate, in the order the candidates were given.
|
|
87
|
+
*/
|
|
88
|
+
export declare function runTuner(opts: RunTunerOptions): Promise<TuningResult[]>;
|
|
89
|
+
/**
|
|
90
|
+
* Pick the best candidate.
|
|
91
|
+
*
|
|
92
|
+
* Ranked by the primary metric, then by Hit@5, MRR and Hit@1 so a tie on the
|
|
93
|
+
* primary metric still resolves to the more useful configuration.
|
|
94
|
+
*
|
|
95
|
+
* @param results - The measured candidates.
|
|
96
|
+
* @param metric - The primary metric (default `ndcg10`).
|
|
97
|
+
* @returns The winning result, or `undefined` when there are no results.
|
|
98
|
+
*/
|
|
99
|
+
export declare function selectBestTuning(results: TuningResult[], metric?: TuningMetric): TuningResult | undefined;
|
|
100
|
+
/**
|
|
101
|
+
* Compare the winner against the candidate that describes the current config.
|
|
102
|
+
*
|
|
103
|
+
* @param best - The winning candidate.
|
|
104
|
+
* @param current - The candidate describing the running configuration.
|
|
105
|
+
* @returns The per-metric deltas (winner minus current).
|
|
106
|
+
*/
|
|
107
|
+
export declare function tuningDelta(best: TuningResult, current: TuningResult | undefined): Record<string, number>;
|
|
108
|
+
/**
|
|
109
|
+
* Write the winning weights into a copy of the configuration.
|
|
110
|
+
*
|
|
111
|
+
* The input config is not mutated — the caller decides whether to persist the
|
|
112
|
+
* result. Only the retrieval section is touched.
|
|
113
|
+
*
|
|
114
|
+
* @param config - The configuration to base the result on.
|
|
115
|
+
* @param best - The winning candidate.
|
|
116
|
+
* @returns A new config with the winning fusion weights applied.
|
|
117
|
+
*/
|
|
118
|
+
export declare function applyTuningToConfig(config: RagConfig, best: TuningCandidate): RagConfig;
|