opencode-rag-plugin 1.18.0 → 1.18.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/ReadMe.md +41 -1
  2. package/dist/cli/commands/init-helpers.d.ts +5 -1
  3. package/dist/cli/commands/init-helpers.js +8 -19
  4. package/dist/cli/commands/init.js +42 -8
  5. package/dist/cli/commands/setup.js +15 -6
  6. package/dist/cli/commands/ui.js +1 -1
  7. package/dist/core/config.d.ts +10 -0
  8. package/dist/core/config.js +6 -1
  9. package/dist/core/interfaces.d.ts +9 -0
  10. package/dist/core/runtime-overrides.d.ts +7 -0
  11. package/dist/core/runtime-overrides.js +15 -0
  12. package/dist/describer/anthropic.d.ts +4 -0
  13. package/dist/describer/anthropic.js +9 -2
  14. package/dist/describer/describer.d.ts +4 -0
  15. package/dist/describer/describer.js +8 -0
  16. package/dist/describer/gemini.d.ts +3 -0
  17. package/dist/describer/gemini.js +8 -2
  18. package/dist/indexer/pipeline.js +40 -0
  19. package/dist/opencode/system-guidance.d.ts +34 -0
  20. package/dist/opencode/system-guidance.js +127 -0
  21. package/dist/plugin.js +74 -35
  22. package/dist/quirks/auto-capture.d.ts +14 -0
  23. package/dist/quirks/auto-capture.js +112 -0
  24. package/dist/quirks/prompts.d.ts +1 -0
  25. package/dist/quirks/prompts.js +13 -0
  26. package/dist/quirks/quirk-store.d.ts +2 -0
  27. package/dist/quirks/quirk-store.js +1 -1
  28. package/dist/vectorstore/lancedb.d.ts +10 -1
  29. package/dist/vectorstore/lancedb.js +28 -2
  30. package/dist/vectorstore/memory.d.ts +2 -0
  31. package/dist/vectorstore/memory.js +3 -0
  32. package/dist/web/api.d.ts +3 -1
  33. package/dist/web/api.js +50 -1
  34. package/dist/web/server.d.ts +2 -1
  35. package/dist/web/server.js +3 -2
  36. package/dist/web/ui/index.html +163 -0
  37. package/package.json +1 -1
package/ReadMe.md CHANGED
@@ -41,13 +41,14 @@ opencode-rag query "authentication middleware"
41
41
  | **OpenCode plugin** | Auto-inject context, read-tool override, TUI settings, Ctrl+Enter to add RAG context, MCP registration on `init` |
42
42
  | **Incremental indexing** | File-hash manifest, background watcher, auto-rebuild on corruption |
43
43
  | **Privacy-first** | All processing stays local (when using Ollama) |
44
- | **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `ui`, `mcp`, `setup`, `eval:sessions`, `eval:analyze`, `eval:compare` |
44
+ | **CLI Tools** | `init`, `index`, `query`, `status`, `list`, `show`, `dump`, `clear`, `describe-image`, `ui`, `mcp`, `setup`, `quirk`, `eval:sessions`, `eval:analyze`, `eval:compare` |
45
45
  | **Proxy-aware** | Corporate proxy support with raw-socket localhost bypass |
46
46
  | **OpenAI / Anthropic / Cohere** | Use alternate embedding providers with API key auto-resolution |
47
47
  | **Evaluation** | Session-level token tracking, RAG-on vs RAG-off comparison, tiktoken BPE counting |
48
48
  | **Documentation mode** | `/doc` slash command: agent adds JSDoc/TSDoc to undocumented files, progress tracked per subdirectory |
49
49
  | **Wiki mode** | `/wiki` slash command: agent maintains a persistent knowledge wiki at `.opencode/wiki/` (ingest, query, lint, seed) |
50
50
  | **Context optimization** | Post-retrieval dedup, per-file chunk limits, adjacent-merge to fit the context window |
51
+ | **Quirk memory** | Persistent experiential memory — agents recall/persist gotchas, preferences, decisions across sessions (`add_quirk` / `recall_quirks`, `opencode-rag quirk`) |
51
52
  | **AGENTS.md directive** | `opencode-rag init` merges a tool-usage directive into `AGENTS.md` via sentinel markers |
52
53
 
53
54
  ## Web UI
@@ -125,6 +126,45 @@ The agent maintains all wiki pages during normal coding sessions (ingest on lear
125
126
 
126
127
  See [Plugin documentation](doc/plugin.md#6-wiki-mode--slash-command-wiki) for the full protocol.
127
128
 
129
+ ## Quirk Memory
130
+
131
+ OpenCodeRAG gives your agent **persistent, cross-session memory** of non-obvious facts — gotchas, preferences, decisions, and environment constraints — that it can recall and extend over time. Quirks are embedded and stored in the same vector store as your code, then recalled via semantic search whenever they're relevant. This lets the agent avoid repeating mistakes and accumulate project knowledge across sessions.
132
+
133
+ **Enabled by default.** Turn it on/off in `opencode-rag.json`:
134
+
135
+ ```json
136
+ {
137
+ "memory": {
138
+ "enabled": true,
139
+ "autoInject": true,
140
+ "minConfidence": 0.5,
141
+ "recallMinScore": 0.3,
142
+ "decay": { "enabled": true, "halfLifeDays": 30 }
143
+ }
144
+ }
145
+ ```
146
+
147
+ **Agent tools:**
148
+
149
+ | Tool | Use when |
150
+ |------|----------|
151
+ | `recall_quirks(query)` | You hit an error or need to remember a gotcha, preference, or decision from past sessions |
152
+ | `add_quirk(content, { type, tags })` | You just discovered a non-obvious fact, workaround, or convention worth remembering |
153
+
154
+ **CLI — manage quirks directly:**
155
+
156
+ ```bash
157
+ opencode-rag quirk add "npm needs --legacy-peer-deps" --type gotcha --tag installation
158
+ opencode-rag quirk list
159
+ opencode-rag quirk lint # flag low-confidence / stale / duplicate quirks
160
+ opencode-rag quirk test "npm needs --legacy-peer-deps"
161
+ # ✓ Quirk has been appended:
162
+ # [gotcha] npm needs --legacy-peer-deps (installation)
163
+ # 99% confidence
164
+ ```
165
+
166
+ When `memory.autoInject` is `true`, relevant quirks are automatically injected into the prompt during retrieval (via the `chat.message` hook). Every `add_quirk` is vetted by an immutable trust monitor that rejects destructive patterns (e.g. `rm -rf`, `force push`, `bypass security`). See [Plugin documentation](doc/plugin.md#9-quirk-memory-experiential-memory) and [CLI Reference: `quirk`](doc/cli.md#quirk).
167
+
128
168
  ## MCP Server (Optional)
129
169
 
130
170
  OpenCodeRAG ships a CLI-based [MCP (Model Context Protocol)](https://spec.modelcontextprotocol.io/) server that exposes semantic code tools to any MCP-compatible client (Claude Desktop, Cursor, etc.).
@@ -83,9 +83,13 @@ export declare function mergeGitignoreContent(existingContent?: string): string;
83
83
  * provided, the section alone is returned as a new file.
84
84
  *
85
85
  * @param existingContent - The current `AGENTS.md` content, or `undefined` if absent.
86
+ * @param opts - Optional settings. `promptEnforcement` controls whether the
87
+ * quirk-capture enforcement rules are included (default `true`).
86
88
  * @returns The merged `AGENTS.md` content with a trailing newline.
87
89
  */
88
- export declare function mergeAgentsMdContent(existingContent?: string): string;
90
+ export declare function mergeAgentsMdContent(existingContent?: string, opts?: {
91
+ promptEnforcement?: boolean;
92
+ }): string;
89
93
  /**
90
94
  * Get the runtime directory path (`~/.opencode`).
91
95
  *
@@ -11,6 +11,7 @@ import { existsSync, mkdirSync, rmSync, symlinkSync, } from "node:fs";
11
11
  import { execSync } from "node:child_process";
12
12
  import { DEFAULT_CONFIG } from "../../core/config.js";
13
13
  import { c } from "../format.js";
14
+ import { BEGIN_MARKER, END_MARKER, buildAgentsMdDirective } from "../../opencode/system-guidance.js";
14
15
  import { getStringRecord, readJsonObject, writeJsonFile, } from "../helpers.js";
15
16
  /**
16
17
  * Build the workspace-local `.opencode/package.json` content.
@@ -252,21 +253,6 @@ export function mergeGitignoreContent(existingContent) {
252
253
  merged.push("# OpenCodeRAG workspace state", ...missing, "");
253
254
  return merged.join("\n");
254
255
  }
255
- const BEGIN_MARKER = "<!-- BEGIN opencode-rag -->";
256
- const END_MARKER = "<!-- END opencode-rag -->";
257
- const AGENTS_MD_SECTION = [
258
- BEGIN_MARKER,
259
- "## Code Navigation",
260
- "",
261
- "ALWAYS use OpenCodeRAG tools before reading or editing:",
262
- "- **Search first** — `search_semantic(query)` instead of grep/glob",
263
- "- **Skeleton before read** — `get_file_skeleton(filePath)` then read specific lines",
264
- "- **Usages before edit** — `find_usages(symbolName)` before modifying any symbol",
265
- "- **Images via describe** — `describe_image(filePath)` — never read raw bytes",
266
- "",
267
- "If no results, run `opencode-rag index`.",
268
- END_MARKER,
269
- ].join("\n");
270
256
  /**
271
257
  * Merge the OpenCodeRAG directive section into an existing `AGENTS.md`.
272
258
  *
@@ -276,23 +262,26 @@ const AGENTS_MD_SECTION = [
276
262
  * provided, the section alone is returned as a new file.
277
263
  *
278
264
  * @param existingContent - The current `AGENTS.md` content, or `undefined` if absent.
265
+ * @param opts - Optional settings. `promptEnforcement` controls whether the
266
+ * quirk-capture enforcement rules are included (default `true`).
279
267
  * @returns The merged `AGENTS.md` content with a trailing newline.
280
268
  */
281
- export function mergeAgentsMdContent(existingContent) {
269
+ export function mergeAgentsMdContent(existingContent, opts) {
270
+ const section = buildAgentsMdDirective({ promptEnforcement: opts?.promptEnforcement ?? true });
282
271
  if (!existingContent) {
283
- return `${AGENTS_MD_SECTION}\n`;
272
+ return `${section}\n`;
284
273
  }
285
274
  const beginIdx = existingContent.indexOf(BEGIN_MARKER);
286
275
  const endIdx = existingContent.indexOf(END_MARKER);
287
276
  if (beginIdx !== -1 && endIdx !== -1 && endIdx > beginIdx) {
288
277
  const before = existingContent.slice(0, beginIdx);
289
278
  const after = existingContent.slice(endIdx + END_MARKER.length);
290
- const merged = `${before}${AGENTS_MD_SECTION}${after}`;
279
+ const merged = `${before}${section}${after}`;
291
280
  return merged.endsWith("\n") ? merged : `${merged}\n`;
292
281
  }
293
282
  const trimmed = existingContent.trimEnd();
294
283
  const separator = trimmed.endsWith(BEGIN_MARKER) ? "" : "\n\n";
295
- return `${trimmed}${separator}${AGENTS_MD_SECTION}\n`;
284
+ return `${trimmed}${separator}${section}\n`;
296
285
  }
297
286
  /**
298
287
  * Get the runtime directory path (`~/.opencode`).
@@ -54,6 +54,11 @@ export function registerInitCommand(program) {
54
54
  else {
55
55
  console.log(` ${c.exists("Exists:")} .opencode/`);
56
56
  }
57
+ // Wiki safety: warn if wiki exists but would be affected (currently init never touches it)
58
+ const wikiDir = path.join(opencodeDir, "wiki");
59
+ if (existsSync(wikiDir)) {
60
+ console.log(` ${c.exists("Preserved:")} .opencode/wiki/ (never modified by init)`);
61
+ }
57
62
  if (!existsSync(pluginsDir)) {
58
63
  mkdirSync(pluginsDir, { recursive: true });
59
64
  console.log(` ${c.created("Created:")} .opencode/plugins/`);
@@ -144,7 +149,18 @@ export function registerInitCommand(program) {
144
149
  }
145
150
  const agentsMdPath = path.join(cwd, "AGENTS.md");
146
151
  const agentsMdExists = existsSync(agentsMdPath);
147
- const nextAgentsMd = mergeAgentsMdContent(agentsMdExists ? readFileSync(agentsMdPath, "utf-8") : undefined);
152
+ // Read effective promptEnforcement from existing config (if present)
153
+ let promptEnforcement = true;
154
+ try {
155
+ if (existsSync(configPath)) {
156
+ const existingCfg = loadConfig(configPath, false);
157
+ promptEnforcement = existingCfg.memory?.promptEnforcement ?? true;
158
+ }
159
+ }
160
+ catch {
161
+ // Malformed or missing config — use default
162
+ }
163
+ const nextAgentsMd = mergeAgentsMdContent(agentsMdExists ? readFileSync(agentsMdPath, "utf-8") : undefined, { promptEnforcement });
148
164
  if (!agentsMdExists || options.force) {
149
165
  writeFileSync(agentsMdPath, nextAgentsMd, "utf-8");
150
166
  console.log(` ${agentsMdExists ? c.updated("Updated:") : c.created("Created:")} AGENTS.md`);
@@ -170,16 +186,34 @@ export function registerInitCommand(program) {
170
186
  console.log(` ${c.exists("Exists:")} .opencode/package.json`);
171
187
  }
172
188
  const configExists = existsSync(configPath);
173
- if (!configExists || options.force) {
174
- if (configExists && options.force) {
175
- copyFileSync(configPath, configPath + ".bak");
176
- console.log(` ${c.dim("Backup:")} ${configPath}.bak`);
177
- }
189
+ if (!configExists) {
178
190
  writeFileSync(configPath, generateDefaultConfigJson(), "utf-8");
179
- console.log(` ${configExists ? c.updated("Updated:") : c.created("Created:")} opencode-rag.json`);
191
+ console.log(` ${c.created("Created:")} opencode-rag.json`);
180
192
  }
181
193
  else {
182
- console.log(` ${c.exists("Exists:")} opencode-rag.json`);
194
+ // NEVER overwrite existing config without interactive confirmation
195
+ let overwrite = false;
196
+ if (process.stdin.isTTY) {
197
+ const readline = await import("node:readline");
198
+ const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
199
+ const answer = await new Promise((resolve) => {
200
+ rl.question(` ${c.warn("opencode-rag.json already exists.")} Overwrite with default config? A backup will be saved. (y/N) `, resolve);
201
+ });
202
+ rl.close();
203
+ overwrite = /^y(?:es)?$/i.test(answer.trim());
204
+ }
205
+ if (overwrite) {
206
+ copyFileSync(configPath, `${configPath}.bak`);
207
+ console.log(` ${c.dim("Backup:")} opencode-rag.json.bak`);
208
+ writeFileSync(configPath, generateDefaultConfigJson(), "utf-8");
209
+ console.log(` ${c.updated("Updated:")} opencode-rag.json`);
210
+ }
211
+ else {
212
+ console.log(` ${c.exists("Exists:")} opencode-rag.json (preserved)`);
213
+ if (!process.stdin.isTTY && options.force) {
214
+ console.log(` ${c.warn("Skipped overwrite:")} non-interactive shell. Edit opencode-rag.json manually or re-run in a TTY.`);
215
+ }
216
+ }
183
217
  }
184
218
  // ── Provider health check + dependency install (parallel) ──
185
219
  const healthPromise = options.skipHealthCheck
@@ -12,16 +12,25 @@ function removeIfExists(targetPath) {
12
12
  }
13
13
  function checkOpenCodeRunning() {
14
14
  try {
15
- const pids = execSync("pgrep -x opencode 2>/dev/null || (Get-Process -Name opencode -ErrorAction SilentlyContinue 2>nul | ForEach-Object { $_.Id })", {
16
- encoding: "utf-8",
17
- timeout: 3_000,
18
- }).trim();
19
- if (pids) {
15
+ if (process.platform === "win32") {
16
+ execSync('tasklist /FI "IMAGENAME eq opencode.exe" 2>nul | find /I "opencode.exe" >nul', {
17
+ timeout: 3_000,
18
+ stdio: "ignore",
19
+ });
20
20
  console.log(`\n ${c.warn("OpenCode is currently running.")} Restart it to load the updated plugin.`);
21
21
  }
22
+ else {
23
+ const pids = execSync("pgrep -x opencode 2>/dev/null", {
24
+ encoding: "utf-8",
25
+ timeout: 3_000,
26
+ }).trim();
27
+ if (pids) {
28
+ console.log(`\n ${c.warn("OpenCode is currently running.")} Restart it to load the updated plugin.`);
29
+ }
30
+ }
22
31
  }
23
32
  catch {
24
- // pgrep/Get-Process aren't available or succeeded silently
33
+ // OpenCode not found or tools unavailable
25
34
  }
26
35
  }
27
36
  export function registerSetupCommand(program) {
@@ -32,7 +32,7 @@ export function registerUiCommand(program) {
32
32
  const port = parseInt(options.port ?? String(config.ui?.port ?? 3210), 10);
33
33
  const openBrowser = options.open !== false && (config.ui?.openBrowser ?? true);
34
34
  const { startWebUi } = await import("../../web/server.js");
35
- const server = await startWebUi(storePath, port, cwd);
35
+ const server = await startWebUi(storePath, port, cwd, config.embedding.vectorDimension ?? 384, config);
36
36
  const url = `http://127.0.0.1:${server.port}`;
37
37
  logCliInfo(logFilePath, "ui", `\n${c.heading("OpenCodeRAG Web UI")}`);
38
38
  logCliInfo(logFilePath, "ui", ` ${c.label("URL:")} ${c.value(url)}`);
@@ -131,6 +131,16 @@ export interface MemoryConfig {
131
131
  minConfidence: number;
132
132
  /** Minimum query-relevance score (0-1) for a quirk to be recalled. */
133
133
  recallMinScore: number;
134
+ /** Automatically extract quirks from each completed agent turn. */
135
+ passiveCapture: boolean;
136
+ /** Upgrade the system-prompt nudge into a mandatory trigger. */
137
+ promptEnforcement: boolean;
138
+ /** Summarize the full session transcript into quirks at session end. */
139
+ sessionEndExtraction: boolean;
140
+ /** Max quirks to auto-capture per turn/session-end pass. */
141
+ autoCaptureMaxPerTurn: number;
142
+ /** Lexical similarity threshold (0-1) above which a candidate is considered a duplicate and skipped. */
143
+ autoCaptureDedupThreshold: number;
134
144
  /** Decay settings for aging quirks. */
135
145
  decay: {
136
146
  /** Whether confidence decays over time. */
@@ -184,7 +184,7 @@ export const DEFAULT_CONFIG = {
184
184
  "Return your changes as a list of file paths with the full new content of the comment block for each modified symbol. Do NOT output the entire file unless asked.",
185
185
  },
186
186
  wikiMode: {
187
- enabled: false,
187
+ enabled: true,
188
188
  systemPrompt: "You are a wiki maintainer for this codebase. You incrementally build and maintain " +
189
189
  "a persistent knowledge wiki at `.opencode/wiki/` — a structured, interlinked collection " +
190
190
  "of markdown files that synthesizes knowledge from the codebase, documentation, and your conversations.\n\n" +
@@ -258,6 +258,11 @@ export const DEFAULT_CONFIG = {
258
258
  autoInject: false,
259
259
  minConfidence: 0.5,
260
260
  recallMinScore: 0.72,
261
+ passiveCapture: false,
262
+ promptEnforcement: true,
263
+ sessionEndExtraction: true,
264
+ autoCaptureMaxPerTurn: 2,
265
+ autoCaptureDedupThreshold: 0.85,
261
266
  decay: {
262
267
  enabled: false,
263
268
  halfLifeDays: 30,
@@ -52,6 +52,13 @@ export interface DescriptionProvider {
52
52
  * @param opts - Optional progress reporting options.
53
53
  */
54
54
  generateBatchDescriptions(chunks: Chunk[], logger?: DescriptionLogger, opts?: BatchDescriptionOptions): Promise<Map<string, string>>;
55
+ /**
56
+ * Generate free-form text from a system prompt and user message.
57
+ * Used by auto-capture to extract quirks from agent transcripts.
58
+ */
59
+ generateText(system: string, user: string, opts?: {
60
+ timeoutMs?: number;
61
+ }): Promise<string>;
55
62
  }
56
63
  /** Explains how a search result score was computed, including vector and keyword contributions. */
57
64
  export interface SearchExplanation {
@@ -176,6 +183,8 @@ export interface VectorStore {
176
183
  getChunksByFilePath(filePath: string): Promise<Chunk[]>;
177
184
  /** Re-open the store, optionally pointing at a new database path. */
178
185
  reopen?(newPath?: string): Promise<void>;
186
+ /** Compact fragments and prune old versions to prevent version-manifest accumulation. */
187
+ optimize?(): Promise<void>;
179
188
  }
180
189
  /** Filter criteria for narrowing search results by file path, language, or kind. */
181
190
  export interface MetadataFilter {
@@ -42,6 +42,13 @@ export interface RuntimeOverrides {
42
42
  model?: string;
43
43
  provider?: string;
44
44
  };
45
+ memory?: {
46
+ passiveCapture?: boolean;
47
+ promptEnforcement?: boolean;
48
+ sessionEndExtraction?: boolean;
49
+ autoCaptureMaxPerTurn?: number;
50
+ autoCaptureDedupThreshold?: number;
51
+ };
45
52
  tui?: {
46
53
  fileListKeybinding?: string;
47
54
  chunksKeybinding?: string;
@@ -122,6 +122,21 @@ export function applyRuntimeOverrides(cfg, overrides) {
122
122
  merged.imageDescription.model = overrides.imageDescription.model;
123
123
  }
124
124
  }
125
+ if (overrides.memory) {
126
+ if (!merged.memory)
127
+ merged.memory = { ...DEFAULT_CONFIG.memory };
128
+ const m = merged.memory;
129
+ if (overrides.memory.passiveCapture !== undefined)
130
+ m.passiveCapture = overrides.memory.passiveCapture;
131
+ if (overrides.memory.promptEnforcement !== undefined)
132
+ m.promptEnforcement = overrides.memory.promptEnforcement;
133
+ if (overrides.memory.sessionEndExtraction !== undefined)
134
+ m.sessionEndExtraction = overrides.memory.sessionEndExtraction;
135
+ if (overrides.memory.autoCaptureMaxPerTurn !== undefined)
136
+ m.autoCaptureMaxPerTurn = overrides.memory.autoCaptureMaxPerTurn;
137
+ if (overrides.memory.autoCaptureDedupThreshold !== undefined)
138
+ m.autoCaptureDedupThreshold = overrides.memory.autoCaptureDedupThreshold;
139
+ }
125
140
  if (overrides.tui) {
126
141
  merged.tui = {
127
142
  ...DEFAULT_CONFIG.tui,
@@ -18,6 +18,10 @@ export declare class AnthropicDescriptionProvider implements DescriptionProvider
18
18
  /** @inheritdoc */
19
19
  generateDescription(chunk: Chunk): Promise<string>;
20
20
  /** @inheritdoc */
21
+ generateText(system: string, user: string, opts?: {
22
+ timeoutMs?: number;
23
+ }): Promise<string>;
24
+ /** @inheritdoc */
21
25
  generateBatchDescriptions(chunks: Chunk[], logger?: DescriptionLogger, opts?: BatchDescriptionOptions): Promise<Map<string, string>>;
22
26
  /**
23
27
  * Sends a request to the Anthropic Messages API with retry and exponential backoff.
@@ -25,6 +25,13 @@ export class AnthropicDescriptionProvider {
25
25
  return this.chatRequest(messages, this.config.timeoutMs ?? 60000);
26
26
  }
27
27
  /** @inheritdoc */
28
+ async generateText(system, user, opts) {
29
+ const messages = [
30
+ { role: "user", content: user },
31
+ ];
32
+ return this.chatRequest(messages, opts?.timeoutMs ?? this.config.timeoutMs ?? 60000, system);
33
+ }
34
+ /** @inheritdoc */
28
35
  async generateBatchDescriptions(chunks, logger, opts) {
29
36
  const log = logger ?? { info: (msg) => process.stderr.write(`${msg}\n`), warn: (msg) => process.stderr.write(`${msg}\n`), debug: (msg) => process.stderr.write(`${msg}\n`) };
30
37
  const concurrency = this.config.batchConcurrency ?? 3;
@@ -60,10 +67,10 @@ export class AnthropicDescriptionProvider {
60
67
  * @returns The trimmed response text from the model.
61
68
  * @throws When all retry attempts are exhausted or the response is empty.
62
69
  */
63
- async chatRequest(messages, timeoutMs) {
70
+ async chatRequest(messages, timeoutMs, systemOverride) {
64
71
  const baseUrl = this.config.baseUrl.replace(/\/+$/, "");
65
72
  const apiKey = this.config.apiKey ?? "";
66
- const systemPrompt = this.config.systemPrompt;
73
+ const systemPrompt = systemOverride ?? this.config.systemPrompt;
67
74
  const body = {
68
75
  model: this.config.model,
69
76
  max_tokens: 4096,
@@ -18,6 +18,10 @@ export declare class LlmDescriptionProvider implements DescriptionProvider {
18
18
  /** @inheritdoc */
19
19
  generateDescription(chunk: Chunk): Promise<string>;
20
20
  /** @inheritdoc */
21
+ generateText(system: string, user: string, opts?: {
22
+ timeoutMs?: number;
23
+ }): Promise<string>;
24
+ /** @inheritdoc */
21
25
  generateBatchDescriptions(chunks: Chunk[], logger?: DescriptionLogger, opts?: BatchDescriptionOptions): Promise<Map<string, string>>;
22
26
  /**
23
27
  * Sends a chat completion request to the LLM API with retry and exponential backoff.
@@ -26,6 +26,14 @@ export class LlmDescriptionProvider {
26
26
  return this.chatRequest(messages, this.config.timeoutMs ?? 60000);
27
27
  }
28
28
  /** @inheritdoc */
29
+ async generateText(system, user, opts) {
30
+ const messages = [
31
+ { role: "system", content: system },
32
+ { role: "user", content: user },
33
+ ];
34
+ return this.chatRequest(messages, opts?.timeoutMs ?? this.config.timeoutMs ?? 60000);
35
+ }
36
+ /** @inheritdoc */
29
37
  async generateBatchDescriptions(chunks, logger, opts) {
30
38
  const log = logger ?? { info: (msg) => process.stderr.write(`${msg}\n`), warn: (msg) => process.stderr.write(`${msg}\n`), debug: (msg) => process.stderr.write(`${msg}\n`) };
31
39
  const concurrency = this.config.batchConcurrency ?? 3;
@@ -12,6 +12,9 @@ export declare class GeminiDescriptionProvider implements DescriptionProvider {
12
12
  private readonly config;
13
13
  constructor(config: DescriptionConfig);
14
14
  generateDescription(chunk: Chunk): Promise<string>;
15
+ generateText(system: string, user: string, opts?: {
16
+ timeoutMs?: number;
17
+ }): Promise<string>;
15
18
  generateBatchDescriptions(chunks: Chunk[], logger?: DescriptionLogger, opts?: BatchDescriptionOptions): Promise<Map<string, string>>;
16
19
  private chatRequest;
17
20
  }
@@ -21,6 +21,12 @@ export class GeminiDescriptionProvider {
21
21
  ];
22
22
  return this.chatRequest(contents, this.config.timeoutMs ?? 60000);
23
23
  }
24
+ async generateText(system, user, opts) {
25
+ const contents = [
26
+ { role: "user", parts: [{ text: user }] },
27
+ ];
28
+ return this.chatRequest(contents, opts?.timeoutMs ?? this.config.timeoutMs ?? 60000, system);
29
+ }
24
30
  async generateBatchDescriptions(chunks, logger, opts) {
25
31
  const log = logger ?? { info: (msg) => process.stderr.write(`${msg}\n`), warn: (msg) => process.stderr.write(`${msg}\n`), debug: (msg) => process.stderr.write(`${msg}\n`) };
26
32
  const concurrency = this.config.batchConcurrency ?? 3;
@@ -46,11 +52,11 @@ export class GeminiDescriptionProvider {
46
52
  log.debug(`[describer] Descriptions generated: ${result.size}/${total}`);
47
53
  return result;
48
54
  }
49
- async chatRequest(contents, timeoutMs) {
55
+ async chatRequest(contents, timeoutMs, systemOverride) {
50
56
  const baseUrl = this.config.baseUrl.replace(/\/+$/, "");
51
57
  const apiKey = this.config.apiKey ?? "";
52
58
  const model = this.config.model;
53
- const systemPrompt = this.config.systemPrompt;
59
+ const systemPrompt = systemOverride ?? this.config.systemPrompt;
54
60
  const allParts = [{ text: systemPrompt }];
55
61
  for (const c of contents) {
56
62
  allParts.push(...c.parts);
@@ -169,7 +169,10 @@ async function runIndexPassInner(options, logger) {
169
169
  const workspaceFiles = await scanWorkspaceFiles(options.cwd, options.config, logger, options.force ? undefined : manifest, filterPaths, options.imageVisionProvider, descCache);
170
170
  const scanSec = ((Date.now() - scanStart) / 1000).toFixed(1);
171
171
  logger.info(`Workspace scan complete: ${workspaceFiles.length} files in ${scanSec}s`);
172
+ logger.info(`Querying vector store for existing chunk count...`);
173
+ const storeCountStart = Date.now();
172
174
  const existingCount = await options.store.count();
175
+ logger.info(`Store has ${existingCount} existing chunks (${((Date.now() - storeCountStart) / 1000).toFixed(1)}s)`);
173
176
  // Detect data loss: if the store has far fewer chunks than the manifest expects,
174
177
  // treat it as a corrupt store (e.g. schema migration dropped the old table).
175
178
  if (!options.force && manifestStatus === "ok" && existingCount > 0) {
@@ -276,6 +279,11 @@ async function runIndexPassInner(options, logger) {
276
279
  logger.debug(`Processing ${workspaceFiles.length} files (${newCount} new, ${modifiedCount} modified, ${unchangedCount} unchanged)...`);
277
280
  const limit = pLimit(options.config.indexing.concurrency);
278
281
  const deferDescriptions = !!options.descriptionProvider;
282
+ const activeCount = workspaceFiles.filter((f) => !f.isEmpty && !f.isTooSmall && (!manifest.files[f.normalizedPath] || manifest.files[f.normalizedPath].hash !== f.hash)).length;
283
+ logger.info(`Chunking ${activeCount} file(s)...`);
284
+ let chunkedDone = 0;
285
+ const chunkProgressInterval = Math.max(1, Math.floor(activeCount / 20));
286
+ const chunkPhaseStart = Date.now();
279
287
  const prepared = await Promise.all(workspaceFiles.map((file) => limit(async () => {
280
288
  const fileLabel = path.relative(options.cwd, file.normalizedPath).replace(/\\/g, "/");
281
289
  const isActive = !file.isEmpty && !file.isTooSmall &&
@@ -284,11 +292,20 @@ async function runIndexPassInner(options, logger) {
284
292
  options.progress?.startFile(fileLabel);
285
293
  }
286
294
  const prep = await prepareFile(file, options.cwd, manifest.files[file.normalizedPath], options.config, options.keywordIndex, options.descriptionProvider, logger, deferDescriptions, descHash);
295
+ if (isActive) {
296
+ chunkedDone++;
297
+ if (chunkedDone % chunkProgressInterval === 0 || chunkedDone === activeCount) {
298
+ const pct = ((chunkedDone / activeCount) * 100).toFixed(1);
299
+ logger.info(` Chunking: ${chunkedDone}/${activeCount} files (${pct}%)`);
300
+ }
301
+ }
287
302
  if (prep.earlyResult && isActive) {
288
303
  options.progress?.finishFile(fileLabel);
289
304
  }
290
305
  return prep;
291
306
  })));
307
+ const chunkPhaseSec = ((Date.now() - chunkPhaseStart) / 1000).toFixed(1);
308
+ logger.info(`Chunking complete: ${activeCount} files in ${chunkPhaseSec}s`);
292
309
  const aborted = () => options.abortSignal?.aborted ?? false;
293
310
  // Shared progress logger: "stage <file> (chunk i/n) — X/total remaining (P%)".
294
311
  const logChunkProgress = (stage, fileLabel, index, count, completed, total) => {
@@ -302,6 +319,7 @@ async function runIndexPassInner(options, logger) {
302
319
  const allChunks = [];
303
320
  const oversizedChunks = [];
304
321
  const maxContentChars = options.config.description?.maxContentChars;
322
+ logger.info(`Description phase: ${deferredPreps.length} files with chunks to describe`);
305
323
  for (const prep of deferredPreps) {
306
324
  for (const chunk of prep.chunks) {
307
325
  if (chunk.metadata.contentType !== "image") {
@@ -469,6 +487,8 @@ async function runIndexPassInner(options, logger) {
469
487
  for (const prep of deferredPreps) {
470
488
  prep.textToEmbed = buildTextsToEmbed(prep.chunks, prep.relPath, prep.metaHeader ?? "", prep.docPrefix ?? "", prep.isImageFile ?? false);
471
489
  }
490
+ const totalDescribedChunks = deferredPreps.reduce((s, p) => s + (p.chunks?.length ?? 0), 0);
491
+ logger.info(`Description phase complete: ${totalDescribedChunks} chunks across ${deferredPreps.length} files`);
472
492
  }
473
493
  }
474
494
  // Cross-file embedding batch: collect all texts into a single queue,
@@ -530,12 +550,15 @@ async function runIndexPassInner(options, logger) {
530
550
  let embeddedDone = 0;
531
551
  const allTexts = embedQueue.map(item => item.text);
532
552
  let allEmbeddings = [];
553
+ const embedPhaseStart = Date.now();
533
554
  if (allTexts.length > 0) {
555
+ logger.info(`Embedding phase: ${allTexts.length} texts in batches of ${batchSize}...`);
534
556
  try {
535
557
  allEmbeddings = await embedBatch(options.embedder, allTexts, batchSize, "document", defaultConcurrency, (completed, total) => {
536
558
  embeddedDone = completed;
537
559
  logChunkProgress("Embedding", "", completed, total, embeddedDone, totalEmbedChunks);
538
560
  });
561
+ logger.info(`Embedding complete: ${allTexts.length} texts in ${((Date.now() - embedPhaseStart) / 1000).toFixed(1)}s`);
539
562
  }
540
563
  catch (err) {
541
564
  logger.warn(` Global embedding failed: ${err.message}`);
@@ -564,6 +587,8 @@ async function runIndexPassInner(options, logger) {
564
587
  }
565
588
  // ── Phase 3: Store + manifest update per file (parallel) ──────────────
566
589
  const filesToStore = prepared.filter((p) => !earlyWorkerResults.has(prepared.indexOf(p)) && p.chunks && (p.textToEmbed?.length ?? 0) > 0).length;
590
+ const storePhaseStart = Date.now();
591
+ logger.info(`Store phase: storing ${filesToStore} file(s) into vector database...`);
567
592
  let storedFiles = 0;
568
593
  const storeLimit = pLimit(options.config.indexing.concurrency);
569
594
  const storeResults = await Promise.all(prepared.map((prep, fi) => storeLimit(async () => {
@@ -648,6 +673,8 @@ async function runIndexPassInner(options, logger) {
648
673
  entry.size = size;
649
674
  }
650
675
  }
676
+ const storePhaseSec = ((Date.now() - storePhaseStart) / 1000).toFixed(1);
677
+ logger.info(`Store phase complete: ${storedFiles} files stored in ${storePhaseSec}s (${stats.totalChunks} total chunks)`);
651
678
  aggregateStats(stats, finalResults);
652
679
  // Update timestamps; advance lastGitCommit ONLY on a complete pass
653
680
  manifest.lastIndexedAt = Date.now();
@@ -696,6 +723,19 @@ async function runIndexPassInner(options, logger) {
696
723
  // a successful swap this points to the new data; after an abort it's the old).
697
724
  await saveManifest(options.storePath, manifest);
698
725
  await options.keywordIndex?.save(options.storePath);
726
+ // Compact fragments and prune old version manifests so countRows() can't
727
+ // hang on accumulated versions from many add/delete cycles.
728
+ if (!aborted()) {
729
+ logger.info("Optimizing vector store (compacting fragments, pruning old versions)...");
730
+ const optimizeStart = Date.now();
731
+ try {
732
+ await effectiveStore.optimize?.();
733
+ logger.info(`Vector store optimized in ${((Date.now() - optimizeStart) / 1000).toFixed(1)}s`);
734
+ }
735
+ catch (err) {
736
+ logger.warn(`Vector store optimization failed: ${err.message}`);
737
+ }
738
+ }
699
739
  // Persist description cache
700
740
  await descCache.save();
701
741
  // Count from the store — after a successful swap, the original handle has
@@ -0,0 +1,34 @@
1
+ /**
2
+ * @fileoverview Single source of truth for the mandatory OpenCodeRAG tool-usage guidance
3
+ * that is injected into both the runtime system prompt (via plugin.ts) and the AGENTS.md
4
+ * directive (via init-helpers.ts).
5
+ */
6
+ /** Sentinel marker that opens the managed section. */
7
+ export declare const BEGIN_MARKER = "<!-- BEGIN opencode-rag -->";
8
+ /** Sentinel marker that closes the managed section. */
9
+ export declare const END_MARKER = "<!-- END opencode-rag -->";
10
+ /**
11
+ * The always-on mandatory guidance lines — tool list, decision tree,
12
+ * proactive triggers, and anti-patterns. Used verbatim by the runtime
13
+ * system prompt injector.
14
+ */
15
+ export declare const MANDATORY_GUIDANCE_LINES: readonly string[];
16
+ /**
17
+ * The conditional quirk-capture enforcement lines. Only included when
18
+ * `memory.promptEnforcement` is true.
19
+ */
20
+ export declare const QUIRK_ENFORCEMENT_LINES: readonly string[];
21
+ /**
22
+ * Build the flat guidance lines array for the runtime system-prompt injector.
23
+ * Returns a new array each call.
24
+ */
25
+ export declare function buildSystemGuidanceLines(opts: {
26
+ promptEnforcement: boolean;
27
+ }): string[];
28
+ /**
29
+ * Build the markdown-formatted AGENTS.md directive section, wrapped in sentinel
30
+ * markers so `mergeAgentsMdContent` can replace it in place on re-runs.
31
+ */
32
+ export declare function buildAgentsMdDirective(opts: {
33
+ promptEnforcement: boolean;
34
+ }): string;