docdex 0.2.23 → 0.2.24

package/assets/agents.md

@@ -1,3 +1,12 @@
+## Priority Directives (Mandatory)
+
+- Always use both memory lobes early: docdex_get_profile + docdex_memory_recall; save new facts via docdex_memory_save and new preferences via docdex_save_preference.
+- Maintain agent-specific profile memory: use your agent_id with docdex_get_profile/docdex_save_preference; store generic self-knowledge (role, tooling, preferences).
+- Use impact analysis for every code change: call HTTP /v1/graph/impact?file=... and account for inbound/outbound deps; if unavailable, state it and proceed cautiously.
+- Apply DAG reasoning for planning: prefer dependency graph facts (impact results and /v1/dag/export) to choose the right change order and scope.
+- Use Docdex tools intentionally: docdex_search/symbols/ast for repo truth; docdex_stats/files/repo_inspect/index for index health.
+- When you do not know something, run docdex_web_research (force_web=true). Web research is encouraged by default for non-repo facts and external APIs.
+
 # Docdex Agent Usage Instructions
 
 > Context for AI Agents: Docdex is your local-first Dual-Lobe Memory and Code Intelligence daemon. Unlike simple vector stores, it provides structural understanding of code (AST/Graph), persistent behavioral profiles (Agent Memory), and gated web enrichment, all strictly scoped to the local machine.
@@ -33,6 +42,7 @@ Docdex v2.1 introduces a strict separation between "facts" and "preferences." Us
 - Content: Your persona, user preferences, coding style, tooling constraints.
 - Example: "Always use Zod for validation," or "User prefers strict TypeScript types."
 - Tools: docdex_save_preference, docdex_get_profile
+- Agent-specific: Each agent should use its own agent_id and store generic self-knowledge (role, tooling, preferences).
 - Usage: Use this to "learn" from corrections. If a user corrects your style, save it here so you do not repeat the mistake in a different repo.
 
 ## Tool Capabilities (MCP & HTTP)
@@ -46,20 +56,6 @@ Standard retrieval. The daemon automatically handles the waterfall (Local -> Web
 | docdex_search | Search code, docs, and ingested libraries. Returns ranked snippets. |
 | docdex_web_research | Explicitly trigger Tier 2 web discovery (DDG + Headless Chrome). Use when you need external docs not present locally. |
 
-Tier 2 discovery providers (in fallback order when DDG HTML fails or is blocked):
-- DuckDuckGo Lite
-- SearXNG JSON (public instance or self-hosted)
-- Google Mobile
-- Brave Search API (requires DOCDEX_BRAVE_API_KEY)
-- Google Custom Search JSON API (requires DOCDEX_GOOGLE_CSE_API_KEY + DOCDEX_GOOGLE_CSE_CX)
-- Bing Web Search API (requires DOCDEX_BING_API_KEY)
-- Tavily API (requires DOCDEX_TAVILY_API_KEY)
-- Exa API (requires DOCDEX_EXA_API_KEY)
-
-Defaults and overrides:
-- Default endpoints: `https://html.duckduckgo.com/html/`, `https://lite.duckduckgo.com/lite/`, `https://searx.be/search` (built-in fallback list), `https://www.google.com/m`, `https://api.search.brave.com/res/v1/web/search`, `https://www.googleapis.com/customsearch/v1`, `https://api.bing.microsoft.com/v7.0/search`, `https://api.tavily.com/search`, `https://api.exa.ai/search`
-- Override envs: `DOCDEX_DDG_BASE_URL`, `DOCDEX_WEB_SEARXNG_URLS` (comma list, or `DOCDEX_SEARXNG_URLS`), `DOCDEX_WEB_GOOGLE_MOBILE_URL`, `DOCDEX_BRAVE_API_KEY` + `DOCDEX_BRAVE_API_URL`, `DOCDEX_GOOGLE_CSE_API_KEY` + `DOCDEX_GOOGLE_CSE_CX` + `DOCDEX_GOOGLE_CSE_API_URL`, `DOCDEX_BING_API_KEY` + `DOCDEX_BING_API_URL`, `DOCDEX_TAVILY_API_KEY` + `DOCDEX_TAVILY_API_URL`, `DOCDEX_EXA_API_KEY` + `DOCDEX_EXA_API_URL`
-
 ### B. Code Intelligence (AST & Graph)
 
 Precision tools for structural analysis. Do not rely on text search for definitions or dependencies.
@@ -80,6 +76,17 @@ Precision tools for structural analysis. Do not rely on text search for definiti
 | docdex_save_preference | Store a global user preference (Style, Tooling, Constraint). |
 | docdex_get_profile | Retrieve global preferences. |
 
+## Quick Tool Map (Often Missed)
+
+- docdex_files: List indexed docs with rel_path/doc_id/token_estimate; use to verify indexing coverage.
+- docdex_stats: Show index size, state dir, and last update time.
+- docdex_repo_inspect: Confirm normalized repo root and repo identity mapping.
+- docdex_index: Reindex the full repo or ingest specific files when stale.
+- docdex_search diff: Limit search to working tree, staged, or ref ranges; filter by paths.
+- docdex_web_research knobs: force_web, skip_local_search, repo_only, no_cache, web_limit, llm_filter_local_results, llm_model.
+- HTTP /v1/initialize: Bind a default repo root for MCP when clients omit project_root.
+- HTTP /v1/dag/export: Export the dependency graph for external analysis.
+
 ## Interaction Patterns
 
 ### 1. Reasoning Workflow
@@ -107,6 +114,21 @@ If the user asks: "Safe to delete getUser?"
 - Action: Call GET /v1/graph/impact?file=src/user.ts
 - Output: Analyze the inbound edges. If the list is not empty, it is unsafe.
 
+### 4. Non-Repo Real-World Queries (Web First)
+
+If the user asks a non-repo, real-world question (weather, news, general facts), immediately call docdex_web_research with force_web=true.
+- Resolve relative dates ("yesterday", "last week") using system time by default.
+- Do not run docdex_search unless the user explicitly wants repo-local context.
+- Assume web access is allowed unless the user forbids it; if the web call fails, report the failure and ask for a source or permission.
+
+### 5. Failure Handling (Missing Results or Errors)
+
+- Ensure project_root or repo_path is set, or call /v1/initialize to bind a default root.
+- Use docdex_repo_inspect to confirm repo identity and normalized root.
+- Use docdex_stats and docdex_files to check whether the index exists and contains files.
+- Reindex with docdex_index (or docdexd index) if the index is stale or empty.
+- Add a repo-local .docdexignore for large generated artifacts or local caches when indexing is slow.
+
 ## Operational Context
 
 ### Repository Identification

package/lib/postinstall_setup.js

@@ -563,7 +563,7 @@ function upsertClaudeInstructions(pathname, instructions) {
   return true;
 }
 
-function upsertContinueInstructions(pathname, instructions) {
+function upsertContinueJsonInstructions(pathname, instructions) {
   const { value } = readJson(pathname);
   if (typeof value !== "object" || value == null || Array.isArray(value)) return false;
   const merged = mergeInstructionText(value.systemMessage, instructions);
@@ -573,6 +573,198 @@ function upsertContinueInstructions(pathname, instructions) {
   return true;
 }
 
+function countLeadingWhitespace(line) {
+  const match = line.match(/^\s*/);
+  return match ? match[0].length : 0;
+}
+
+function isYamlTopLevelKey(line, baseIndent) {
+  const indent = countLeadingWhitespace(line);
+  if (indent > baseIndent) return false;
+  const trimmed = line.trimStart();
+  if (!trimmed || trimmed.startsWith("#") || trimmed.startsWith("-")) return false;
+  return trimmed.includes(":");
+}
+
+function hasYamlContent(lines) {
+  return lines.some((line) => {
+    const trimmed = line.trim();
+    return trimmed && !trimmed.startsWith("#");
+  });
+}
+
+function splitInlineYamlList(value) {
+  const trimmed = String(value || "").trim();
+  if (trimmed === "[]") return [];
+  if (!trimmed.startsWith("[") || !trimmed.endsWith("]")) return null;
+  const inner = trimmed.slice(1, -1);
+  const items = [];
+  let current = "";
+  let inSingle = false;
+  let inDouble = false;
+  let escaped = false;
+  for (const ch of inner) {
+    if (escaped) {
+      current += ch;
+      escaped = false;
+      continue;
+    }
+    if (ch === "\\\\") {
+      escaped = true;
+      current += ch;
+      continue;
+    }
+    if (ch === "'" && !inDouble) {
+      inSingle = !inSingle;
+      current += ch;
+      continue;
+    }
+    if (ch === '"' && !inSingle) {
+      inDouble = !inDouble;
+      current += ch;
+      continue;
+    }
+    if (ch === "," && !inSingle && !inDouble) {
+      const next = current.trim();
+      if (next) items.push(next);
+      current = "";
+      continue;
+    }
+    current += ch;
+  }
+  const next = current.trim();
+  if (next) items.push(next);
+  return items;
+}
+
+function inlineRulesToItems(value, itemIndent) {
+  const trimmed = String(value || "").trim();
+  if (!trimmed) return [];
+  const prefix = " ".repeat(itemIndent);
+  const split = splitInlineYamlList(trimmed);
+  if (split) {
+    return split.map((item) => [`${prefix}- ${item}`]).filter((item) => item[0].trim() !== `${prefix}-`);
+  }
+  return [[`${prefix}- ${trimmed}`]];
+}
+
+function buildYamlRuleBlock(itemIndent, instructions) {
+  const prefix = " ".repeat(itemIndent);
+  const contentPrefix = " ".repeat(itemIndent + 2);
+  const lines = [`${prefix}- |`];
+  for (const line of String(instructions).split(/\r?\n/)) {
+    lines.push(`${contentPrefix}${line}`);
+  }
+  return lines;
+}
+
+function rewriteContinueYamlRules(source, instructions, addDocdex) {
+  const lines = String(source || "").split(/\r?\n/);
+  const ruleLineRe = /^(\s*)rules\s*:(.*)$/;
+  let rulesIndex = -1;
+  let rulesIndent = 0;
+  let rulesInline = "";
+  for (let i = 0; i < lines.length; i += 1) {
+    const match = lines[i].match(ruleLineRe);
+    if (!match) continue;
+    rulesIndex = i;
+    rulesIndent = match[1]?.length || 0;
+    rulesInline = (match[2] || "").trim();
+    if (rulesInline.includes("#")) {
+      rulesInline = rulesInline.split("#")[0].trim();
+    }
+    if (rulesInline.startsWith("#")) rulesInline = "";
+    break;
+  }
+
+  if (rulesIndex === -1) {
+    if (!addDocdex) return null;
+    const trimmed = String(source || "").trimEnd();
+    const docdexBlock = buildYamlRuleBlock(2, instructions);
+    const prefix = trimmed ? `${trimmed}\n\n` : "";
+    return `${prefix}rules:\n${docdexBlock.join("\n")}`;
+  }
+
+  let endIndex = lines.length;
+  for (let i = rulesIndex + 1; i < lines.length; i += 1) {
+    if (isYamlTopLevelKey(lines[i], rulesIndent)) {
+      endIndex = i;
+      break;
+    }
+  }
+  const blockLines = lines.slice(rulesIndex + 1, endIndex);
+  const preLines = [];
+  const items = [];
+  let currentItem = [];
+  let itemIndent = null;
+  let startedItems = false;
+
+  for (const line of blockLines) {
+    const trimmed = line.trimStart();
+    const indent = countLeadingWhitespace(line);
+    const isItem = trimmed.startsWith("-") && indent > rulesIndent;
+    if (isItem) {
+      if (itemIndent == null) itemIndent = indent;
+      if (indent === itemIndent) {
+        if (startedItems && currentItem.length) {
+          items.push(currentItem);
+          currentItem = [];
+        }
+        startedItems = true;
+      }
+      currentItem.push(line);
+      continue;
+    }
+    if (startedItems) {
+      currentItem.push(line);
+    } else {
+      preLines.push(line);
+    }
+  }
+  if (currentItem.length) items.push(currentItem);
+
+  const inferredIndent = itemIndent == null ? rulesIndent + 2 : itemIndent;
+  if (!items.length && rulesInline) {
+    items.push(...inlineRulesToItems(rulesInline, inferredIndent));
+    rulesInline = "";
+  }
+
+  const keptItems = items.filter((item) => {
+    const text = item.join("\n");
+    return !(text.includes(DOCDEX_INFO_START_PREFIX) && text.includes(DOCDEX_INFO_END));
+  });
+
+  if (addDocdex) {
+    keptItems.push(buildYamlRuleBlock(inferredIndent, instructions));
+  }
+
+  const removeRulesBlock =
+    !addDocdex && !keptItems.length && !hasYamlContent(preLines) && !rulesInline;
+
+  const output = [];
+  output.push(...lines.slice(0, rulesIndex));
+  if (!removeRulesBlock) {
+    output.push(`${" ".repeat(rulesIndent)}rules:`);
+    output.push(...preLines);
+    for (const item of keptItems) {
+      output.push(...item);
+    }
+  }
+  output.push(...lines.slice(endIndex));
+  const next = output.join("\n");
+  return next === source ? null : next;
+}
+
+function upsertContinueYamlRules(pathname, instructions) {
+  if (!fs.existsSync(pathname)) return false;
+  const normalized = normalizeInstructionText(instructions);
+  if (!normalized) return false;
+  const current = fs.readFileSync(pathname, "utf8");
+  const updated = rewriteContinueYamlRules(current, normalized, true);
+  if (!updated) return false;
+  return writeTextFile(pathname, updated);
+}
+
 function upsertZedInstructions(pathname, instructions) {
   const { value } = readJson(pathname);
   if (typeof value !== "object" || value == null || Array.isArray(value)) return false;
@@ -586,12 +778,56 @@ function upsertZedInstructions(pathname, instructions) {
   return true;
 }
 
-function upsertVsCodeInstructions(pathname, instructionsPath) {
+function upsertVsCodeInstructionKey(value, key, instructions) {
+  const existing = typeof value[key] === "string" ? value[key] : "";
+  const merged = mergeInstructionText(existing, instructions);
+  if (!merged || merged === existing) return false;
+  value[key] = merged;
+  return true;
+}
+
+function upsertVsCodeInstructionLocations(value, instructionsDir) {
+  const key = "chat.instructionsFilesLocations";
+  const location = String(instructionsDir);
+  if (value[key] && typeof value[key] === "object" && !Array.isArray(value[key])) {
+    if (value[key][location] === true) return false;
+    value[key][location] = true;
+    return true;
+  }
+  if (Array.isArray(value[key])) {
+    if (value[key].some((entry) => entry === location)) return false;
+    value[key].push(location);
+    return true;
+  }
+  if (typeof value[key] === "string") {
+    if (value[key] === location) return false;
+    value[key] = [value[key], location];
+    return true;
+  }
+  value[key] = { [location]: true };
+  return true;
+}
+
+function upsertVsCodeInstructions(pathname, instructions, instructionsDir) {
   const { value } = readJson(pathname);
   if (typeof value !== "object" || value == null || Array.isArray(value)) return false;
-  const key = "copilot.chat.codeGeneration.instructions";
-  if (value[key] === instructionsPath) return false;
-  value[key] = instructionsPath;
+  const normalized = normalizeInstructionText(instructions);
+  if (!normalized) return false;
+  let updated = false;
+  if (upsertVsCodeInstructionKey(value, "github.copilot.chat.codeGeneration.instructions", instructions)) {
+    updated = true;
+  }
+  if (upsertVsCodeInstructionKey(value, "copilot.chat.codeGeneration.instructions", instructions)) {
+    updated = true;
+  }
+  if (value["github.copilot.chat.codeGeneration.useInstructionFiles"] !== true) {
+    value["github.copilot.chat.codeGeneration.useInstructionFiles"] = true;
+    updated = true;
+  }
+  if (upsertVsCodeInstructionLocations(value, instructionsDir)) {
+    updated = true;
+  }
+  if (!updated) return false;
   writeJson(pathname, value);
   return true;
 }
@@ -921,6 +1157,12 @@ function clientInstructionPaths() {
   const appData = process.env.APPDATA || path.join(home, "AppData", "Roaming");
   const userProfile = process.env.USERPROFILE || home;
   const vscodeGlobalInstructions = path.join(home, ".vscode", "global_instructions.md");
+  const vscodeInstructionsDir = path.join(home, ".vscode", "instructions");
+  const vscodeInstructionsFile = path.join(vscodeInstructionsDir, "docdex.md");
+  const continueRoot = path.join(userProfile, ".continue");
+  const continueJson = path.join(continueRoot, "config.json");
+  const continueYaml = path.join(continueRoot, "config.yaml");
+  const continueYml = path.join(continueRoot, "config.yml");
   const windsurfGlobalRules = path.join(userProfile, ".codeium", "windsurf", "memories", "global_rules.md");
   const rooRules = path.join(home, ".roo", "rules", "docdex.md");
   const pearaiAgent = path.join(home, ".config", "pearai", "agent.md");
@@ -932,10 +1174,14 @@ function clientInstructionPaths() {
     case "win32":
       return {
         claude: path.join(appData, "Claude", "claude_desktop_config.json"),
-        continue: path.join(userProfile, ".continue", "config.json"),
+        continue: continueJson,
+        continueYaml,
+        continueYml,
         zed: path.join(appData, "Zed", "settings.json"),
         vscodeSettings: path.join(appData, "Code", "User", "settings.json"),
         vscodeGlobalInstructions,
+        vscodeInstructionsDir,
+        vscodeInstructionsFile,
         windsurfGlobalRules,
         rooRules,
         pearaiAgent,
@@ -947,10 +1193,14 @@ function clientInstructionPaths() {
     case "darwin":
       return {
         claude: path.join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json"),
-        continue: path.join(home, ".continue", "config.json"),
+        continue: continueJson,
+        continueYaml,
+        continueYml,
         zed: path.join(home, ".config", "zed", "settings.json"),
         vscodeSettings: path.join(home, "Library", "Application Support", "Code", "User", "settings.json"),
         vscodeGlobalInstructions,
+        vscodeInstructionsDir,
+        vscodeInstructionsFile,
         windsurfGlobalRules,
         rooRules,
         pearaiAgent,
@@ -962,10 +1212,14 @@ function clientInstructionPaths() {
     default:
       return {
         claude: path.join(home, ".config", "Claude", "claude_desktop_config.json"),
-        continue: path.join(home, ".continue", "config.json"),
+        continue: continueJson,
+        continueYaml,
+        continueYml,
         zed: path.join(home, ".config", "zed", "settings.json"),
         vscodeSettings: path.join(home, ".config", "Code", "User", "settings.json"),
         vscodeGlobalInstructions,
+        vscodeInstructionsDir,
+        vscodeInstructionsFile,
         windsurfGlobalRules,
         rooRules,
         pearaiAgent,
@@ -1045,8 +1299,15 @@ function applyAgentInstructions({ logger } = {}) {
       upsertPromptFile(paths.vscodeGlobalInstructions, instructions, { prepend: true })
     );
   }
-  if (paths.vscodeSettings && paths.vscodeGlobalInstructions) {
-    safeApply("vscode-settings", () => upsertVsCodeInstructions(paths.vscodeSettings, paths.vscodeGlobalInstructions));
+  if (paths.vscodeInstructionsFile) {
+    safeApply("vscode-instructions-file", () =>
+      upsertPromptFile(paths.vscodeInstructionsFile, instructions, { prepend: true })
+    );
+  }
+  if (paths.vscodeSettings && paths.vscodeInstructionsDir) {
+    safeApply("vscode-settings", () =>
+      upsertVsCodeInstructions(paths.vscodeSettings, instructions, paths.vscodeInstructionsDir)
+    );
   }
   if (paths.windsurfGlobalRules) {
     safeApply("windsurf", () => upsertPromptFile(paths.windsurfGlobalRules, instructions, { prepend: true }));
@@ -1060,18 +1321,25 @@ function applyAgentInstructions({ logger } = {}) {
   if (paths.claude) {
     safeApply("claude", () => upsertClaudeInstructions(paths.claude, instructions));
   }
-  if (paths.continue) {
-    safeApply("continue", () => upsertContinueInstructions(paths.continue, instructions));
+  const continueYamlExists =
+    (paths.continueYaml && fs.existsSync(paths.continueYaml)) ||
+    (paths.continueYml && fs.existsSync(paths.continueYml));
+  if (continueYamlExists) {
+    if (paths.continueYaml && fs.existsSync(paths.continueYaml)) {
+      safeApply("continue-yaml", () => upsertContinueYamlRules(paths.continueYaml, instructions));
+    }
+    if (paths.continueYml && fs.existsSync(paths.continueYml)) {
+      safeApply("continue-yml", () => upsertContinueYamlRules(paths.continueYml, instructions));
+    }
+    if (paths.continue && fs.existsSync(paths.continue)) {
+      safeApply("continue-json", () => upsertContinueJsonInstructions(paths.continue, instructions));
+    }
+  } else if (paths.continue) {
+    safeApply("continue-json", () => upsertContinueJsonInstructions(paths.continue, instructions));
   }
   if (paths.zed) {
     safeApply("zed", () => upsertZedInstructions(paths.zed, instructions));
   }
-  if (paths.aiderConfig) {
-    safeApply("aider", () => upsertYamlInstruction(paths.aiderConfig, "system-prompt", instructions));
-  }
-  if (paths.gooseConfig) {
-    safeApply("goose", () => upsertYamlInstruction(paths.gooseConfig, "instructions", instructions));
-  }
   if (paths.openInterpreterConfig) {
     safeApply("open-interpreter", () =>
       upsertYamlInstruction(paths.openInterpreterConfig, "system_message", instructions)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "docdex",
-  "version": "0.2.23",
+  "version": "0.2.24",
   "mcpName": "io.github.bekirdag/docdex",
   "description": "Local-first documentation and code indexer with HTTP/MCP search, AST, and agent memory.",
   "bin": {