karajan-code 2.22.0 → 2.24.0

package/README.md

@@ -345,6 +345,49 @@ Opt out: set `telemetry: false` in `~/.karajan/kj.config.yml`
 
 ## Recent releases
 
+> **v2.24.0 released** — Minor. **RAG Camino C — pre-loop auto-retrieval** (KJC-PCS-0049). After v2.23.0 taught the agents that `kj_rag_query` exists, Karajan now injects prior context for them automatically: a new pre-loop stage queries the vec store with the task description and prepends the top-K chunks to the task before any LLM call.
+>
+> Opt-in: `config.rag.preload.enabled = false` by default. Five guards before the retrieval fires (`disabled`, `no-task`, empty corpus, no hits, error). The stage never throws — best-effort enrichment, opt-out by default, opt-back-out on failure. When all guards pass, the task receives an extra block:
+>
+> ```markdown
+> ## Prior context from RAG (auto-retrieved, top N)
+>
+> ### [plan · AUTH-1] /path/to/plan.json
+>
+> <chunk text, truncated 600 chars>
+> ```
+>
+> Because `task` flows through `runPlanningPhases` to researcher / architect / planner via parameter passing, **one mutation feeds six consumers** — no per-stage prompt wrapper, no fan-out through `pipelineFlags`. Triage runs first because its `roleOverrides` gate the rest of the pipeline; the RAG stage runs right after, before `domainCurator`.
+>
+> Workflow:
+>
+> ```bash
+> kj onboard
+> kj rag index
+> yq -i '.rag.preload.enabled = true' ~/.karajan/kj.config.yml
+> kj run task.md  # researcher/architect/planner/coder all see prior context
+> ```
+>
+> **Coming in v2.25.0+**: Camino B (`/kj-rag-query` slash command for Skills hosts), Camino D (Brain decisor heuristic deciding **when** to pre-fetch based on task complexity — refinement on top of C), chokidar watcher for live re-indexing, AST source chunker, BM25 + cosine hybrid scoring.
+>
+> **v2.23.0 released** — Minor. **RAG exposed to agents and humans alike** (KJC-PCS-0049, Steps 7+8+Camino A). After v2.22.0's CLI MVP, the corpus is now reachable from three more places.
+>
+> 1. **MCP tools** `kj_rag_query` + `kj_rag_index` (PR #815). Any MCP-connected agent — Claude Desktop, Cursor, Claude Code, Karajan's own roles — can call them. Tool count 25 → 27. Empty store responds `empty: true` so agents have a deterministic recovery signal.
+> 2. **HU Board RAG panel** (PR #816). New input + scope dropdown (All / Plans / Onboarding / Code) + Search button + results pane between the preflight panel and the kanban. `POST /api/rag/query` backs it.
+> 3. **Role templates teach agents about the tool** (PR #817). `templates/roles/{coder,researcher,architect,planner,spec-reviewer}.md` each gain a tailored 'Prior context (RAG, opt-in)' section. Shared rule: when the store is empty, proceed without retrieval — do NOT block, do NOT ask the human to seed.
+>
+> Workflow:
+>
+> ```bash
+> kj onboard
+> kj rag index
+> kj plan generate task.md --use-onboarding
+> # From here on, every agent invocation can call kj_rag_query via MCP.
+> # Humans get the panel on the Board.
+> ```
+>
+> **Coming in v2.24.0+**: Camino B (`/kj-rag-query` slash command for hosts in Skills mode without MCP), Camino C (automatic pre-loop stage that pre-fetches retrieval and prepends it to the coder/researcher/architect prompt without the agent having to ask), Camino D (Brain decisor heuristic for when retrieval is worth the tokens), chokidar watcher for live re-indexing, AST-aware source chunker, BM25 + cosine hybrid scoring.
+>
 > **v2.22.0 released** — Minor. **Project RAG MVP** (KJC-PCS-0049). Karajan now indexes its plans + onboarding briefs (and optionally project sources) into a local vector store and lets you query them semantically from the CLI. Six PRs.
 >
 > | Step | PR | Module |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "karajan-code",
-  "version": "2.22.0",
+  "version": "2.24.0",
   "description": "Local multi-agent coding orchestrator with TDD, SonarQube, and code review pipeline",
   "type": "module",
   "license": "AGPL-3.0",

package/packages/hu-board/public/app.js

@@ -590,6 +590,18 @@ async function renderBoard() {
           Comprobando estado del proyecto…
         </div>
       </div>
+      <div id="rag-panel" class="rag-panel" style="margin:8px 0;display:flex;gap:8px;align-items:flex-start;flex-wrap:wrap;">
+        <input id="rag-query" type="text" placeholder="🔍 RAG search: ask anything about this project's plans / onboarding / code…"
+               style="flex:1;min-width:260px;padding:6px 10px;font-size:0.85rem;background:var(--bg-primary);border:1px solid var(--border);color:var(--text);border-radius:var(--radius-sm);" />
+        <select id="rag-scope" style="padding:6px 10px;font-size:0.85rem;background:var(--bg-primary);border:1px solid var(--border);color:var(--text);border-radius:var(--radius-sm);">
+          <option value="all">All</option><option value="plans">Plans</option><option value="onboarding">Onboarding</option><option value="code">Code</option>
+        </select>
+        <button id="rag-search-btn" class="control-btn"
+                style="padding:6px 12px;font-size:0.85rem;background:var(--color-blue,#3b82f6);color:#fff;border:none;border-radius:var(--radius-sm);cursor:pointer;font-weight:600;">
+          Search
+        </button>
+        <div id="rag-results" style="flex-basis:100%;font-size:0.8rem;color:var(--text-muted);"></div>
+      </div>
       <div id="plan-rollup-banner"></div>
       <div class="kanban">
         ${visibleColumns.map((c) => renderKanbanColumn(c.title, c.cls, c.rows)).join('')}
@@ -694,6 +706,39 @@ async function renderBoard() {
         }
       });
     }
+
+    // RAG search panel (KJC-PCS-0049 Step 8) — input + scope + render hits.
+    const ragInput = document.getElementById('rag-query');
+    const ragScope = document.getElementById('rag-scope');
+    const ragBtn = document.getElementById('rag-search-btn');
+    const ragResults = document.getElementById('rag-results');
+    async function runRagSearch() {
+      const text = ragInput?.value?.trim();
+      if (!text) return;
+      ragResults.innerHTML = '<em>Searching…</em>';
+      try {
+        const res = await fetch('/api/rag/query', {
+          method: 'POST', headers: { 'Content-Type': 'application/json' },
+          body: JSON.stringify({ text, topK: 5, scope: ragScope?.value || 'all' }),
+        });
+        const body = await res.json();
+        if (!res.ok) { ragResults.innerHTML = `<span style="color:var(--color-red);">Error: ${esc(body.error || res.status)}</span>`; return; }
+        if (body.empty) { ragResults.innerHTML = '<em>No chunks indexed yet. Run <code>kj rag index</code> in this project\'s terminal first.</em>'; return; }
+        if (body.hits.length === 0) { ragResults.innerHTML = '<em>No hits.</em>'; return; }
+        ragResults.innerHTML = body.hits.map((h) => {
+          const label = h.metadata?.hu_id || h.metadata?.symbol || h.metadata?.headingPath?.join(' > ') || 'block';
+          const snippet = h.text.length > 240 ? `${esc(h.text.slice(0, 240))}…` : esc(h.text);
+          return `<div style="border-left:3px solid var(--color-blue,#3b82f6);padding:6px 10px;margin:6px 0;background:var(--bg-primary);">
+            <div style="font-size:0.75rem;color:var(--text-muted);margin-bottom:4px;">[${esc(h.kind)} · ${esc(label)} · score=${h.score.toFixed(4)}] ${esc(h.source)}</div>
+            <div style="font-size:0.85rem;white-space:pre-wrap;">${snippet}</div>
+          </div>`;
+        }).join('');
+      } catch (err) {
+        ragResults.innerHTML = `<span style="color:var(--color-red);">Error: ${esc(err.message)}</span>`;
+      }
+    }
+    ragBtn?.addEventListener('click', runRagSearch);
+    ragInput?.addEventListener('keydown', (e) => { if (e.key === 'Enter') runRagSearch(); });
   } catch (err) {
     app.innerHTML = `<div class="empty-state"><div class="empty-state__title">Error loading board</div><div class="empty-state__text">${esc(err.message)}</div></div>`;
   }

package/packages/hu-board/src/routes/api.js

@@ -1279,4 +1279,31 @@ router.post('/sync', (_req, res) => {
   }
 });
 
+/**
+ * POST /api/rag/query - Semantic search over the indexed RAG corpus
+ * (KJC-PCS-0049 Step 8). Body: { text, topK?, scope? }. Response:
+ * { hits: [...], empty, topK, scope }. Empty store carries empty=true
+ * so the UI can show an actionable hint instead of just empty results.
+ */
+router.post('/rag/query', async (req, res) => {
+  const { text, topK = 5, scope = 'all' } = req.body || {};
+  if (typeof text !== 'string' || text.length === 0) {
+    return res.status(400).json({ error: "'text' is required (non-empty string)" });
+  }
+  try {
+    const { openVecStore, countChunks } = await import('../../../../src/rag/vec-store.js');
+    const { OllamaEmbedder } = await import('../../../../src/rag/embedder.js');
+    const { query } = await import('../../../../src/rag/retriever.js');
+    const db = openVecStore({ dim: 768 });
+    try {
+      if (countChunks(db) === 0) return res.json({ hits: [], empty: true, topK, scope });
+      const embedder = new OllamaEmbedder({});
+      const hits = await query(db, embedder, text, { topK: Number(topK), scope });
+      res.json({ hits, empty: false, topK, scope });
+    } finally { db.close(); }
+  } catch (err) {
+    res.status(500).json({ error: err.message });
+  }
+});
+
 export default router;

package/src/mcp/handlers/rag-handler.js

@@ -0,0 +1,57 @@
+// KJC-PCS-0049 Step 7 — MCP handlers calling src/rag/* directly. Imports
+// from src/commands/* are forbidden by the layer-boundaries test
+// (MCP and CLI are peer layers); we hit the pure rag/* modules instead.
+import { countChunks, openVecStore } from "../../rag/vec-store.js";
+import { OllamaEmbedder } from "../../rag/embedder.js";
+import { indexProject } from "../../rag/indexer.js";
+import { query } from "../../rag/retriever.js";
+import { getKarajanHome } from "../../utils/paths.js";
+import { resolveProjectDir, buildConfig, responseText, failPayload } from "../shared-helpers.js";
+
+const silentLogger = {
+  info: () => {}, warn: () => {}, error: () => {}, debug: () => {}, setContext: () => {},
+};
+
+function makeEmbedder(config) {
+  const cfg = config?.rag?.embedder || {};
+  return new OllamaEmbedder({ url: cfg.url, model: cfg.model, dim: cfg.dim });
+}
+
+export async function handleRagQuery(args, server) {
+  const text = args?.text;
+  if (typeof text !== "string" || text.length === 0) {
+    return failPayload("kj_rag_query: 'text' is required and must be a non-empty string");
+  }
+  try {
+    const projectDir = await resolveProjectDir(server, args?.projectDir);
+    const config = await buildConfig({ ...args, projectDir }, "rag-query");
+    const topK = Number(args?.topK) || 5;
+    const scope = args?.scope || "all";
+    const db = openVecStore({ dim: config?.rag?.embedder?.dim || 768 });
+    try {
+      if (countChunks(db) === 0) return responseText({ hits: [], empty: true, topK, scope });
+      const hits = await query(db, makeEmbedder(config), text, { topK, scope });
+      return responseText({ hits, empty: false, topK, scope });
+    } finally { db.close(); }
+  } catch (err) {
+    return failPayload(`kj_rag_query failed: ${err.message}`);
+  }
+}
+
+export async function handleRagIndex(args, server) {
+  try {
+    const projectDir = await resolveProjectDir(server, args?.projectDir);
+    const config = await buildConfig({ ...args, projectDir }, "rag-index");
+    const db = openVecStore({ dim: config?.rag?.embedder?.dim || 768 });
+    try {
+      const totals = await indexProject(projectDir, {
+        db, embedder: makeEmbedder(config),
+        karajanHome: getKarajanHome(), logger: silentLogger,
+        withSources: Boolean(args?.withSources),
+      });
+      return responseText(totals);
+    } finally { db.close(); }
+  } catch (err) {
+    return failPayload(`kj_rag_index failed: ${err.message}`);
+  }
+}

package/src/mcp/server-handlers.js

@@ -39,6 +39,7 @@ import {
   handleScan, handleBoard, handleUndo, handleClean
 } from "./handlers/management-handlers.js";
 import { handleHu, handleSkills, handleSuggest } from "./handlers/hu-handlers.js";
+import { handleRagQuery, handleRagIndex } from "./handlers/rag-handler.js";
 
 export async function handleToolCall(name, args, server, extra) {
   const a = asObject(args);
@@ -68,6 +69,8 @@ export async function handleToolCall(name, args, server, extra) {
     kj_skills:      (a) => handleSkills(a),
     kj_undo:        (a, server) => handleUndo(a, server),
     kj_clean:       (a) => handleClean(a),
+    kj_rag_query:   (a, server) => handleRagQuery(a, server),
+    kj_rag_index:   (a, server) => handleRagIndex(a, server),
   }[name];
   if (handler) {
     return handler(a, server, extra);

package/src/mcp/tools.js

@@ -403,5 +403,30 @@ export const tools = [
         huDays: { type: "number", description: "Keep HU story batches for this many days. Default: 14." }
       }
     }
+  },
+  {
+    name: "kj_rag_query",
+    description: "Run a semantic search across the indexed Karajan plans, onboarding briefs and (optionally) project source code. Returns the top-K nearest chunks with their metadata so the agent can quote prior decisions, locate where a concept was last implemented, or pull architectural context without re-walking the repo. The store is empty until `kj_rag_index` (or `kj rag index` CLI) has been run at least once on this project; in that case the response carries `empty: true`.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        text: { type: "string", description: "The natural-language query." },
+        topK: { type: "number", description: "Max number of hits to return. Default: 5." },
+        scope: { type: "string", description: "Filter by chunk kind: 'plans' | 'code' | 'onboarding' | 'all' (default)." },
+        projectDir: { type: "string", description: "Absolute path to the project directory" }
+      },
+      required: ["text"]
+    }
+  },
+  {
+    name: "kj_rag_index",
+    description: "(Re)index this project's Karajan-managed assets — every plan-*.json under ~/.karajan/plans/<slug>/ plus the onboarding brief at ~/.karajan/onboarding/<slug>.md — into the local vector store. Idempotent: a re-run replaces the chunks for each source, never appends duplicates. Pass `withSources: true` to also index the project's JS/TS files (skips node_modules / .git / dist / etc.). Required as a one-time bootstrap before kj_rag_query returns anything; safe to re-run after every `kj plan generate`.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        withSources: { type: "boolean", description: "Also index .js/.ts files under projectDir. Default false." },
+        projectDir: { type: "string", description: "Absolute path to the project directory" }
+      }
+    }
   }
 ];

package/src/orchestrator/drivers/pre-loop.js

@@ -40,6 +40,7 @@ import {
   runResearcherStage, runArchitectStage, runPlannerStage,
   runDiscoverStage, runHuReviewerStage,
 } from "../pre-loop-stages.js";
+import { runRagContextStage } from "../stages/rag-context-stage.js";
 // TSK-0336: triage goes through the StageRegistry / StageExecutor contract
 // so canRun / execute / onFailure are actually exercised in production.
 // The registry is a singleton with TriageStage / CoderStage / ReviewerStage
@@ -123,6 +124,20 @@ export async function runPreLoopStages({ config, logger, emitter, eventBase, ses
   applyTriageOverrides(pipelineFlags, triageResult.roleOverrides);
   await persistStage("triage", triageResult.stageResult);
 
+  // --- RAG preload (KJC-PCS-0049 Camino C, opt-in) ---
+  // Runs only when config.rag.preload.enabled === true. Queries the local
+  // vector store with the task description, prepends the top-K chunks to
+  // the task as a markdown block so researcher/architect/planner/coder
+  // all see prior context without having to call kj_rag_query themselves.
+  // Empty store / network / etc. degrade to no-op via the stage itself.
+  if (!resumeSkip("ragPreload")) {
+    const ragRes = await runRagContextStage({ config, logger, emitter, eventBase, task });
+    if (!ragRes.skipped && ragRes.contextBlock) {
+      task = `${task}\n\n${ragRes.contextBlock}`;
+      await persistStage("ragPreload", { hits: ragRes.hits.length, scope: config?.rag?.preload?.scope || "all" });
+    }
+  }
+
   // --- Brain decisor (opt-in, intent-driven routing) ---
   //
   // When enabled, translates triage output + task text + CLI overrides into a

package/src/orchestrator/stages/rag-context-stage.js

@@ -0,0 +1,60 @@
+// KJC-PCS-0049 Camino C — pre-loop RAG retrieval stage.
+// Runs between triage and domainCurator. When `config.rag.preload.enabled`
+// is true and the corpus has chunks, queries the retriever with the
+// task text and returns a markdown block ready to prepend to the task.
+// Failures degrade gracefully (warn + continue); the rest of the pipeline
+// never sees an exception from this stage.
+import { emitProgress, makeEvent } from "../../utils/events.js";
+import { openVecStore, countChunks } from "../../rag/vec-store.js";
+import { OllamaEmbedder } from "../../rag/embedder.js";
+import { query } from "../../rag/retriever.js";
+
+const DEFAULT_TOP_K = 5;
+const DEFAULT_SCOPE = "all";
+const MAX_CHUNK_CHARS = 600;
+
+/**
+ * Runs the RAG preload stage. Returns:
+ *   { skipped: true, reason }                  — when disabled / empty / error
+ *   { skipped: false, contextBlock, hits }     — markdown block + raw hits
+ *
+ * The caller decides whether to mutate `task` with the contextBlock. We
+ * keep that responsibility at the call site so this module stays pure.
+ */
+export async function runRagContextStage({ config, logger, emitter, eventBase, task }) {
+  const preload = config?.rag?.preload;
+  if (!preload?.enabled) return { skipped: true, reason: "disabled" };
+  if (typeof task !== "string" || task.length === 0) return { skipped: true, reason: "no-task" };
+  try {
+    const dim = config?.rag?.embedder?.dim || 768;
+    const db = openVecStore({ dim });
+    try {
+      if (countChunks(db) === 0) {
+        logger?.info?.("[rag-preload] corpus is empty, skipping (run `kj rag index` to seed)");
+        return { skipped: true, reason: "empty" };
+      }
+      const topK = preload.topK || DEFAULT_TOP_K;
+      const scope = preload.scope || DEFAULT_SCOPE;
+      const cfg = config?.rag?.embedder || {};
+      const embedder = new OllamaEmbedder({ url: cfg.url, model: cfg.model, dim });
+      const hits = await query(db, embedder, task, { topK, scope });
+      if (hits.length === 0) return { skipped: true, reason: "no-hits" };
+      const lines = hits.map((h) => {
+        const label = h.metadata?.hu_id || h.metadata?.symbol || h.metadata?.headingPath?.join(" > ") || "block";
+        const text = h.text.length > MAX_CHUNK_CHARS ? `${h.text.slice(0, MAX_CHUNK_CHARS)}…` : h.text;
+        return `### [${h.kind} · ${label}] ${h.source}\n\n${text}`;
+      });
+      const contextBlock = `## Prior context from RAG (auto-retrieved, top ${hits.length})\n\n${lines.join("\n\n")}`;
+      emitProgress(emitter, makeEvent("rag-preload:hits", { ...eventBase, stage: "rag-preload" }, {
+        message: `RAG preload: injected ${hits.length} chunk(s) into the task context`,
+        detail: { topK, scope, hits: hits.length },
+      }));
+      return { skipped: false, contextBlock, hits };
+    } finally {
+      db.close();
+    }
+  } catch (err) {
+    logger?.warn?.(`[rag-preload] failed (non-blocking): ${err.message}`);
+    return { skipped: true, reason: "error", error: err.message };
+  }
+}

package/templates/roles/architect.md

@@ -70,3 +70,7 @@ Karajan projects MAY enforce a CI gate that fails any PR whose net delta exceeds
 - The gate counts the SUM of every changed file, not per-file. Tests count too. 5 files × 40 LOC = 200 = on the limit.
 - Excluded from the count: lockfiles, snapshots, `dist/`, `node_modules/`, generated `tests/_diet/`, `public/docs/`. Source + tests count.
 - Token-economy: oversized PRs get rejected at CI and the work is redone — partitioning upfront saves the round-trip.
+
+## Prior context (RAG, opt-in)
+
+If a design decision touches an area the project already shaped (layering, patterns, contract style), call the `kj_rag_query` MCP tool with `{ text, topK: 3, scope: "all" }` and align your `tradeoffs` / `patterns` with what the prior plans already used. When the response carries `empty: true`, the corpus has not been indexed — proceed without it; do NOT block on retrieval.

package/templates/roles/coder.md

@@ -110,3 +110,7 @@ Return a JSON object:
   "summary": "Human-readable summary of changes"
 }
 ```
+
+## Prior context (RAG, opt-in)
+
+If you need to know how the project handled a similar concern before (auth, error model, retry policy, naming convention…), call the `kj_rag_query` MCP tool with `{ text, topK: 3, scope: "all" }`. It returns the closest chunks from prior plans + onboarding brief + indexed sources. When the response carries `empty: true`, the corpus has not been indexed yet — proceed without it; do NOT block on retrieval, and do NOT ask the human to run `kj rag index`. Use sparingly: one query per concern, not per file.

package/templates/roles/planner.md

@@ -55,3 +55,7 @@ Karajan projects MAY enforce a CI gate that fails any PR whose net delta exceeds
 - The gate counts the SUM of every changed file, not per-file. Tests count too. 5 files × 40 LOC = 200 = on the limit.
 - Excluded from the count: lockfiles, snapshots, `dist/`, `node_modules/`, generated `tests/_diet/`, `public/docs/`. Source + tests count.
 - Token-economy: oversized PRs get rejected at CI and the work is redone — partitioning upfront saves the round-trip.
+
+## Prior context (RAG, opt-in)
+
+Before emitting blocked_by graphs, dependencies, or reuse markers, query the local RAG corpus via the `kj_rag_query` MCP tool with `{ text, topK: 5, scope: "plans" }` to see how previous plans wired similar HUs. If a prior plan already produced the utility/spike the new plan would need, mark `reuse: ["<that-hu-id>"]` instead of re-implementing. When `empty: true`, the corpus has not been indexed — proceed without retrieval; do NOT block on it.

package/templates/roles/researcher.md

@@ -35,3 +35,7 @@ You are the **Researcher** in a multi-role AI pipeline. Your job is to investiga
   "summary": "Research complete: 5 files affected, 2 risks identified"
 }
 ```
+
+## Prior context (RAG, opt-in)
+
+Before re-walking the repo, query the local RAG corpus via the `kj_rag_query` MCP tool with `{ text, topK: 5, scope: "plans" }` to surface decisions, hot files and ADR mentions captured in past plans + onboarding brief. When the response carries `empty: true`, the corpus has not been indexed yet — proceed with file-level research; do NOT block on retrieval. Quote retrieved evidence as bullet items under your `affects` list when relevant; do NOT paste raw chunks into the summary.

package/templates/roles/spec-reviewer.md

@@ -36,3 +36,7 @@ is the WORST of any finding, or `ok` if findings is empty.
 
 Direct, accionable. No moralising. `suggestion` must be concrete (a paste-able
 sentence), not vague advice.
+
+## Prior context (RAG, opt-in)
+
+When evaluating whether the spec duplicates a prior plan's scope or contradicts an ADR, call the `kj_rag_query` MCP tool with `{ text, topK: 3, scope: "all" }` using a phrase from the spec as the query. If retrieved chunks describe overlapping HUs already approved, surface the conflict via a finding with `kind: "scope_overlap"` and reference the prior HU id. When `empty: true`, the corpus has not been indexed — proceed without retrieval; do NOT block on it.