karajan-code 2.23.0 → 2.25.0

package/README.md

@@ -345,6 +345,39 @@ Opt out: set `telemetry: false` in `~/.karajan/kj.config.yml`
 
 ## Recent releases
 
+> **v2.25.0 released** — Minor. **RAG Camino B + Camino D** (KJC-PCS-0049). Closes the consumer-surface plan. Skills hosts can now invoke RAG via `/kj-rag-query` without MCP, and the pre-loop retrieval stage from v2.24.0 only fires when triage signals make it worthwhile.
+>
+> Camino B — `/kj-rag-query <text> [--scope <s>] [--top-k <n>]` slash command shipped by `kj init` to `.claude/commands/`. Thin wrapper over `kj rag query`; passes flags through, surfaces `empty:true` as a one-line hint, renders hits as background context rather than raw JSON. For Claude Code / Cursor instances loaded without MCP.
+>
+> Camino D — Brain decisor heuristic in `src/orchestrator/stages/rag-preload-decisor.js`. New `config.rag.preload.policy`: `always` (v2.24.0 behaviour, kept for back-compat), `never` (benchmarking), `auto` (default). In `auto` mode, retrieval fires when triage decomposes, level ∈ {complex, high, epic}, task body ≥ 200 chars, or `config.rag.preload.brownfield` is set. Otherwise the stage persists `{ skipped: true, reason: 'auto:low-value' }` and the pipeline pays no retrieval cost on trivial tasks.
+>
+> **Coming in v2.26.0+**: chokidar watcher for live re-indexing, AST source chunker (tree-sitter / `@babel/parser`), BM25 + cosine hybrid scoring.
+>
+> **v2.24.0 released** — Minor. **RAG Camino C — pre-loop auto-retrieval** (KJC-PCS-0049). After v2.23.0 taught the agents that `kj_rag_query` exists, Karajan now injects prior context for them automatically: a new pre-loop stage queries the vec store with the task description and prepends the top-K chunks to the task before any LLM call.
+>
+> Opt-in: `config.rag.preload.enabled = false` by default. Five guards before the retrieval fires (`disabled`, `no-task`, empty corpus, no hits, error). The stage never throws — best-effort enrichment, opt-out by default, opt-back-out on failure. When all guards pass, the task receives an extra block:
+>
+> ```markdown
+> ## Prior context from RAG (auto-retrieved, top N)
+>
+> ### [plan · AUTH-1] /path/to/plan.json
+>
+> <chunk text, truncated 600 chars>
+> ```
+>
+> Because `task` flows through `runPlanningPhases` to researcher / architect / planner via parameter passing, **one mutation feeds six consumers** — no per-stage prompt wrapper, no fan-out through `pipelineFlags`. Triage runs first because its `roleOverrides` gate the rest of the pipeline; the RAG stage runs right after, before `domainCurator`.
+>
+> Workflow:
+>
+> ```bash
+> kj onboard
+> kj rag index
+> yq -i '.rag.preload.enabled = true' ~/.karajan/kj.config.yml
+> kj run task.md  # researcher/architect/planner/coder all see prior context
+> ```
+>
+> **Coming in v2.25.0+**: Camino B (`/kj-rag-query` slash command for Skills hosts), Camino D (Brain decisor heuristic deciding **when** to pre-fetch based on task complexity — refinement on top of C), chokidar watcher for live re-indexing, AST source chunker, BM25 + cosine hybrid scoring.
+>
 > **v2.23.0 released** — Minor. **RAG exposed to agents and humans alike** (KJC-PCS-0049, Steps 7+8+Camino A). After v2.22.0's CLI MVP, the corpus is now reachable from three more places.
 >
 > 1. **MCP tools** `kj_rag_query` + `kj_rag_index` (PR #815). Any MCP-connected agent — Claude Desktop, Cursor, Claude Code, Karajan's own roles — can call them. Tool count 25 → 27. Empty store responds `empty: true` so agents have a deterministic recovery signal.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "karajan-code",
-  "version": "2.23.0",
+  "version": "2.25.0",
   "description": "Local multi-agent coding orchestrator with TDD, SonarQube, and code review pipeline",
   "type": "module",
   "license": "AGPL-3.0",

package/src/commands/init.js

@@ -544,7 +544,7 @@ async function installSkills(logger, interactive) {
 
   if (installed > 0) {
     logger.info(`Installed ${installed} Karajan skill(s) in .claude/commands/`);
-    logger.info("Available as slash commands: /kj-run, /kj-code, /kj-review, /kj-test, /kj-security, /kj-discover, /kj-architect, /kj-sonar");
+    logger.info("Available as slash commands: /kj-run, /kj-code, /kj-review, /kj-test, /kj-security, /kj-discover, /kj-architect, /kj-sonar, /kj-rag-query");
   } else {
     logger.info("All skills already installed.");
   }

package/src/orchestrator/drivers/pre-loop.js

@@ -40,6 +40,8 @@ import {
   runResearcherStage, runArchitectStage, runPlannerStage,
   runDiscoverStage, runHuReviewerStage,
 } from "../pre-loop-stages.js";
+import { runRagContextStage } from "../stages/rag-context-stage.js";
+import { shouldPreloadRag } from "../stages/rag-preload-decisor.js";
 // TSK-0336: triage goes through the StageRegistry / StageExecutor contract
 // so canRun / execute / onFailure are actually exercised in production.
 // The registry is a singleton with TriageStage / CoderStage / ReviewerStage
@@ -123,6 +125,34 @@ export async function runPreLoopStages({ config, logger, emitter, eventBase, ses
   applyTriageOverrides(pipelineFlags, triageResult.roleOverrides);
   await persistStage("triage", triageResult.stageResult);
 
+  // --- RAG preload (KJC-PCS-0049 Camino C, opt-in) ---
+  // Runs only when config.rag.preload.enabled === true. Queries the local
+  // vector store with the task description, prepends the top-K chunks to
+  // the task as a markdown block so researcher/architect/planner/coder
+  // all see prior context without having to call kj_rag_query themselves.
+  // Empty store / network / etc. degrade to no-op via the stage itself.
+  if (!resumeSkip("ragPreload")) {
+    // KJC-TSK-0434 (Camino D): Brain decisor heuristic — skip retrieval
+    // when triage + task shape say it would be low-value. Defaults to
+    // policy=auto; `always`/`never` are escape hatches.
+    const decision = shouldPreloadRag({ triage: triageResult.stageResult, task, config });
+    if (!decision.pull) {
+      await persistStage("ragPreload", { skipped: true, reason: decision.reason });
+    } else {
+      const ragRes = await runRagContextStage({ config, logger, emitter, eventBase, task });
+      if (!ragRes.skipped && ragRes.contextBlock) {
+        task = `${task}\n\n${ragRes.contextBlock}`;
+        await persistStage("ragPreload", {
+          hits: ragRes.hits.length,
+          scope: config?.rag?.preload?.scope || "all",
+          decisor: decision.reason,
+        });
+      } else {
+        await persistStage("ragPreload", { skipped: true, reason: ragRes.reason, decisor: decision.reason });
+      }
+    }
+  }
+
   // --- Brain decisor (opt-in, intent-driven routing) ---
   //
   // When enabled, translates triage output + task text + CLI overrides into a

package/src/orchestrator/stages/rag-context-stage.js

@@ -0,0 +1,60 @@
+// KJC-PCS-0049 Camino C — pre-loop RAG retrieval stage.
+// Runs between triage and domainCurator. When `config.rag.preload.enabled`
+// is true and the corpus has chunks, queries the retriever with the
+// task text and returns a markdown block ready to prepend to the task.
+// Failures degrade gracefully (warn + continue); the rest of the pipeline
+// never sees an exception from this stage.
+import { emitProgress, makeEvent } from "../../utils/events.js";
+import { openVecStore, countChunks } from "../../rag/vec-store.js";
+import { OllamaEmbedder } from "../../rag/embedder.js";
+import { query } from "../../rag/retriever.js";
+
+const DEFAULT_TOP_K = 5;
+const DEFAULT_SCOPE = "all";
+const MAX_CHUNK_CHARS = 600;
+
+/**
+ * Runs the RAG preload stage. Returns:
+ *   { skipped: true, reason }                  — when disabled / empty / error
+ *   { skipped: false, contextBlock, hits }     — markdown block + raw hits
+ *
+ * The caller decides whether to mutate `task` with the contextBlock. We
+ * keep that responsibility at the call site so this module stays pure.
+ */
+export async function runRagContextStage({ config, logger, emitter, eventBase, task }) {
+  const preload = config?.rag?.preload;
+  if (!preload?.enabled) return { skipped: true, reason: "disabled" };
+  if (typeof task !== "string" || task.length === 0) return { skipped: true, reason: "no-task" };
+  try {
+    const dim = config?.rag?.embedder?.dim || 768;
+    const db = openVecStore({ dim });
+    try {
+      if (countChunks(db) === 0) {
+        logger?.info?.("[rag-preload] corpus is empty, skipping (run `kj rag index` to seed)");
+        return { skipped: true, reason: "empty" };
+      }
+      const topK = preload.topK || DEFAULT_TOP_K;
+      const scope = preload.scope || DEFAULT_SCOPE;
+      const cfg = config?.rag?.embedder || {};
+      const embedder = new OllamaEmbedder({ url: cfg.url, model: cfg.model, dim });
+      const hits = await query(db, embedder, task, { topK, scope });
+      if (hits.length === 0) return { skipped: true, reason: "no-hits" };
+      const lines = hits.map((h) => {
+        const label = h.metadata?.hu_id || h.metadata?.symbol || h.metadata?.headingPath?.join(" > ") || "block";
+        const text = h.text.length > MAX_CHUNK_CHARS ? `${h.text.slice(0, MAX_CHUNK_CHARS)}…` : h.text;
+        return `### [${h.kind} · ${label}] ${h.source}\n\n${text}`;
+      });
+      const contextBlock = `## Prior context from RAG (auto-retrieved, top ${hits.length})\n\n${lines.join("\n\n")}`;
+      emitProgress(emitter, makeEvent("rag-preload:hits", { ...eventBase, stage: "rag-preload" }, {
+        message: `RAG preload: injected ${hits.length} chunk(s) into the task context`,
+        detail: { topK, scope, hits: hits.length },
+      }));
+      return { skipped: false, contextBlock, hits };
+    } finally {
+      db.close();
+    }
+  } catch (err) {
+    logger?.warn?.(`[rag-preload] failed (non-blocking): ${err.message}`);
+    return { skipped: true, reason: "error", error: err.message };
+  }
+}

package/src/orchestrator/stages/rag-preload-decisor.js

@@ -0,0 +1,58 @@
+// KJC-TSK-0434 — RAG Camino D: Brain decisor heuristic for retrieval.
+// Decides WHETHER to pay the cost of running the RAG preload stage based
+// on signals from triage + task shape, instead of pre-fetching every run.
+//
+// Returns `{ pull: boolean, reason: string }`. The pre-loop driver
+// skips `runRagContextStage` when `pull === false`.
+//
+// Policies (config.rag.preload.policy):
+//   - "always"  : pull regardless (back-compat with v2.24.0 behaviour)
+//   - "never"   : never pull (useful for benchmarking, debugging)
+//   - "auto"    : heuristic — pull only when triage signals complexity,
+//                 decomposition, or when the task body is long enough
+//                 to suggest non-trivial scope. Default.
+
+const DEFAULT_POLICY = "auto";
+const COMPLEX_LEVELS = new Set(["complex", "high", "epic"]);
+const LONG_TASK_CHARS = 200;
+
+/**
+ * @param {object} params
+ * @param {object} [params.triage]   - triage stage result (may be null)
+ * @param {string} [params.task]     - the user task text
+ * @param {object} [params.config]   - full karajan config
+ * @returns {{ pull: boolean, reason: string }}
+ */
+export function shouldPreloadRag({ triage, task, config } = {}) {
+  const policy = config?.rag?.preload?.policy || DEFAULT_POLICY;
+
+  if (policy === "always") return { pull: true, reason: "policy:always" };
+  if (policy === "never") return { pull: false, reason: "policy:never" };
+
+  // policy === "auto" — apply heuristic.
+  // Brownfield is an explicit user signal — if onboarding produced a
+  // brief and config marks the project as brownfield, prior context is
+  // almost always worth pulling.
+  if (config?.rag?.preload?.brownfield === true) {
+    return { pull: true, reason: "auto:brownfield" };
+  }
+
+  // Triage said this is big enough to decompose → multi-HU run benefits
+  // from prior context fed to researcher/architect/planner.
+  if (triage?.shouldDecompose === true) {
+    return { pull: true, reason: "auto:decompose" };
+  }
+
+  // Triage classified the task as complex / high-effort.
+  if (triage?.level && COMPLEX_LEVELS.has(String(triage.level).toLowerCase())) {
+    return { pull: true, reason: "auto:complex" };
+  }
+
+  // Long task body usually means a brief with context, motivation and
+  // non-trivial scope — worth checking the RAG.
+  if (typeof task === "string" && task.length >= LONG_TASK_CHARS) {
+    return { pull: true, reason: "auto:long-task" };
+  }
+
+  return { pull: false, reason: "auto:low-value" };
+}

package/templates/skills/kj-rag-query.md

@@ -0,0 +1,50 @@
+# kj-rag-query — Query Project RAG index
+
+Query the Project RAG (vector store + embedder) and return the top-K most relevant chunks of the indexed project: plans, onboarding briefs, and (optionally) source code.
+
+Use this when you need prior context about the project before deciding, refactoring, planning, or coding — without scanning the filesystem manually.
+
+## Your task
+
+$ARGUMENTS
+
+## How it works
+
+This skill is a thin wrapper over the Karajan CLI:
+
+```
+kj rag query "<text>" [--scope <all|plans|onboarding|code>] [--top-k <n>] [--json]
+```
+
+Parse `$ARGUMENTS` as the natural-language query plus optional flags:
+
+- `--scope <s>`: restrict retrieval to one source bucket. Default: `all`.
+  - `plans` → indexed plan JSON files (HU titles, scope, blocked_by).
+  - `onboarding` → onboarding briefs from `~/.karajan/onboarding/`.
+  - `code` → indexed source files (only present if user ran `kj rag index --with-sources`).
+- `--top-k <n>`: number of chunks to return. Default: 3. Max recommended: 10.
+- `--json`: emit raw JSON (use when piping into another tool).
+
+If `$ARGUMENTS` is empty, print a one-line usage hint and stop.
+
+## Behavior contract
+
+1. **Empty store**: if the CLI replies with `empty: true`, do NOT block the conversation. Tell the user once: "RAG index is empty — run `kj rag index` to seed it" and continue without prior context.
+2. **Zero hits**: if the store has chunks but no match scored above threshold, say "no relevant prior context" and continue.
+3. **Hits returned**: render each as a short block — `[score] source — snippet` — and use them as background context for the user's actual question. Do NOT dump raw JSON unless `--json` was passed.
+4. **Never re-validate flags**. Pass `--scope`/`--top-k` straight to the CLI; the CLI is the source of truth.
+
+## Examples
+
+User: `/kj-rag-query how does the retriever rank chunks`
+→ `kj rag query "how does the retriever rank chunks"`
+
+User: `/kj-rag-query auth flow --scope plans --top-k 5`
+→ `kj rag query "auth flow" --scope plans --top-k 5`
+
+User: `/kj-rag-query --scope onboarding architecture`
+→ `kj rag query "architecture" --scope onboarding`
+
+## Why this exists (Camino B)
+
+Hosts that load Karajan as Skills (Claude Code without MCP, Cursor without MCP) cannot reach the `kj_rag_query` MCP tool. This skill bridges that gap with the same retrieval contract.