karajan-code 2.20.0 → 2.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -0
- package/package.json +2 -1
- package/scripts/esbuild-sea.config.mjs +31 -1
- package/src/cli/register-meta.js +34 -0
- package/src/cli/register-plan.js +2 -0
- package/src/commands/onboard.js +35 -0
- package/src/commands/plan/generate.js +19 -1
- package/src/commands/rag.js +66 -0
- package/src/onboarder/cache.js +24 -0
- package/src/onboarder/collectors/index.js +119 -0
- package/src/rag/chunker.js +119 -0
- package/src/rag/embedder.js +81 -0
- package/src/rag/indexer.js +116 -0
- package/src/rag/retriever.js +47 -0
- package/src/rag/vec-store.js +113 -0
- package/src/roles/onboarder-role.js +61 -0
- package/templates/roles/onboarder.md +36 -0
package/README.md
CHANGED
|
@@ -345,6 +345,45 @@ Opt out: set `telemetry: false` in `~/.karajan/kj.config.yml`
|
|
|
345
345
|
|
|
346
346
|
## Recent releases
|
|
347
347
|
|
|
348
|
+
> **v2.22.0 released** — Minor. **Project RAG MVP** (KJC-PCS-0049). Karajan now indexes its plans + onboarding briefs (and optionally project sources) into a local vector store and lets you query them semantically from the CLI. Six PRs.
|
|
349
|
+
>
|
|
350
|
+
> | Step | PR | Module |
|
|
351
|
+
> |---|---|---|
|
|
352
|
+
> | 1 | #808 | Vector store on `better-sqlite3` + `sqlite-vec` at `~/.karajan/rag.db` |
|
|
353
|
+
> | 2 | #809 | Ollama embedder adapter (`nomic-embed-text`, dim 768, `localhost:11434`) |
|
|
354
|
+
> | 3 | #810 | Three chunkers: markdown heading hierarchy, plan JSON per-HU, JS/TS export-symbol |
|
|
355
|
+
> | 4 | #811 | Indexer (`indexFile` + `indexProject`), idempotent, embedder failures = warn + continue |
|
|
356
|
+
> | 5 | #812 | Retriever + ranking, kind boost breaks ties (plan +0.05, onboarding +0.03, code 0) |
|
|
357
|
+
> | 6 | #813 | `kj rag` CLI: `index [--with-sources]` + `query <text> [--scope] [--top-k]` |
|
|
358
|
+
>
|
|
359
|
+
> ```bash
|
|
360
|
+
> cd ~/your-project
|
|
361
|
+
> kj onboard # Architecture Brief
|
|
362
|
+
> kj plan generate task.md -y # Plans
|
|
363
|
+
> kj rag index # Seed the vec store
|
|
364
|
+
> kj rag query "how did I handle auth in module X?"
|
|
365
|
+
> ```
|
|
366
|
+
>
|
|
367
|
+
> **Coming in v2.23.0**: MCP tool `kj_rag_query` (other agents query the RAG), HU Board search panel, chokidar watcher for live re-indexing, AST-aware source chunker, BM25 + cosine hybrid scoring.
|
|
368
|
+
>
|
|
369
|
+
> The SEA binary stubs out `src/rag/*` + `src/commands/rag.js` (same pattern as the HU Board) — `kj rag` requires `npm install -g karajan-code`.
|
|
370
|
+
>
|
|
371
|
+
> **v2.21.0 released** — Minor. **Brownfield Onboarder role**. Karajan now ships a dedicated path to analyze any existing codebase and produce a Markdown Architecture Brief that the planner can consume as automatic context. Closes KJC-TSK-0384 (3 PRs).
|
|
372
|
+
>
|
|
373
|
+
> KJC-TSK-0384 (PRs #804 + #805 + #806): **`kj onboard`** runs five deterministic collectors over a project root — directory walk (ignoring `node_modules` / `.git` / `dist` / `build`), git log (commits, branches, hot files via `--name-only` over the last 200 commits), 18 well-known config patterns + `package.json` scripts, ADR-style filenames under `docs/adr/`, `docs/adrs/`, `docs/architecture/`, plus a one-shot bundle. Then optionally synthesises a Markdown Architecture Brief via the new OnboarderRole. Output lands at `~/.karajan/onboarding/<slug>.md`. Flags: `--no-synth` (skip the LLM call, dump the raw collectors — useful for CI), `--output <path>` (override default target). Greenfield projects produce `# Project is greenfield` instead of erroring.
|
|
374
|
+
>
|
|
375
|
+
> **New `--use-onboarding` flag on `kj plan generate`**: reads the cached Architecture Brief and prepends it to the planner context under a `## Architecture Brief (from kj onboard)` heading. Silent on cache miss without the flag; loud `warn` when the flag is set but no cache exists, so a missed `kj onboard` invocation surfaces immediately.
|
|
376
|
+
>
|
|
377
|
+
> Workflow:
|
|
378
|
+
>
|
|
379
|
+
> ```bash
|
|
380
|
+
> kj onboard # produces ~/.karajan/onboarding/<slug>.md once
|
|
381
|
+
> kj plan generate task.md \
|
|
382
|
+
> --use-onboarding # next plan uses it as context
|
|
383
|
+
> ```
|
|
384
|
+
>
|
|
385
|
+
> **What's next** — The **Project RAG** epic (KJC-PCS-0049) starts in v2.22.0: vector store + Ollama embedder + indexer + retriever + CLI / MCP / HU Board consumers. Onboarder is its prerequisite (the brief feeds the indexer's first pass).
|
|
386
|
+
>
|
|
348
387
|
> **v2.20.0 released** — Minor. **HU Board polish + UX papercuts** cluster: 5 cards closed across two net-new features, two PG housekeeping syncs for work that had already landed, and one docs refresh.
|
|
349
388
|
>
|
|
350
389
|
> KJC-TSK-0397 (PR #801): **`kj plan generate` now prepends a `[PREFLIGHT-000]` HU to every plan** and gates every functional HU on it. The HU's acceptance tests are stack-aware shell commands — Node gets `node --version` + `npm install` + conditional `npm test` / `npm run lint`; Python gets `python --version` + `pip install -r requirements.txt` (or `poetry install`) + `pytest --collect-only`; Firebase projects get `firebase projects:list`; GCP projects get `gcloud auth list`; everyone gets `git status --porcelain`. Idempotent: a plan that already has a HU titled `PREFLIGHT-000` / "verificar entorno" is left untouched. Opt out per-invocation with `--no-preflight-hu`. The point: Karajan owns the plumbing, the user no longer has to remember to add a 'verify env' step to every task file.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "karajan-code",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.22.0",
|
|
4
4
|
"description": "Local multi-agent coding orchestrator with TDD, SonarQube, and code review pipeline",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "AGPL-3.0",
|
|
@@ -80,6 +80,7 @@
|
|
|
80
80
|
"js-yaml": "^4.1.0",
|
|
81
81
|
"knip": "^6.14.1",
|
|
82
82
|
"madge": "^8.0.0",
|
|
83
|
+
"sqlite-vec": "^0.1.9",
|
|
83
84
|
"valibot": "^1.3.1"
|
|
84
85
|
},
|
|
85
86
|
"devDependencies": {
|
|
@@ -139,6 +139,36 @@ const huBoardStubPlugin = {
|
|
|
139
139
|
},
|
|
140
140
|
};
|
|
141
141
|
|
|
142
|
+
// Same reasoning as the HU Board stub: src/rag/* + src/commands/rag.js
|
|
143
|
+
// depend on better-sqlite3 + sqlite-vec, native modules that esbuild
|
|
144
|
+
// cannot bundle. The SEA binary stubs them out and tells the user to
|
|
145
|
+
// install via npm to get RAG. KJC-PCS-0049 / Step 6.
|
|
146
|
+
const ragStubPlugin = {
|
|
147
|
+
name: "rag-stub",
|
|
148
|
+
setup(build) {
|
|
149
|
+
build.onResolve({ filter: /[\\/](rag[\\/][^\\/]+|commands[\\/]rag)\.js$/ }, (args) => ({
|
|
150
|
+
path: args.path, namespace: "rag-stub",
|
|
151
|
+
}));
|
|
152
|
+
build.onLoad({ filter: /.*/, namespace: "rag-stub" }, () => ({
|
|
153
|
+
contents: `
|
|
154
|
+
const NOT_AVAILABLE = "The RAG subsystem is not available in this standalone binary.\\n" +
|
|
155
|
+
"Install karajan-code from npm to get \\\`kj rag\\\`:\\n" +
|
|
156
|
+
" npm install -g karajan-code\\n";
|
|
157
|
+
function notAvailable() { throw new Error(NOT_AVAILABLE); }
|
|
158
|
+
module.exports = {
|
|
159
|
+
__esModule: false,
|
|
160
|
+
ragIndexCommand: notAvailable, ragQueryCommand: notAvailable,
|
|
161
|
+
openVecStore: notAvailable, OllamaEmbedder: notAvailable,
|
|
162
|
+
indexFile: notAvailable, indexProject: notAvailable,
|
|
163
|
+
query: notAvailable, chunkMarkdown: notAvailable, chunkPlan: notAvailable, chunkSource: notAvailable,
|
|
164
|
+
default: notAvailable,
|
|
165
|
+
};
|
|
166
|
+
`,
|
|
167
|
+
loader: "js",
|
|
168
|
+
}));
|
|
169
|
+
},
|
|
170
|
+
};
|
|
171
|
+
|
|
142
172
|
/** @type {import('esbuild').BuildOptions} */
|
|
143
173
|
export const seaBuildOptions = {
|
|
144
174
|
entryPoints: ["src/cli.js"],
|
|
@@ -169,6 +199,6 @@ export const seaBuildOptions = {
|
|
|
169
199
|
// throws → collector returns available:false. npm installs get knip
|
|
170
200
|
// resolved normally from node_modules.
|
|
171
201
|
external: ["better-sqlite3", "madge", "knip", "oxc-parser", "oxc-resolver"],
|
|
172
|
-
plugins: [seaTransformPlugin, huBoardStubPlugin],
|
|
202
|
+
plugins: [seaTransformPlugin, huBoardStubPlugin, ragStubPlugin],
|
|
173
203
|
logLevel: "info",
|
|
174
204
|
};
|
package/src/cli/register-meta.js
CHANGED
|
@@ -2,6 +2,8 @@ import { discoverCommand } from "../commands/discover.js";
|
|
|
2
2
|
import { triageCommand } from "../commands/triage.js";
|
|
3
3
|
import { researcherCommand } from "../commands/researcher.js";
|
|
4
4
|
import { architectCommand } from "../commands/architect.js";
|
|
5
|
+
import { onboardCommand } from "../commands/onboard.js";
|
|
6
|
+
import { ragIndexCommand, ragQueryCommand } from "../commands/rag.js";
|
|
5
7
|
import { auditCommand } from "../commands/audit.js";
|
|
6
8
|
import { resumeCommand } from "../commands/resume.js";
|
|
7
9
|
import { boardCommand } from "../commands/board.js";
|
|
@@ -86,6 +88,38 @@ export function registerMeta(program, { pkgVersion }) {
|
|
|
86
88
|
});
|
|
87
89
|
});
|
|
88
90
|
|
|
91
|
+
program
|
|
92
|
+
.command("onboard")
|
|
93
|
+
.description("Analyze an existing codebase (brownfield) and produce an Architecture Brief")
|
|
94
|
+
.option("--no-synth", "Skip the LLM synthesis step, dump raw collectors only")
|
|
95
|
+
.option("--output <path>", "Override the default ~/.karajan/onboarding/<slug>.md target")
|
|
96
|
+
.action(async (flags) => {
|
|
97
|
+
await withConfig(pkgVersion, "onboard", flags, async ({ config, logger }) => {
|
|
98
|
+
await onboardCommand({ config, logger, flags });
|
|
99
|
+
});
|
|
100
|
+
});
|
|
101
|
+
|
|
102
|
+
const rag = program.command("rag").description("Retrieval-augmented search over Karajan plans, onboarding briefs and project code");
|
|
103
|
+
rag.command("index")
|
|
104
|
+
.description("Index plans + onboarding (and optionally project sources) into the local vector store")
|
|
105
|
+
.option("--with-sources", "Also index the projectDir's JS/TS files")
|
|
106
|
+
.option("--json", "Output the totals as JSON")
|
|
107
|
+
.action(async (flags) => {
|
|
108
|
+
await withConfig(pkgVersion, "rag-index", flags, async ({ config, logger }) => {
|
|
109
|
+
await ragIndexCommand({ config, logger, flags });
|
|
110
|
+
});
|
|
111
|
+
});
|
|
112
|
+
rag.command("query <text>")
|
|
113
|
+
.description("Run a semantic query against the indexed RAG corpus")
|
|
114
|
+
.option("--scope <scope>", "plans | code | onboarding | all (default: all)", "all")
|
|
115
|
+
.option("--top-k <n>", "Number of hits to return (default: 5)", "5")
|
|
116
|
+
.option("--json", "Output the hits as JSON")
|
|
117
|
+
.action(async (text, flags) => {
|
|
118
|
+
await withConfig(pkgVersion, "rag-query", flags, async ({ config, logger }) => {
|
|
119
|
+
await ragQueryCommand({ text, config, logger, flags });
|
|
120
|
+
});
|
|
121
|
+
});
|
|
122
|
+
|
|
89
123
|
program
|
|
90
124
|
.command("audit")
|
|
91
125
|
.description("Analyze codebase health (read-only)")
|
package/src/cli/register-plan.js
CHANGED
|
@@ -43,6 +43,7 @@ export function registerPlan(program, { pkgVersion }) {
|
|
|
43
43
|
.option("--skip-spec-review", "Bypass the spec-reviewer pre-pipeline audit")
|
|
44
44
|
.option("--quick", "Sketch mode — skip every quality pass after the initial planner call")
|
|
45
45
|
.option("--no-preflight-hu", "Skip auto-injection of [PREFLIGHT-000] HU at the top of the generated plan")
|
|
46
|
+
.option("--use-onboarding", "Inject the cached Architecture Brief (~/.karajan/onboarding/<slug>.md) into the planner context")
|
|
46
47
|
.option("-y, --yes", "Skip the project-name prompt (use the auto-derived default).")
|
|
47
48
|
.option("--no-interactive", "Force non-interactive mode (no prompts, use defaults).")
|
|
48
49
|
.action(async (task, flags) => {
|
|
@@ -102,6 +103,7 @@ export function registerPlan(program, { pkgVersion }) {
|
|
|
102
103
|
yes: Boolean(flags.yes),
|
|
103
104
|
interactive: flags.interactive,
|
|
104
105
|
preflightHu: flags.preflightHu !== false, // KJC-TSK-0397
|
|
106
|
+
useOnboarding: Boolean(flags.useOnboarding), // KJC-TSK-0384 PR 3
|
|
105
107
|
},
|
|
106
108
|
});
|
|
107
109
|
});
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
// KJC-TSK-0384 PR 2 — `kj onboard` runs collectors + optional LLM synth.
|
|
2
|
+
import { writeFile, mkdir, readFile } from "node:fs/promises";
|
|
3
|
+
import { existsSync } from "node:fs";
|
|
4
|
+
import path from "node:path";
|
|
5
|
+
import { collectAll } from "../onboarder/collectors/index.js";
|
|
6
|
+
import { OnboarderRole } from "../roles/onboarder-role.js";
|
|
7
|
+
import { resolveRole } from "../config.js";
|
|
8
|
+
import { getKarajanHome } from "../utils/paths.js";
|
|
9
|
+
|
|
10
|
+
export function briefPath(projectDir) {
|
|
11
|
+
const slug = path.basename(projectDir).replace(/[^a-zA-Z0-9._-]/g, "-").toLowerCase() || "project";
|
|
12
|
+
return path.join(getKarajanHome(), "onboarding", `${slug}.md`);
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
export async function onboardCommand({ config, logger, flags = {} }) {
|
|
16
|
+
const projectDir = config?.projectDir || process.cwd();
|
|
17
|
+
const out = flags.output || briefPath(projectDir);
|
|
18
|
+
logger.info(`[onboard] collecting from ${projectDir}`);
|
|
19
|
+
const bundle = await collectAll(projectDir);
|
|
20
|
+
await mkdir(path.dirname(out), { recursive: true });
|
|
21
|
+
if (flags.noSynth) {
|
|
22
|
+
await writeFile(out, `# Onboarding bundle — ${projectDir}\n\n\`\`\`json\n${JSON.stringify(bundle, null, 2)}\n\`\`\`\n`, "utf8");
|
|
23
|
+
logger.info(`[onboard] wrote raw bundle to ${out} (--no-synth)`);
|
|
24
|
+
return { path: out, bundle, brief: null };
|
|
25
|
+
}
|
|
26
|
+
const roleCfg = resolveRole(config, "onboarder") || resolveRole(config, "architect");
|
|
27
|
+
const templatePath = new URL("../../templates/roles/onboarder.md", import.meta.url).pathname;
|
|
28
|
+
const instructions = existsSync(templatePath) ? await readFile(templatePath, "utf8") : "";
|
|
29
|
+
const role = new OnboarderRole({ instructions, config, ...roleCfg });
|
|
30
|
+
const result = await role.run({ bundle });
|
|
31
|
+
const brief = result?.result?.brief || "# Project is greenfield\n\nNo synth output.\n";
|
|
32
|
+
await writeFile(out, brief, "utf8");
|
|
33
|
+
logger.info(`[onboard] wrote Architecture Brief to ${out}`);
|
|
34
|
+
return { path: out, bundle, brief };
|
|
35
|
+
}
|
|
@@ -15,6 +15,7 @@ import { buildStandbyState } from "../../brain/standby-store.js";
|
|
|
15
15
|
import { printResumeHint } from "../../utils/display/resume-hint.js";
|
|
16
16
|
import { runSpecReview } from "../../spec-review/run-spec-review.js";
|
|
17
17
|
import { prependPreflightHu } from "../../plan/preflight-hu.js";
|
|
18
|
+
import { readCachedBrief } from "../../onboarder/cache.js";
|
|
18
19
|
import { formatPlan, formatHuTable } from "./_shared.js";
|
|
19
20
|
|
|
20
21
|
/**
|
|
@@ -56,12 +57,29 @@ async function planGenerateImpl({ task, config, logger, json, context, runLog, f
|
|
|
56
57
|
task = reviewResult.finalSpec; // downstream runs the refined spec
|
|
57
58
|
}
|
|
58
59
|
|
|
60
|
+
// KJC-TSK-0384 PR 3: --use-onboarding folds the cached Architecture Brief
|
|
61
|
+
// from `kj onboard` into the planner's context. Silent when no brief is
|
|
62
|
+
// cached so users without one don't see noise; loud warn when they passed
|
|
63
|
+
// the flag explicitly but the cache is empty so the missed wiring surfaces.
|
|
64
|
+
let resolvedContext = context;
|
|
65
|
+
if (flags?.useOnboarding) {
|
|
66
|
+
const brief = await readCachedBrief(config?.projectDir || process.cwd());
|
|
67
|
+
if (brief.found) {
|
|
68
|
+
resolvedContext = [
|
|
69
|
+
resolvedContext, brief.content ? `## Architecture Brief (from kj onboard)\n\n${brief.content}` : null,
|
|
70
|
+
].filter(Boolean).join("\n\n");
|
|
71
|
+
runLog.logText(`[planner] onboarding brief injected from ${brief.path}`);
|
|
72
|
+
} else {
|
|
73
|
+
logger?.warn?.(`--use-onboarding passed but no brief at ${brief.path}. Run 'kj onboard' first.`);
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
|
|
59
77
|
const plannerRole = resolveRole(config, "planner");
|
|
60
78
|
await assertAgentsAvailable([plannerRole.provider]);
|
|
61
79
|
runLog.logText(`[planner] provider=${plannerRole.provider}`);
|
|
62
80
|
|
|
63
81
|
const planner = createAgent(plannerRole.provider, config, logger);
|
|
64
|
-
const prompt = buildPlannerPrompt({ task, context });
|
|
82
|
+
const prompt = buildPlannerPrompt({ task, context: resolvedContext });
|
|
65
83
|
const silenceTimeoutMs = Number(config?.session?.max_agent_silence_minutes) > 0
|
|
66
84
|
? Math.round(Number(config.session.max_agent_silence_minutes) * 60 * 1000)
|
|
67
85
|
: undefined;
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 6 — `kj rag` command group. Two subcommands:
|
|
2
|
+
// kj rag index [--project <slug>] [--with-sources]
|
|
3
|
+
// kj rag query <text> [--scope plans|code|onboarding|all] [--top-k N] [--json]
|
|
4
|
+
// Closes the v2.22.0 RAG MVP end-to-end from the terminal.
|
|
5
|
+
import { openVecStore, countChunks } from "../rag/vec-store.js";
|
|
6
|
+
import { OllamaEmbedder } from "../rag/embedder.js";
|
|
7
|
+
import { indexProject } from "../rag/indexer.js";
|
|
8
|
+
import { query } from "../rag/retriever.js";
|
|
9
|
+
import { getKarajanHome } from "../utils/paths.js";
|
|
10
|
+
|
|
11
|
+
function makeEmbedder(config) {
|
|
12
|
+
const cfg = config?.rag?.embedder || {};
|
|
13
|
+
return new OllamaEmbedder({ url: cfg.url, model: cfg.model, dim: cfg.dim });
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
function openDb(config) {
|
|
17
|
+
return openVecStore({ dim: config?.rag?.embedder?.dim || 768 });
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export async function ragIndexCommand({ config, logger, flags = {} }) {
|
|
21
|
+
const projectDir = config?.projectDir || process.cwd();
|
|
22
|
+
const db = openDb(config);
|
|
23
|
+
try {
|
|
24
|
+
const totals = await indexProject(projectDir, {
|
|
25
|
+
db, embedder: makeEmbedder(config),
|
|
26
|
+
karajanHome: getKarajanHome(), logger,
|
|
27
|
+
withSources: Boolean(flags.withSources),
|
|
28
|
+
});
|
|
29
|
+
if (flags.json) {
|
|
30
|
+
process.stdout.write(`${JSON.stringify(totals)}\n`);
|
|
31
|
+
} else {
|
|
32
|
+
logger.info(`[rag] indexed ${totals.indexed} chunk(s) across ${totals.files} file(s) (${totals.failed} failed)`);
|
|
33
|
+
}
|
|
34
|
+
return totals;
|
|
35
|
+
} finally {
|
|
36
|
+
db.close();
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
export async function ragQueryCommand({ text, config, logger, flags = {} }) {
|
|
41
|
+
if (!text) throw new Error("kj rag query: text argument required");
|
|
42
|
+
const db = openDb(config);
|
|
43
|
+
try {
|
|
44
|
+
if (countChunks(db) === 0) {
|
|
45
|
+
logger.warn("[rag] No chunks indexed yet. Run 'kj rag index' first.");
|
|
46
|
+
if (flags.json) process.stdout.write("[]\n");
|
|
47
|
+
return [];
|
|
48
|
+
}
|
|
49
|
+
const topK = Math.max(1, Number(flags.topK) || 5);
|
|
50
|
+
const scope = flags.scope || "all";
|
|
51
|
+
const hits = await query(db, makeEmbedder(config), text, { topK, scope });
|
|
52
|
+
if (flags.json) {
|
|
53
|
+
process.stdout.write(`${JSON.stringify(hits)}\n`);
|
|
54
|
+
} else {
|
|
55
|
+
for (const h of hits) {
|
|
56
|
+
const label = h.metadata?.hu_id || h.metadata?.symbol || h.metadata?.headingPath?.join(" > ") || "block";
|
|
57
|
+
logger.info(`[${h.kind} · ${label} · score=${h.score.toFixed(4)}] ${h.source}`);
|
|
58
|
+
logger.info(h.text.length > 240 ? `${h.text.slice(0, 240)}…` : h.text);
|
|
59
|
+
logger.info("");
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
return hits;
|
|
63
|
+
} finally {
|
|
64
|
+
db.close();
|
|
65
|
+
}
|
|
66
|
+
}
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
// KJC-TSK-0384 PR 3 — read the cached Architecture Brief deterministically.
|
|
2
|
+
// briefPath() is exported by src/commands/onboard.js so the writer and
|
|
3
|
+
// reader share the same slug rule.
|
|
4
|
+
import { existsSync } from "node:fs";
|
|
5
|
+
import { readFile } from "node:fs/promises";
|
|
6
|
+
|
|
7
|
+
import { briefPath } from "../commands/onboard.js";
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* Read the cached Architecture Brief for `projectDir`. Returns:
|
|
11
|
+
* { found: true, path, content } — when the file exists
|
|
12
|
+
* { found: false, path } — when it does not
|
|
13
|
+
* Never throws.
|
|
14
|
+
*/
|
|
15
|
+
export async function readCachedBrief(projectDir) {
|
|
16
|
+
const p = briefPath(projectDir);
|
|
17
|
+
if (!existsSync(p)) return { found: false, path: p };
|
|
18
|
+
try {
|
|
19
|
+
const content = await readFile(p, "utf8");
|
|
20
|
+
return { found: true, path: p, content };
|
|
21
|
+
} catch {
|
|
22
|
+
return { found: false, path: p };
|
|
23
|
+
}
|
|
24
|
+
}
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
// Onboarder collectors — deterministic, zero-LLM extractors. Each pure,
|
|
2
|
+
// JSON-serialisable, fail-soft (returns null on the slot, never throws).
|
|
3
|
+
// Output bundle is consumed by PR 2 synthesis or dumped via `--no-synth`.
|
|
4
|
+
import { execFileSync } from "node:child_process";
|
|
5
|
+
import { existsSync, readFileSync, statSync } from "node:fs";
|
|
6
|
+
import { readdir } from "node:fs/promises";
|
|
7
|
+
import { join, relative } from "node:path";
|
|
8
|
+
|
|
9
|
+
import { detectProjectStack } from "../../utils/stack-detect.js";
|
|
10
|
+
|
|
11
|
+
const IGNORED_DIRS = new Set(["node_modules", ".git", "dist", "build", "coverage", ".karajan", ".next", "__pycache__"]);
|
|
12
|
+
|
|
13
|
+
function safeRun(cmd, args, opts = {}) {
|
|
14
|
+
try { return execFileSync(cmd, args, { encoding: "utf8", stdio: ["pipe", "pipe", "pipe"], ...opts }).trim(); }
|
|
15
|
+
catch { return null; }
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
// Tree summary: immediate dirs with rough byte count. Ignores noise dirs.
|
|
19
|
+
export async function collectTree(projectDir, { maxDepth = 2 } = {}) {
|
|
20
|
+
async function walk(dir, depth) {
|
|
21
|
+
if (depth > maxDepth) return [];
|
|
22
|
+
let entries;
|
|
23
|
+
try { entries = await readdir(dir, { withFileTypes: true }); } catch { return []; }
|
|
24
|
+
const out = [];
|
|
25
|
+
for (const ent of entries) {
|
|
26
|
+
if (ent.name.startsWith(".") && ent.name !== ".github") continue;
|
|
27
|
+
if (IGNORED_DIRS.has(ent.name)) continue;
|
|
28
|
+
const sub = join(dir, ent.name);
|
|
29
|
+
if (ent.isDirectory()) {
|
|
30
|
+
const children = await walk(sub, depth + 1);
|
|
31
|
+
let bytes = 0;
|
|
32
|
+
for (const c of children) bytes += c.bytes || 0;
|
|
33
|
+
out.push({ path: relative(projectDir, sub), kind: "dir", bytes, children });
|
|
34
|
+
} else if (ent.isFile()) {
|
|
35
|
+
try { out.push({ path: relative(projectDir, sub), kind: "file", bytes: statSync(sub).size }); } catch { /* race */ }
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
return out;
|
|
39
|
+
}
|
|
40
|
+
return walk(projectDir, 0);
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
// Git log + branches + hot files. Returns null on non-git (greenfield).
|
|
44
|
+
export function collectGitHistory(projectDir, { maxHotFiles = 10 } = {}) {
|
|
45
|
+
const inside = safeRun("git", ["rev-parse", "--is-inside-work-tree"], { cwd: projectDir });
|
|
46
|
+
if (inside !== "true") return null;
|
|
47
|
+
const commitsRaw = safeRun("git", ["log", "--oneline", "-n", "100"], { cwd: projectDir }) || "";
|
|
48
|
+
const branchesRaw = safeRun("git", ["branch", "--list", "--all", "--format=%(refname:short)"], { cwd: projectDir }) || "";
|
|
49
|
+
const hotRaw = safeRun("git", ["log", "--name-only", "--pretty=format:", "-n", "200"], { cwd: projectDir }) || "";
|
|
50
|
+
const counts = {};
|
|
51
|
+
for (const line of hotRaw.split("\n").map((s) => s.trim()).filter(Boolean)) {
|
|
52
|
+
counts[line] = (counts[line] || 0) + 1;
|
|
53
|
+
}
|
|
54
|
+
const hotFiles = Object.entries(counts)
|
|
55
|
+
.sort((a, b) => b[1] - a[1]).slice(0, maxHotFiles)
|
|
56
|
+
.map(([file, touches]) => ({ file, touches }));
|
|
57
|
+
return {
|
|
58
|
+
commitCount: commitsRaw ? commitsRaw.split("\n").length : 0,
|
|
59
|
+
branches: branchesRaw.split("\n").filter(Boolean),
|
|
60
|
+
hotFiles,
|
|
61
|
+
headSha: safeRun("git", ["rev-parse", "HEAD"], { cwd: projectDir }) || null,
|
|
62
|
+
};
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
// Presence-check of known config files + package.json scripts.
|
|
66
|
+
export function collectConfigs(projectDir) {
|
|
67
|
+
const configFiles = [
|
|
68
|
+
"package.json", "pyproject.toml", "Cargo.toml", "go.mod",
|
|
69
|
+
"tsconfig.json", "jsconfig.json",
|
|
70
|
+
".eslintrc.js", ".eslintrc.json", "eslint.config.js",
|
|
71
|
+
"vitest.config.js", "vitest.config.ts", "jest.config.js", "jest.config.ts",
|
|
72
|
+
"firebase.json", ".gcloudignore", "Dockerfile", "docker-compose.yml",
|
|
73
|
+
".github/workflows", ".gitlab-ci.yml", "Makefile",
|
|
74
|
+
];
|
|
75
|
+
const present = configFiles.filter((f) => existsSync(join(projectDir, f)));
|
|
76
|
+
let scripts = null;
|
|
77
|
+
try {
|
|
78
|
+
const pkg = JSON.parse(readFileSync(join(projectDir, "package.json"), "utf8"));
|
|
79
|
+
scripts = pkg.scripts || null;
|
|
80
|
+
} catch { /* not a node project */ }
|
|
81
|
+
return { present, scripts };
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
// Discover ADR files under docs/adr/, docs/adrs/, docs/architecture/, root.
|
|
85
|
+
export async function collectAdrs(projectDir) {
|
|
86
|
+
const dirs = ["docs/adr", "docs/adrs", "docs/architecture", "."];
|
|
87
|
+
const found = [];
|
|
88
|
+
for (const d of dirs) {
|
|
89
|
+
const abs = join(projectDir, d);
|
|
90
|
+
if (!existsSync(abs)) continue;
|
|
91
|
+
try {
|
|
92
|
+
const entries = await readdir(abs);
|
|
93
|
+
for (const f of entries) {
|
|
94
|
+
if (/^adr-?\d|^\d{4}-.+\.md$/i.test(f) || /architecture.*\.md$/i.test(f)) {
|
|
95
|
+
found.push(relative(projectDir, join(abs, f)));
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
} catch { /* unreadable */ }
|
|
99
|
+
}
|
|
100
|
+
return found;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
// One-shot bundle: every slot is independent; missing slots → null/[].
|
|
104
|
+
export async function collectAll(projectDir) {
|
|
105
|
+
const [stack, tree, adrs] = await Promise.all([
|
|
106
|
+
detectProjectStack(projectDir).catch(() => null),
|
|
107
|
+
collectTree(projectDir).catch(() => []),
|
|
108
|
+
collectAdrs(projectDir).catch(() => []),
|
|
109
|
+
]);
|
|
110
|
+
return {
|
|
111
|
+
projectDir,
|
|
112
|
+
stack,
|
|
113
|
+
tree,
|
|
114
|
+
git: collectGitHistory(projectDir),
|
|
115
|
+
configs: collectConfigs(projectDir),
|
|
116
|
+
adrs,
|
|
117
|
+
collectedAt: new Date().toISOString(),
|
|
118
|
+
};
|
|
119
|
+
}
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 3 — Chunkers for the RAG indexer.
|
|
2
|
+
// Each chunker is pure: takes text + opts, returns
|
|
3
|
+
// `[{ text, metadata: { source, kind, ...extras } }]`. The indexer
|
|
4
|
+
// (Step 4) hands every chunk to the embedder + vec store; the
|
|
5
|
+
// retriever (Step 5) reads metadata back to enrich results.
|
|
6
|
+
|
|
7
|
+
const DEFAULT_CHUNK_LIMIT = 800;
|
|
8
|
+
const DEFAULT_OVERLAP = 100;
|
|
9
|
+
|
|
10
|
+
// ---------- shared splitter ---------------------------------------
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* Split a long string into overlapping windows. Used by every chunker
|
|
14
|
+
* that produces a single section too big for the embedder's context.
|
|
15
|
+
*/
|
|
16
|
+
function windowText(text, { limit = DEFAULT_CHUNK_LIMIT, overlap = DEFAULT_OVERLAP } = {}) {
|
|
17
|
+
if (text.length <= limit) return [text];
|
|
18
|
+
const step = Math.max(1, limit - overlap);
|
|
19
|
+
const out = [];
|
|
20
|
+
for (let i = 0; i < text.length; i += step) {
|
|
21
|
+
out.push(text.slice(i, i + limit));
|
|
22
|
+
if (i + limit >= text.length) break;
|
|
23
|
+
}
|
|
24
|
+
return out;
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
// ---------- markdown chunker (plans / onboarding briefs) ----------
|
|
28
|
+
|
|
29
|
+
const HEADING_RE = /^(#{1,6})\s+(.+)$/gm;
|
|
30
|
+
|
|
31
|
+
/**
|
|
32
|
+
* Chunk a markdown document by heading hierarchy. The headingPath
|
|
33
|
+
* metadata tracks ancestors so a retrieval result can show "from
|
|
34
|
+
* '# Plan' › '## Auth' › '### Login'".
|
|
35
|
+
*/
|
|
36
|
+
export function chunkMarkdown(text, { path = "<inline>", kind = "plan", limit = DEFAULT_CHUNK_LIMIT, overlap = DEFAULT_OVERLAP } = {}) {
|
|
37
|
+
const sections = [];
|
|
38
|
+
const stack = []; // [{ depth, title }]
|
|
39
|
+
let cursor = 0;
|
|
40
|
+
let pending = { headingPath: [], start: 0 };
|
|
41
|
+
HEADING_RE.lastIndex = 0;
|
|
42
|
+
let m;
|
|
43
|
+
while ((m = HEADING_RE.exec(text)) !== null) {
|
|
44
|
+
if (m.index > cursor) {
|
|
45
|
+
pending.body = text.slice(pending.start, m.index).trim();
|
|
46
|
+
if (pending.body) sections.push({ ...pending });
|
|
47
|
+
}
|
|
48
|
+
const depth = m[1].length;
|
|
49
|
+
const title = m[2].trim();
|
|
50
|
+
while (stack.length > 0 && stack[stack.length - 1].depth >= depth) stack.pop();
|
|
51
|
+
stack.push({ depth, title });
|
|
52
|
+
pending = { headingPath: stack.map((s) => s.title), start: HEADING_RE.lastIndex };
|
|
53
|
+
cursor = HEADING_RE.lastIndex;
|
|
54
|
+
}
|
|
55
|
+
const tail = text.slice(pending.start).trim();
|
|
56
|
+
if (tail) sections.push({ ...pending, body: tail });
|
|
57
|
+
const chunks = [];
|
|
58
|
+
for (const sec of sections) {
|
|
59
|
+
const pieces = windowText(sec.body, { limit, overlap });
|
|
60
|
+
for (const piece of pieces) {
|
|
61
|
+
chunks.push({ text: piece, metadata: { source: path, kind, headingPath: sec.headingPath } });
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
return chunks;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
// ---------- plan chunker (one chunk per HU) -----------------------
|
|
68
|
+
|
|
69
|
+
/**
|
|
70
|
+
* Chunk a Karajan plan JSON. One chunk per HU; an extra chunk for the
|
|
71
|
+
* plan-level summary if present. The retriever uses hu_id to deep-link
|
|
72
|
+
* back to the board.
|
|
73
|
+
*/
|
|
74
|
+
export function chunkPlan(plan, { path = "<plan>", limit = DEFAULT_CHUNK_LIMIT } = {}) {
|
|
75
|
+
const out = [];
|
|
76
|
+
if (plan?.summary) {
|
|
77
|
+
const pieces = windowText(String(plan.summary), { limit });
|
|
78
|
+
for (const piece of pieces) {
|
|
79
|
+
out.push({ text: piece, metadata: { source: path, kind: "plan", section: "summary" } });
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
for (const hu of plan?.hus || []) {
|
|
83
|
+
const parts = [hu.title, hu.description, hu.scope].filter(Boolean).join("\n\n");
|
|
84
|
+
if (!parts.trim()) continue;
|
|
85
|
+
for (const piece of windowText(parts, { limit })) {
|
|
86
|
+
out.push({ text: piece, metadata: { source: path, kind: "plan", hu_id: hu.id || null, title: hu.title || null } });
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
return out;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// ---------- source chunker (export-symbol granularity) ------------
|
|
93
|
+
|
|
94
|
+
const EXPORT_RE = /^export\s+(?:async\s+)?(?:function|class|const|let)\s+(\w+)/gm;
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Chunk a JS/TS source file by top-level export symbol. Falls back to
|
|
98
|
+
* the windowed splitter on files without explicit exports.
|
|
99
|
+
*/
|
|
100
|
+
export function chunkSource(text, { path = "<src>", limit = DEFAULT_CHUNK_LIMIT, overlap = DEFAULT_OVERLAP } = {}) {
|
|
101
|
+
const marks = [];
|
|
102
|
+
EXPORT_RE.lastIndex = 0;
|
|
103
|
+
let m;
|
|
104
|
+
while ((m = EXPORT_RE.exec(text)) !== null) marks.push({ idx: m.index, symbol: m[1] });
|
|
105
|
+
if (marks.length === 0) {
|
|
106
|
+
return windowText(text, { limit, overlap })
|
|
107
|
+
.map((piece) => ({ text: piece, metadata: { source: path, kind: "code", symbol: null } }));
|
|
108
|
+
}
|
|
109
|
+
const out = [];
|
|
110
|
+
for (let i = 0; i < marks.length; i += 1) {
|
|
111
|
+
const start = marks[i].idx;
|
|
112
|
+
const end = i + 1 < marks.length ? marks[i + 1].idx : text.length;
|
|
113
|
+
const body = text.slice(start, end);
|
|
114
|
+
for (const piece of windowText(body, { limit, overlap })) {
|
|
115
|
+
out.push({ text: piece, metadata: { source: path, kind: "code", symbol: marks[i].symbol } });
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
return out;
|
|
119
|
+
}
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 2 — Ollama embedder adapter for the RAG pipeline.
|
|
2
|
+
// Talks to a local Ollama server (POST /api/embeddings) and returns
|
|
3
|
+
// Float32Array vectors. Cero deps externas; usa fetch global.
|
|
4
|
+
//
|
|
5
|
+
// Defaults:
|
|
6
|
+
// url = process.env.KJ_OLLAMA_URL or "http://localhost:11434"
|
|
7
|
+
// model = process.env.KJ_OLLAMA_EMBED_MODEL or "nomic-embed-text"
|
|
8
|
+
// dim = 768 (matches nomic-embed-text)
|
|
9
|
+
// timeoutMs = 30000
|
|
10
|
+
|
|
11
|
+
const DEFAULT_URL = "http://localhost:11434";
|
|
12
|
+
const DEFAULT_MODEL = "nomic-embed-text";
|
|
13
|
+
const DEFAULT_DIM = 768;
|
|
14
|
+
const DEFAULT_TIMEOUT_MS = 30000;
|
|
15
|
+
|
|
16
|
+
export class OllamaEmbedderError extends Error {
|
|
17
|
+
constructor(message, { cause, status } = {}) {
|
|
18
|
+
super(message);
|
|
19
|
+
this.name = "OllamaEmbedderError";
|
|
20
|
+
if (cause) this.cause = cause;
|
|
21
|
+
if (status != null) this.status = status;
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
export class OllamaEmbedder {
|
|
26
|
+
constructor({
|
|
27
|
+
url = process.env.KJ_OLLAMA_URL || DEFAULT_URL,
|
|
28
|
+
model = process.env.KJ_OLLAMA_EMBED_MODEL || DEFAULT_MODEL,
|
|
29
|
+
dim = DEFAULT_DIM,
|
|
30
|
+
timeoutMs = DEFAULT_TIMEOUT_MS,
|
|
31
|
+
fetchFn = globalThis.fetch,
|
|
32
|
+
} = {}) {
|
|
33
|
+
this.url = url.replace(/\/$/, "");
|
|
34
|
+
this.model = model;
|
|
35
|
+
this.dim = dim;
|
|
36
|
+
this.timeoutMs = timeoutMs;
|
|
37
|
+
this.fetch = fetchFn;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
/** Single-text embedding. Returns Float32Array(dim). */
|
|
41
|
+
async embed(text) {
|
|
42
|
+
if (typeof text !== "string" || text.length === 0) {
|
|
43
|
+
throw new OllamaEmbedderError("embed: text must be a non-empty string");
|
|
44
|
+
}
|
|
45
|
+
const ctrl = new AbortController();
|
|
46
|
+
const timer = setTimeout(() => ctrl.abort(), this.timeoutMs);
|
|
47
|
+
let res;
|
|
48
|
+
try {
|
|
49
|
+
res = await this.fetch(`${this.url}/api/embeddings`, {
|
|
50
|
+
method: "POST",
|
|
51
|
+
headers: { "Content-Type": "application/json" },
|
|
52
|
+
body: JSON.stringify({ model: this.model, prompt: text }),
|
|
53
|
+
signal: ctrl.signal,
|
|
54
|
+
});
|
|
55
|
+
} catch (err) {
|
|
56
|
+
throw new OllamaEmbedderError(`Ollama embed request failed (${this.url}): ${err.message}`, { cause: err });
|
|
57
|
+
} finally {
|
|
58
|
+
clearTimeout(timer);
|
|
59
|
+
}
|
|
60
|
+
if (!res.ok) {
|
|
61
|
+
throw new OllamaEmbedderError(`Ollama embed HTTP ${res.status} from ${this.url}`, { status: res.status });
|
|
62
|
+
}
|
|
63
|
+
const body = await res.json();
|
|
64
|
+
const arr = body?.embedding;
|
|
65
|
+
if (!Array.isArray(arr) || arr.length === 0) {
|
|
66
|
+
throw new OllamaEmbedderError("Ollama response missing 'embedding' array");
|
|
67
|
+
}
|
|
68
|
+
if (arr.length !== this.dim) {
|
|
69
|
+
throw new OllamaEmbedderError(`Ollama dim mismatch: got ${arr.length}, expected ${this.dim} for model ${this.model}`);
|
|
70
|
+
}
|
|
71
|
+
return Float32Array.from(arr);
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/** Batch embedding (sequential — Ollama /api/embeddings is single-text). */
|
|
75
|
+
async embedBatch(texts) {
|
|
76
|
+
if (!Array.isArray(texts)) throw new OllamaEmbedderError("embedBatch: texts must be an array");
|
|
77
|
+
const out = [];
|
|
78
|
+
for (const t of texts) out.push(await this.embed(t));
|
|
79
|
+
return out;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 4 — Indexer for the RAG pipeline. Wires the three
|
|
2
|
+
// previous modules together: chunker → embedder → vec store. The CLI
|
|
3
|
+
// (Step 6) drives this; Step 5's retriever reads what gets persisted
|
|
4
|
+
// here. A chokidar watcher version stays out of scope for v2.22.0 —
|
|
5
|
+
// indexProject() is on-demand from `kj rag index`, which keeps the
|
|
6
|
+
// failure mode simple (re-run to refresh).
|
|
7
|
+
import { existsSync, readFileSync } from "node:fs";
|
|
8
|
+
import { readdir } from "node:fs/promises";
|
|
9
|
+
import { extname, join, relative } from "node:path";
|
|
10
|
+
|
|
11
|
+
import { chunkMarkdown, chunkPlan, chunkSource } from "./chunker.js";
|
|
12
|
+
import { insertChunk, deleteChunksBySource } from "./vec-store.js";
|
|
13
|
+
|
|
14
|
+
const ONBOARDING_DIR = "onboarding", PLANS_DIR = "plans";
|
|
15
|
+
|
|
16
|
+
function detectKind(path) {
|
|
17
|
+
const ext = extname(path).toLowerCase();
|
|
18
|
+
const base = path.split("/").pop() || "";
|
|
19
|
+
if (ext === ".json" && (/^plan-.+\.json$/i.test(base) || path.includes(`/${PLANS_DIR}/`))) return "plan";
|
|
20
|
+
if (ext === ".md") return path.includes(`/${ONBOARDING_DIR}/`) ? "onboarding" : "plan";
|
|
21
|
+
if ([".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx"].includes(ext)) return "code";
|
|
22
|
+
return null;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
function chunksFor(path, kind) {
|
|
26
|
+
const text = readFileSync(path, "utf8");
|
|
27
|
+
if (kind === "plan" && extname(path) === ".json") {
|
|
28
|
+
try { return chunkPlan(JSON.parse(text), { path }); }
|
|
29
|
+
catch { return []; }
|
|
30
|
+
}
|
|
31
|
+
if (kind === "plan" || kind === "onboarding") {
|
|
32
|
+
return chunkMarkdown(text, { path, kind });
|
|
33
|
+
}
|
|
34
|
+
if (kind === "code") return chunkSource(text, { path });
|
|
35
|
+
return [];
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
/**
|
|
39
|
+
* Index a single file. Returns `{ indexed, failed }`. Idempotent —
|
|
40
|
+
* pre-existing chunks for this `source` path are deleted before the
|
|
41
|
+
* fresh batch is written, so a second call against the same file
|
|
42
|
+
* leaves the store with exactly the latest chunks (no duplicates).
|
|
43
|
+
*/
|
|
44
|
+
export async function indexFile(path, { db, embedder, logger = console } = {}) {
|
|
45
|
+
if (!existsSync(path)) { logger.warn?.(`[rag-indexer] file not found: ${path}`); return { indexed: 0, failed: 0 }; }
|
|
46
|
+
const kind = detectKind(path);
|
|
47
|
+
if (!kind) { logger.warn?.(`[rag-indexer] unknown kind: ${path}`); return { indexed: 0, failed: 0 }; }
|
|
48
|
+
const chunks = chunksFor(path, kind);
|
|
49
|
+
if (!chunks.length) return { indexed: 0, failed: 0 };
|
|
50
|
+
deleteChunksBySource(db, path);
|
|
51
|
+
let indexed = 0;
|
|
52
|
+
let failed = 0;
|
|
53
|
+
for (const ch of chunks) {
|
|
54
|
+
try {
|
|
55
|
+
const embedding = await embedder.embed(ch.text);
|
|
56
|
+
insertChunk(db, { source: path, kind: ch.metadata.kind || kind, text: ch.text, metadata: ch.metadata, embedding });
|
|
57
|
+
indexed += 1;
|
|
58
|
+
} catch (err) {
|
|
59
|
+
failed += 1;
|
|
60
|
+
logger.warn?.(`[rag-indexer] embed failed for ${path} (${chunkLabel(ch)}): ${err.message}`);
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
logger.info?.(`[rag-indexer] ${relative(process.cwd(), path)} → ${indexed} chunk(s) (${failed} failed)`);
|
|
64
|
+
return { indexed, failed };
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
function chunkLabel(ch) {
|
|
68
|
+
return ch.metadata?.hu_id || ch.metadata?.symbol || ch.metadata?.headingPath?.join(" > ") || "block";
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
async function listFiles(dir, predicate) {
|
|
72
|
+
const out = [];
|
|
73
|
+
async function walk(d) {
|
|
74
|
+
let ents; try { ents = await readdir(d, { withFileTypes: true }); } catch { return; }
|
|
75
|
+
for (const e of ents) {
|
|
76
|
+
const p = join(d, e.name);
|
|
77
|
+
if (e.isDirectory()) await walk(p); else if (e.isFile() && predicate(p)) out.push(p);
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
await walk(dir); return out;
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/**
|
|
84
|
+
* Index every Karajan-managed asset for the project: every plan-*.json
|
|
85
|
+
* under `<karajanHome>/plans/<slug>/`, the onboarding brief under
|
|
86
|
+
* `<karajanHome>/onboarding/`, and (optionally) source files under the
|
|
87
|
+
* projectDir. Sources are gated behind `withSources: true` because
|
|
88
|
+
* indexing tens of thousands of files is expensive — the CLI exposes
|
|
89
|
+
* it as `--with-sources`.
|
|
90
|
+
*/
|
|
91
|
+
export async function indexProject(projectDir, { db, embedder, karajanHome, logger = console, withSources = false } = {}) {
|
|
92
|
+
const totals = { indexed: 0, failed: 0, files: 0 };
|
|
93
|
+
const slug = projectDir.split("/").pop()?.replace(/[^a-zA-Z0-9._-]/g, "-").toLowerCase() || "project";
|
|
94
|
+
const planRoot = join(karajanHome, PLANS_DIR, slug);
|
|
95
|
+
if (existsSync(planRoot)) {
|
|
96
|
+
const plans = await listFiles(planRoot, (p) => /plan-.*\.json$/.test(p));
|
|
97
|
+
for (const p of plans) {
|
|
98
|
+
const r = await indexFile(p, { db, embedder, logger });
|
|
99
|
+
totals.indexed += r.indexed; totals.failed += r.failed; totals.files += 1;
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
const onboarding = join(karajanHome, ONBOARDING_DIR, `${slug}.md`);
|
|
103
|
+
if (existsSync(onboarding)) {
|
|
104
|
+
const r = await indexFile(onboarding, { db, embedder, logger });
|
|
105
|
+
totals.indexed += r.indexed; totals.failed += r.failed; totals.files += 1;
|
|
106
|
+
}
|
|
107
|
+
if (withSources) {
|
|
108
|
+
const SKIP = new Set(["node_modules", ".git", "dist", "build", "coverage", ".karajan", ".next"]);
|
|
109
|
+
const sources = await listFiles(projectDir, (p) => /\.(js|mjs|cjs|ts|tsx|jsx)$/.test(p) && !p.split("/").some((seg) => SKIP.has(seg)));
|
|
110
|
+
for (const s of sources) {
|
|
111
|
+
const r = await indexFile(s, { db, embedder, logger });
|
|
112
|
+
totals.indexed += r.indexed; totals.failed += r.failed; totals.files += 1;
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
return totals;
|
|
116
|
+
}
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 5 — Retriever for the RAG pipeline. Composes the
|
|
2
|
+
// embedder (Step 2) + vec store (Step 1) into a one-call query API.
|
|
3
|
+
// The CLI (Step 6) and the future MCP tool are the only consumers.
|
|
4
|
+
|
|
5
|
+
import { searchSimilar } from "./vec-store.js";
|
|
6
|
+
|
|
7
|
+
// Kind boosts for rerank: a plan chunk is more "intentional" than a
|
|
8
|
+
// raw code chunk, an onboarding brief sits between them. Tunable via
|
|
9
|
+
// the third argument; the defaults are conservative — they break ties
|
|
10
|
+
// between equidistant chunks but won't reorder by big distance gaps.
|
|
11
|
+
const DEFAULT_KIND_BOOST = { plan: 0.05, onboarding: 0.03, code: 0 };
|
|
12
|
+
|
|
13
|
+
/**
|
|
14
|
+
* Run a natural-language query against the indexed RAG corpus. Returns
|
|
15
|
+
* the top-K chunks ranked by adjusted distance (lower = closer).
|
|
16
|
+
*
|
|
17
|
+
* query(db, embedder, "how did I handle auth in module X?",
|
|
18
|
+
* { topK: 5, scope: 'plans' })
|
|
19
|
+
*
|
|
20
|
+
* @param {Database} db — opened via openVecStore()
|
|
21
|
+
* @param {OllamaEmbedder} embedder
|
|
22
|
+
* @param {string} text — the query
|
|
23
|
+
* @param {Object} [opts]
|
|
24
|
+
* @param {number} [opts.topK=5]
|
|
25
|
+
* @param {'plans'|'code'|'onboarding'|'all'} [opts.scope='all']
|
|
26
|
+
* @param {Object} [opts.kindBoost]
|
|
27
|
+
*/
|
|
28
|
+
export async function query(db, embedder, text, { topK = 5, scope = "all", kindBoost = DEFAULT_KIND_BOOST } = {}) {
|
|
29
|
+
if (!text || typeof text !== "string") throw new Error("query: text must be a non-empty string");
|
|
30
|
+
const queryEmbedding = await embedder.embed(text);
|
|
31
|
+
// Over-fetch (topK * 2) so the rerank step has room to reshuffle
|
|
32
|
+
// without dropping a relevant chunk past the cut. Clamped at 50 to
|
|
33
|
+
// avoid loading the entire store on tiny topK calls.
|
|
34
|
+
const fetchK = Math.min(50, topK * 2);
|
|
35
|
+
const scopeKind = scope === "plans" ? "plan" : scope === "code" ? "code" : scope === "onboarding" ? "onboarding" : null;
|
|
36
|
+
const raw = searchSimilar(db, queryEmbedding, fetchK, { kind: scopeKind });
|
|
37
|
+
if (raw.length === 0) return [];
|
|
38
|
+
// Adjust each chunk's distance by its kind boost. The vec store returns
|
|
39
|
+
// cosine distance (smaller = closer); subtracting the boost makes
|
|
40
|
+
// higher-priority kinds appear "closer" without altering the source
|
|
41
|
+
// truth in the DB.
|
|
42
|
+
const reranked = raw.map((r) => ({
|
|
43
|
+
...r,
|
|
44
|
+
score: r.distance - (kindBoost[r.kind] || 0),
|
|
45
|
+
})).sort((a, b) => a.score - b.score).slice(0, topK);
|
|
46
|
+
return reranked;
|
|
47
|
+
}
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
// KJC-PCS-0049 Step 1 — Vector store sobre better-sqlite3 + sqlite-vec.
|
|
2
|
+
// El RAG indexer (Step 4) escribe; el retriever (Step 5) lee. Una sola
|
|
3
|
+
// DB en `~/.karajan/rag.db` por instalación.
|
|
4
|
+
import { mkdirSync, existsSync } from "node:fs";
|
|
5
|
+
import { dirname, join } from "node:path";
|
|
6
|
+
import Database from "better-sqlite3";
|
|
7
|
+
import * as sqliteVec from "sqlite-vec";
|
|
8
|
+
|
|
9
|
+
import { getKarajanHome } from "../utils/paths.js";
|
|
10
|
+
|
|
11
|
+
const DEFAULT_DIM = 768;
|
|
12
|
+
|
|
13
|
+
export function dbPath() {
|
|
14
|
+
return process.env.KJ_RAG_DB || join(getKarajanHome(), "rag.db");
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
/**
|
|
18
|
+
* Open the rag.db, load the sqlite-vec extension, and ensure the
|
|
19
|
+
* chunks + vec_chunks tables exist. Idempotent: a second call against
|
|
20
|
+
* the same file is a no-op past the schema creation.
|
|
21
|
+
*
|
|
22
|
+
* Returns the open Database handle. Caller owns close().
|
|
23
|
+
*
|
|
24
|
+
* @param {Object} [opts]
|
|
25
|
+
* @param {number} [opts.dim=768] — embedding dimension. Must match
|
|
26
|
+
* the embedder configured for the project (Step 2). Changing it
|
|
27
|
+
* later requires a fresh DB; the constant is captured in the
|
|
28
|
+
* virtual table DDL.
|
|
29
|
+
*/
|
|
30
|
+
export function openVecStore({ dim = DEFAULT_DIM, path = dbPath() } = {}) {
|
|
31
|
+
if (!existsSync(dirname(path))) mkdirSync(dirname(path), { recursive: true });
|
|
32
|
+
const db = new Database(path);
|
|
33
|
+
sqliteVec.load(db);
|
|
34
|
+
db.pragma("journal_mode = WAL");
|
|
35
|
+
db.pragma("foreign_keys = ON");
|
|
36
|
+
db.exec(`
|
|
37
|
+
CREATE TABLE IF NOT EXISTS chunks (
|
|
38
|
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
39
|
+
source TEXT NOT NULL,
|
|
40
|
+
kind TEXT NOT NULL CHECK (kind IN ('plan', 'code', 'onboarding')),
|
|
41
|
+
text TEXT NOT NULL,
|
|
42
|
+
metadata TEXT,
|
|
43
|
+
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
44
|
+
);
|
|
45
|
+
CREATE INDEX IF NOT EXISTS chunks_by_source ON chunks(source);
|
|
46
|
+
CREATE INDEX IF NOT EXISTS chunks_by_kind ON chunks(kind);
|
|
47
|
+
`);
|
|
48
|
+
db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(embedding float[${dim}]);`);
|
|
49
|
+
return db;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
/**
|
|
53
|
+
* Persist one chunk. The chunks.id and vec_chunks.rowid stay in sync
|
|
54
|
+
* because the virtual table rowid is set explicitly to the freshly
|
|
55
|
+
* minted chunks.id. Returns that id.
|
|
56
|
+
*/
|
|
57
|
+
export function insertChunk(db, { source, kind, text, metadata = null, embedding }) {
|
|
58
|
+
if (!Array.isArray(embedding) && !ArrayBuffer.isView(embedding)) {
|
|
59
|
+
throw new Error("insertChunk: embedding must be an array or typed array of floats");
|
|
60
|
+
}
|
|
61
|
+
const meta = metadata == null ? null : (typeof metadata === "string" ? metadata : JSON.stringify(metadata));
|
|
62
|
+
const ins = db.prepare("INSERT INTO chunks (source, kind, text, metadata) VALUES (?, ?, ?, ?)");
|
|
63
|
+
const info = ins.run(source, kind, text, meta);
|
|
64
|
+
// sqlite-vec's vec0 virtual table requires the rowid to arrive as a BigInt
|
|
65
|
+
// even for values that comfortably fit in a Number. Better-sqlite3 returns
|
|
66
|
+
// lastInsertRowid as BigInt only when safeIntegers mode is on; force it.
|
|
67
|
+
const id = typeof info.lastInsertRowid === "bigint" ? info.lastInsertRowid : BigInt(info.lastInsertRowid);
|
|
68
|
+
const buf = embedding instanceof Float32Array ? embedding : Float32Array.from(embedding);
|
|
69
|
+
db.prepare("INSERT INTO vec_chunks(rowid, embedding) VALUES (?, ?)").run(id, Buffer.from(buf.buffer));
|
|
70
|
+
return Number(id);
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Cosine-ordered nearest-neighbour search. Returns `[{ id, source, kind,
|
|
75
|
+
* text, metadata, distance }]` for the topK best matches; lower
|
|
76
|
+
* distance = closer. Returns `[]` when the store is empty.
|
|
77
|
+
*/
|
|
78
|
+
export function searchSimilar(db, embedding, topK = 5, { kind = null } = {}) {
|
|
79
|
+
const buf = embedding instanceof Float32Array ? embedding : Float32Array.from(embedding);
|
|
80
|
+
const kindClause = kind ? "AND c.kind = ?" : "";
|
|
81
|
+
const sql = `
|
|
82
|
+
SELECT c.id, c.source, c.kind, c.text, c.metadata, v.distance
|
|
83
|
+
FROM vec_chunks v
|
|
84
|
+
JOIN chunks c ON c.id = v.rowid
|
|
85
|
+
WHERE v.embedding MATCH ? AND k = ? ${kindClause}
|
|
86
|
+
ORDER BY v.distance
|
|
87
|
+
`;
|
|
88
|
+
const params = [Buffer.from(buf.buffer), topK];
|
|
89
|
+
if (kind) params.push(kind);
|
|
90
|
+
const rows = db.prepare(sql).all(...params);
|
|
91
|
+
return rows.map((r) => ({ ...r, metadata: r.metadata ? safeParse(r.metadata) : null }));
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
function safeParse(s) { try { return JSON.parse(s); } catch { return s; } }
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Remove every chunk that came from a given source path. Both tables
|
|
98
|
+
* are touched explicitly because vec0 virtual tables do not honour
|
|
99
|
+
* SQLite foreign-key cascades.
|
|
100
|
+
*/
|
|
101
|
+
export function deleteChunksBySource(db, source) {
|
|
102
|
+
const ids = db.prepare("SELECT id FROM chunks WHERE source = ?").all(source).map((r) => r.id);
|
|
103
|
+
if (ids.length === 0) return 0;
|
|
104
|
+
const placeholders = ids.map(() => "?").join(",");
|
|
105
|
+
db.prepare(`DELETE FROM vec_chunks WHERE rowid IN (${placeholders})`).run(...ids);
|
|
106
|
+
db.prepare(`DELETE FROM chunks WHERE id IN (${placeholders})`).run(...ids);
|
|
107
|
+
return ids.length;
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
export function countChunks(db, { kind = null } = {}) {
|
|
111
|
+
if (kind) return db.prepare("SELECT COUNT(*) AS n FROM chunks WHERE kind = ?").get(kind).n;
|
|
112
|
+
return db.prepare("SELECT COUNT(*) AS n FROM chunks").get().n;
|
|
113
|
+
}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
// KJC-TSK-0384 PR 2 — Onboarder role. Wraps collectors bundle + LLM
|
|
2
|
+
// synthesis into a Markdown Architecture Brief.
|
|
3
|
+
import { AgentRole } from "./agent-role.js";
|
|
4
|
+
|
|
5
|
+
const FENCE_RE = /^```(?:markdown|md)?\s*\n([\s\S]*?)\n```\s*$/;
|
|
6
|
+
|
|
7
|
+
export class OnboarderRole extends AgentRole {
|
|
8
|
+
constructor(opts) {
|
|
9
|
+
super({ ...opts, name: "onboarder" });
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
extractInput(input) {
|
|
13
|
+
if (typeof input === "string") return { bundle: { projectDir: input } };
|
|
14
|
+
return { bundle: input?.bundle || input || { projectDir: "" } };
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
async buildPrompt({ bundle }) {
|
|
18
|
+
const instructions = this.instructions || "";
|
|
19
|
+
const prompt = [
|
|
20
|
+
instructions,
|
|
21
|
+
"",
|
|
22
|
+
"## Bundle (deterministic collectors output)",
|
|
23
|
+
"",
|
|
24
|
+
"```json",
|
|
25
|
+
JSON.stringify(bundle, null, 2),
|
|
26
|
+
"```",
|
|
27
|
+
"",
|
|
28
|
+
"Produce the Architecture Brief now.",
|
|
29
|
+
].join("\n");
|
|
30
|
+
return { prompt };
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
parseOutput(raw) {
|
|
34
|
+
if (!raw || typeof raw !== "string") return { brief: "" };
|
|
35
|
+
const trimmed = raw.trim();
|
|
36
|
+
const fenced = trimmed.match(FENCE_RE);
|
|
37
|
+
return { brief: fenced ? fenced[1].trim() : trimmed };
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
buildSuccessResult(parsed, provider) {
|
|
41
|
+
return { brief: parsed.brief, provider };
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
buildSummary(parsed) {
|
|
45
|
+
const lines = (parsed.brief || "").split("\n").length;
|
|
46
|
+
return `Architecture brief composed (${lines} line${lines === 1 ? "" : "s"})`;
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
handleParseNull(agentResult, provider) {
|
|
50
|
+
return {
|
|
51
|
+
ok: true,
|
|
52
|
+
result: { brief: agentResult.output || "", provider },
|
|
53
|
+
summary: "Architecture brief composed (raw)",
|
|
54
|
+
usage: agentResult.usage,
|
|
55
|
+
};
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
handleParseError(_err, agentResult, provider) {
|
|
59
|
+
return this.handleParseNull(agentResult, provider);
|
|
60
|
+
}
|
|
61
|
+
}
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Onboarder
|
|
2
|
+
|
|
3
|
+
Onboarder role of Karajan. Take a deterministic collectors bundle from a
|
|
4
|
+
brownfield project and emit a concise Markdown **Architecture Brief** for
|
|
5
|
+
downstream roles (planner, architect, researcher, coder).
|
|
6
|
+
|
|
7
|
+
## Mode
|
|
8
|
+
|
|
9
|
+
Read-only. NO Bash/Write/Edit/MultiEdit/NotebookEdit. Output the Markdown
|
|
10
|
+
brief as plain text in your response, NOT via tools.
|
|
11
|
+
|
|
12
|
+
If the project is greenfield (no git, no configs, empty tree) emit
|
|
13
|
+
`# Project is greenfield` plus whatever signal exists. NEVER error out.
|
|
14
|
+
|
|
15
|
+
## Input
|
|
16
|
+
|
|
17
|
+
JSON bundle with `projectDir`, `stack`, `tree`, `git` (or null), `configs`,
|
|
18
|
+
`adrs`. Slots may be null/empty.
|
|
19
|
+
|
|
20
|
+
## Output
|
|
21
|
+
|
|
22
|
+
Markdown only — skip any section without data:
|
|
23
|
+
|
|
24
|
+
```markdown
|
|
25
|
+
# Architecture Brief — <projectDir>
|
|
26
|
+
## Stack
|
|
27
|
+
## Layout
|
|
28
|
+
## Configs
|
|
29
|
+
## Git history
|
|
30
|
+
## ADRs
|
|
31
|
+
## Conventions inferred
|
|
32
|
+
## Recommendations for new work
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Be conservative: only assert what the bundle shows. "Likely" / "appears to"
|
|
36
|
+
are fine; invention is not. Aim for ~40–80 lines total.
|