pi-knowledge 0.5.4 → 0.5.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +5 -2
- package/dist/index.js +59 -4
- package/dist/src/engine.d.ts +3 -0
- package/dist/src/engine.js +35 -19
- package/dist/src/model-worker-client.d.ts +3 -0
- package/dist/src/model-worker-client.js +44 -1
- package/dist/src/runtime-config.d.ts +7 -0
- package/dist/src/runtime-config.js +36 -0
- package/dist/src/search/tuning.d.ts +41 -0
- package/dist/src/search/tuning.js +190 -0
- package/docs/configuration.md +18 -5
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,22 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.5.6] - 2026-07-23
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Added runtime search tuning profiles (`auto`, `low_token`, `precision`, `recall`, `long_context`, `code`, `docs`, `balanced`) with diagnostics that report the applied result limit, snippet length, hybrid threshold, candidate pool, adaptive context, and deep rerank breadth.
|
|
7
|
+
|
|
8
|
+
### Changed
|
|
9
|
+
- Added bounded environment overrides for search defaults so slow local model users can reduce result count, lengthen snippets, tighten hybrid precision, or tune candidate breadth without editing source.
|
|
10
|
+
|
|
11
|
+
## [0.5.5] - 2026-07-23
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
- Added `knowledge_configure` so agents can validate and persist a Node 22+ executable for the isolated local model worker without relying on environment variables injected after OMP startup.
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
- Auto-discovers Codex bundled Windows Node runtimes under `%LOCALAPPDATA%\OpenAI\Codex\runtimes\cua_node\*\bin\node.exe` for OMP sessions without global `node` on `PATH`.
|
|
18
|
+
- Reuses persisted runtime config before falling back to process or PATH discovery, so one successful configuration survives future OMP sessions.
|
|
19
|
+
|
|
3
20
|
## [0.5.4] - 2026-07-23
|
|
4
21
|
|
|
5
22
|
### Fixed
|
package/README.md
CHANGED
|
@@ -110,6 +110,7 @@ omp install ./pi-knowledge
|
|
|
110
110
|
| `knowledge_show` | List all knowledge bases with stats |
|
|
111
111
|
| `knowledge_status` | Show engine status with health diagnostics (stale, orphans, coverage) |
|
|
112
112
|
| `knowledge_doctor` | Diagnose health score, skipped files, stuck jobs, stale data, and recommended fixes |
|
|
113
|
+
| `knowledge_configure` | Persist runtime prerequisites such as a Windows `node.exe` path for the isolated local model worker |
|
|
113
114
|
| `knowledge_clear` | Remove all knowledge bases after `confirm: true` |
|
|
114
115
|
| `knowledge_export` | Export a KB to shareable JSONL file |
|
|
115
116
|
| `knowledge_import` | Import a KB from JSONL (re-embeds content) |
|
|
@@ -124,6 +125,8 @@ omp install ./pi-knowledge
|
|
|
124
125
|
- `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
|
|
125
126
|
- `code`, `config`, `errors`, `docs`, `decision`: intent aliases for agents. `code`, `config`, and `errors` bias toward exact lexical/symbol evidence; `docs` and `decision` keep the intended source type explicit.
|
|
126
127
|
|
|
128
|
+
Search profiles tune result count, snippet length, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth. `profile: "auto"` selects a runtime profile from query shape, mode, and KB source type; explicit tool parameters still win over profile defaults. Use `profile: "low_token"` for slow local models that need fewer, stricter, longer results in one call; use `precision` for identifiers/errors, `recall` for broad discovery, `long_context` for prose/specs, and `balanced` for compatibility.
|
|
129
|
+
|
|
127
130
|
Mode selection contract:
|
|
128
131
|
|
|
129
132
|
- Start with `hybrid` for most project questions that contain useful lexical anchors.
|
|
@@ -133,7 +136,7 @@ Mode selection contract:
|
|
|
133
136
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
134
137
|
- If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
|
|
135
138
|
|
|
136
|
-
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, and source freshness where available.
|
|
139
|
+
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, and source freshness where available.
|
|
137
140
|
|
|
138
141
|
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and code symbol names while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, or functions, and reduces duplicate-looking chunk hits.
|
|
139
142
|
Symbol/config/heading metadata is also rebuilt during `knowledge_update`; older KBs created before this feature may show a doctor action recommending update.
|
|
@@ -175,7 +178,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
175
178
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
176
179
|
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
177
180
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
178
|
-
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE` |
|
|
181
|
+
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
179
182
|
| Release fixtures | `PI_KNOWLEDGE_E2E_PDF`, `PI_KNOWLEDGE_E2E_DOCX` |
|
|
180
183
|
|
|
181
184
|
## Pi and OMP Support
|
package/dist/index.js
CHANGED
|
@@ -248,6 +248,41 @@ export default function (pi) {
|
|
|
248
248
|
};
|
|
249
249
|
},
|
|
250
250
|
});
|
|
251
|
+
pi.registerTool({
|
|
252
|
+
name: "knowledge_configure",
|
|
253
|
+
label: "Knowledge Configure",
|
|
254
|
+
description: "Configure pi-knowledge runtime settings such as the local model worker Node executable",
|
|
255
|
+
promptSnippet: "Configure local pi-knowledge runtime prerequisites before indexing",
|
|
256
|
+
promptGuidelines: [
|
|
257
|
+
"Use knowledge_configure when knowledge_doctor or knowledge_add reports that local embeddings cannot find a usable Node executable",
|
|
258
|
+
"On Windows, pass the full node.exe path when the user provides one; paths with spaces are accepted",
|
|
259
|
+
"If node_path is omitted, the tool auto-discovers a usable Node 22+ executable and persists it for future sessions",
|
|
260
|
+
"Do not switch users to API embeddings unless they explicitly choose a network embedding provider",
|
|
261
|
+
],
|
|
262
|
+
parameters: Type.Object({
|
|
263
|
+
node_path: Type.Optional(Type.String({
|
|
264
|
+
description: "Optional full path to a Node 22+ executable for the isolated local embedding model worker, such as C:\\Program Files\\nodejs\\node.exe",
|
|
265
|
+
})),
|
|
266
|
+
}),
|
|
267
|
+
async execute(_id, params, _signal) {
|
|
268
|
+
if (_signal?.aborted)
|
|
269
|
+
throw new Error("Cancelled");
|
|
270
|
+
const { configureModelWorkerNodePath } = await import(__rewriteRelativeImportExtension(runtimeModule("./src/model-worker-client")));
|
|
271
|
+
if (_signal?.aborted)
|
|
272
|
+
throw new Error("Cancelled");
|
|
273
|
+
const nodePath = typeof params.node_path === "string" && params.node_path.trim() ? params.node_path : undefined;
|
|
274
|
+
const configuredPath = configureModelWorkerNodePath(nodePath);
|
|
275
|
+
return {
|
|
276
|
+
content: [
|
|
277
|
+
{
|
|
278
|
+
type: "text",
|
|
279
|
+
text: `Configured local model worker Node executable: ${configuredPath}`,
|
|
280
|
+
},
|
|
281
|
+
],
|
|
282
|
+
details: { node_path: configuredPath },
|
|
283
|
+
};
|
|
284
|
+
},
|
|
285
|
+
});
|
|
251
286
|
pi.registerTool({
|
|
252
287
|
name: "knowledge_add",
|
|
253
288
|
label: "Knowledge Add",
|
|
@@ -328,6 +363,7 @@ export default function (pi) {
|
|
|
328
363
|
"Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording or hybrid returns no lexical matches",
|
|
329
364
|
"Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
|
|
330
365
|
"Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
|
|
366
|
+
"Use profile 'low_token' for slow local models that need fewer, stricter, longer snippets; use profile 'precision' for identifiers/errors and 'recall' or 'long_context' for broad prose/document research",
|
|
331
367
|
"If a search returns no results or obviously weak results, retry once with a different mode before concluding the KB lacks the answer",
|
|
332
368
|
"If top results are repetitive, retry with diversity 'strong' or mode 'adaptive' instead of increasing the limit first",
|
|
333
369
|
],
|
|
@@ -346,7 +382,17 @@ export default function (pi) {
|
|
|
346
382
|
Type.Literal("errors"),
|
|
347
383
|
Type.Literal("decision"),
|
|
348
384
|
])),
|
|
349
|
-
|
|
385
|
+
profile: Type.Optional(Type.Union([
|
|
386
|
+
Type.Literal("auto"),
|
|
387
|
+
Type.Literal("balanced"),
|
|
388
|
+
Type.Literal("low_token"),
|
|
389
|
+
Type.Literal("precision"),
|
|
390
|
+
Type.Literal("recall"),
|
|
391
|
+
Type.Literal("long_context"),
|
|
392
|
+
Type.Literal("code"),
|
|
393
|
+
Type.Literal("docs"),
|
|
394
|
+
])),
|
|
395
|
+
limit: Type.Optional(Type.Number({ description: "Max results (default comes from search profile/env)" })),
|
|
350
396
|
kb_id: Type.Optional(Type.String({ description: "Limit search to a specific KB by ID or exact name" })),
|
|
351
397
|
offset: Type.Optional(Type.Number({ description: "Pagination offset" })),
|
|
352
398
|
file_type: Type.Optional(Type.String({ description: "Filter by file type (e.g. typescript, markdown, python)" })),
|
|
@@ -356,15 +402,18 @@ export default function (pi) {
|
|
|
356
402
|
}),
|
|
357
403
|
async execute(_id, params, _signal) {
|
|
358
404
|
const { engine } = await ensureInitialized();
|
|
359
|
-
const { query, mode, limit, kb_id, offset, file_type, path_pattern, diversity, diagnostics } = params;
|
|
405
|
+
const { query, mode, profile, limit, kb_id, offset, file_type, path_pattern, diversity, diagnostics } = params;
|
|
360
406
|
const filters = file_type || path_pattern ? { file_type, path_pattern } : undefined;
|
|
361
|
-
const response = await engine.search(query, { mode, limit, kb_id, offset, filters, diversity }, _signal);
|
|
407
|
+
const response = await engine.search(query, { mode, profile, limit, kb_id, offset, filters, diversity }, _signal);
|
|
362
408
|
if (response.results.length === 0) {
|
|
363
409
|
const details = [
|
|
364
410
|
"No results found.",
|
|
365
411
|
response.mode_used ? `Mode used: ${response.mode_used}` : "",
|
|
366
412
|
response.retry_modes?.length ? `Retried: ${response.retry_modes.join(", ")}` : "",
|
|
367
413
|
response.warnings?.length ? `Warnings: ${response.warnings.join(" | ")}` : "",
|
|
414
|
+
diagnostics && response.tuning
|
|
415
|
+
? `Tuning: profile=${response.tuning.requested_profile}->${response.tuning.selected_profile}, limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.candidate_limit}`
|
|
416
|
+
: "",
|
|
368
417
|
response.suggestions?.length ? `Suggestions: ${response.suggestions.join(" | ")}` : "",
|
|
369
418
|
]
|
|
370
419
|
.filter(Boolean)
|
|
@@ -376,6 +425,9 @@ export default function (pi) {
|
|
|
376
425
|
output += ` — mode: ${response.mode_used}`;
|
|
377
426
|
if (response.retry_modes?.length)
|
|
378
427
|
output += ` — retried: ${response.retry_modes.join(", ")}`;
|
|
428
|
+
if (diagnostics && response.tuning) {
|
|
429
|
+
output += ` — profile: ${response.tuning.requested_profile}->${response.tuning.selected_profile}`;
|
|
430
|
+
}
|
|
379
431
|
output += ":\n\n";
|
|
380
432
|
if (response.warnings?.length) {
|
|
381
433
|
output = `Warnings:\n- ${response.warnings.join("\n- ")}\n\n${output}`;
|
|
@@ -388,7 +440,10 @@ export default function (pi) {
|
|
|
388
440
|
const provenance = diagnostics && r.provenance
|
|
389
441
|
? `\nProvenance: chunk=${r.provenance.chunk_id}, hash=${r.provenance.chunk_hash.slice(0, 12)}, reason=${r.provenance.match_reason}, indexed_at=${r.provenance.indexed_at}, source_mtime=${r.provenance.source_mtime ?? "unknown"}, stale=${r.provenance.stale}`
|
|
390
442
|
: "";
|
|
391
|
-
|
|
443
|
+
const tuning = diagnostics && response.tuning && i === 0
|
|
444
|
+
? `\nTuning: limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.candidate_limit}`
|
|
445
|
+
: "";
|
|
446
|
+
return `[${i + 1}] ${r.file_path} (${r.kb_name}, score: ${r.score.toFixed(3)})\n${r.snippet}${diag}${provenance}${tuning}`;
|
|
392
447
|
})
|
|
393
448
|
.join("\n\n");
|
|
394
449
|
return { content: [{ type: "text", text: output }], details: diagnostics ? response : undefined };
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -1,10 +1,12 @@
|
|
|
1
1
|
import { type DiagnosticResult } from "./diagnostics/health.ts";
|
|
2
2
|
import { createSkippedScanStats } from "./indexer/chunker.ts";
|
|
3
3
|
import { type RankingDiagnostics } from "./search/ranking.ts";
|
|
4
|
+
import { type SearchProfile, type SearchTuningSummary } from "./search/tuning.ts";
|
|
4
5
|
import { type KnowledgeBase, type KnowledgeSymbol } from "./storage/sqlite.ts";
|
|
5
6
|
export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
6
7
|
export interface SearchOptions {
|
|
7
8
|
mode?: SearchMode;
|
|
9
|
+
profile?: SearchProfile;
|
|
8
10
|
limit?: number;
|
|
9
11
|
offset?: number;
|
|
10
12
|
kb_id?: string;
|
|
@@ -43,6 +45,7 @@ export interface SearchResponse {
|
|
|
43
45
|
mode_used?: SearchMode;
|
|
44
46
|
retry_modes?: SearchMode[];
|
|
45
47
|
suggestions?: string[];
|
|
48
|
+
tuning?: SearchTuningSummary;
|
|
46
49
|
}
|
|
47
50
|
export type ProgressCallback = (msg: string) => void;
|
|
48
51
|
export interface AddOptions {
|
package/dist/src/engine.js
CHANGED
|
@@ -9,14 +9,12 @@ import { shutdownModelWorker } from "./model-worker-client.js";
|
|
|
9
9
|
import { searchBM25 } from "./search/bm25.js";
|
|
10
10
|
import { weightedScoreFusion } from "./search/fusion.js";
|
|
11
11
|
import { normalizedQueryText, tokenizeForSearch } from "./search/query.js";
|
|
12
|
-
import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence,
|
|
12
|
+
import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence, normalizeFileTypeFilter, queryCoverage, scoreChunkForQuery, } from "./search/ranking.js";
|
|
13
13
|
import { disposeReranker, prepareRerankerForShutdown, rerank } from "./search/reranker.js";
|
|
14
|
+
import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js";
|
|
14
15
|
import { searchVectorFile } from "./search/vector.js";
|
|
15
16
|
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBStatus, } from "./storage/sqlite.js";
|
|
16
17
|
export const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
17
|
-
const ADAPTIVE_CONTEXT_LINES = 80;
|
|
18
|
-
const ADAPTIVE_MAX_CONTEXT_CHARS = 6_000;
|
|
19
|
-
const ADAPTIVE_NEIGHBOR_TARGET = 5;
|
|
20
18
|
const INDEX_EMBED_BATCH_SIZE = 64;
|
|
21
19
|
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
22
20
|
function isCancellationError(error) {
|
|
@@ -256,7 +254,7 @@ function buildQuerySnippet(content, query, maxLength = 240) {
|
|
|
256
254
|
const suffix = end < content.length ? "..." : "";
|
|
257
255
|
return `${prefix}${content.slice(start, end)}${suffix}`;
|
|
258
256
|
}
|
|
259
|
-
function buildAdaptiveContext(seed, chunks, queryTokens) {
|
|
257
|
+
function buildAdaptiveContext(seed, chunks, queryTokens, options) {
|
|
260
258
|
const scored = chunks
|
|
261
259
|
.map((chunk) => {
|
|
262
260
|
const distance = chunk.id === seed.id
|
|
@@ -268,7 +266,7 @@ function buildAdaptiveContext(seed, chunks, queryTokens) {
|
|
|
268
266
|
return { chunk, score: seedBoost + proximity + coverage };
|
|
269
267
|
})
|
|
270
268
|
.sort((a, b) => b.score - a.score)
|
|
271
|
-
.slice(0,
|
|
269
|
+
.slice(0, options.neighborTarget);
|
|
272
270
|
if (!scored.some((item) => item.chunk.id === seed.id))
|
|
273
271
|
scored.push({ chunk: seed, score: Number.MAX_SAFE_INTEGER });
|
|
274
272
|
const selectedIds = new Set(scored.map((item) => item.chunk.id));
|
|
@@ -279,9 +277,9 @@ function buildAdaptiveContext(seed, chunks, queryTokens) {
|
|
|
279
277
|
const sourceChunkIds = [];
|
|
280
278
|
let totalLength = 0;
|
|
281
279
|
for (const chunk of ordered) {
|
|
282
|
-
if (totalLength >=
|
|
280
|
+
if (totalLength >= options.maxContextChars)
|
|
283
281
|
break;
|
|
284
|
-
const remaining =
|
|
282
|
+
const remaining = options.maxContextChars - totalLength;
|
|
285
283
|
const text = chunk.content.slice(0, remaining);
|
|
286
284
|
parts.push(text);
|
|
287
285
|
sourceChunkIds.push(chunk.id);
|
|
@@ -1109,12 +1107,14 @@ export class KnowledgeEngine {
|
|
|
1109
1107
|
const attempts = [primaryMode, ...fallbackModesFor(primaryMode)];
|
|
1110
1108
|
const tried = [];
|
|
1111
1109
|
const warnings = [];
|
|
1110
|
+
let lastTuning;
|
|
1112
1111
|
for (const mode of attempts) {
|
|
1113
1112
|
if (tried.includes(mode))
|
|
1114
1113
|
continue;
|
|
1115
1114
|
tried.push(mode);
|
|
1116
1115
|
throwIfAborted(signal);
|
|
1117
1116
|
const response = await this.search(query, { ...options, mode }, signal);
|
|
1117
|
+
lastTuning = response.tuning;
|
|
1118
1118
|
if (response.warnings)
|
|
1119
1119
|
warnings.push(...response.warnings);
|
|
1120
1120
|
if (!isWeakAutoResponse(query, response, primaryMode, mode)) {
|
|
@@ -1136,16 +1136,16 @@ export class KnowledgeEngine {
|
|
|
1136
1136
|
warnings: warnings.length > 0 ? [...new Set(warnings)] : undefined,
|
|
1137
1137
|
mode_used: tried.at(-1),
|
|
1138
1138
|
retry_modes: tried.slice(0, -1),
|
|
1139
|
+
tuning: lastTuning,
|
|
1139
1140
|
suggestions: [
|
|
1140
1141
|
"No results after auto mode fallback. Check knowledge_status, try a more exact term, or rebuild the KB if indexing rules changed.",
|
|
1141
1142
|
],
|
|
1142
1143
|
};
|
|
1143
1144
|
}
|
|
1144
1145
|
const db = this.db;
|
|
1145
|
-
const { mode = "hybrid",
|
|
1146
|
+
const { mode = "hybrid", offset = 0, kb_id, filters, diversity = "balanced" } = options;
|
|
1146
1147
|
const resolvedMode = retrievalModeFor(mode);
|
|
1147
1148
|
const retrievalMode = resolvedMode === "adaptive" ? "hybrid" : resolvedMode;
|
|
1148
|
-
const candidateLimit = Math.max(50, offset + limit * 12);
|
|
1149
1149
|
const normalizedQuery = normalizedQueryText(query);
|
|
1150
1150
|
const queryTokens = tokenizeForSimilarity(query);
|
|
1151
1151
|
const normalizedFileType = normalizeFileTypeFilter(filters?.file_type);
|
|
@@ -1159,6 +1159,17 @@ export class KnowledgeEngine {
|
|
|
1159
1159
|
warnings.push(`"${kb.name}" is ${kb.status}; search skipped until indexing is ready`);
|
|
1160
1160
|
}
|
|
1161
1161
|
}
|
|
1162
|
+
const tuning = resolveSearchTuning({
|
|
1163
|
+
query,
|
|
1164
|
+
mode,
|
|
1165
|
+
profile: options.profile,
|
|
1166
|
+
kbSourceTypes: kbs.map((kb) => kb.source_type),
|
|
1167
|
+
});
|
|
1168
|
+
const limit = typeof options.limit === "number" && Number.isFinite(options.limit) && options.limit > 0
|
|
1169
|
+
? Math.trunc(options.limit)
|
|
1170
|
+
: tuning.defaultLimit;
|
|
1171
|
+
const candidateLimit = Math.max(tuning.candidateMin, offset + limit * tuning.candidateMultiplier);
|
|
1172
|
+
const tuningSummary = summarizeSearchTuning(tuning, limit, candidateLimit);
|
|
1162
1173
|
if (kbs.length === 0) {
|
|
1163
1174
|
return {
|
|
1164
1175
|
results: [],
|
|
@@ -1166,6 +1177,7 @@ export class KnowledgeEngine {
|
|
|
1166
1177
|
has_more: false,
|
|
1167
1178
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1168
1179
|
mode_used: mode,
|
|
1180
|
+
tuning: tuningSummary,
|
|
1169
1181
|
};
|
|
1170
1182
|
}
|
|
1171
1183
|
const allResults = [];
|
|
@@ -1240,7 +1252,7 @@ export class KnowledgeEngine {
|
|
|
1240
1252
|
const chunk = getChunkById(db, result.chunkId);
|
|
1241
1253
|
if (!chunk)
|
|
1242
1254
|
return false;
|
|
1243
|
-
return result.score >=
|
|
1255
|
+
return result.score >= tuning.minHybridScore && hasEnoughLexicalEvidence(chunk, queryTokens);
|
|
1244
1256
|
});
|
|
1245
1257
|
}
|
|
1246
1258
|
scored.sort((a, b) => b.score - a.score);
|
|
@@ -1260,14 +1272,13 @@ export class KnowledgeEngine {
|
|
|
1260
1272
|
}
|
|
1261
1273
|
if (mode === "deep" && filtered.length > 0) {
|
|
1262
1274
|
const candidates = filtered
|
|
1263
|
-
.slice(0,
|
|
1275
|
+
.slice(0, tuning.deepRerankCandidates)
|
|
1264
1276
|
.map((r) => {
|
|
1265
1277
|
const chunk = getChunkById(db, r.chunkId);
|
|
1266
1278
|
return chunk ? { chunkId: r.chunkId, content: chunk.content } : null;
|
|
1267
1279
|
})
|
|
1268
1280
|
.filter(Boolean);
|
|
1269
|
-
const reranked = await rerank(query, candidates, Math.max(limit *
|
|
1270
|
-
throwIfAborted(signal);
|
|
1281
|
+
const reranked = await rerank(query, candidates, Math.max(limit * tuning.deepRerankTopKMultiplier, limit), signal);
|
|
1271
1282
|
const ranked = [];
|
|
1272
1283
|
for (const r of reranked) {
|
|
1273
1284
|
throwIfAborted(signal);
|
|
@@ -1281,7 +1292,7 @@ export class KnowledgeEngine {
|
|
|
1281
1292
|
ranking: scoreChunkForQuery(r.score, chunk, queryTokens),
|
|
1282
1293
|
score: r.score,
|
|
1283
1294
|
content: chunk.content,
|
|
1284
|
-
snippet: buildQuerySnippet(chunk.content, query),
|
|
1295
|
+
snippet: buildQuerySnippet(chunk.content, query, tuning.snippetMaxLength),
|
|
1285
1296
|
startLine: chunk.start_line,
|
|
1286
1297
|
endLine: chunk.end_line,
|
|
1287
1298
|
sourceChunkIds: [chunk.id],
|
|
@@ -1319,6 +1330,7 @@ export class KnowledgeEngine {
|
|
|
1319
1330
|
has_more: offset + limit < diversified.length,
|
|
1320
1331
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1321
1332
|
mode_used: mode,
|
|
1333
|
+
tuning: tuningSummary,
|
|
1322
1334
|
};
|
|
1323
1335
|
}
|
|
1324
1336
|
const ranked = [];
|
|
@@ -1329,15 +1341,18 @@ export class KnowledgeEngine {
|
|
|
1329
1341
|
continue;
|
|
1330
1342
|
const kb = getKB(db, chunk.kb_id);
|
|
1331
1343
|
if (mode === "adaptive") {
|
|
1332
|
-
const contextChunks = getChunksByFile(db, chunk.kb_id, chunk.file_path, Math.max(1, chunk.start_line -
|
|
1333
|
-
const context = buildAdaptiveContext(chunk, contextChunks.length > 0 ? contextChunks : [chunk], queryTokens
|
|
1344
|
+
const contextChunks = getChunksByFile(db, chunk.kb_id, chunk.file_path, Math.max(1, chunk.start_line - tuning.adaptiveContextLines), chunk.end_line + tuning.adaptiveContextLines);
|
|
1345
|
+
const context = buildAdaptiveContext(chunk, contextChunks.length > 0 ? contextChunks : [chunk], queryTokens, {
|
|
1346
|
+
maxContextChars: tuning.adaptiveMaxContextChars,
|
|
1347
|
+
neighborTarget: tuning.adaptiveNeighborTarget,
|
|
1348
|
+
});
|
|
1334
1349
|
pushAdaptiveCandidate(ranked, {
|
|
1335
1350
|
chunk,
|
|
1336
1351
|
kbName: kb?.name ?? "unknown",
|
|
1337
1352
|
ranking: rankingByChunkId.get(r.chunkId),
|
|
1338
1353
|
score: r.score,
|
|
1339
1354
|
content: context.content,
|
|
1340
|
-
snippet: buildQuerySnippet(context.content, query),
|
|
1355
|
+
snippet: buildQuerySnippet(context.content, query, tuning.snippetMaxLength),
|
|
1341
1356
|
startLine: context.startLine,
|
|
1342
1357
|
endLine: context.endLine,
|
|
1343
1358
|
sourceChunkIds: context.sourceChunkIds,
|
|
@@ -1350,7 +1365,7 @@ export class KnowledgeEngine {
|
|
|
1350
1365
|
ranking: rankingByChunkId.get(r.chunkId),
|
|
1351
1366
|
score: r.score,
|
|
1352
1367
|
content: chunk.content,
|
|
1353
|
-
snippet: buildQuerySnippet(chunk.content, query),
|
|
1368
|
+
snippet: buildQuerySnippet(chunk.content, query, tuning.snippetMaxLength),
|
|
1354
1369
|
startLine: chunk.start_line,
|
|
1355
1370
|
endLine: chunk.end_line,
|
|
1356
1371
|
sourceChunkIds: [chunk.id],
|
|
@@ -1390,6 +1405,7 @@ export class KnowledgeEngine {
|
|
|
1390
1405
|
has_more: offset + limit < total,
|
|
1391
1406
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1392
1407
|
mode_used: mode,
|
|
1408
|
+
tuning: tuningSummary,
|
|
1393
1409
|
suggestions: results.length === 0
|
|
1394
1410
|
? [
|
|
1395
1411
|
"Try mode 'fast' for exact symbols or mode 'semantic' for conceptual wording.",
|
|
@@ -1,3 +1,6 @@
|
|
|
1
|
+
export declare function validateModelWorkerNodePath(nodePath: string): string;
|
|
2
|
+
export declare function resolveModelWorkerNodePath(): string;
|
|
3
|
+
export declare function configureModelWorkerNodePath(nodePath?: string): string;
|
|
1
4
|
export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", signal?: AbortSignal): Promise<Float32Array[]>;
|
|
2
5
|
export interface RerankWorkerCandidate {
|
|
3
6
|
chunkId: string;
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
import { execFileSync, fork, spawn } from "node:child_process";
|
|
2
|
-
import { existsSync } from "node:fs";
|
|
2
|
+
import { existsSync, readdirSync, statSync } from "node:fs";
|
|
3
3
|
import { basename, delimiter, join } from "node:path";
|
|
4
4
|
import { StringDecoder } from "node:string_decoder";
|
|
5
5
|
import { fileURLToPath } from "node:url";
|
|
6
|
+
import { getConfiguredNodePath, writeRuntimeConfig } from "./runtime-config.js";
|
|
6
7
|
let worker = null;
|
|
7
8
|
let nextRequestId = 1;
|
|
8
9
|
let resolvedNodeExecPath = null;
|
|
@@ -59,6 +60,25 @@ function appendIfPresent(candidates, value, suffix) {
|
|
|
59
60
|
return;
|
|
60
61
|
candidates.push(suffix ? join(normalized, suffix) : normalized);
|
|
61
62
|
}
|
|
63
|
+
function appendCodexNodeCandidates(candidates) {
|
|
64
|
+
const localAppData = process.env.LOCALAPPDATA ? normalizeExecutableCandidate(process.env.LOCALAPPDATA) : "";
|
|
65
|
+
if (!localAppData)
|
|
66
|
+
return;
|
|
67
|
+
const runtimeRoot = join(localAppData, "OpenAI", "Codex", "runtimes", "cua_node");
|
|
68
|
+
if (!existsSync(runtimeRoot))
|
|
69
|
+
return;
|
|
70
|
+
const discovered = readdirSync(runtimeRoot, { withFileTypes: true })
|
|
71
|
+
.filter((entry) => entry.isDirectory())
|
|
72
|
+
.map((entry) => {
|
|
73
|
+
const nodePath = join(runtimeRoot, entry.name, "bin", "node.exe");
|
|
74
|
+
const mtimeMs = existsSync(nodePath) ? statSync(nodePath).mtimeMs : 0;
|
|
75
|
+
return { nodePath, mtimeMs };
|
|
76
|
+
})
|
|
77
|
+
.filter((candidate) => candidate.mtimeMs > 0)
|
|
78
|
+
.sort((a, b) => b.mtimeMs - a.mtimeMs);
|
|
79
|
+
for (const candidate of discovered)
|
|
80
|
+
candidates.push(candidate.nodePath);
|
|
81
|
+
}
|
|
62
82
|
function getWindowsNodeCandidates() {
|
|
63
83
|
const candidates = [];
|
|
64
84
|
appendIfPresent(candidates, process.env.NVM_SYMLINK, "node.exe");
|
|
@@ -66,6 +86,7 @@ function getWindowsNodeCandidates() {
|
|
|
66
86
|
appendIfPresent(candidates, process.env.ProgramFiles, join("nodejs", "node.exe"));
|
|
67
87
|
appendIfPresent(candidates, process.env["ProgramFiles(x86)"], join("nodejs", "node.exe"));
|
|
68
88
|
appendIfPresent(candidates, process.env.LOCALAPPDATA, join("Programs", "nodejs", "node.exe"));
|
|
89
|
+
appendCodexNodeCandidates(candidates);
|
|
69
90
|
for (const directory of pathEnvValue().split(delimiter)) {
|
|
70
91
|
appendIfPresent(candidates, directory, "node.exe");
|
|
71
92
|
}
|
|
@@ -100,6 +121,16 @@ function getNodeExecPath() {
|
|
|
100
121
|
throw new Error(`PI_KNOWLEDGE_NODE_PATH does not point to a usable Node ${MIN_NODE_MAJOR}+ executable: ${toError(error).message}`);
|
|
101
122
|
}
|
|
102
123
|
}
|
|
124
|
+
const persisted = getConfiguredNodePath();
|
|
125
|
+
if (persisted) {
|
|
126
|
+
try {
|
|
127
|
+
resolvedNodeExecPath = validateNodeExecPath(normalizeExecutableCandidate(persisted));
|
|
128
|
+
return resolvedNodeExecPath;
|
|
129
|
+
}
|
|
130
|
+
catch {
|
|
131
|
+
// Persisted paths can point at removable Codex runtimes; continue with auto-discovery.
|
|
132
|
+
}
|
|
133
|
+
}
|
|
103
134
|
const execName = basename(process.execPath).toLowerCase();
|
|
104
135
|
const currentProcessLooksLikeNode = execName === "node" || execName === "node.exe";
|
|
105
136
|
if (currentProcessLooksLikeNode) {
|
|
@@ -120,6 +151,18 @@ function getNodeExecPath() {
|
|
|
120
151
|
}
|
|
121
152
|
throw new Error(`Node ${MIN_NODE_MAJOR}+ is required for local pi-knowledge embeddings, but no usable node executable was found. Install Node ${MIN_NODE_MAJOR}+ or set PI_KNOWLEDGE_NODE_PATH to the full node.exe path.`);
|
|
122
153
|
}
|
|
154
|
+
export function validateModelWorkerNodePath(nodePath) {
|
|
155
|
+
return validateNodeExecPath(normalizeExecutableCandidate(nodePath));
|
|
156
|
+
}
|
|
157
|
+
export function resolveModelWorkerNodePath() {
|
|
158
|
+
return getNodeExecPath();
|
|
159
|
+
}
|
|
160
|
+
export function configureModelWorkerNodePath(nodePath) {
|
|
161
|
+
const resolved = nodePath ? validateModelWorkerNodePath(nodePath) : getNodeExecPath();
|
|
162
|
+
writeRuntimeConfig({ node_path: resolved });
|
|
163
|
+
resolvedNodeExecPath = resolved;
|
|
164
|
+
return resolved;
|
|
165
|
+
}
|
|
123
166
|
function appendWorkerStderr(current, chunk) {
|
|
124
167
|
const next = `${current}${chunk.toString("utf-8")}`;
|
|
125
168
|
return next.length > WORKER_STDERR_TAIL_CHARS ? next.slice(-WORKER_STDERR_TAIL_CHARS) : next;
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
export type RuntimeConfig = {
|
|
2
|
+
node_path?: string;
|
|
3
|
+
updated_at?: number;
|
|
4
|
+
};
|
|
5
|
+
export declare function readRuntimeConfig(knowledgeDir?: string): RuntimeConfig;
|
|
6
|
+
export declare function getConfiguredNodePath(): string | undefined;
|
|
7
|
+
export declare function writeRuntimeConfig(update: RuntimeConfig, knowledgeDir?: string): RuntimeConfig;
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
|
2
|
+
import { join } from "node:path";
|
|
3
|
+
import { getDefaultKnowledgeDir } from "./storage/sqlite.js";
|
|
4
|
+
const CONFIG_FILE = "config.json";
|
|
5
|
+
function runtimeConfigPath(knowledgeDir = getDefaultKnowledgeDir()) {
|
|
6
|
+
return join(knowledgeDir, CONFIG_FILE);
|
|
7
|
+
}
|
|
8
|
+
function isRuntimeConfig(value) {
|
|
9
|
+
if (typeof value !== "object" || value === null)
|
|
10
|
+
return false;
|
|
11
|
+
if ("node_path" in value && value.node_path !== undefined && typeof value.node_path !== "string")
|
|
12
|
+
return false;
|
|
13
|
+
return !("updated_at" in value) || value.updated_at === undefined || typeof value.updated_at === "number";
|
|
14
|
+
}
|
|
15
|
+
export function readRuntimeConfig(knowledgeDir = getDefaultKnowledgeDir()) {
|
|
16
|
+
const configPath = runtimeConfigPath(knowledgeDir);
|
|
17
|
+
if (!existsSync(configPath))
|
|
18
|
+
return {};
|
|
19
|
+
const parsed = JSON.parse(readFileSync(configPath, "utf8"));
|
|
20
|
+
if (!isRuntimeConfig(parsed))
|
|
21
|
+
throw new Error(`Invalid pi-knowledge runtime config: ${configPath}`);
|
|
22
|
+
return parsed;
|
|
23
|
+
}
|
|
24
|
+
export function getConfiguredNodePath() {
|
|
25
|
+
return readRuntimeConfig().node_path?.trim() || undefined;
|
|
26
|
+
}
|
|
27
|
+
export function writeRuntimeConfig(update, knowledgeDir = getDefaultKnowledgeDir()) {
|
|
28
|
+
mkdirSync(knowledgeDir, { recursive: true });
|
|
29
|
+
const existing = readRuntimeConfig(knowledgeDir);
|
|
30
|
+
const next = { ...existing, ...update, updated_at: Date.now() };
|
|
31
|
+
const configPath = runtimeConfigPath(knowledgeDir);
|
|
32
|
+
const tempPath = `${configPath}.${process.pid}.tmp`;
|
|
33
|
+
writeFileSync(tempPath, `${JSON.stringify(next, null, 2)}\n`);
|
|
34
|
+
renameSync(tempPath, configPath);
|
|
35
|
+
return next;
|
|
36
|
+
}
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
export type SearchProfile = "auto" | "balanced" | "low_token" | "precision" | "recall" | "long_context" | "code" | "docs";
|
|
2
|
+
export type RuntimeProfile = Exclude<SearchProfile, "auto">;
|
|
3
|
+
type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
4
|
+
export interface SearchTuning {
|
|
5
|
+
requestedProfile: SearchProfile;
|
|
6
|
+
selectedProfile: RuntimeProfile;
|
|
7
|
+
defaultLimit: number;
|
|
8
|
+
snippetMaxLength: number;
|
|
9
|
+
minHybridScore: number;
|
|
10
|
+
candidateMin: number;
|
|
11
|
+
candidateMultiplier: number;
|
|
12
|
+
adaptiveContextLines: number;
|
|
13
|
+
adaptiveMaxContextChars: number;
|
|
14
|
+
adaptiveNeighborTarget: number;
|
|
15
|
+
deepRerankCandidates: number;
|
|
16
|
+
deepRerankTopKMultiplier: number;
|
|
17
|
+
}
|
|
18
|
+
export interface SearchTuningSummary {
|
|
19
|
+
requested_profile: SearchProfile;
|
|
20
|
+
selected_profile: RuntimeProfile;
|
|
21
|
+
limit: number;
|
|
22
|
+
snippet_max_length: number;
|
|
23
|
+
min_hybrid_score: number;
|
|
24
|
+
candidate_limit: number;
|
|
25
|
+
candidate_min: number;
|
|
26
|
+
candidate_multiplier: number;
|
|
27
|
+
adaptive_context_lines: number;
|
|
28
|
+
adaptive_max_context_chars: number;
|
|
29
|
+
adaptive_neighbor_target: number;
|
|
30
|
+
deep_rerank_candidates: number;
|
|
31
|
+
deep_rerank_top_k_multiplier: number;
|
|
32
|
+
}
|
|
33
|
+
interface SearchTuningInput {
|
|
34
|
+
query: string;
|
|
35
|
+
mode: SearchModeForTuning;
|
|
36
|
+
profile?: SearchProfile;
|
|
37
|
+
kbSourceTypes: string[];
|
|
38
|
+
}
|
|
39
|
+
export declare function resolveSearchTuning(input: SearchTuningInput): SearchTuning;
|
|
40
|
+
export declare function summarizeSearchTuning(tuning: SearchTuning, limit: number, candidateLimit: number): SearchTuningSummary;
|
|
41
|
+
export {};
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
import { MIN_HYBRID_SCORE } from "./ranking.js";
|
|
2
|
+
const PROFILE_DEFAULTS = {
|
|
3
|
+
balanced: {
|
|
4
|
+
defaultLimit: 10,
|
|
5
|
+
snippetMaxLength: 240,
|
|
6
|
+
minHybridScore: MIN_HYBRID_SCORE,
|
|
7
|
+
candidateMin: 50,
|
|
8
|
+
candidateMultiplier: 12,
|
|
9
|
+
adaptiveContextLines: 80,
|
|
10
|
+
adaptiveMaxContextChars: 6_000,
|
|
11
|
+
adaptiveNeighborTarget: 5,
|
|
12
|
+
deepRerankCandidates: 30,
|
|
13
|
+
deepRerankTopKMultiplier: 3,
|
|
14
|
+
},
|
|
15
|
+
low_token: {
|
|
16
|
+
defaultLimit: 5,
|
|
17
|
+
snippetMaxLength: 768,
|
|
18
|
+
minHybridScore: 0.4,
|
|
19
|
+
candidateMin: 35,
|
|
20
|
+
candidateMultiplier: 8,
|
|
21
|
+
adaptiveContextLines: 80,
|
|
22
|
+
adaptiveMaxContextChars: 6_000,
|
|
23
|
+
adaptiveNeighborTarget: 5,
|
|
24
|
+
deepRerankCandidates: 20,
|
|
25
|
+
deepRerankTopKMultiplier: 2,
|
|
26
|
+
},
|
|
27
|
+
precision: {
|
|
28
|
+
defaultLimit: 5,
|
|
29
|
+
snippetMaxLength: 400,
|
|
30
|
+
minHybridScore: 0.4,
|
|
31
|
+
candidateMin: 30,
|
|
32
|
+
candidateMultiplier: 6,
|
|
33
|
+
adaptiveContextLines: 60,
|
|
34
|
+
adaptiveMaxContextChars: 4_000,
|
|
35
|
+
adaptiveNeighborTarget: 4,
|
|
36
|
+
deepRerankCandidates: 20,
|
|
37
|
+
deepRerankTopKMultiplier: 2,
|
|
38
|
+
},
|
|
39
|
+
recall: {
|
|
40
|
+
defaultLimit: 15,
|
|
41
|
+
snippetMaxLength: 320,
|
|
42
|
+
minHybridScore: 0.12,
|
|
43
|
+
candidateMin: 75,
|
|
44
|
+
candidateMultiplier: 16,
|
|
45
|
+
adaptiveContextLines: 100,
|
|
46
|
+
adaptiveMaxContextChars: 8_000,
|
|
47
|
+
adaptiveNeighborTarget: 6,
|
|
48
|
+
deepRerankCandidates: 40,
|
|
49
|
+
deepRerankTopKMultiplier: 3,
|
|
50
|
+
},
|
|
51
|
+
long_context: {
|
|
52
|
+
defaultLimit: 5,
|
|
53
|
+
snippetMaxLength: 1_000,
|
|
54
|
+
minHybridScore: 0.25,
|
|
55
|
+
candidateMin: 50,
|
|
56
|
+
candidateMultiplier: 10,
|
|
57
|
+
adaptiveContextLines: 120,
|
|
58
|
+
adaptiveMaxContextChars: 10_000,
|
|
59
|
+
adaptiveNeighborTarget: 7,
|
|
60
|
+
deepRerankCandidates: 30,
|
|
61
|
+
deepRerankTopKMultiplier: 2,
|
|
62
|
+
},
|
|
63
|
+
code: {
|
|
64
|
+
defaultLimit: 10,
|
|
65
|
+
snippetMaxLength: 320,
|
|
66
|
+
minHybridScore: 0.25,
|
|
67
|
+
candidateMin: 50,
|
|
68
|
+
candidateMultiplier: 12,
|
|
69
|
+
adaptiveContextLines: 80,
|
|
70
|
+
adaptiveMaxContextChars: 6_000,
|
|
71
|
+
adaptiveNeighborTarget: 5,
|
|
72
|
+
deepRerankCandidates: 30,
|
|
73
|
+
deepRerankTopKMultiplier: 3,
|
|
74
|
+
},
|
|
75
|
+
docs: {
|
|
76
|
+
defaultLimit: 8,
|
|
77
|
+
snippetMaxLength: 768,
|
|
78
|
+
minHybridScore: 0.22,
|
|
79
|
+
candidateMin: 60,
|
|
80
|
+
candidateMultiplier: 12,
|
|
81
|
+
adaptiveContextLines: 120,
|
|
82
|
+
adaptiveMaxContextChars: 9_000,
|
|
83
|
+
adaptiveNeighborTarget: 7,
|
|
84
|
+
deepRerankCandidates: 30,
|
|
85
|
+
deepRerankTopKMultiplier: 3,
|
|
86
|
+
},
|
|
87
|
+
};
|
|
88
|
+
const SEARCH_PROFILES = new Set([
|
|
89
|
+
"auto",
|
|
90
|
+
"balanced",
|
|
91
|
+
"low_token",
|
|
92
|
+
"precision",
|
|
93
|
+
"recall",
|
|
94
|
+
"long_context",
|
|
95
|
+
"code",
|
|
96
|
+
"docs",
|
|
97
|
+
]);
|
|
98
|
+
function parseProfile(value) {
|
|
99
|
+
const normalized = value?.trim().toLowerCase();
|
|
100
|
+
if (!normalized)
|
|
101
|
+
return undefined;
|
|
102
|
+
return SEARCH_PROFILES.has(normalized) ? normalized : undefined;
|
|
103
|
+
}
|
|
104
|
+
function envNumber(name) {
|
|
105
|
+
const raw = process.env[name]?.trim();
|
|
106
|
+
if (!raw)
|
|
107
|
+
return undefined;
|
|
108
|
+
const value = Number(raw);
|
|
109
|
+
return Number.isFinite(value) ? value : undefined;
|
|
110
|
+
}
|
|
111
|
+
function clampInt(value, fallback, min, max) {
|
|
112
|
+
if (value === undefined)
|
|
113
|
+
return fallback;
|
|
114
|
+
return Math.min(max, Math.max(min, Math.trunc(value)));
|
|
115
|
+
}
|
|
116
|
+
function clampNumber(value, fallback, min, max) {
|
|
117
|
+
if (value === undefined)
|
|
118
|
+
return fallback;
|
|
119
|
+
return Math.min(max, Math.max(min, value));
|
|
120
|
+
}
|
|
121
|
+
function looksExactOrCodeQuery(query) {
|
|
122
|
+
const trimmed = query.trim();
|
|
123
|
+
if (trimmed.length === 0)
|
|
124
|
+
return false;
|
|
125
|
+
const lower = trimmed.toLowerCase();
|
|
126
|
+
if (/^[A-Z][A-Z0-9_]{2,}$/.test(trimmed))
|
|
127
|
+
return true;
|
|
128
|
+
if (/^[\w./-]+\.(ts|tsx|js|jsx|json|md|py|go|rs|java|yml|yaml|toml)$/.test(lower))
|
|
129
|
+
return true;
|
|
130
|
+
return /[A-Za-z_$][\w$]*\([^)]*\)|[A-Za-z_$][\w$]*\.[A-Za-z_$][\w$]*|\b[A-Z][A-Za-z0-9_$]*[A-Z][A-Za-z0-9_$]*\b|\b[a-z]+[A-Z][A-Za-z0-9_$]*\b/.test(trimmed);
|
|
131
|
+
}
|
|
132
|
+
function looksErrorQuery(query) {
|
|
133
|
+
return /\b(error|exception|failed|failure|stack|trace|enoent|eacces|timeout|abort|cancelled|錯誤|失敗)\b/i.test(query);
|
|
134
|
+
}
|
|
135
|
+
function looksLongContextQuery(query) {
|
|
136
|
+
const wordCount = query.trim().split(/\s+/).filter(Boolean).length;
|
|
137
|
+
return (wordCount >= 10 ||
|
|
138
|
+
/\b(explain|why|how|architecture|design|overview|docs?|spec|manual|guide)\b|流程|架構|概念|為什麼|如何|文件|規格/i.test(query));
|
|
139
|
+
}
|
|
140
|
+
function chooseRuntimeProfile(input) {
|
|
141
|
+
if (input.mode === "code" || input.mode === "config" || input.mode === "errors" || input.mode === "fast") {
|
|
142
|
+
return "precision";
|
|
143
|
+
}
|
|
144
|
+
if (input.mode === "docs" || input.mode === "decision" || input.mode === "adaptive")
|
|
145
|
+
return "long_context";
|
|
146
|
+
if (input.mode === "semantic")
|
|
147
|
+
return "recall";
|
|
148
|
+
if (looksErrorQuery(input.query) || looksExactOrCodeQuery(input.query))
|
|
149
|
+
return "precision";
|
|
150
|
+
const hasDocumentSource = input.kbSourceTypes.some((sourceType) => sourceType === "url" || sourceType === "text");
|
|
151
|
+
if (looksLongContextQuery(input.query) || hasDocumentSource)
|
|
152
|
+
return "long_context";
|
|
153
|
+
return "balanced";
|
|
154
|
+
}
|
|
155
|
+
export function resolveSearchTuning(input) {
|
|
156
|
+
const requestedProfile = input.profile ?? parseProfile(process.env.PI_KNOWLEDGE_SEARCH_PROFILE) ?? "auto";
|
|
157
|
+
const selectedProfile = requestedProfile === "auto" ? chooseRuntimeProfile(input) : requestedProfile;
|
|
158
|
+
const base = PROFILE_DEFAULTS[selectedProfile];
|
|
159
|
+
return {
|
|
160
|
+
requestedProfile,
|
|
161
|
+
selectedProfile,
|
|
162
|
+
defaultLimit: clampInt(envNumber("PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT"), base.defaultLimit, 1, 50),
|
|
163
|
+
snippetMaxLength: clampInt(envNumber("PI_KNOWLEDGE_SNIPPET_MAX_LENGTH"), base.snippetMaxLength, 80, 4_000),
|
|
164
|
+
minHybridScore: clampNumber(envNumber("PI_KNOWLEDGE_MIN_HYBRID_SCORE"), base.minHybridScore, 0, 1),
|
|
165
|
+
candidateMin: clampInt(envNumber("PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN"), base.candidateMin, 10, 500),
|
|
166
|
+
candidateMultiplier: clampInt(envNumber("PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER"), base.candidateMultiplier, 2, 30),
|
|
167
|
+
adaptiveContextLines: clampInt(envNumber("PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES"), base.adaptiveContextLines, 10, 500),
|
|
168
|
+
adaptiveMaxContextChars: clampInt(envNumber("PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS"), base.adaptiveMaxContextChars, 1_000, 50_000),
|
|
169
|
+
adaptiveNeighborTarget: clampInt(envNumber("PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET"), base.adaptiveNeighborTarget, 1, 20),
|
|
170
|
+
deepRerankCandidates: clampInt(envNumber("PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES"), base.deepRerankCandidates, 5, 100),
|
|
171
|
+
deepRerankTopKMultiplier: clampInt(envNumber("PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER"), base.deepRerankTopKMultiplier, 1, 10),
|
|
172
|
+
};
|
|
173
|
+
}
|
|
174
|
+
export function summarizeSearchTuning(tuning, limit, candidateLimit) {
|
|
175
|
+
return {
|
|
176
|
+
requested_profile: tuning.requestedProfile,
|
|
177
|
+
selected_profile: tuning.selectedProfile,
|
|
178
|
+
limit,
|
|
179
|
+
snippet_max_length: tuning.snippetMaxLength,
|
|
180
|
+
min_hybrid_score: tuning.minHybridScore,
|
|
181
|
+
candidate_limit: candidateLimit,
|
|
182
|
+
candidate_min: tuning.candidateMin,
|
|
183
|
+
candidate_multiplier: tuning.candidateMultiplier,
|
|
184
|
+
adaptive_context_lines: tuning.adaptiveContextLines,
|
|
185
|
+
adaptive_max_context_chars: tuning.adaptiveMaxContextChars,
|
|
186
|
+
adaptive_neighbor_target: tuning.adaptiveNeighborTarget,
|
|
187
|
+
deep_rerank_candidates: tuning.deepRerankCandidates,
|
|
188
|
+
deep_rerank_top_k_multiplier: tuning.deepRerankTopKMultiplier,
|
|
189
|
+
};
|
|
190
|
+
}
|
package/docs/configuration.md
CHANGED
|
@@ -30,7 +30,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
30
30
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
31
31
|
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
32
32
|
|
|
33
|
-
Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches common Node, Volta, NVM, and `PATH` locations before failing. If worker startup still fails,
|
|
33
|
+
Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
|
|
34
34
|
|
|
35
35
|
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended.
|
|
36
36
|
|
|
@@ -41,13 +41,26 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
41
41
|
| `PI_KNOWLEDGE_WATCH` | unset | Set to `true` to start file watchers for directory KBs. The polling fallback remains available when native `fs.watch` fails. |
|
|
42
42
|
| `PI_KNOWLEDGE_AUTO_INJECT` | unset | Set to `true` to auto-search KB context before model calls. This is opt-in. |
|
|
43
43
|
| `PI_KNOWLEDGE_STALE_INDEXING_MS` | built-in stale threshold | Override the stale indexing threshold used by diagnostics and `knowledge_doctor`. |
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
`
|
|
44
|
+
| `PI_KNOWLEDGE_SEARCH_PROFILE` | `auto` | Select the default search tuning profile. Values: `auto`, `balanced`, `low_token`, `precision`, `recall`, `long_context`, `code`, `docs`. Tool `profile` parameters override this. |
|
|
45
|
+
| `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT` | profile default | Override the result limit used only when `knowledge_search` does not pass `limit`. Clamped to 1-50. |
|
|
46
|
+
| `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH` | profile default | Override returned snippet length. Clamped to 80-4000 characters. |
|
|
47
|
+
| `PI_KNOWLEDGE_MIN_HYBRID_SCORE` | profile default | Override the hybrid confidence gate. Higher values reduce noisy results; lower values increase recall. Clamped to 0-1. |
|
|
48
|
+
| `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN` | profile default | Override the minimum internal retrieval candidate pool. Clamped to 10-500. |
|
|
49
|
+
| `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER` | profile default | Override the internal candidate multiplier applied to the effective result limit. Clamped to 2-30. |
|
|
50
|
+
| `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES` | profile default | Override surrounding lines fetched for `adaptive` context expansion. Clamped to 10-500. |
|
|
51
|
+
| `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS` | profile default | Override maximum adaptive context characters per result. Clamped to 1000-50000. |
|
|
52
|
+
| `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET` | profile default | Override the number of nearby/query-relevant chunks considered for adaptive windows. Clamped to 1-20. |
|
|
53
|
+
| `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES` | profile default | Override how many filtered candidates `deep` mode sends to the reranker. Clamped to 5-100. |
|
|
54
|
+
| `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` | profile default | Override deep rerank top-k multiplier relative to the effective result limit. Clamped to 1-10. |
|
|
55
|
+
|
|
56
|
+
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool size, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
57
|
+
|
|
58
|
+
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording.
|
|
48
59
|
|
|
49
60
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
50
61
|
|
|
62
|
+
`knowledge_configure` writes runtime configuration to the knowledge directory's `config.json`. It validates Node 22+ before persisting `node_path`, so agents can configure a Codex- or user-installed `node.exe` once without relying on environment variables being injected into an already-running OMP process.
|
|
63
|
+
|
|
51
64
|
`knowledge_symbol_search` uses a lightweight declaration-pattern index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find common code symbols, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; for methods, uncommon syntax, or empty important lookups, use `knowledge_search` mode `fast`/`adaptive` and check `knowledge_doctor` for stale or missing symbol metadata. Older file/directory/URL KBs without symbol metadata should be updated before relying on symbol lookup. Imported portable KBs and inline text KBs do not retain an active source path; rebuild their symbols by re-importing or re-adding the content.
|
|
52
65
|
|
|
53
66
|
## Test and Release Fixtures
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.6",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|