pi-knowledge 0.14.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +4 -2
- package/dist/index.js +158 -24
- package/dist/src/engine.d.ts +50 -1
- package/dist/src/engine.js +390 -70
- package/dist/src/search/tuning.d.ts +2 -0
- package/dist/src/search/tuning.js +2 -0
- package/docs/configuration.md +4 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.15.0] - 2026-11-15
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- Added deterministic, engine-owned zero-result classifications for `knowledge_search` and `knowledge_symbol_search`, with structured reasons/actions, searched and skipped KB summaries, stage-specific candidate counts, and bounded `auto` attempt details (#24).
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
- Replaced hypothetical zero-result guidance with concise fact-first actions while keeping miss classification free of source diagnostics scans and absolute source-path enumeration.
|
|
12
|
+
- Split search diagnostics into retrieved, score-threshold-passed, eligibility-passed, metadata-passed, and diversity candidate counts so threshold, lexical-evidence, and file/path rejection remain distinct.
|
|
13
|
+
|
|
5
14
|
## [0.14.0] - 2026-11-12
|
|
6
15
|
|
|
7
16
|
### Added
|
package/README.md
CHANGED
|
@@ -242,11 +242,13 @@ Mode selection contract:
|
|
|
242
242
|
- Use `semantic` when the query is conceptual and exact terms may differ from indexed wording; use `semantic_hybrid` when weak lexical overlap should not block vector-backed fused results.
|
|
243
243
|
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, AST-related sibling methods where indexed, or enough context to make a safe edit.
|
|
244
244
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
245
|
-
- If results are empty
|
|
245
|
+
- If results are empty, follow the returned classified cause and action. `auto` records its attempted modes; do not retry a mode already listed.
|
|
246
246
|
|
|
247
247
|
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. In `adaptive` mode, code chunks with AST metadata can prefer same-parent/sibling chunks inside the same bounded file window; this is metadata-driven context expansion, not caller/callee analysis. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, source freshness, and adaptive `source_chunk_ids` where available.
|
|
248
248
|
|
|
249
|
-
|
|
249
|
+
Zero-result responses are self-classifying. `knowledge_search` distinguishes missing or unavailable KBs, KBs with no indexed chunks, embedding incompatibility, retrieval misses, score-threshold rejection, lexical-evidence rejection, metadata filtering, empty reranker output, and out-of-range pages. `knowledge_symbol_search` distinguishes missing symbol metadata, exact-name misses, symbol filters, unavailable KBs, and pagination. Both return concise two-line guidance plus a structured `details.miss` object; classification uses request and search-pipeline facts without running a source diagnostics scan.
|
|
250
|
+
|
|
251
|
+
`limit` must be an integer from 1 to 50. `offset` must be a non-negative integer, and `offset + limit` cannot exceed the bounded 800-candidate search window. Each selected KB may contribute candidates up to the profile-derived retrieval limit, but merged candidates are globally ranked and capped before deep reranking, adaptive expansion, or diversity work. Diagnostics expose per-KB retrieval breadth, retrieved, score-threshold-passed, eligibility-passed, metadata-passed, and diversity counts plus whether the global pool was truncated.
|
|
250
252
|
|
|
251
253
|
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. Before embedding, the complete contextual input is checked against the active provider budget; only oversized chunks are split again along structural boundaries.
|
|
252
254
|
Symbol/config/heading metadata and embedding-safe chunk boundaries are rebuilt during `knowledge_update`. `knowledge_doctor` reports `reindex_for_embedding_budget` for older KBs: file, directory, and URL sources can use `knowledge_update`; imported or inline-text sources should be re-imported or re-added from the original source.
|
package/dist/index.js
CHANGED
|
@@ -68,6 +68,155 @@ function scanOptionsFromSourceOptions(raw) {
|
|
|
68
68
|
return {};
|
|
69
69
|
}
|
|
70
70
|
}
|
|
71
|
+
const SEARCH_MISS_ACTION_TEXT = {
|
|
72
|
+
add_knowledge_base: "Use knowledge_add to index a directory, file, URL, or text source.",
|
|
73
|
+
fix_kb_id: "Fix kb_id or omit it to search all knowledge bases.",
|
|
74
|
+
wait_for_indexing: "Retry after indexing completes.",
|
|
75
|
+
fix_knowledge_base: "Run knowledge_doctor and follow the action for the affected knowledge base.",
|
|
76
|
+
fix_embedding: "Run knowledge_doctor to resolve the embedding compatibility problem.",
|
|
77
|
+
try_semantic: 'Try mode "semantic", or rephrase with terms from the target files.',
|
|
78
|
+
try_semantic_hybrid: 'Try mode "semantic_hybrid", or rephrase with terms from the target files.',
|
|
79
|
+
relax_filter: "Relax the file type or path filter.",
|
|
80
|
+
reset_offset: "Retry with offset 0.",
|
|
81
|
+
rephrase: "Rephrase with exact terms from the target files.",
|
|
82
|
+
};
|
|
83
|
+
const SYMBOL_MISS_ACTION_TEXT = {
|
|
84
|
+
add_knowledge_base: "Use knowledge_add to index a directory, file, URL, or text source.",
|
|
85
|
+
fix_kb_id: "Fix kb_id or omit it to search all knowledge bases.",
|
|
86
|
+
wait_for_indexing: "Retry after indexing completes.",
|
|
87
|
+
fix_knowledge_base: "Run knowledge_doctor and follow the action for the affected knowledge base.",
|
|
88
|
+
rebuild_symbol_index: "Run knowledge_update for a source-backed KB, or re-add an inline/imported source to rebuild symbol metadata.",
|
|
89
|
+
broaden_symbol_search: "Retry with exact=false.",
|
|
90
|
+
relax_filter: "Relax the kind or file_pattern filter.",
|
|
91
|
+
reset_offset: "Retry with offset 0.",
|
|
92
|
+
use_text_search: 'Try knowledge_search mode "fast" for source text or "adaptive" for surrounding context.',
|
|
93
|
+
};
|
|
94
|
+
function inlineLabel(value, fallback) {
|
|
95
|
+
const normalized = value
|
|
96
|
+
.replace(/[\p{Cc}\p{Cf}]+/gu, " ")
|
|
97
|
+
.replace(/\s+/g, " ")
|
|
98
|
+
.trim()
|
|
99
|
+
.replaceAll('"', "'");
|
|
100
|
+
return normalized && normalized.length <= 80 ? `"${normalized}"` : fallback;
|
|
101
|
+
}
|
|
102
|
+
function requestedKbSuffix(requestedKbId) {
|
|
103
|
+
return requestedKbId ? ` in ${inlineLabel(requestedKbId, "the requested KB")}` : "";
|
|
104
|
+
}
|
|
105
|
+
function compactFilter(miss) {
|
|
106
|
+
const filters = [];
|
|
107
|
+
if (miss.file_type)
|
|
108
|
+
filters.push(`file_type ${inlineLabel(miss.file_type, "with the requested value")}`);
|
|
109
|
+
if (miss.path_pattern)
|
|
110
|
+
filters.push(`path_pattern ${inlineLabel(miss.path_pattern, "with the requested value")}`);
|
|
111
|
+
return filters.join(" and ") || "the requested metadata filters";
|
|
112
|
+
}
|
|
113
|
+
function formatSearchMiss(miss) {
|
|
114
|
+
const scope = requestedKbSuffix(miss.requested_kb_id);
|
|
115
|
+
const indexedChunks = miss.searched_kbs.reduce((count, kb) => count + kb.chunk_count, 0);
|
|
116
|
+
const attemptedModes = miss.attempts?.map((attempt) => attempt.mode);
|
|
117
|
+
const attemptSuffix = attemptedModes && attemptedModes.length > 1 ? ` Modes tried: ${attemptedModes.join(", ")}.` : "";
|
|
118
|
+
let cause;
|
|
119
|
+
switch (miss.reason) {
|
|
120
|
+
case "no_knowledge_bases":
|
|
121
|
+
cause = "No results: no knowledge bases are indexed.";
|
|
122
|
+
break;
|
|
123
|
+
case "knowledge_base_not_found":
|
|
124
|
+
cause = `No results: knowledge base ${inlineLabel(miss.requested_kb_id ?? "", "from the requested kb_id")} does not exist.`;
|
|
125
|
+
break;
|
|
126
|
+
case "no_searchable_knowledge_bases": {
|
|
127
|
+
const indexing = miss.skipped_kbs.find((kb) => kb.status === "indexing");
|
|
128
|
+
const errored = miss.skipped_kbs.find((kb) => kb.status === "error");
|
|
129
|
+
if (miss.action === "wait_for_indexing" && indexing) {
|
|
130
|
+
cause = `No results yet: ${inlineLabel(indexing.name, "the selected KB")} is still indexing (${indexing.chunk_count} chunks persisted).`;
|
|
131
|
+
}
|
|
132
|
+
else if (errored) {
|
|
133
|
+
cause = `No results: ${inlineLabel(errored.name, "the selected KB")} is in error state and is not searchable.`;
|
|
134
|
+
}
|
|
135
|
+
else if (indexing) {
|
|
136
|
+
cause = `No results: one or more selected knowledge bases are unavailable while ${inlineLabel(indexing.name, "one KB")} indexes.`;
|
|
137
|
+
}
|
|
138
|
+
else {
|
|
139
|
+
cause = "No results: no selected knowledge base is currently searchable.";
|
|
140
|
+
}
|
|
141
|
+
break;
|
|
142
|
+
}
|
|
143
|
+
case "no_indexed_chunks":
|
|
144
|
+
cause = "No results: the searchable knowledge bases contain no indexed chunks.";
|
|
145
|
+
break;
|
|
146
|
+
case "embedding_incompatible":
|
|
147
|
+
cause = "No results: the selected knowledge bases are incompatible with the active embedding configuration.";
|
|
148
|
+
break;
|
|
149
|
+
case "no_retrieval_candidates":
|
|
150
|
+
cause = `No results: no retrieval candidates matched${scope} (${indexedChunks} indexed chunks).${attemptSuffix}`;
|
|
151
|
+
break;
|
|
152
|
+
case "threshold_filtered": {
|
|
153
|
+
const threshold = miss.min_hybrid_score === undefined ? "the score threshold" : `the ${miss.min_hybrid_score} score threshold`;
|
|
154
|
+
cause = `No results: ${miss.retrieved_candidates} candidates were retrieved, but none passed ${threshold}.${attemptSuffix}`;
|
|
155
|
+
break;
|
|
156
|
+
}
|
|
157
|
+
case "evidence_filtered":
|
|
158
|
+
cause = `No results: ${miss.threshold_candidates} candidates passed the score threshold, but none had enough lexical evidence.${attemptSuffix}`;
|
|
159
|
+
break;
|
|
160
|
+
case "metadata_filtered":
|
|
161
|
+
cause = `No results: ${miss.eligible_candidates} eligible candidates matched the query, but none matched ${compactFilter(miss)}.`;
|
|
162
|
+
break;
|
|
163
|
+
case "reranker_empty":
|
|
164
|
+
cause = `No results: ${miss.filtered_candidates} candidates reached deep reranking, but the reranker returned none.`;
|
|
165
|
+
break;
|
|
166
|
+
case "page_out_of_range":
|
|
167
|
+
cause = `No results on this page: the requested offset exceeds ${miss.total_results} available results.`;
|
|
168
|
+
break;
|
|
169
|
+
case "no_match":
|
|
170
|
+
cause = `No results: candidates were processed but no result remained${scope}.${attemptSuffix}`;
|
|
171
|
+
break;
|
|
172
|
+
}
|
|
173
|
+
return `${cause}\n${SEARCH_MISS_ACTION_TEXT[miss.action]}`;
|
|
174
|
+
}
|
|
175
|
+
function formatSymbolMiss(miss) {
|
|
176
|
+
const scope = requestedKbSuffix(miss.requested_kb_id);
|
|
177
|
+
let cause;
|
|
178
|
+
switch (miss.reason) {
|
|
179
|
+
case "no_knowledge_bases":
|
|
180
|
+
cause = "No symbols found: no knowledge bases are indexed.";
|
|
181
|
+
break;
|
|
182
|
+
case "knowledge_base_not_found":
|
|
183
|
+
cause = `No symbols found: knowledge base ${inlineLabel(miss.requested_kb_id ?? "", "from the requested kb_id")} does not exist.`;
|
|
184
|
+
break;
|
|
185
|
+
case "no_searchable_knowledge_bases": {
|
|
186
|
+
const indexing = miss.skipped_kbs.find((kb) => kb.status === "indexing");
|
|
187
|
+
const errored = miss.skipped_kbs.find((kb) => kb.status === "error");
|
|
188
|
+
if (miss.action === "wait_for_indexing" && indexing) {
|
|
189
|
+
cause = `No symbols yet: ${inlineLabel(indexing.name, "the selected KB")} is still indexing (${indexing.chunk_count} chunks persisted).`;
|
|
190
|
+
}
|
|
191
|
+
else if (errored) {
|
|
192
|
+
cause = `No symbols found: ${inlineLabel(errored.name, "the selected KB")} is in error state and is not searchable.`;
|
|
193
|
+
}
|
|
194
|
+
else if (indexing) {
|
|
195
|
+
cause = `No symbols found: one or more selected knowledge bases are unavailable while ${inlineLabel(indexing.name, "one KB")} indexes.`;
|
|
196
|
+
}
|
|
197
|
+
else {
|
|
198
|
+
cause = "No symbols found: no selected knowledge base is currently searchable.";
|
|
199
|
+
}
|
|
200
|
+
break;
|
|
201
|
+
}
|
|
202
|
+
case "symbol_index_empty":
|
|
203
|
+
cause = `No symbols are indexed${scope}.`;
|
|
204
|
+
break;
|
|
205
|
+
case "exact_name_not_found":
|
|
206
|
+
cause = `No symbol matched the exact normalized name${scope}.`;
|
|
207
|
+
break;
|
|
208
|
+
case "metadata_filtered":
|
|
209
|
+
cause = `No symbols matched the requested kind or file_pattern${scope}; ${miss.unfiltered_matches} unfiltered matches exist.`;
|
|
210
|
+
break;
|
|
211
|
+
case "page_out_of_range":
|
|
212
|
+
cause = "No symbols on this page: the requested offset exceeds the available matches.";
|
|
213
|
+
break;
|
|
214
|
+
case "no_match":
|
|
215
|
+
cause = `No indexed symbol matched${scope}.`;
|
|
216
|
+
break;
|
|
217
|
+
}
|
|
218
|
+
return `${cause}\n${SYMBOL_MISS_ACTION_TEXT[miss.action]}`;
|
|
219
|
+
}
|
|
71
220
|
function runtimeModule(modulePath) {
|
|
72
221
|
return `${modulePath}${RUNTIME_EXTENSION}`;
|
|
73
222
|
}
|
|
@@ -370,7 +519,7 @@ export default function (pi) {
|
|
|
370
519
|
"Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
|
|
371
520
|
"Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
|
|
372
521
|
"Use profile 'low_token' for slow local models that need fewer, stricter, longer snippets; use profile 'precision' for identifiers/errors and 'recall' or 'long_context' for broad prose/document research",
|
|
373
|
-
"If a search returns no results
|
|
522
|
+
"If a search returns no results, follow its classified action; do not repeat modes listed in the auto attempt summary",
|
|
374
523
|
"If top results are repetitive, retry with diversity 'strong' or mode 'adaptive' instead of increasing the limit first",
|
|
375
524
|
],
|
|
376
525
|
parameters: Type.Object({
|
|
@@ -423,19 +572,9 @@ export default function (pi) {
|
|
|
423
572
|
const filters = file_type || path_pattern ? { file_type, path_pattern } : undefined;
|
|
424
573
|
const response = await engine.search(query, { mode, profile, limit, kb_id, offset, filters, diversity }, _signal);
|
|
425
574
|
if (response.results.length === 0) {
|
|
426
|
-
|
|
427
|
-
"
|
|
428
|
-
|
|
429
|
-
response.retry_modes?.length ? `Retried: ${response.retry_modes.join(", ")}` : "",
|
|
430
|
-
response.warnings?.length ? `Warnings: ${response.warnings.join(" | ")}` : "",
|
|
431
|
-
diagnostics && response.tuning
|
|
432
|
-
? `Tuning: profile=${response.tuning.requested_profile}->${response.tuning.selected_profile}, limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.diversity_candidates}/${response.tuning.candidate_limit}, retrieval_per_kb=${response.tuning.retrieval_limit_per_kb}, retrieved=${response.tuning.retrieved_candidates}, filtered=${response.tuning.filtered_candidates}, truncated=${response.tuning.candidate_pool_truncated}`
|
|
433
|
-
: "",
|
|
434
|
-
response.suggestions?.length ? `Suggestions: ${response.suggestions.join(" | ")}` : "",
|
|
435
|
-
]
|
|
436
|
-
.filter(Boolean)
|
|
437
|
-
.join("\n");
|
|
438
|
-
return { content: [{ type: "text", text: details }], details: diagnostics ? response : undefined };
|
|
575
|
+
if (!response.miss)
|
|
576
|
+
throw new Error("Search returned no results without a miss classification");
|
|
577
|
+
return { content: [{ type: "text", text: formatSearchMiss(response.miss) }], details: response };
|
|
439
578
|
}
|
|
440
579
|
let output = `${response.total_count} results (showing ${response.results.length})`;
|
|
441
580
|
if (response.mode_used)
|
|
@@ -458,7 +597,7 @@ export default function (pi) {
|
|
|
458
597
|
? `\nProvenance: chunk=${r.provenance.chunk_id}, hash=${r.provenance.chunk_hash.slice(0, 12)}, reason=${r.provenance.match_reason}, indexed_at=${r.provenance.indexed_at}, source_mtime=${r.provenance.source_mtime ?? "unknown"}, stale=${r.provenance.stale}`
|
|
459
598
|
: "";
|
|
460
599
|
const tuning = diagnostics && response.tuning && i === 0
|
|
461
|
-
? `\nTuning: limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.diversity_candidates}/${response.tuning.candidate_limit}, retrieval_per_kb=${response.tuning.retrieval_limit_per_kb}, retrieved=${response.tuning.retrieved_candidates},
|
|
600
|
+
? `\nTuning: limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.diversity_candidates}/${response.tuning.candidate_limit}, retrieval_per_kb=${response.tuning.retrieval_limit_per_kb}, retrieved=${response.tuning.retrieved_candidates}, threshold=${response.tuning.threshold_candidates}, eligible=${response.tuning.eligible_candidates}, metadata=${response.tuning.filtered_candidates}, truncated=${response.tuning.candidate_pool_truncated}`
|
|
462
601
|
: "";
|
|
463
602
|
return `[${i + 1}] ${r.file_path} (${r.kb_name}, score: ${r.score.toFixed(3)})\n${r.snippet}${diag}${provenance}${tuning}`;
|
|
464
603
|
})
|
|
@@ -475,7 +614,7 @@ export default function (pi) {
|
|
|
475
614
|
promptGuidelines: [
|
|
476
615
|
"Use knowledge_symbol_search before knowledge_search when looking for a function, class, env var, config key, route, or Markdown heading",
|
|
477
616
|
"Use exact=true when the user provides a precise symbol or key",
|
|
478
|
-
"This is a lightweight declaration-pattern index, not full LSP;
|
|
617
|
+
"This is a lightweight declaration-pattern index, not full LSP; on a miss, follow its classified action and use knowledge_search mode 'fast' or 'adaptive' only when directed to source text",
|
|
479
618
|
"After finding a symbol, use knowledge_search mode 'adaptive' with the file path or symbol name when surrounding implementation context is needed",
|
|
480
619
|
],
|
|
481
620
|
parameters: Type.Object({
|
|
@@ -512,14 +651,9 @@ export default function (pi) {
|
|
|
512
651
|
exact: params.exact === true,
|
|
513
652
|
}, _signal);
|
|
514
653
|
if (response.results.length === 0) {
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
type: "text",
|
|
519
|
-
text: "No symbols found. This may mean the symbol is absent, symbol metadata is missing/stale, or the lightweight extractor does not cover this syntax. For important lookups, run knowledge_doctor or fallback to knowledge_search mode 'fast'/'adaptive'.",
|
|
520
|
-
},
|
|
521
|
-
],
|
|
522
|
-
};
|
|
654
|
+
if (!response.miss)
|
|
655
|
+
throw new Error("Symbol search returned no results without a miss classification");
|
|
656
|
+
return { content: [{ type: "text", text: formatSymbolMiss(response.miss) }], details: response };
|
|
523
657
|
}
|
|
524
658
|
const lines = response.results.map((result, index) => {
|
|
525
659
|
const signature = result.signature ? ` — ${result.signature}` : "";
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -38,6 +38,40 @@ export interface SearchResult {
|
|
|
38
38
|
};
|
|
39
39
|
}
|
|
40
40
|
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.ts";
|
|
41
|
+
export type SearchMissReason = "no_knowledge_bases" | "knowledge_base_not_found" | "no_searchable_knowledge_bases" | "no_indexed_chunks" | "embedding_incompatible" | "no_retrieval_candidates" | "threshold_filtered" | "evidence_filtered" | "metadata_filtered" | "reranker_empty" | "page_out_of_range" | "no_match";
|
|
42
|
+
export type SearchMissAction = "add_knowledge_base" | "fix_kb_id" | "wait_for_indexing" | "fix_knowledge_base" | "fix_embedding" | "try_semantic" | "try_semantic_hybrid" | "relax_filter" | "reset_offset" | "rephrase";
|
|
43
|
+
export interface SearchKnowledgeBaseSummary {
|
|
44
|
+
id: string;
|
|
45
|
+
name: string;
|
|
46
|
+
status: KnowledgeBase["status"];
|
|
47
|
+
chunk_count: number;
|
|
48
|
+
}
|
|
49
|
+
export interface SearchAttemptSummary {
|
|
50
|
+
mode: SearchMode;
|
|
51
|
+
reason?: SearchMissReason;
|
|
52
|
+
retrieved_candidates: number;
|
|
53
|
+
threshold_candidates: number;
|
|
54
|
+
eligible_candidates: number;
|
|
55
|
+
filtered_candidates: number;
|
|
56
|
+
result_count: number;
|
|
57
|
+
}
|
|
58
|
+
export interface SearchMiss {
|
|
59
|
+
reason: SearchMissReason;
|
|
60
|
+
action: SearchMissAction;
|
|
61
|
+
requested_kb_id?: string;
|
|
62
|
+
file_type?: string;
|
|
63
|
+
path_pattern?: string;
|
|
64
|
+
min_hybrid_score?: number;
|
|
65
|
+
searched_kbs: SearchKnowledgeBaseSummary[];
|
|
66
|
+
skipped_kbs: SearchKnowledgeBaseSummary[];
|
|
67
|
+
retrieved_candidates: number;
|
|
68
|
+
threshold_candidates: number;
|
|
69
|
+
eligible_candidates: number;
|
|
70
|
+
filtered_candidates: number;
|
|
71
|
+
ranked_candidates: number;
|
|
72
|
+
total_results: number;
|
|
73
|
+
attempts?: SearchAttemptSummary[];
|
|
74
|
+
}
|
|
41
75
|
export interface SearchResponse {
|
|
42
76
|
results: SearchResult[];
|
|
43
77
|
total_count: number;
|
|
@@ -45,8 +79,8 @@ export interface SearchResponse {
|
|
|
45
79
|
warnings?: string[];
|
|
46
80
|
mode_used?: SearchMode;
|
|
47
81
|
retry_modes?: SearchMode[];
|
|
48
|
-
suggestions?: string[];
|
|
49
82
|
tuning?: SearchTuningSummary;
|
|
83
|
+
miss?: SearchMiss;
|
|
50
84
|
}
|
|
51
85
|
export type ProgressCallback = (msg: string) => void;
|
|
52
86
|
export interface AddOptions {
|
|
@@ -96,6 +130,20 @@ export interface SymbolSearchOptions {
|
|
|
96
130
|
offset?: number;
|
|
97
131
|
exact?: boolean;
|
|
98
132
|
}
|
|
133
|
+
export type SymbolSearchMissReason = "no_knowledge_bases" | "knowledge_base_not_found" | "no_searchable_knowledge_bases" | "symbol_index_empty" | "exact_name_not_found" | "metadata_filtered" | "page_out_of_range" | "no_match";
|
|
134
|
+
export type SymbolSearchMissAction = "add_knowledge_base" | "fix_kb_id" | "wait_for_indexing" | "fix_knowledge_base" | "rebuild_symbol_index" | "broaden_symbol_search" | "relax_filter" | "reset_offset" | "use_text_search";
|
|
135
|
+
export interface SymbolSearchMiss {
|
|
136
|
+
reason: SymbolSearchMissReason;
|
|
137
|
+
action: SymbolSearchMissAction;
|
|
138
|
+
requested_kb_id?: string;
|
|
139
|
+
kind?: KnowledgeSymbol["kind"];
|
|
140
|
+
file_pattern?: string;
|
|
141
|
+
exact: boolean;
|
|
142
|
+
searched_kbs: SearchKnowledgeBaseSummary[];
|
|
143
|
+
skipped_kbs: SearchKnowledgeBaseSummary[];
|
|
144
|
+
indexed_symbols: number;
|
|
145
|
+
unfiltered_matches: number;
|
|
146
|
+
}
|
|
99
147
|
export interface SymbolSearchResponse {
|
|
100
148
|
results: Array<{
|
|
101
149
|
name: string;
|
|
@@ -112,6 +160,7 @@ export interface SymbolSearchResponse {
|
|
|
112
160
|
}>;
|
|
113
161
|
total_count: number;
|
|
114
162
|
has_more: boolean;
|
|
163
|
+
miss?: SymbolSearchMiss;
|
|
115
164
|
}
|
|
116
165
|
export declare const CURRENT_URL_STRUCTURE_POLICY = 1;
|
|
117
166
|
export declare const CURRENT_EMBEDDING_INPUT_POLICY = 2;
|
package/dist/src/engine.js
CHANGED
|
@@ -150,6 +150,181 @@ function kbTrustMultiplier(kb) {
|
|
|
150
150
|
multiplier *= 0.98;
|
|
151
151
|
return multiplier;
|
|
152
152
|
}
|
|
153
|
+
function summarizeKnowledgeBases(kbs) {
|
|
154
|
+
return kbs.map((kb) => ({
|
|
155
|
+
id: kb.id,
|
|
156
|
+
name: kb.name,
|
|
157
|
+
status: kb.status,
|
|
158
|
+
chunk_count: kb.chunk_count,
|
|
159
|
+
}));
|
|
160
|
+
}
|
|
161
|
+
function searchMissAction(reason, mode, availableKnowledgeBases) {
|
|
162
|
+
switch (reason) {
|
|
163
|
+
case "no_knowledge_bases":
|
|
164
|
+
return "add_knowledge_base";
|
|
165
|
+
case "knowledge_base_not_found":
|
|
166
|
+
return "fix_kb_id";
|
|
167
|
+
case "no_searchable_knowledge_bases":
|
|
168
|
+
return availableKnowledgeBases.length > 0 && availableKnowledgeBases.every((kb) => kb.status === "indexing")
|
|
169
|
+
? "wait_for_indexing"
|
|
170
|
+
: "fix_knowledge_base";
|
|
171
|
+
case "no_indexed_chunks":
|
|
172
|
+
return availableKnowledgeBases.some((kb) => kb.status === "indexing")
|
|
173
|
+
? "wait_for_indexing"
|
|
174
|
+
: "fix_knowledge_base";
|
|
175
|
+
case "embedding_incompatible":
|
|
176
|
+
return "fix_embedding";
|
|
177
|
+
case "no_retrieval_candidates":
|
|
178
|
+
return mode === "fast" || mode === "code" || mode === "config" || mode === "errors"
|
|
179
|
+
? "try_semantic"
|
|
180
|
+
: mode === "semantic" || mode === "semantic_hybrid"
|
|
181
|
+
? "rephrase"
|
|
182
|
+
: "try_semantic_hybrid";
|
|
183
|
+
case "threshold_filtered":
|
|
184
|
+
return mode === "semantic_hybrid" ? "rephrase" : "try_semantic_hybrid";
|
|
185
|
+
case "evidence_filtered":
|
|
186
|
+
return "try_semantic_hybrid";
|
|
187
|
+
case "metadata_filtered":
|
|
188
|
+
return "relax_filter";
|
|
189
|
+
case "page_out_of_range":
|
|
190
|
+
return "reset_offset";
|
|
191
|
+
case "reranker_empty":
|
|
192
|
+
case "no_match":
|
|
193
|
+
return "rephrase";
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
function classifySearchMiss(facts) {
|
|
197
|
+
const skippedKnowledgeBases = facts.availableKnowledgeBases.filter((kb) => kb.status !== "ready" && kb.status !== "stale");
|
|
198
|
+
let reason;
|
|
199
|
+
if (facts.allKnowledgeBases.length === 0) {
|
|
200
|
+
reason = "no_knowledge_bases";
|
|
201
|
+
}
|
|
202
|
+
else if (facts.requestedKbId &&
|
|
203
|
+
!facts.availableKnowledgeBases.some((kb) => kb.id === facts.requestedKbId || kb.name === facts.requestedKbId)) {
|
|
204
|
+
reason = "knowledge_base_not_found";
|
|
205
|
+
}
|
|
206
|
+
else if (facts.searchedKnowledgeBases.length === 0) {
|
|
207
|
+
reason = "no_searchable_knowledge_bases";
|
|
208
|
+
}
|
|
209
|
+
else if (facts.searchedKnowledgeBases.every((kb) => kb.chunk_count === 0)) {
|
|
210
|
+
reason = "no_indexed_chunks";
|
|
211
|
+
}
|
|
212
|
+
else if (facts.retrievedCandidates === 0 &&
|
|
213
|
+
facts.embeddingIncompatibleCount === facts.searchedKnowledgeBases.filter((kb) => kb.chunk_count > 0).length) {
|
|
214
|
+
reason = "embedding_incompatible";
|
|
215
|
+
}
|
|
216
|
+
else if (facts.retrievedCandidates === 0) {
|
|
217
|
+
reason = "no_retrieval_candidates";
|
|
218
|
+
}
|
|
219
|
+
else if (facts.thresholdCandidates === 0) {
|
|
220
|
+
reason = "threshold_filtered";
|
|
221
|
+
}
|
|
222
|
+
else if (facts.eligibleCandidates === 0) {
|
|
223
|
+
reason = "evidence_filtered";
|
|
224
|
+
}
|
|
225
|
+
else if (facts.filteredCandidates === 0 && (facts.fileType || facts.pathPattern)) {
|
|
226
|
+
reason = "metadata_filtered";
|
|
227
|
+
}
|
|
228
|
+
else if (facts.mode === "deep" && facts.rankedCandidates === 0) {
|
|
229
|
+
reason = "reranker_empty";
|
|
230
|
+
}
|
|
231
|
+
else if (facts.totalResults > 0 && facts.offset >= facts.totalResults) {
|
|
232
|
+
reason = "page_out_of_range";
|
|
233
|
+
}
|
|
234
|
+
else {
|
|
235
|
+
reason = "no_match";
|
|
236
|
+
}
|
|
237
|
+
return {
|
|
238
|
+
reason,
|
|
239
|
+
action: searchMissAction(reason, facts.mode, facts.availableKnowledgeBases),
|
|
240
|
+
requested_kb_id: facts.requestedKbId,
|
|
241
|
+
min_hybrid_score: facts.minimumScore,
|
|
242
|
+
file_type: facts.fileType,
|
|
243
|
+
path_pattern: facts.pathPattern,
|
|
244
|
+
searched_kbs: summarizeKnowledgeBases(facts.searchedKnowledgeBases),
|
|
245
|
+
skipped_kbs: summarizeKnowledgeBases(skippedKnowledgeBases),
|
|
246
|
+
retrieved_candidates: facts.retrievedCandidates,
|
|
247
|
+
threshold_candidates: facts.thresholdCandidates,
|
|
248
|
+
eligible_candidates: facts.eligibleCandidates,
|
|
249
|
+
filtered_candidates: facts.filteredCandidates,
|
|
250
|
+
ranked_candidates: facts.rankedCandidates,
|
|
251
|
+
total_results: facts.totalResults,
|
|
252
|
+
};
|
|
253
|
+
}
|
|
254
|
+
function searchAttemptSummary(mode, response) {
|
|
255
|
+
return {
|
|
256
|
+
mode,
|
|
257
|
+
reason: response.miss?.reason,
|
|
258
|
+
retrieved_candidates: response.tuning?.retrieved_candidates ?? 0,
|
|
259
|
+
threshold_candidates: response.tuning?.threshold_candidates ?? 0,
|
|
260
|
+
eligible_candidates: response.tuning?.eligible_candidates ?? 0,
|
|
261
|
+
filtered_candidates: response.tuning?.filtered_candidates ?? 0,
|
|
262
|
+
result_count: response.results.length,
|
|
263
|
+
};
|
|
264
|
+
}
|
|
265
|
+
function symbolMissAction(reason, availableKnowledgeBases) {
|
|
266
|
+
switch (reason) {
|
|
267
|
+
case "no_knowledge_bases":
|
|
268
|
+
return "add_knowledge_base";
|
|
269
|
+
case "knowledge_base_not_found":
|
|
270
|
+
return "fix_kb_id";
|
|
271
|
+
case "no_searchable_knowledge_bases":
|
|
272
|
+
return availableKnowledgeBases.length > 0 && availableKnowledgeBases.every((kb) => kb.status === "indexing")
|
|
273
|
+
? "wait_for_indexing"
|
|
274
|
+
: "fix_knowledge_base";
|
|
275
|
+
case "symbol_index_empty":
|
|
276
|
+
return "rebuild_symbol_index";
|
|
277
|
+
case "exact_name_not_found":
|
|
278
|
+
return "broaden_symbol_search";
|
|
279
|
+
case "metadata_filtered":
|
|
280
|
+
return "relax_filter";
|
|
281
|
+
case "page_out_of_range":
|
|
282
|
+
return "reset_offset";
|
|
283
|
+
case "no_match":
|
|
284
|
+
return "use_text_search";
|
|
285
|
+
}
|
|
286
|
+
}
|
|
287
|
+
function classifySymbolSearchMiss(facts) {
|
|
288
|
+
const skippedKnowledgeBases = facts.availableKnowledgeBases.filter((kb) => kb.status !== "ready" && kb.status !== "stale");
|
|
289
|
+
let reason;
|
|
290
|
+
if (facts.allKnowledgeBases.length === 0) {
|
|
291
|
+
reason = "no_knowledge_bases";
|
|
292
|
+
}
|
|
293
|
+
else if (facts.requestedKbId &&
|
|
294
|
+
!facts.availableKnowledgeBases.some((kb) => kb.id === facts.requestedKbId || kb.name === facts.requestedKbId)) {
|
|
295
|
+
reason = "knowledge_base_not_found";
|
|
296
|
+
}
|
|
297
|
+
else if (facts.searchedKnowledgeBases.length === 0) {
|
|
298
|
+
reason = "no_searchable_knowledge_bases";
|
|
299
|
+
}
|
|
300
|
+
else if (facts.indexedSymbols === 0) {
|
|
301
|
+
reason = "symbol_index_empty";
|
|
302
|
+
}
|
|
303
|
+
else if (facts.totalResults > 0 && facts.offset >= facts.totalResults) {
|
|
304
|
+
reason = "page_out_of_range";
|
|
305
|
+
}
|
|
306
|
+
else if ((facts.kind || facts.filePattern) && facts.unfilteredMatches > 0) {
|
|
307
|
+
reason = "metadata_filtered";
|
|
308
|
+
}
|
|
309
|
+
else if (facts.exact) {
|
|
310
|
+
reason = "exact_name_not_found";
|
|
311
|
+
}
|
|
312
|
+
else {
|
|
313
|
+
reason = "no_match";
|
|
314
|
+
}
|
|
315
|
+
return {
|
|
316
|
+
reason,
|
|
317
|
+
action: symbolMissAction(reason, facts.availableKnowledgeBases),
|
|
318
|
+
requested_kb_id: facts.requestedKbId,
|
|
319
|
+
kind: facts.kind,
|
|
320
|
+
file_pattern: facts.filePattern,
|
|
321
|
+
exact: facts.exact,
|
|
322
|
+
searched_kbs: summarizeKnowledgeBases(facts.searchedKnowledgeBases),
|
|
323
|
+
skipped_kbs: summarizeKnowledgeBases(skippedKnowledgeBases),
|
|
324
|
+
indexed_symbols: facts.indexedSymbols,
|
|
325
|
+
unfiltered_matches: facts.unfilteredMatches,
|
|
326
|
+
};
|
|
327
|
+
}
|
|
153
328
|
function formatBytes(bytes) {
|
|
154
329
|
if (bytes < 1024)
|
|
155
330
|
return `${bytes} B`;
|
|
@@ -1163,43 +1338,82 @@ export class KnowledgeEngine {
|
|
|
1163
1338
|
const requestedMode = options.mode ?? "hybrid";
|
|
1164
1339
|
if (requestedMode === "auto") {
|
|
1165
1340
|
const primaryMode = chooseAutoMode(query);
|
|
1166
|
-
const
|
|
1341
|
+
const modes = [primaryMode, ...fallbackModesFor(primaryMode)];
|
|
1167
1342
|
const tried = [];
|
|
1343
|
+
const attemptSummaries = [];
|
|
1168
1344
|
const warnings = [];
|
|
1169
|
-
let
|
|
1170
|
-
|
|
1345
|
+
let lastMissResponse;
|
|
1346
|
+
let bestWeakResponse;
|
|
1347
|
+
for (const mode of modes) {
|
|
1171
1348
|
if (tried.includes(mode))
|
|
1172
1349
|
continue;
|
|
1173
1350
|
tried.push(mode);
|
|
1174
1351
|
throwIfAborted(signal);
|
|
1175
1352
|
const response = await this.search(query, { ...options, mode }, signal);
|
|
1176
|
-
|
|
1353
|
+
attemptSummaries.push(searchAttemptSummary(mode, response));
|
|
1177
1354
|
if (response.warnings)
|
|
1178
1355
|
warnings.push(...response.warnings);
|
|
1356
|
+
if (response.miss?.reason === "no_knowledge_bases" ||
|
|
1357
|
+
response.miss?.reason === "knowledge_base_not_found" ||
|
|
1358
|
+
response.miss?.reason === "no_searchable_knowledge_bases" ||
|
|
1359
|
+
response.miss?.reason === "no_indexed_chunks") {
|
|
1360
|
+
return {
|
|
1361
|
+
...response,
|
|
1362
|
+
warnings: warnings.length > 0 ? [...new Set(warnings)] : response.warnings,
|
|
1363
|
+
mode_used: mode,
|
|
1364
|
+
retry_modes: tried.slice(0, -1),
|
|
1365
|
+
miss: { ...response.miss, attempts: attemptSummaries },
|
|
1366
|
+
};
|
|
1367
|
+
}
|
|
1179
1368
|
if (!isWeakAutoResponse(query, response, primaryMode, mode)) {
|
|
1180
1369
|
return {
|
|
1181
1370
|
...response,
|
|
1182
1371
|
warnings: warnings.length > 0 ? [...new Set(warnings)] : response.warnings,
|
|
1183
1372
|
mode_used: mode,
|
|
1184
1373
|
retry_modes: tried.slice(0, -1),
|
|
1185
|
-
suggestions: response.suggestions,
|
|
1186
1374
|
};
|
|
1187
1375
|
}
|
|
1376
|
+
if (response.results.length > 0) {
|
|
1377
|
+
if (!bestWeakResponse || response.results.length > bestWeakResponse.response.results.length) {
|
|
1378
|
+
bestWeakResponse = { mode, response };
|
|
1379
|
+
}
|
|
1380
|
+
}
|
|
1381
|
+
else {
|
|
1382
|
+
lastMissResponse = { mode, response };
|
|
1383
|
+
}
|
|
1188
1384
|
if (tried.length >= 3)
|
|
1189
1385
|
break;
|
|
1190
1386
|
}
|
|
1191
|
-
|
|
1192
|
-
|
|
1193
|
-
|
|
1194
|
-
|
|
1195
|
-
|
|
1196
|
-
|
|
1197
|
-
|
|
1198
|
-
|
|
1199
|
-
|
|
1200
|
-
|
|
1201
|
-
|
|
1202
|
-
|
|
1387
|
+
if (lastMissResponse?.response.miss) {
|
|
1388
|
+
const attemptedModes = new Set(attemptSummaries.map((attempt) => attempt.mode));
|
|
1389
|
+
let action = lastMissResponse.response.miss.action;
|
|
1390
|
+
if (action === "try_semantic" && attemptedModes.has("semantic")) {
|
|
1391
|
+
action = attemptedModes.has("semantic_hybrid") ? "rephrase" : "try_semantic_hybrid";
|
|
1392
|
+
}
|
|
1393
|
+
else if (action === "try_semantic_hybrid" && attemptedModes.has("semantic_hybrid")) {
|
|
1394
|
+
action = "rephrase";
|
|
1395
|
+
}
|
|
1396
|
+
return {
|
|
1397
|
+
...lastMissResponse.response,
|
|
1398
|
+
warnings: warnings.length > 0 ? [...new Set(warnings)] : lastMissResponse.response.warnings,
|
|
1399
|
+
mode_used: lastMissResponse.mode,
|
|
1400
|
+
retry_modes: tried.filter((mode) => mode !== lastMissResponse.mode),
|
|
1401
|
+
miss: {
|
|
1402
|
+
...lastMissResponse.response.miss,
|
|
1403
|
+
action,
|
|
1404
|
+
attempts: attemptSummaries,
|
|
1405
|
+
},
|
|
1406
|
+
};
|
|
1407
|
+
}
|
|
1408
|
+
if (bestWeakResponse) {
|
|
1409
|
+
return {
|
|
1410
|
+
...bestWeakResponse.response,
|
|
1411
|
+
warnings: warnings.length > 0 ? [...new Set(warnings)] : bestWeakResponse.response.warnings,
|
|
1412
|
+
mode_used: bestWeakResponse.mode,
|
|
1413
|
+
retry_modes: tried.filter((mode) => mode !== bestWeakResponse.mode),
|
|
1414
|
+
};
|
|
1415
|
+
}
|
|
1416
|
+
throw new Error("Auto search exhausted without a result or miss classification");
|
|
1203
1417
|
}
|
|
1204
1418
|
const db = this.db;
|
|
1205
1419
|
const { mode = "hybrid", offset: requestedOffset, kb_id, filters, diversity = "balanced" } = options;
|
|
@@ -1209,8 +1423,9 @@ export class KnowledgeEngine {
|
|
|
1209
1423
|
const queryTokens = tokenizeForSimilarity(query);
|
|
1210
1424
|
const normalizedFileType = normalizeFileTypeFilter(filters?.file_type);
|
|
1211
1425
|
const warnings = [];
|
|
1426
|
+
const allKBs = listKBs(db);
|
|
1212
1427
|
const selectedKB = kb_id ? (getKB(db, kb_id) ?? getKBByName(db, kb_id)) : undefined;
|
|
1213
|
-
const availableKBs = kb_id ? [selectedKB].filter(Boolean) :
|
|
1428
|
+
const availableKBs = kb_id ? [selectedKB].filter(Boolean) : allKBs;
|
|
1214
1429
|
const kbs = availableKBs.filter((kb) => kb.status === "ready" || kb.status === "stale");
|
|
1215
1430
|
const kbById = new Map(kbs.map((kb) => [kb.id, kb]));
|
|
1216
1431
|
for (const kb of availableKBs) {
|
|
@@ -1236,11 +1451,30 @@ export class KnowledgeEngine {
|
|
|
1236
1451
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1237
1452
|
mode_used: mode,
|
|
1238
1453
|
tuning: tuningSummary,
|
|
1454
|
+
miss: classifySearchMiss({
|
|
1455
|
+
allKnowledgeBases: allKBs,
|
|
1456
|
+
availableKnowledgeBases: availableKBs,
|
|
1457
|
+
searchedKnowledgeBases: kbs,
|
|
1458
|
+
requestedKbId: kb_id,
|
|
1459
|
+
mode,
|
|
1460
|
+
fileType: filters?.file_type,
|
|
1461
|
+
pathPattern: filters?.path_pattern,
|
|
1462
|
+
minimumScore: tuning.minHybridScore,
|
|
1463
|
+
offset,
|
|
1464
|
+
retrievedCandidates: 0,
|
|
1465
|
+
thresholdCandidates: 0,
|
|
1466
|
+
eligibleCandidates: 0,
|
|
1467
|
+
filteredCandidates: 0,
|
|
1468
|
+
rankedCandidates: 0,
|
|
1469
|
+
totalResults: 0,
|
|
1470
|
+
embeddingIncompatibleCount: 0,
|
|
1471
|
+
}),
|
|
1239
1472
|
};
|
|
1240
1473
|
}
|
|
1241
1474
|
const allResults = [];
|
|
1242
1475
|
const vectorsByChunkId = new Map();
|
|
1243
1476
|
let queryEmbedding;
|
|
1477
|
+
const embeddingIncompatibleKbIds = new Set();
|
|
1244
1478
|
for (const kb of kbs) {
|
|
1245
1479
|
throwIfAborted(signal);
|
|
1246
1480
|
if (kb.chunk_count === 0)
|
|
@@ -1258,6 +1492,7 @@ export class KnowledgeEngine {
|
|
|
1258
1492
|
throwIfAborted(signal);
|
|
1259
1493
|
const compatibility = inspectEmbeddingCompatibility(kb.embedding_signature, kb.embedding_dimension, queryEmbeddingConfig, queryVec.length);
|
|
1260
1494
|
if (!compatibility.compatible) {
|
|
1495
|
+
embeddingIncompatibleKbIds.add(kb.id);
|
|
1261
1496
|
warnings.push(embeddingMismatchWarning(kb, compatibility));
|
|
1262
1497
|
continue;
|
|
1263
1498
|
}
|
|
@@ -1283,6 +1518,7 @@ export class KnowledgeEngine {
|
|
|
1283
1518
|
vectorsByChunkId.set(chunkId, vector);
|
|
1284
1519
|
}
|
|
1285
1520
|
else {
|
|
1521
|
+
embeddingIncompatibleKbIds.add(kb.id);
|
|
1286
1522
|
warnings.push(embeddingMismatchWarning(kb, compatibility));
|
|
1287
1523
|
}
|
|
1288
1524
|
const fused = weightedScoreFusion(bm25Results, vecResults);
|
|
@@ -1307,30 +1543,30 @@ export class KnowledgeEngine {
|
|
|
1307
1543
|
result.score = ranking.adjusted_score;
|
|
1308
1544
|
}
|
|
1309
1545
|
}
|
|
1310
|
-
let
|
|
1546
|
+
let thresholded = unique.filter((result) => rankingByChunkId.has(result.chunkId));
|
|
1547
|
+
if (retrievalMode !== "fast" && retrievalMode !== "semantic") {
|
|
1548
|
+
thresholded = thresholded.filter((result) => result.score >= tuning.minHybridScore);
|
|
1549
|
+
}
|
|
1550
|
+
const thresholdCandidates = thresholded.length;
|
|
1551
|
+
let eligible = thresholded;
|
|
1311
1552
|
if (retrievalMode === "fast") {
|
|
1312
|
-
|
|
1553
|
+
eligible = thresholded.filter((result) => {
|
|
1313
1554
|
const chunk = getChunkById(db, result.chunkId);
|
|
1314
|
-
|
|
1315
|
-
return false;
|
|
1316
|
-
return hasAnyLexicalEvidence(buildChunkEmbeddingText(chunk), queryTokens);
|
|
1555
|
+
return chunk ? hasAnyLexicalEvidence(buildChunkEmbeddingText(chunk), queryTokens) : false;
|
|
1317
1556
|
});
|
|
1318
1557
|
}
|
|
1319
|
-
else if (retrievalMode !== "semantic") {
|
|
1320
|
-
|
|
1558
|
+
else if (retrievalMode !== "semantic" && retrievalMode !== "semantic_hybrid") {
|
|
1559
|
+
eligible = thresholded.filter((result) => {
|
|
1321
1560
|
const chunk = getChunkById(db, result.chunkId);
|
|
1322
|
-
|
|
1323
|
-
return false;
|
|
1324
|
-
if (retrievalMode === "semantic_hybrid")
|
|
1325
|
-
return result.score >= tuning.minHybridScore;
|
|
1326
|
-
return result.score >= tuning.minHybridScore && hasEnoughLexicalEvidence(chunk, queryTokens);
|
|
1561
|
+
return chunk ? hasEnoughLexicalEvidence(chunk, queryTokens) : false;
|
|
1327
1562
|
});
|
|
1328
1563
|
}
|
|
1329
|
-
|
|
1564
|
+
eligible.sort((a, b) => b.score - a.score);
|
|
1565
|
+
const eligibleCandidates = eligible.length;
|
|
1330
1566
|
// Apply metadata filters post-retrieval
|
|
1331
|
-
let filtered =
|
|
1567
|
+
let filtered = eligible;
|
|
1332
1568
|
if (normalizedFileType || filters?.path_pattern) {
|
|
1333
|
-
filtered =
|
|
1569
|
+
filtered = eligible.filter((r) => {
|
|
1334
1570
|
const chunk = getChunkById(db, r.chunkId);
|
|
1335
1571
|
if (!chunk)
|
|
1336
1572
|
return false;
|
|
@@ -1402,19 +1638,42 @@ export class KnowledgeEngine {
|
|
|
1402
1638
|
},
|
|
1403
1639
|
};
|
|
1404
1640
|
});
|
|
1641
|
+
const deepTuning = {
|
|
1642
|
+
...tuningSummary,
|
|
1643
|
+
retrieved_candidates: unique.length,
|
|
1644
|
+
threshold_candidates: thresholdCandidates,
|
|
1645
|
+
eligible_candidates: eligibleCandidates,
|
|
1646
|
+
filtered_candidates: filtered.length,
|
|
1647
|
+
diversity_candidates: ranked.length,
|
|
1648
|
+
candidate_pool_truncated: candidatePoolTruncated,
|
|
1649
|
+
};
|
|
1405
1650
|
return {
|
|
1406
1651
|
results,
|
|
1407
1652
|
total_count: diversified.length,
|
|
1408
1653
|
has_more: offset + limit < diversified.length,
|
|
1409
1654
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1410
1655
|
mode_used: mode,
|
|
1411
|
-
tuning:
|
|
1412
|
-
|
|
1413
|
-
|
|
1414
|
-
|
|
1415
|
-
|
|
1416
|
-
|
|
1417
|
-
|
|
1656
|
+
tuning: deepTuning,
|
|
1657
|
+
miss: results.length === 0
|
|
1658
|
+
? classifySearchMiss({
|
|
1659
|
+
allKnowledgeBases: allKBs,
|
|
1660
|
+
availableKnowledgeBases: availableKBs,
|
|
1661
|
+
searchedKnowledgeBases: kbs,
|
|
1662
|
+
requestedKbId: kb_id,
|
|
1663
|
+
mode,
|
|
1664
|
+
fileType: filters?.file_type,
|
|
1665
|
+
pathPattern: filters?.path_pattern,
|
|
1666
|
+
minimumScore: tuning.minHybridScore,
|
|
1667
|
+
offset,
|
|
1668
|
+
retrievedCandidates: unique.length,
|
|
1669
|
+
thresholdCandidates,
|
|
1670
|
+
eligibleCandidates,
|
|
1671
|
+
filteredCandidates: filtered.length,
|
|
1672
|
+
rankedCandidates: ranked.length,
|
|
1673
|
+
totalResults: diversified.length,
|
|
1674
|
+
embeddingIncompatibleCount: embeddingIncompatibleKbIds.size,
|
|
1675
|
+
})
|
|
1676
|
+
: undefined,
|
|
1418
1677
|
};
|
|
1419
1678
|
}
|
|
1420
1679
|
const ranked = [];
|
|
@@ -1483,24 +1742,41 @@ export class KnowledgeEngine {
|
|
|
1483
1742
|
};
|
|
1484
1743
|
})(),
|
|
1485
1744
|
}));
|
|
1745
|
+
const responseTuning = {
|
|
1746
|
+
...tuningSummary,
|
|
1747
|
+
retrieved_candidates: unique.length,
|
|
1748
|
+
threshold_candidates: thresholdCandidates,
|
|
1749
|
+
eligible_candidates: eligibleCandidates,
|
|
1750
|
+
filtered_candidates: filtered.length,
|
|
1751
|
+
diversity_candidates: ranked.length,
|
|
1752
|
+
candidate_pool_truncated: candidatePoolTruncated,
|
|
1753
|
+
};
|
|
1486
1754
|
return {
|
|
1487
1755
|
results,
|
|
1488
1756
|
total_count: total,
|
|
1489
1757
|
has_more: offset + limit < total,
|
|
1490
1758
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1491
1759
|
mode_used: mode,
|
|
1492
|
-
tuning:
|
|
1493
|
-
|
|
1494
|
-
|
|
1495
|
-
|
|
1496
|
-
|
|
1497
|
-
|
|
1498
|
-
|
|
1499
|
-
|
|
1500
|
-
|
|
1501
|
-
|
|
1502
|
-
|
|
1503
|
-
|
|
1760
|
+
tuning: responseTuning,
|
|
1761
|
+
miss: results.length === 0
|
|
1762
|
+
? classifySearchMiss({
|
|
1763
|
+
allKnowledgeBases: allKBs,
|
|
1764
|
+
availableKnowledgeBases: availableKBs,
|
|
1765
|
+
searchedKnowledgeBases: kbs,
|
|
1766
|
+
requestedKbId: kb_id,
|
|
1767
|
+
mode,
|
|
1768
|
+
fileType: filters?.file_type,
|
|
1769
|
+
pathPattern: filters?.path_pattern,
|
|
1770
|
+
minimumScore: tuning.minHybridScore,
|
|
1771
|
+
offset,
|
|
1772
|
+
retrievedCandidates: unique.length,
|
|
1773
|
+
thresholdCandidates,
|
|
1774
|
+
eligibleCandidates,
|
|
1775
|
+
filteredCandidates: filtered.length,
|
|
1776
|
+
rankedCandidates: ranked.length,
|
|
1777
|
+
totalResults: total,
|
|
1778
|
+
embeddingIncompatibleCount: embeddingIncompatibleKbIds.size,
|
|
1779
|
+
})
|
|
1504
1780
|
: undefined,
|
|
1505
1781
|
};
|
|
1506
1782
|
}
|
|
@@ -1591,11 +1867,32 @@ export class KnowledgeEngine {
|
|
|
1591
1867
|
throw new Error("Engine not initialized");
|
|
1592
1868
|
throwIfAborted(signal);
|
|
1593
1869
|
const db = this.db;
|
|
1870
|
+
const allKBs = listKBs(db);
|
|
1594
1871
|
const selectedKB = options.kb_id ? (getKB(db, options.kb_id) ?? getKBByName(db, options.kb_id)) : undefined;
|
|
1595
|
-
|
|
1596
|
-
|
|
1872
|
+
const availableKBs = options.kb_id ? [selectedKB].filter(Boolean) : allKBs;
|
|
1873
|
+
const searchedKBs = availableKBs.filter((kb) => kb.status === "ready" || kb.status === "stale");
|
|
1597
1874
|
const limit = Math.max(1, Math.min(200, options.limit ?? 20));
|
|
1598
1875
|
const offset = Math.max(0, options.offset ?? 0);
|
|
1876
|
+
if (searchedKBs.length === 0) {
|
|
1877
|
+
return {
|
|
1878
|
+
results: [],
|
|
1879
|
+
total_count: 0,
|
|
1880
|
+
has_more: false,
|
|
1881
|
+
miss: classifySymbolSearchMiss({
|
|
1882
|
+
allKnowledgeBases: allKBs,
|
|
1883
|
+
availableKnowledgeBases: availableKBs,
|
|
1884
|
+
searchedKnowledgeBases: searchedKBs,
|
|
1885
|
+
requestedKbId: options.kb_id,
|
|
1886
|
+
kind: options.kind,
|
|
1887
|
+
filePattern: options.file_pattern,
|
|
1888
|
+
exact: options.exact === true,
|
|
1889
|
+
offset,
|
|
1890
|
+
totalResults: 0,
|
|
1891
|
+
indexedSymbols: 0,
|
|
1892
|
+
unfilteredMatches: 0,
|
|
1893
|
+
}),
|
|
1894
|
+
};
|
|
1895
|
+
}
|
|
1599
1896
|
const symbolOptions = {
|
|
1600
1897
|
kbId: selectedKB?.id,
|
|
1601
1898
|
kind: options.kind,
|
|
@@ -1611,25 +1908,48 @@ export class KnowledgeEngine {
|
|
|
1611
1908
|
const page = symbols.slice(0, limit);
|
|
1612
1909
|
const total = countSymbols(db, query, symbolOptions);
|
|
1613
1910
|
throwIfAborted(signal);
|
|
1911
|
+
const results = page.map((symbol) => {
|
|
1912
|
+
const kb = getKB(db, symbol.kb_id);
|
|
1913
|
+
return {
|
|
1914
|
+
name: symbol.name,
|
|
1915
|
+
kind: symbol.kind,
|
|
1916
|
+
file_path: symbol.file_path,
|
|
1917
|
+
file_type: symbol.file_type,
|
|
1918
|
+
kb_name: kb?.name ?? "unknown",
|
|
1919
|
+
start_line: symbol.start_line,
|
|
1920
|
+
end_line: symbol.end_line,
|
|
1921
|
+
signature: symbol.signature ?? undefined,
|
|
1922
|
+
container_name: symbol.container_name ?? undefined,
|
|
1923
|
+
text: symbol.text,
|
|
1924
|
+
indexed_at: symbol.indexed_at,
|
|
1925
|
+
};
|
|
1926
|
+
});
|
|
1927
|
+
let miss;
|
|
1928
|
+
if (results.length === 0) {
|
|
1929
|
+
const indexedSymbols = countSymbols(db, "", { kbId: selectedKB?.id });
|
|
1930
|
+
const unfilteredMatches = total === 0 && (options.kind || options.file_pattern)
|
|
1931
|
+
? countSymbols(db, query, { kbId: selectedKB?.id, exact: options.exact })
|
|
1932
|
+
: total;
|
|
1933
|
+
throwIfAborted(signal);
|
|
1934
|
+
miss = classifySymbolSearchMiss({
|
|
1935
|
+
allKnowledgeBases: allKBs,
|
|
1936
|
+
availableKnowledgeBases: availableKBs,
|
|
1937
|
+
searchedKnowledgeBases: searchedKBs,
|
|
1938
|
+
requestedKbId: options.kb_id,
|
|
1939
|
+
kind: options.kind,
|
|
1940
|
+
filePattern: options.file_pattern,
|
|
1941
|
+
exact: options.exact === true,
|
|
1942
|
+
offset,
|
|
1943
|
+
totalResults: total,
|
|
1944
|
+
indexedSymbols,
|
|
1945
|
+
unfilteredMatches,
|
|
1946
|
+
});
|
|
1947
|
+
}
|
|
1614
1948
|
return {
|
|
1615
|
-
results
|
|
1616
|
-
const kb = getKB(db, symbol.kb_id);
|
|
1617
|
-
return {
|
|
1618
|
-
name: symbol.name,
|
|
1619
|
-
kind: symbol.kind,
|
|
1620
|
-
file_path: symbol.file_path,
|
|
1621
|
-
file_type: symbol.file_type,
|
|
1622
|
-
kb_name: kb?.name ?? "unknown",
|
|
1623
|
-
start_line: symbol.start_line,
|
|
1624
|
-
end_line: symbol.end_line,
|
|
1625
|
-
signature: symbol.signature ?? undefined,
|
|
1626
|
-
container_name: symbol.container_name ?? undefined,
|
|
1627
|
-
text: symbol.text,
|
|
1628
|
-
indexed_at: symbol.indexed_at,
|
|
1629
|
-
};
|
|
1630
|
-
}),
|
|
1949
|
+
results,
|
|
1631
1950
|
total_count: total,
|
|
1632
1951
|
has_more: symbols.length > limit,
|
|
1952
|
+
miss,
|
|
1633
1953
|
};
|
|
1634
1954
|
}
|
|
1635
1955
|
doctor(signal) {
|
|
@@ -24,6 +24,8 @@ export interface SearchTuningSummary {
|
|
|
24
24
|
candidate_limit: number;
|
|
25
25
|
retrieval_limit_per_kb: number;
|
|
26
26
|
retrieved_candidates: number;
|
|
27
|
+
threshold_candidates: number;
|
|
28
|
+
eligible_candidates: number;
|
|
27
29
|
filtered_candidates: number;
|
|
28
30
|
diversity_candidates: number;
|
|
29
31
|
candidate_pool_truncated: boolean;
|
|
@@ -196,6 +196,8 @@ export function summarizeSearchTuning(tuning, limit, candidateLimit) {
|
|
|
196
196
|
candidate_limit: candidateLimit,
|
|
197
197
|
retrieval_limit_per_kb: candidateLimit,
|
|
198
198
|
retrieved_candidates: 0,
|
|
199
|
+
threshold_candidates: 0,
|
|
200
|
+
eligible_candidates: 0,
|
|
199
201
|
filtered_candidates: 0,
|
|
200
202
|
diversity_candidates: 0,
|
|
201
203
|
candidate_pool_truncated: false,
|
package/docs/configuration.md
CHANGED
|
@@ -86,15 +86,17 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
86
86
|
| `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES` | profile default | Override how many filtered candidates `deep` mode sends to the reranker. Clamped to 5-100. |
|
|
87
87
|
| `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` | profile default | Override deep rerank top-k multiplier relative to the effective result limit. Clamped to 1-10. |
|
|
88
88
|
|
|
89
|
-
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, global candidate limit, per-KB retrieval limit, retrieved/
|
|
89
|
+
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, global candidate limit, per-KB retrieval limit, retrieved/threshold-passed/eligibility-passed/metadata-passed/diversity candidate counts, global-pool truncation, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
90
90
|
|
|
91
91
|
`knowledge_search` supports `file_type`, substring `path_pattern`, `profile`, integer `limit` from 1 to 50, and non-negative integer `offset`. `offset + limit` cannot exceed the 800-candidate global search window. Each KB retrieves up to the profile-derived candidate limit, then merged results are ranked, filtered, and globally capped before deep reranking, adaptive expansion, or diversity. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` for pure vector search, or `semantic_hybrid` when weak lexical overlap should not block vector-backed weighted fusion. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks without claiming caller/callee graph semantics.
|
|
92
92
|
|
|
93
|
+
Empty `knowledge_search` responses always include structured `details.miss` with a deterministic reason, action, searched/skipped KB summaries, stage counts, requested filters, and bounded `auto` attempt summaries. Empty `knowledge_symbol_search` responses expose the corresponding symbol miss contract. The default text is a two-line rendering of these facts. Miss classification does not run source diagnostics, enumerate absolute source paths, change ranking, or mutate a KB.
|
|
94
|
+
|
|
93
95
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `reindex_for_embedding_budget`, `fix_embedding_config`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
94
96
|
|
|
95
97
|
`knowledge_configure` writes runtime configuration to the knowledge directory's `config.json`. It validates Node 22+ before persisting `node_path`, so agents can configure a Codex- or user-installed `node.exe` once without relying on environment variables being injected into an already-running OMP process.
|
|
96
98
|
|
|
97
|
-
`knowledge_symbol_search` uses a lightweight symbol index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find AST-backed methods/functions/classes/interfaces/types/variables for supported code files, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; it does not answer caller/callee or rename-safety questions.
|
|
99
|
+
`knowledge_symbol_search` uses a lightweight symbol index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find AST-backed methods/functions/classes/interfaces/types/variables for supported code files, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; it does not answer caller/callee or rename-safety questions. Empty lookups distinguish missing KBs, unavailable KBs, empty symbol metadata, exact-name misses, metadata filters, pagination, and ordinary no-match results. If the symbol index itself is empty, use `knowledge_update` for source-backed KBs; imported portable KBs and inline text KBs do not retain an active source path and must be re-imported or re-added from the original content.
|
|
98
100
|
|
|
99
101
|
## Test and Release Fixtures
|
|
100
102
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.15.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|