pi-knowledge 0.10.2 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +4 -3
- package/dist/index.js +2 -1
- package/dist/src/embedding/provider.d.ts +2 -0
- package/dist/src/embedding/provider.js +27 -8
- package/dist/src/engine.d.ts +1 -1
- package/dist/src/engine.js +5 -2
- package/dist/src/model-worker-client.d.ts +1 -1
- package/dist/src/model-worker-client.js +4 -2
- package/dist/src/model-worker.js +12 -2
- package/dist/src/search/tuning.d.ts +1 -1
- package/dist/src/search/tuning.js +1 -1
- package/docs/configuration.md +3 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.11.0] - 2026-09-23
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Added `PI_KNOWLEDGE_EMBEDDING_PREFIXES=on|off` so OpenAI-compatible and local embeddings can preserve or omit `query: ` / `passage: ` input prefixes while keeping default E5 behavior compatible with existing KB signatures (#18).
|
|
7
|
+
- Added `semantic_hybrid` search mode for vector-backed weighted fusion without the default hybrid lexical-evidence gate, giving semantic and multilingual queries an explicit recall-oriented path while preserving `hybrid` precision defaults (#17).
|
|
8
|
+
|
|
3
9
|
## [0.10.2] - 2026-09-19
|
|
4
10
|
|
|
5
11
|
### Fixed
|
package/README.md
CHANGED
|
@@ -67,7 +67,7 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
|
|
|
67
67
|
- **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and AST-derived code structure such as symbols, scope, parent symbol, and signatures with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
|
|
68
68
|
- **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
|
|
69
69
|
- **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and bounded adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results. Code KBs with AST metadata can also use same-parent/sibling structure during adaptive expansion.
|
|
70
|
-
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
70
|
+
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `semantic_hybrid`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
71
71
|
- **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
|
|
72
72
|
|
|
73
73
|
This is intentionally not a heavy ColBERT-style late-interaction index yet (Khattab and Zaharia 2020). The current product chooses lightweight local embeddings, BM25, query-aware ranking, optional cross-encoder reranking, streamed vector scans, and health diagnostics for commercial usefulness with low setup cost.
|
|
@@ -125,6 +125,7 @@ omp install ./pi-knowledge
|
|
|
125
125
|
- `fast`: BM25 keyword search for exact symbols, commands, and identifiers.
|
|
126
126
|
- `semantic`: vector search for conceptual matches.
|
|
127
127
|
- `hybrid`: lexical-anchored BM25 + vector search with normalized weighted score fusion. It requires keyword evidence to avoid low-confidence semantic false positives.
|
|
128
|
+
- `semantic_hybrid`: vector search plus optional BM25 weighted fusion without the hybrid lexical-evidence gate; use explicitly for semantic/multilingual queries where lexical overlap is weak but fused scoring is still useful.
|
|
128
129
|
- `deep`: hybrid retrieval followed by cross-encoder reranking.
|
|
129
130
|
- `adaptive`: hybrid retrieval followed by query-time contextual window expansion around seed chunks. It keeps the matched seed, prefers nearby/query-relevant neighboring chunks, and collapses overlapping windows from the same file.
|
|
130
131
|
- `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
|
|
@@ -136,7 +137,7 @@ Mode selection contract:
|
|
|
136
137
|
|
|
137
138
|
- Start with `hybrid` for most project questions that contain useful lexical anchors.
|
|
138
139
|
- Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
|
|
139
|
-
- Use `semantic` when the query is conceptual
|
|
140
|
+
- Use `semantic` when the query is conceptual and exact terms may differ from indexed wording; use `semantic_hybrid` when weak lexical overlap should not block vector-backed fused results.
|
|
140
141
|
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, AST-related sibling methods where indexed, or enough context to make a safe edit.
|
|
141
142
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
142
143
|
- If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
|
|
@@ -185,7 +186,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
185
186
|
|------|-----------------------|
|
|
186
187
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
187
188
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
188
|
-
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
189
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
189
190
|
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
190
191
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
191
192
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
package/dist/index.js
CHANGED
|
@@ -364,7 +364,7 @@ export default function (pi) {
|
|
|
364
364
|
"Use knowledge_search to find relevant context before answering domain questions",
|
|
365
365
|
"Default to mode 'hybrid' for most project questions with useful lexical anchors; it fuses BM25 and semantic vectors but still requires keyword evidence",
|
|
366
366
|
"Use mode 'fast' for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings",
|
|
367
|
-
"Use mode 'semantic' for
|
|
367
|
+
"Use mode 'semantic' for pure vector search when exact terms may differ from indexed wording; use mode 'semantic_hybrid' when you want vector recall plus BM25/fusion scoring without the hybrid lexical evidence gate",
|
|
368
368
|
"Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
|
|
369
369
|
"Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
|
|
370
370
|
"Use profile 'low_token' for slow local models that need fewer, stricter, longer snippets; use profile 'precision' for identifiers/errors and 'recall' or 'long_context' for broad prose/document research",
|
|
@@ -378,6 +378,7 @@ export default function (pi) {
|
|
|
378
378
|
Type.Literal("fast"),
|
|
379
379
|
Type.Literal("semantic"),
|
|
380
380
|
Type.Literal("hybrid"),
|
|
381
|
+
Type.Literal("semantic_hybrid"),
|
|
381
382
|
Type.Literal("deep"),
|
|
382
383
|
Type.Literal("adaptive"),
|
|
383
384
|
Type.Literal("code"),
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
export type EmbeddingProvider = "local" | "openai";
|
|
2
2
|
export type EmbeddingPrefix = "query" | "passage";
|
|
3
|
+
export type EmbeddingPrefixStrategy = "on" | "off";
|
|
3
4
|
export interface EmbeddingConfig {
|
|
4
5
|
provider: EmbeddingProvider;
|
|
5
6
|
model: string;
|
|
@@ -7,6 +8,7 @@ export interface EmbeddingConfig {
|
|
|
7
8
|
maxChars: number;
|
|
8
9
|
queryPrefix: "query";
|
|
9
10
|
documentPrefix: "passage";
|
|
11
|
+
prefixes: EmbeddingPrefixStrategy;
|
|
10
12
|
pooling: "mean";
|
|
11
13
|
normalize: true;
|
|
12
14
|
}
|
|
@@ -13,13 +13,14 @@ export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
|
|
|
13
13
|
const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
|
|
14
14
|
const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
|
|
15
15
|
const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
|
|
16
|
-
function localEmbeddingConfig(maxChars) {
|
|
16
|
+
function localEmbeddingConfig(maxChars, prefixes) {
|
|
17
17
|
return {
|
|
18
18
|
provider: "local",
|
|
19
19
|
model: DEFAULT_LOCAL_EMBEDDING_MODEL,
|
|
20
20
|
maxChars,
|
|
21
21
|
queryPrefix: "query",
|
|
22
22
|
documentPrefix: "passage",
|
|
23
|
+
prefixes,
|
|
23
24
|
pooling: "mean",
|
|
24
25
|
normalize: true,
|
|
25
26
|
};
|
|
@@ -65,6 +66,14 @@ function waitForNoActiveRuns() {
|
|
|
65
66
|
function providerInstanceHash(value) {
|
|
66
67
|
return createHash("sha256").update(value).digest("hex").slice(0, 12);
|
|
67
68
|
}
|
|
69
|
+
function resolveEmbeddingPrefixStrategy(env) {
|
|
70
|
+
const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_PREFIXES)?.toLowerCase();
|
|
71
|
+
if (!raw)
|
|
72
|
+
return "on";
|
|
73
|
+
if (raw === "on" || raw === "off")
|
|
74
|
+
return raw;
|
|
75
|
+
throw new Error(`Unsupported embedding prefix strategy: ${raw}. Use PI_KNOWLEDGE_EMBEDDING_PREFIXES=on or off.`);
|
|
76
|
+
}
|
|
68
77
|
export function resolveEmbeddingConfig(env = process.env) {
|
|
69
78
|
const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING) ?? `local:${DEFAULT_LOCAL_EMBEDDING_MODEL}`;
|
|
70
79
|
const separator = raw.indexOf(":");
|
|
@@ -72,12 +81,13 @@ export function resolveEmbeddingConfig(env = process.env) {
|
|
|
72
81
|
const model = separator >= 0 ? raw.slice(separator + 1).trim() : "";
|
|
73
82
|
const maxCharsValue = Number(cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS) ?? DEFAULT_API_MAX_EMBED_CHARS);
|
|
74
83
|
const maxChars = Number.isFinite(maxCharsValue) && maxCharsValue > 0 ? Math.trunc(maxCharsValue) : DEFAULT_API_MAX_EMBED_CHARS;
|
|
84
|
+
const prefixes = resolveEmbeddingPrefixStrategy(env);
|
|
75
85
|
if (provider === "local") {
|
|
76
86
|
const localModel = model || DEFAULT_LOCAL_EMBEDDING_MODEL;
|
|
77
87
|
if (localModel !== DEFAULT_LOCAL_EMBEDDING_MODEL) {
|
|
78
88
|
throw new Error(`Unsupported local embedding model: ${localModel}`);
|
|
79
89
|
}
|
|
80
|
-
return localEmbeddingConfig(maxChars);
|
|
90
|
+
return localEmbeddingConfig(maxChars, prefixes);
|
|
81
91
|
}
|
|
82
92
|
if (provider === "openai") {
|
|
83
93
|
return {
|
|
@@ -87,6 +97,7 @@ export function resolveEmbeddingConfig(env = process.env) {
|
|
|
87
97
|
maxChars,
|
|
88
98
|
queryPrefix: "query",
|
|
89
99
|
documentPrefix: "passage",
|
|
100
|
+
prefixes,
|
|
90
101
|
pooling: "mean",
|
|
91
102
|
normalize: true,
|
|
92
103
|
};
|
|
@@ -98,7 +109,7 @@ export function embeddingConfigLabel(config) {
|
|
|
98
109
|
}
|
|
99
110
|
export function embeddingSignature(config, dimension) {
|
|
100
111
|
const base = config.baseUrl ? `:base-sha256=${providerInstanceHash(config.baseUrl)}` : "";
|
|
101
|
-
|
|
112
|
+
const parts = [
|
|
102
113
|
`${config.provider}:${config.model}${base}`,
|
|
103
114
|
`dim=${dimension}`,
|
|
104
115
|
`apiMax=${config.provider === "openai" ? config.maxChars : "none"}`,
|
|
@@ -106,7 +117,10 @@ export function embeddingSignature(config, dimension) {
|
|
|
106
117
|
`normalize=${config.normalize}`,
|
|
107
118
|
`q=${config.queryPrefix}`,
|
|
108
119
|
`d=${config.documentPrefix}`,
|
|
109
|
-
]
|
|
120
|
+
];
|
|
121
|
+
if (config.prefixes === "off")
|
|
122
|
+
parts.push("prefixes=off");
|
|
123
|
+
return parts.join(":");
|
|
110
124
|
}
|
|
111
125
|
export async function dispose() {
|
|
112
126
|
clearIdleTimer();
|
|
@@ -124,9 +138,14 @@ export async function prepareForShutdown() {
|
|
|
124
138
|
clearIdleTimer();
|
|
125
139
|
await waitForNoActiveRuns();
|
|
126
140
|
}
|
|
141
|
+
function applyEmbeddingPrefix(text, prefix, config) {
|
|
142
|
+
return config.prefixes === "on" ? `${prefix}: ${text}` : text;
|
|
143
|
+
}
|
|
127
144
|
async function embedViaAPI(texts, prefix, config, signal) {
|
|
128
|
-
const
|
|
129
|
-
|
|
145
|
+
const safeTexts = texts.map((text) => {
|
|
146
|
+
const embeddingText = applyEmbeddingPrefix(text, prefix, config);
|
|
147
|
+
return embeddingText.length > config.maxChars ? embeddingText.slice(0, config.maxChars) : embeddingText;
|
|
148
|
+
});
|
|
130
149
|
const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
|
|
131
150
|
if (!config.baseUrl)
|
|
132
151
|
throw new Error("OpenAI embedding base URL is not configured");
|
|
@@ -162,12 +181,12 @@ export async function embedTextsWithConfig(texts, prefix, signal, options = {})
|
|
|
162
181
|
if (!API_FALLBACK_TO_LOCAL || !options.allowApiFallback)
|
|
163
182
|
throw error;
|
|
164
183
|
console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
|
|
165
|
-
actualConfig = localEmbeddingConfig(config.maxChars);
|
|
184
|
+
actualConfig = localEmbeddingConfig(config.maxChars, config.prefixes);
|
|
166
185
|
}
|
|
167
186
|
}
|
|
168
187
|
beginRun();
|
|
169
188
|
try {
|
|
170
|
-
return { vectors: await embedInModelWorker(texts, prefix, signal), config: actualConfig };
|
|
189
|
+
return { vectors: await embedInModelWorker(texts, prefix, actualConfig.prefixes, signal), config: actualConfig };
|
|
171
190
|
}
|
|
172
191
|
finally {
|
|
173
192
|
endRun();
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { createSkippedScanStats } from "./indexer/chunker.ts";
|
|
|
3
3
|
import { type RankingDiagnostics } from "./search/ranking.ts";
|
|
4
4
|
import { type SearchProfile, type SearchTuningSummary } from "./search/tuning.ts";
|
|
5
5
|
import { type KnowledgeBase, type KnowledgeSymbol } from "./storage/sqlite.ts";
|
|
6
|
-
export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
6
|
+
export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
7
7
|
export interface SearchOptions {
|
|
8
8
|
mode?: SearchMode;
|
|
9
9
|
profile?: SearchProfile;
|
package/dist/src/engine.js
CHANGED
|
@@ -1371,8 +1371,9 @@ export class KnowledgeEngine {
|
|
|
1371
1371
|
vectorsByChunkId.set(chunkId, vector);
|
|
1372
1372
|
}
|
|
1373
1373
|
else {
|
|
1374
|
+
const semanticHybrid = retrievalMode === "semantic_hybrid";
|
|
1374
1375
|
const bm25Results = searchBM25(db, normalizedQuery || query, candidateLimit, kb.id);
|
|
1375
|
-
if (bm25Results.length === 0)
|
|
1376
|
+
if (bm25Results.length === 0 && !semanticHybrid)
|
|
1376
1377
|
continue;
|
|
1377
1378
|
let vecResults = [];
|
|
1378
1379
|
const { vector: queryVec, config: queryEmbeddingConfig } = await embedQueryWithConfig(query, signal);
|
|
@@ -1422,6 +1423,8 @@ export class KnowledgeEngine {
|
|
|
1422
1423
|
const chunk = getChunkById(db, result.chunkId);
|
|
1423
1424
|
if (!chunk)
|
|
1424
1425
|
return false;
|
|
1426
|
+
if (retrievalMode === "semantic_hybrid")
|
|
1427
|
+
return result.score >= tuning.minHybridScore;
|
|
1425
1428
|
return result.score >= tuning.minHybridScore && hasEnoughLexicalEvidence(chunk, queryTokens);
|
|
1426
1429
|
});
|
|
1427
1430
|
}
|
|
@@ -1578,7 +1581,7 @@ export class KnowledgeEngine {
|
|
|
1578
1581
|
tuning: tuningSummary,
|
|
1579
1582
|
suggestions: results.length === 0
|
|
1580
1583
|
? [
|
|
1581
|
-
"Try mode 'fast' for exact symbols
|
|
1584
|
+
"Try mode 'fast' for exact symbols, mode 'semantic' for pure vector search, or mode 'semantic_hybrid' when weak lexical overlap should not block fused vector results.",
|
|
1582
1585
|
"Run knowledge_status if the KB should contain this answer.",
|
|
1583
1586
|
]
|
|
1584
1587
|
: undefined,
|
|
@@ -2,7 +2,7 @@ import type { HfRerankerConfig } from "./search/reranker-config.ts";
|
|
|
2
2
|
export declare function validateModelWorkerNodePath(nodePath: string): string;
|
|
3
3
|
export declare function resolveModelWorkerNodePath(): string;
|
|
4
4
|
export declare function configureModelWorkerNodePath(nodePath?: string): string;
|
|
5
|
-
export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", signal?: AbortSignal): Promise<Float32Array[]>;
|
|
5
|
+
export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", prefixesOrSignal?: "on" | "off" | AbortSignal, signal?: AbortSignal): Promise<Float32Array[]>;
|
|
6
6
|
export interface RerankWorkerCandidate {
|
|
7
7
|
chunkId: string;
|
|
8
8
|
content: string;
|
|
@@ -385,8 +385,10 @@ function getWorker() {
|
|
|
385
385
|
IpcModelWorkerTransport.create(workerPath, nodeExecPath) ?? new StdioModelWorkerTransport(workerPath, nodeExecPath);
|
|
386
386
|
return worker;
|
|
387
387
|
}
|
|
388
|
-
export async function embedInModelWorker(texts, prefix, signal) {
|
|
389
|
-
const
|
|
388
|
+
export async function embedInModelWorker(texts, prefix, prefixesOrSignal, signal) {
|
|
389
|
+
const prefixes = prefixesOrSignal === "off" ? "off" : "on";
|
|
390
|
+
const requestSignal = typeof prefixesOrSignal === "object" ? prefixesOrSignal : signal;
|
|
391
|
+
const result = await getWorker().request({ type: "embed", texts, prefix, prefixes }, requestSignal);
|
|
390
392
|
if (!Array.isArray(result))
|
|
391
393
|
throw new Error("Invalid embedding worker response");
|
|
392
394
|
return result.map((vector) => {
|
package/dist/src/model-worker.js
CHANGED
|
@@ -66,11 +66,17 @@ async function loadRerankerPipeline(config) {
|
|
|
66
66
|
rerankerPipeline = { key, pipe: loaded };
|
|
67
67
|
return loaded;
|
|
68
68
|
}
|
|
69
|
+
function applyEmbeddingPrefix(text, prefix, prefixes) {
|
|
70
|
+
return prefixes === "on" ? `${prefix}: ${text}` : text;
|
|
71
|
+
}
|
|
69
72
|
async function handleEmbed(request) {
|
|
70
73
|
const pipe = await loadEmbeddingPipeline();
|
|
71
74
|
const vectors = [];
|
|
72
75
|
for (const text of request.texts) {
|
|
73
|
-
const output = await pipe(
|
|
76
|
+
const output = await pipe(applyEmbeddingPrefix(text, request.prefix, request.prefixes), {
|
|
77
|
+
pooling: "mean",
|
|
78
|
+
normalize: true,
|
|
79
|
+
});
|
|
74
80
|
vectors.push(Array.from(output.data));
|
|
75
81
|
}
|
|
76
82
|
return vectors;
|
|
@@ -110,7 +116,11 @@ function parseWorkerRequest(message) {
|
|
|
110
116
|
if (!("prefix" in message) || (message.prefix !== "query" && message.prefix !== "passage")) {
|
|
111
117
|
throw new Error("Invalid embed prefix");
|
|
112
118
|
}
|
|
113
|
-
|
|
119
|
+
const prefixes = "prefixes" in message && (message.prefixes === "on" || message.prefixes === "off") ? message.prefixes : "on";
|
|
120
|
+
if ("prefixes" in message && message.prefixes !== "on" && message.prefixes !== "off") {
|
|
121
|
+
throw new Error("Invalid embed prefix strategy");
|
|
122
|
+
}
|
|
123
|
+
return { id: message.id, type: "embed", texts: message.texts, prefix: message.prefix, prefixes };
|
|
114
124
|
}
|
|
115
125
|
if (message.type === "rerank") {
|
|
116
126
|
if (!("query" in message) || typeof message.query !== "string")
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
export type SearchProfile = "auto" | "balanced" | "low_token" | "precision" | "recall" | "long_context" | "code" | "docs";
|
|
2
2
|
export type RuntimeProfile = Exclude<SearchProfile, "auto">;
|
|
3
|
-
type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
3
|
+
type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
4
4
|
export interface SearchTuning {
|
|
5
5
|
requestedProfile: SearchProfile;
|
|
6
6
|
selectedProfile: RuntimeProfile;
|
|
@@ -143,7 +143,7 @@ function chooseRuntimeProfile(input) {
|
|
|
143
143
|
}
|
|
144
144
|
if (input.mode === "docs" || input.mode === "decision" || input.mode === "adaptive")
|
|
145
145
|
return "long_context";
|
|
146
|
-
if (input.mode === "semantic")
|
|
146
|
+
if (input.mode === "semantic" || input.mode === "semantic_hybrid")
|
|
147
147
|
return "recall";
|
|
148
148
|
if (looksErrorQuery(input.query) || looksExactOrCodeQuery(input.query))
|
|
149
149
|
return "precision";
|
package/docs/configuration.md
CHANGED
|
@@ -25,6 +25,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
25
25
|
| `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
|
|
26
26
|
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
27
27
|
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
28
|
+
| `PI_KNOWLEDGE_EMBEDDING_PREFIXES` | `on` | Controls whether `query: ` and `passage: ` prefixes are prepended before embedding. Values: `on`, `off`. Keep `on` for the default E5 local model; use `off` only for models or self-hosted APIs that are evaluated without E5-style prefixes. |
|
|
28
29
|
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
29
30
|
| `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
|
|
30
31
|
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
|
|
@@ -32,7 +33,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
32
33
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
33
34
|
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
34
35
|
|
|
35
|
-
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character settings or after upgrading older KBs with no signature metadata.
|
|
36
|
+
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, non-default prefix strategy, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character/prefix settings or after upgrading older KBs with no signature metadata.
|
|
36
37
|
|
|
37
38
|
## Reranker
|
|
38
39
|
|
|
@@ -84,7 +85,7 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
84
85
|
|
|
85
86
|
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool size, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
86
87
|
|
|
87
|
-
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic`
|
|
88
|
+
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` for pure vector search, or `semantic_hybrid` when weak lexical overlap should not block vector-backed weighted fusion. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks, but it remains metadata-driven context expansion rather than caller/callee analysis.
|
|
88
89
|
|
|
89
90
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
90
91
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|