pi-knowledge 0.10.1 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,17 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.11.0] - 2026-09-23
4
+
5
+ ### Added
6
+ - Added `PI_KNOWLEDGE_EMBEDDING_PREFIXES=on|off` so OpenAI-compatible and local embeddings can preserve or omit `query: ` / `passage: ` input prefixes while keeping default E5 behavior compatible with existing KB signatures (#18).
7
+ - Added `semantic_hybrid` search mode for vector-backed weighted fusion without the default hybrid lexical-evidence gate, giving semantic and multilingual queries an explicit recall-oriented path while preserving `hybrid` precision defaults (#17).
8
+
9
+ ## [0.10.2] - 2026-09-19
10
+
11
+ ### Fixed
12
+ - Allowed OpenAI-compatible embedding endpoints to run without an API key and added `PI_KNOWLEDGE_EMBEDDING_API_KEY` as an embedding-specific bearer token override (#15).
13
+ - Added `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` so self-hosted embedding servers can use request batch sizes below the default 64 chunk limit (#16).
14
+
3
15
  ## [0.10.1] - 2026-09-10
4
16
 
5
17
  ### Fixed
package/README.md CHANGED
@@ -67,7 +67,7 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
67
67
  - **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and AST-derived code structure such as symbols, scope, parent symbol, and signatures with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
68
68
  - **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
69
69
  - **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and bounded adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results. Code KBs with AST metadata can also use same-parent/sibling structure during adaptive expansion.
70
- - **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
70
+ - **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `semantic_hybrid`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
71
71
  - **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
72
72
 
73
73
  This is intentionally not a heavy ColBERT-style late-interaction index yet (Khattab and Zaharia 2020). The current product chooses lightweight local embeddings, BM25, query-aware ranking, optional cross-encoder reranking, streamed vector scans, and health diagnostics for commercial usefulness with low setup cost.
@@ -125,6 +125,7 @@ omp install ./pi-knowledge
125
125
  - `fast`: BM25 keyword search for exact symbols, commands, and identifiers.
126
126
  - `semantic`: vector search for conceptual matches.
127
127
  - `hybrid`: lexical-anchored BM25 + vector search with normalized weighted score fusion. It requires keyword evidence to avoid low-confidence semantic false positives.
128
+ - `semantic_hybrid`: vector search plus optional BM25 weighted fusion without the hybrid lexical-evidence gate; use explicitly for semantic/multilingual queries where lexical overlap is weak but fused scoring is still useful.
128
129
  - `deep`: hybrid retrieval followed by cross-encoder reranking.
129
130
  - `adaptive`: hybrid retrieval followed by query-time contextual window expansion around seed chunks. It keeps the matched seed, prefers nearby/query-relevant neighboring chunks, and collapses overlapping windows from the same file.
130
131
  - `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
@@ -136,7 +137,7 @@ Mode selection contract:
136
137
 
137
138
  - Start with `hybrid` for most project questions that contain useful lexical anchors.
138
139
  - Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
139
- - Use `semantic` when the query is conceptual, exact terms may differ from indexed wording, or hybrid returns no lexical matches.
140
+ - Use `semantic` when the query is conceptual and exact terms may differ from indexed wording; use `semantic_hybrid` when weak lexical overlap should not block vector-backed fused results.
140
141
  - Use `adaptive` when the answer needs nearby code, neighboring documentation sections, AST-related sibling methods where indexed, or enough context to make a safe edit.
141
142
  - Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
142
143
  - If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
@@ -166,13 +167,15 @@ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BA
166
167
  ```bash
167
168
  export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
168
169
  export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
169
- export OPENAI_API_KEY=local-placeholder
170
+ # Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
170
171
  ```
171
172
 
172
173
  API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
173
174
 
174
175
  API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
175
176
 
177
+ For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
178
+
176
179
  Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
177
180
 
178
181
  ## Configuration
@@ -183,7 +186,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
183
186
  |------|-----------------------|
184
187
  | Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
185
188
  | Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
186
- | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
189
+ | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
187
190
  | Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
188
191
  | Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
189
192
  | Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
package/dist/index.js CHANGED
@@ -364,7 +364,7 @@ export default function (pi) {
364
364
  "Use knowledge_search to find relevant context before answering domain questions",
365
365
  "Default to mode 'hybrid' for most project questions with useful lexical anchors; it fuses BM25 and semantic vectors but still requires keyword evidence",
366
366
  "Use mode 'fast' for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings",
367
- "Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording or hybrid returns no lexical matches",
367
+ "Use mode 'semantic' for pure vector search when exact terms may differ from indexed wording; use mode 'semantic_hybrid' when you want vector recall plus BM25/fusion scoring without the hybrid lexical evidence gate",
368
368
  "Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
369
369
  "Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
370
370
  "Use profile 'low_token' for slow local models that need fewer, stricter, longer snippets; use profile 'precision' for identifiers/errors and 'recall' or 'long_context' for broad prose/document research",
@@ -378,6 +378,7 @@ export default function (pi) {
378
378
  Type.Literal("fast"),
379
379
  Type.Literal("semantic"),
380
380
  Type.Literal("hybrid"),
381
+ Type.Literal("semantic_hybrid"),
381
382
  Type.Literal("deep"),
382
383
  Type.Literal("adaptive"),
383
384
  Type.Literal("code"),
@@ -1,5 +1,6 @@
1
1
  export type EmbeddingProvider = "local" | "openai";
2
2
  export type EmbeddingPrefix = "query" | "passage";
3
+ export type EmbeddingPrefixStrategy = "on" | "off";
3
4
  export interface EmbeddingConfig {
4
5
  provider: EmbeddingProvider;
5
6
  model: string;
@@ -7,6 +8,7 @@ export interface EmbeddingConfig {
7
8
  maxChars: number;
8
9
  queryPrefix: "query";
9
10
  documentPrefix: "passage";
11
+ prefixes: EmbeddingPrefixStrategy;
10
12
  pooling: "mean";
11
13
  normalize: true;
12
14
  }
@@ -13,13 +13,14 @@ export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
13
13
  const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
14
14
  const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
15
15
  const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
16
- function localEmbeddingConfig(maxChars) {
16
+ function localEmbeddingConfig(maxChars, prefixes) {
17
17
  return {
18
18
  provider: "local",
19
19
  model: DEFAULT_LOCAL_EMBEDDING_MODEL,
20
20
  maxChars,
21
21
  queryPrefix: "query",
22
22
  documentPrefix: "passage",
23
+ prefixes,
23
24
  pooling: "mean",
24
25
  normalize: true,
25
26
  };
@@ -65,6 +66,14 @@ function waitForNoActiveRuns() {
65
66
  function providerInstanceHash(value) {
66
67
  return createHash("sha256").update(value).digest("hex").slice(0, 12);
67
68
  }
69
+ function resolveEmbeddingPrefixStrategy(env) {
70
+ const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_PREFIXES)?.toLowerCase();
71
+ if (!raw)
72
+ return "on";
73
+ if (raw === "on" || raw === "off")
74
+ return raw;
75
+ throw new Error(`Unsupported embedding prefix strategy: ${raw}. Use PI_KNOWLEDGE_EMBEDDING_PREFIXES=on or off.`);
76
+ }
68
77
  export function resolveEmbeddingConfig(env = process.env) {
69
78
  const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING) ?? `local:${DEFAULT_LOCAL_EMBEDDING_MODEL}`;
70
79
  const separator = raw.indexOf(":");
@@ -72,12 +81,13 @@ export function resolveEmbeddingConfig(env = process.env) {
72
81
  const model = separator >= 0 ? raw.slice(separator + 1).trim() : "";
73
82
  const maxCharsValue = Number(cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS) ?? DEFAULT_API_MAX_EMBED_CHARS);
74
83
  const maxChars = Number.isFinite(maxCharsValue) && maxCharsValue > 0 ? Math.trunc(maxCharsValue) : DEFAULT_API_MAX_EMBED_CHARS;
84
+ const prefixes = resolveEmbeddingPrefixStrategy(env);
75
85
  if (provider === "local") {
76
86
  const localModel = model || DEFAULT_LOCAL_EMBEDDING_MODEL;
77
87
  if (localModel !== DEFAULT_LOCAL_EMBEDDING_MODEL) {
78
88
  throw new Error(`Unsupported local embedding model: ${localModel}`);
79
89
  }
80
- return localEmbeddingConfig(maxChars);
90
+ return localEmbeddingConfig(maxChars, prefixes);
81
91
  }
82
92
  if (provider === "openai") {
83
93
  return {
@@ -87,6 +97,7 @@ export function resolveEmbeddingConfig(env = process.env) {
87
97
  maxChars,
88
98
  queryPrefix: "query",
89
99
  documentPrefix: "passage",
100
+ prefixes,
90
101
  pooling: "mean",
91
102
  normalize: true,
92
103
  };
@@ -98,7 +109,7 @@ export function embeddingConfigLabel(config) {
98
109
  }
99
110
  export function embeddingSignature(config, dimension) {
100
111
  const base = config.baseUrl ? `:base-sha256=${providerInstanceHash(config.baseUrl)}` : "";
101
- return [
112
+ const parts = [
102
113
  `${config.provider}:${config.model}${base}`,
103
114
  `dim=${dimension}`,
104
115
  `apiMax=${config.provider === "openai" ? config.maxChars : "none"}`,
@@ -106,7 +117,10 @@ export function embeddingSignature(config, dimension) {
106
117
  `normalize=${config.normalize}`,
107
118
  `q=${config.queryPrefix}`,
108
119
  `d=${config.documentPrefix}`,
109
- ].join(":");
120
+ ];
121
+ if (config.prefixes === "off")
122
+ parts.push("prefixes=off");
123
+ return parts.join(":");
110
124
  }
111
125
  export async function dispose() {
112
126
  clearIdleTimer();
@@ -124,18 +138,24 @@ export async function prepareForShutdown() {
124
138
  clearIdleTimer();
125
139
  await waitForNoActiveRuns();
126
140
  }
141
+ function applyEmbeddingPrefix(text, prefix, config) {
142
+ return config.prefixes === "on" ? `${prefix}: ${text}` : text;
143
+ }
127
144
  async function embedViaAPI(texts, prefix, config, signal) {
128
- const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
129
- const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
130
- const apiKey = process.env.OPENAI_API_KEY;
131
- if (!apiKey)
132
- throw new Error("OPENAI_API_KEY required for openai embedding");
145
+ const safeTexts = texts.map((text) => {
146
+ const embeddingText = applyEmbeddingPrefix(text, prefix, config);
147
+ return embeddingText.length > config.maxChars ? embeddingText.slice(0, config.maxChars) : embeddingText;
148
+ });
149
+ const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
133
150
  if (!config.baseUrl)
134
151
  throw new Error("OpenAI embedding base URL is not configured");
135
152
  const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
153
+ const headers = { "Content-Type": "application/json" };
154
+ if (apiKey)
155
+ headers.Authorization = `Bearer ${apiKey}`;
136
156
  const res = await fetch(endpoint, {
137
157
  method: "POST",
138
- headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
158
+ headers,
139
159
  body: JSON.stringify({ input: safeTexts, model: config.model }),
140
160
  signal,
141
161
  });
@@ -161,12 +181,12 @@ export async function embedTextsWithConfig(texts, prefix, signal, options = {})
161
181
  if (!API_FALLBACK_TO_LOCAL || !options.allowApiFallback)
162
182
  throw error;
163
183
  console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
164
- actualConfig = localEmbeddingConfig(config.maxChars);
184
+ actualConfig = localEmbeddingConfig(config.maxChars, config.prefixes);
165
185
  }
166
186
  }
167
187
  beginRun();
168
188
  try {
169
- return { vectors: await embedInModelWorker(texts, prefix, signal), config: actualConfig };
189
+ return { vectors: await embedInModelWorker(texts, prefix, actualConfig.prefixes, signal), config: actualConfig };
170
190
  }
171
191
  finally {
172
192
  endRun();
@@ -3,7 +3,7 @@ import { createSkippedScanStats } from "./indexer/chunker.ts";
3
3
  import { type RankingDiagnostics } from "./search/ranking.ts";
4
4
  import { type SearchProfile, type SearchTuningSummary } from "./search/tuning.ts";
5
5
  import { type KnowledgeBase, type KnowledgeSymbol } from "./storage/sqlite.ts";
6
- export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
6
+ export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
7
7
  export interface SearchOptions {
8
8
  mode?: SearchMode;
9
9
  profile?: SearchProfile;
@@ -15,8 +15,18 @@ import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js"
15
15
  import { searchVectorFile } from "./search/vector.js";
16
16
  import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
17
17
  export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
18
- const INDEX_EMBED_BATCH_SIZE = 64;
18
+ const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
19
19
  const VECTOR_REDUNDANCY_WEIGHT = 0.35;
20
+ function resolveIndexEmbedBatchSize(env = process.env) {
21
+ const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
22
+ if (!raw)
23
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
24
+ const value = Number(raw);
25
+ if (!Number.isFinite(value))
26
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
27
+ const batchSize = Math.trunc(value);
28
+ return batchSize > 0 ? batchSize : DEFAULT_INDEX_EMBED_BATCH_SIZE;
29
+ }
20
30
  function isCancellationError(error) {
21
31
  return error instanceof Error && error.message === "Cancelled";
22
32
  }
@@ -704,6 +714,7 @@ export class KnowledgeEngine {
704
714
  throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
705
715
  }
706
716
  const embeddingConfig = resolveEmbeddingConfig();
717
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
707
718
  const kb = createKB(db, {
708
719
  name,
709
720
  source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
@@ -759,7 +770,7 @@ export class KnowledgeEngine {
759
770
  return;
760
771
  if (signal?.aborted)
761
772
  throw new Error("Cancelled");
762
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
773
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
763
774
  reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
764
775
  const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
765
776
  if (signal?.aborted)
@@ -774,7 +785,7 @@ export class KnowledgeEngine {
774
785
  };
775
786
  const addChunks = async (chunks, processedFiles, totalFiles) => {
776
787
  pendingChunks.push(...chunks);
777
- while (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE) {
788
+ while (pendingChunks.length >= embeddingBatchSize) {
778
789
  await flushPending(processedFiles, totalFiles);
779
790
  }
780
791
  };
@@ -927,6 +938,7 @@ export class KnowledgeEngine {
927
938
  startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
928
939
  const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
929
940
  const embeddingConfig = resolveEmbeddingConfig();
941
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
930
942
  const embeddingModel = embeddingConfigLabel(embeddingConfig);
931
943
  const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
932
944
  let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
@@ -991,7 +1003,7 @@ export class KnowledgeEngine {
991
1003
  return;
992
1004
  if (signal?.aborted)
993
1005
  throw new Error("Cancelled");
994
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1006
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
995
1007
  const elapsed = Date.now() - startedAt;
996
1008
  const message = `Embedding update batch: ${addedCount} new chunks stored, ${scannedFiles} files scanned, elapsed ${formatDuration(elapsed)}`;
997
1009
  updateIndexingJob(this.db, kb.id, {
@@ -1045,7 +1057,7 @@ export class KnowledgeEngine {
1045
1057
  continue;
1046
1058
  }
1047
1059
  pendingChunks.push(chunk);
1048
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1060
+ if (pendingChunks.length >= embeddingBatchSize)
1049
1061
  await flushPending();
1050
1062
  }
1051
1063
  };
@@ -1359,8 +1371,9 @@ export class KnowledgeEngine {
1359
1371
  vectorsByChunkId.set(chunkId, vector);
1360
1372
  }
1361
1373
  else {
1374
+ const semanticHybrid = retrievalMode === "semantic_hybrid";
1362
1375
  const bm25Results = searchBM25(db, normalizedQuery || query, candidateLimit, kb.id);
1363
- if (bm25Results.length === 0)
1376
+ if (bm25Results.length === 0 && !semanticHybrid)
1364
1377
  continue;
1365
1378
  let vecResults = [];
1366
1379
  const { vector: queryVec, config: queryEmbeddingConfig } = await embedQueryWithConfig(query, signal);
@@ -1410,6 +1423,8 @@ export class KnowledgeEngine {
1410
1423
  const chunk = getChunkById(db, result.chunkId);
1411
1424
  if (!chunk)
1412
1425
  return false;
1426
+ if (retrievalMode === "semantic_hybrid")
1427
+ return result.score >= tuning.minHybridScore;
1413
1428
  return result.score >= tuning.minHybridScore && hasEnoughLexicalEvidence(chunk, queryTokens);
1414
1429
  });
1415
1430
  }
@@ -1566,7 +1581,7 @@ export class KnowledgeEngine {
1566
1581
  tuning: tuningSummary,
1567
1582
  suggestions: results.length === 0
1568
1583
  ? [
1569
- "Try mode 'fast' for exact symbols or mode 'semantic' for conceptual wording.",
1584
+ "Try mode 'fast' for exact symbols, mode 'semantic' for pure vector search, or mode 'semantic_hybrid' when weak lexical overlap should not block fused vector results.",
1570
1585
  "Run knowledge_status if the KB should contain this answer.",
1571
1586
  ]
1572
1587
  : undefined,
@@ -1789,6 +1804,7 @@ export class KnowledgeEngine {
1789
1804
  const { createWriteStream } = await import("node:fs");
1790
1805
  const tempOutputPath = tempVectorPath(outputPath);
1791
1806
  const stream = createWriteStream(tempOutputPath, { encoding: "utf-8" });
1807
+ const progressBatchSize = resolveIndexEmbedBatchSize();
1792
1808
  let count = 0;
1793
1809
  const header = JSON.stringify({
1794
1810
  name: kb.name,
@@ -1809,7 +1825,7 @@ export class KnowledgeEngine {
1809
1825
  metadata_json: chunk.metadata_json,
1810
1826
  }));
1811
1827
  count++;
1812
- if (count % INDEX_EMBED_BATCH_SIZE === 0)
1828
+ if (count % progressBatchSize === 0)
1813
1829
  onProgress?.(`Exported ${count}/${kb.chunk_count} chunks...`);
1814
1830
  }
1815
1831
  await finishWriteStream(stream);
@@ -1838,6 +1854,7 @@ export class KnowledgeEngine {
1838
1854
  const stream = createReadStream(inputPath, { encoding: "utf-8" });
1839
1855
  const lines = createInterface({ input: stream, crlfDelay: Infinity });
1840
1856
  const embeddingConfig = resolveEmbeddingConfig();
1857
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
1841
1858
  let header;
1842
1859
  let kb;
1843
1860
  let vectorWriter;
@@ -1850,7 +1867,7 @@ export class KnowledgeEngine {
1850
1867
  if (!this.db || !kb || !vectorWriter || pendingChunks.length === 0)
1851
1868
  return;
1852
1869
  throwIfAborted(signal);
1853
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1870
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
1854
1871
  const symbols = pendingSymbols.splice(0);
1855
1872
  const total = header?.chunk_count;
1856
1873
  const message = `Embedding import batch: ${inserted}/${total ?? "unknown"} chunks stored`;
@@ -1919,7 +1936,7 @@ export class KnowledgeEngine {
1919
1936
  chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
1920
1937
  pendingChunks.push(chunk);
1921
1938
  pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
1922
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1939
+ if (pendingChunks.length >= embeddingBatchSize)
1923
1940
  await flushPending();
1924
1941
  }
1925
1942
  if (!header)
@@ -2,7 +2,7 @@ import type { HfRerankerConfig } from "./search/reranker-config.ts";
2
2
  export declare function validateModelWorkerNodePath(nodePath: string): string;
3
3
  export declare function resolveModelWorkerNodePath(): string;
4
4
  export declare function configureModelWorkerNodePath(nodePath?: string): string;
5
- export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", signal?: AbortSignal): Promise<Float32Array[]>;
5
+ export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", prefixesOrSignal?: "on" | "off" | AbortSignal, signal?: AbortSignal): Promise<Float32Array[]>;
6
6
  export interface RerankWorkerCandidate {
7
7
  chunkId: string;
8
8
  content: string;
@@ -385,8 +385,10 @@ function getWorker() {
385
385
  IpcModelWorkerTransport.create(workerPath, nodeExecPath) ?? new StdioModelWorkerTransport(workerPath, nodeExecPath);
386
386
  return worker;
387
387
  }
388
- export async function embedInModelWorker(texts, prefix, signal) {
389
- const result = await getWorker().request({ type: "embed", texts, prefix }, signal);
388
+ export async function embedInModelWorker(texts, prefix, prefixesOrSignal, signal) {
389
+ const prefixes = prefixesOrSignal === "off" ? "off" : "on";
390
+ const requestSignal = typeof prefixesOrSignal === "object" ? prefixesOrSignal : signal;
391
+ const result = await getWorker().request({ type: "embed", texts, prefix, prefixes }, requestSignal);
390
392
  if (!Array.isArray(result))
391
393
  throw new Error("Invalid embedding worker response");
392
394
  return result.map((vector) => {
@@ -66,11 +66,17 @@ async function loadRerankerPipeline(config) {
66
66
  rerankerPipeline = { key, pipe: loaded };
67
67
  return loaded;
68
68
  }
69
+ function applyEmbeddingPrefix(text, prefix, prefixes) {
70
+ return prefixes === "on" ? `${prefix}: ${text}` : text;
71
+ }
69
72
  async function handleEmbed(request) {
70
73
  const pipe = await loadEmbeddingPipeline();
71
74
  const vectors = [];
72
75
  for (const text of request.texts) {
73
- const output = await pipe(`${request.prefix}: ${text}`, { pooling: "mean", normalize: true });
76
+ const output = await pipe(applyEmbeddingPrefix(text, request.prefix, request.prefixes), {
77
+ pooling: "mean",
78
+ normalize: true,
79
+ });
74
80
  vectors.push(Array.from(output.data));
75
81
  }
76
82
  return vectors;
@@ -110,7 +116,11 @@ function parseWorkerRequest(message) {
110
116
  if (!("prefix" in message) || (message.prefix !== "query" && message.prefix !== "passage")) {
111
117
  throw new Error("Invalid embed prefix");
112
118
  }
113
- return { id: message.id, type: "embed", texts: message.texts, prefix: message.prefix };
119
+ const prefixes = "prefixes" in message && (message.prefixes === "on" || message.prefixes === "off") ? message.prefixes : "on";
120
+ if ("prefixes" in message && message.prefixes !== "on" && message.prefixes !== "off") {
121
+ throw new Error("Invalid embed prefix strategy");
122
+ }
123
+ return { id: message.id, type: "embed", texts: message.texts, prefix: message.prefix, prefixes };
114
124
  }
115
125
  if (message.type === "rerank") {
116
126
  if (!("query" in message) || typeof message.query !== "string")
@@ -1,6 +1,6 @@
1
1
  export type SearchProfile = "auto" | "balanced" | "low_token" | "precision" | "recall" | "long_context" | "code" | "docs";
2
2
  export type RuntimeProfile = Exclude<SearchProfile, "auto">;
3
- type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
3
+ type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
4
4
  export interface SearchTuning {
5
5
  requestedProfile: SearchProfile;
6
6
  selectedProfile: RuntimeProfile;
@@ -143,7 +143,7 @@ function chooseRuntimeProfile(input) {
143
143
  }
144
144
  if (input.mode === "docs" || input.mode === "decision" || input.mode === "adaptive")
145
145
  return "long_context";
146
- if (input.mode === "semantic")
146
+ if (input.mode === "semantic" || input.mode === "semantic_hybrid")
147
147
  return "recall";
148
148
  if (looksErrorQuery(input.query) || looksExactOrCodeQuery(input.query))
149
149
  return "precision";
@@ -21,16 +21,19 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
21
21
  | Variable | Default | Purpose |
22
22
  |----------|---------|---------|
23
23
  | `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
24
- | `OPENAI_API_KEY` | unset | API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Some local OpenAI-compatible servers accept a placeholder value. |
24
+ | `PI_KNOWLEDGE_EMBEDDING_API_KEY` | unset | API key for OpenAI-compatible embedding APIs. Takes precedence over `OPENAI_API_KEY`; omit for trusted local endpoints that do not require auth. |
25
+ | `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
25
26
  | `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
26
27
  | `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
28
+ | `PI_KNOWLEDGE_EMBEDDING_PREFIXES` | `on` | Controls whether `query: ` and `passage: ` prefixes are prepended before embedding. Values: `on`, `off`. Keep `on` for the default E5 local model; use `off` only for models or self-hosted APIs that are evaluated without E5-style prefixes. |
27
29
  | `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
30
+ | `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
28
31
  | `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
29
32
  | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
30
33
  | `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
31
34
  | `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
32
35
 
33
- Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character settings or after upgrading older KBs with no signature metadata.
36
+ Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, non-default prefix strategy, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character/prefix settings or after upgrading older KBs with no signature metadata.
34
37
 
35
38
  ## Reranker
36
39
 
@@ -59,7 +62,7 @@ For Hugging Face Text Embeddings Inference rerankers, set `PI_KNOWLEDGE_RERANKER
59
62
 
60
63
  Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
61
64
 
62
- API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB.
65
+ API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, context-window errors, or server batch limits and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB. OpenAI-compatible embedding requests send `Authorization: Bearer <key>` only when `PI_KNOWLEDGE_EMBEDDING_API_KEY` or `OPENAI_API_KEY` is configured.
63
66
 
64
67
  ## Runtime Features and Diagnostics
65
68
 
@@ -82,7 +85,7 @@ API embedding failures intentionally surface by default. Silent fallback can hid
82
85
 
83
86
  Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool size, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
84
87
 
85
- `knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks, but it remains metadata-driven retrieval rather than LSP, caller/callee, or code graph analysis.
88
+ `knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` for pure vector search, or `semantic_hybrid` when weak lexical overlap should not block vector-backed weighted fusion. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks, but it remains metadata-driven context expansion rather than caller/callee analysis.
86
89
 
87
90
  `knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
88
91
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.10.1",
3
+ "version": "0.11.0",
4
4
  "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",