pi-knowledge 0.10.0 → 0.10.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,17 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.10.2] - 2026-09-19
4
+
5
+ ### Fixed
6
+ - Allowed OpenAI-compatible embedding endpoints to run without an API key and added `PI_KNOWLEDGE_EMBEDDING_API_KEY` as an embedding-specific bearer token override (#15).
7
+ - Added `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` so self-hosted embedding servers can use request batch sizes below the default 64 chunk limit (#16).
8
+
9
+ ## [0.10.1] - 2026-09-10
10
+
11
+ ### Fixed
12
+ - Repaired `knowledge_update` recovery for knowledge bases whose SQLite chunks are ahead of a short or truncated vector file; update now rebuilds or re-embeds missing vectors instead of leaving the KB in `error` (#13).
13
+ - Aligned the published Tree-sitter runtime and grammar dependency graph so clean npm installs no longer emit peer override warnings for `tree-sitter@0.25.1` incompatibilities (#14).
14
+
3
15
  ## [0.10.0] - 2026-08-27
4
16
 
5
17
  ### Added
package/README.md CHANGED
@@ -83,6 +83,11 @@ pi install npm:pi-knowledge
83
83
  # Install for OMP
84
84
  omp install npm:pi-knowledge
85
85
 
86
+ # On minimal Ubuntu/Debian servers, install native build tools first because
87
+ # OMP's Bun-based npm install compiles better-sqlite3 from source.
88
+ sudo apt-get update && sudo apt-get install -y build-essential python3
89
+ omp install npm:pi-knowledge
90
+
86
91
  # Or from source
87
92
  pi install ./pi-knowledge
88
93
  omp install ./pi-knowledge
@@ -161,13 +166,15 @@ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BA
161
166
  ```bash
162
167
  export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
163
168
  export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
164
- export OPENAI_API_KEY=local-placeholder
169
+ # Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
165
170
  ```
166
171
 
167
172
  API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
168
173
 
169
174
  API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
170
175
 
176
+ For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
177
+
171
178
  Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
172
179
 
173
180
  ## Configuration
@@ -178,7 +185,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
178
185
  |------|-----------------------|
179
186
  | Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
180
187
  | Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
181
- | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
188
+ | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
182
189
  | Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
183
190
  | Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
184
191
  | Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
@@ -127,15 +127,16 @@ export async function prepareForShutdown() {
127
127
  async function embedViaAPI(texts, prefix, config, signal) {
128
128
  const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
129
129
  const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
130
- const apiKey = process.env.OPENAI_API_KEY;
131
- if (!apiKey)
132
- throw new Error("OPENAI_API_KEY required for openai embedding");
130
+ const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
133
131
  if (!config.baseUrl)
134
132
  throw new Error("OpenAI embedding base URL is not configured");
135
133
  const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
134
+ const headers = { "Content-Type": "application/json" };
135
+ if (apiKey)
136
+ headers.Authorization = `Bearer ${apiKey}`;
136
137
  const res = await fetch(endpoint, {
137
138
  method: "POST",
138
- headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
139
+ headers,
139
140
  body: JSON.stringify({ input: safeTexts, model: config.model }),
140
141
  signal,
141
142
  });
@@ -15,14 +15,29 @@ import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js"
15
15
  import { searchVectorFile } from "./search/vector.js";
16
16
  import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
17
17
  export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
18
- const INDEX_EMBED_BATCH_SIZE = 64;
18
+ const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
19
19
  const VECTOR_REDUNDANCY_WEIGHT = 0.35;
20
+ function resolveIndexEmbedBatchSize(env = process.env) {
21
+ const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
22
+ if (!raw)
23
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
24
+ const value = Number(raw);
25
+ if (!Number.isFinite(value))
26
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
27
+ const batchSize = Math.trunc(value);
28
+ return batchSize > 0 ? batchSize : DEFAULT_INDEX_EMBED_BATCH_SIZE;
29
+ }
20
30
  function isCancellationError(error) {
21
31
  return error instanceof Error && error.message === "Cancelled";
22
32
  }
23
33
  function tempVectorPath(vectorPath) {
24
34
  return `${vectorPath}.${Date.now()}.${Math.random().toString(16).slice(2)}.tmp`;
25
35
  }
36
+ function assertEmbeddingBatchSize(vectors, expected, operation) {
37
+ if (vectors.length !== expected) {
38
+ throw new Error(`Embedding provider returned ${vectors.length} vectors for ${expected} ${operation} chunks`);
39
+ }
40
+ }
26
41
  function retrievalModeFor(mode) {
27
42
  if (mode === "auto")
28
43
  return "hybrid";
@@ -699,6 +714,7 @@ export class KnowledgeEngine {
699
714
  throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
700
715
  }
701
716
  const embeddingConfig = resolveEmbeddingConfig();
717
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
702
718
  const kb = createKB(db, {
703
719
  name,
704
720
  source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
@@ -754,11 +770,12 @@ export class KnowledgeEngine {
754
770
  return;
755
771
  if (signal?.aborted)
756
772
  throw new Error("Cancelled");
757
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
773
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
758
774
  reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
759
775
  const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
760
776
  if (signal?.aborted)
761
777
  throw new Error("Cancelled");
778
+ assertEmbeddingBatchSize(vectors, batch.length, "add");
762
779
  persistEmbeddingMetadata(db, kb.id, embeddingConfig, vectors);
763
780
  insertChunks(db, kb.id, batch);
764
781
  writer.append(vectors);
@@ -768,7 +785,7 @@ export class KnowledgeEngine {
768
785
  };
769
786
  const addChunks = async (chunks, processedFiles, totalFiles) => {
770
787
  pendingChunks.push(...chunks);
771
- while (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE) {
788
+ while (pendingChunks.length >= embeddingBatchSize) {
772
789
  await flushPending(processedFiles, totalFiles);
773
790
  }
774
791
  };
@@ -921,9 +938,10 @@ export class KnowledgeEngine {
921
938
  startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
922
939
  const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
923
940
  const embeddingConfig = resolveEmbeddingConfig();
941
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
924
942
  const embeddingModel = embeddingConfigLabel(embeddingConfig);
925
943
  const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
926
- const canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
944
+ let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
927
945
  let replacementVectorPath;
928
946
  let addedVectorPath;
929
947
  let addedVectorWriter;
@@ -943,6 +961,24 @@ export class KnowledgeEngine {
943
961
  existingHashes.set(chunk.content_hash, entries);
944
962
  existingIndex++;
945
963
  }
964
+ if (canReuseExistingVectors) {
965
+ const oldVectorReader = openVectorReader(vectorPath);
966
+ try {
967
+ if (!oldVectorReader ||
968
+ oldVectorReader.count !== existingIndex ||
969
+ oldVectorReader.dim !== kb.embedding_dimension) {
970
+ canReuseExistingVectors = false;
971
+ const vectorCount = oldVectorReader?.count ?? 0;
972
+ const vectorDimension = oldVectorReader?.dim ?? 0;
973
+ const message = `Stored vector file for "${kb.name}" is incomplete or incompatible (${vectorCount} vectors/${vectorDimension}d for ${existingIndex} chunks/${kb.embedding_dimension}d); rebuilding all vectors`;
974
+ updateIndexingJob(this.db, kb.id, { phase: "embedding", message });
975
+ onProgress?.(message);
976
+ }
977
+ }
978
+ finally {
979
+ oldVectorReader?.close();
980
+ }
981
+ }
946
982
  const reusableHashes = canReuseExistingVectors
947
983
  ? existingHashes
948
984
  : new Map();
@@ -967,7 +1003,7 @@ export class KnowledgeEngine {
967
1003
  return;
968
1004
  if (signal?.aborted)
969
1005
  throw new Error("Cancelled");
970
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1006
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
971
1007
  const elapsed = Date.now() - startedAt;
972
1008
  const message = `Embedding update batch: ${addedCount} new chunks stored, ${scannedFiles} files scanned, elapsed ${formatDuration(elapsed)}`;
973
1009
  updateIndexingJob(this.db, kb.id, {
@@ -983,6 +1019,7 @@ export class KnowledgeEngine {
983
1019
  const newVectors = await embedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
984
1020
  if (signal?.aborted)
985
1021
  throw new Error("Cancelled");
1022
+ assertEmbeddingBatchSize(newVectors, batch.length, "update");
986
1023
  addedVectorWriter.append(newVectors);
987
1024
  for (let i = 0; i < batch.length; i++) {
988
1025
  const indexes = newVectorIndexByHash.get(batch[i].content_hash) ?? [];
@@ -1020,7 +1057,7 @@ export class KnowledgeEngine {
1020
1057
  continue;
1021
1058
  }
1022
1059
  pendingChunks.push(chunk);
1023
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1060
+ if (pendingChunks.length >= embeddingBatchSize)
1024
1061
  await flushPending();
1025
1062
  }
1026
1063
  };
@@ -1122,6 +1159,29 @@ export class KnowledgeEngine {
1122
1159
  const indexes = indexesByHash.get(hash);
1123
1160
  return indexes?.shift();
1124
1161
  };
1162
+ const repairMissingVector = async (chunk) => {
1163
+ const message = `Re-embedding chunk with missing stored vector: ${chunk.id}`;
1164
+ if (!this.db)
1165
+ throw new Error("Engine not initialized");
1166
+ updateIndexingJob(this.db, kb.id, {
1167
+ phase: "embedding",
1168
+ message,
1169
+ processed_files: scannedFiles,
1170
+ processed_chunks: finalChunkCount,
1171
+ added_chunks: addedCount,
1172
+ removed_chunks: idsToRemove.length,
1173
+ unchanged_chunks: unchanged,
1174
+ });
1175
+ onProgress?.(message);
1176
+ const repairedVectors = await embedDocuments([buildChunkEmbeddingText(chunk)], signal);
1177
+ if (signal?.aborted)
1178
+ throw new Error("Cancelled");
1179
+ assertEmbeddingBatchSize(repairedVectors, 1, "repair");
1180
+ const [repairedVector] = repairedVectors;
1181
+ if (!repairedVector)
1182
+ throw new Error(`Embedding provider returned no repaired vector for chunk: ${chunk.id}`);
1183
+ return repairedVector;
1184
+ };
1125
1185
  try {
1126
1186
  for (const chunk of iterateChunksByKB(this.db, kb.id)) {
1127
1187
  if (signal?.aborted)
@@ -1130,13 +1190,13 @@ export class KnowledgeEngine {
1130
1190
  continue;
1131
1191
  const oldVectorIndex = takeVectorIndex(oldVectorIndexByHash, chunk.content_hash);
1132
1192
  const newVectorIndex = takeVectorIndex(newVectorIndexByHash, chunk.content_hash);
1133
- const vector = oldVectorReader && oldVectorIndex !== undefined
1193
+ let vector = oldVectorReader && oldVectorIndex !== undefined
1134
1194
  ? oldVectorReader.read(oldVectorIndex)
1135
1195
  : newVectorReader && newVectorIndex !== undefined
1136
1196
  ? newVectorReader.read(newVectorIndex)
1137
1197
  : undefined;
1138
1198
  if (!vector)
1139
- throw new Error(`Missing vector while rebuilding knowledge base: ${chunk.id}`);
1199
+ vector = await repairMissingVector(chunk);
1140
1200
  finalEmbeddingDimension ??= vector.length;
1141
1201
  vectorWriter.append([vector]);
1142
1202
  finalChunkCount++;
@@ -1741,6 +1801,7 @@ export class KnowledgeEngine {
1741
1801
  const { createWriteStream } = await import("node:fs");
1742
1802
  const tempOutputPath = tempVectorPath(outputPath);
1743
1803
  const stream = createWriteStream(tempOutputPath, { encoding: "utf-8" });
1804
+ const progressBatchSize = resolveIndexEmbedBatchSize();
1744
1805
  let count = 0;
1745
1806
  const header = JSON.stringify({
1746
1807
  name: kb.name,
@@ -1761,7 +1822,7 @@ export class KnowledgeEngine {
1761
1822
  metadata_json: chunk.metadata_json,
1762
1823
  }));
1763
1824
  count++;
1764
- if (count % INDEX_EMBED_BATCH_SIZE === 0)
1825
+ if (count % progressBatchSize === 0)
1765
1826
  onProgress?.(`Exported ${count}/${kb.chunk_count} chunks...`);
1766
1827
  }
1767
1828
  await finishWriteStream(stream);
@@ -1790,6 +1851,7 @@ export class KnowledgeEngine {
1790
1851
  const stream = createReadStream(inputPath, { encoding: "utf-8" });
1791
1852
  const lines = createInterface({ input: stream, crlfDelay: Infinity });
1792
1853
  const embeddingConfig = resolveEmbeddingConfig();
1854
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
1793
1855
  let header;
1794
1856
  let kb;
1795
1857
  let vectorWriter;
@@ -1802,7 +1864,7 @@ export class KnowledgeEngine {
1802
1864
  if (!this.db || !kb || !vectorWriter || pendingChunks.length === 0)
1803
1865
  return;
1804
1866
  throwIfAborted(signal);
1805
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1867
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
1806
1868
  const symbols = pendingSymbols.splice(0);
1807
1869
  const total = header?.chunk_count;
1808
1870
  const message = `Embedding import batch: ${inserted}/${total ?? "unknown"} chunks stored`;
@@ -1816,6 +1878,7 @@ export class KnowledgeEngine {
1816
1878
  onProgress?.(message);
1817
1879
  const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
1818
1880
  throwIfAborted(signal);
1881
+ assertEmbeddingBatchSize(vectors, batch.length, "import");
1819
1882
  persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
1820
1883
  insertChunks(this.db, kb.id, batch);
1821
1884
  insertSymbols(this.db, kb.id, symbols);
@@ -1870,7 +1933,7 @@ export class KnowledgeEngine {
1870
1933
  chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
1871
1934
  pendingChunks.push(chunk);
1872
1935
  pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
1873
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1936
+ if (pendingChunks.length >= embeddingBatchSize)
1874
1937
  await flushPending();
1875
1938
  }
1876
1939
  if (!header)
@@ -21,10 +21,12 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
21
21
  | Variable | Default | Purpose |
22
22
  |----------|---------|---------|
23
23
  | `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
24
- | `OPENAI_API_KEY` | unset | API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Some local OpenAI-compatible servers accept a placeholder value. |
24
+ | `PI_KNOWLEDGE_EMBEDDING_API_KEY` | unset | API key for OpenAI-compatible embedding APIs. Takes precedence over `OPENAI_API_KEY`; omit for trusted local endpoints that do not require auth. |
25
+ | `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
25
26
  | `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
26
27
  | `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
27
28
  | `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
29
+ | `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
28
30
  | `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
29
31
  | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
30
32
  | `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
@@ -59,7 +61,7 @@ For Hugging Face Text Embeddings Inference rerankers, set `PI_KNOWLEDGE_RERANKER
59
61
 
60
62
  Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
61
63
 
62
- API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB.
64
+ API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, context-window errors, or server batch limits and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB. OpenAI-compatible embedding requests send `Authorization: Bearer <key>` only when `PI_KNOWLEDGE_EMBEDDING_API_KEY` or `OPENAI_API_KEY` is configured.
63
65
 
64
66
  ## Runtime Features and Diagnostics
65
67
 
@@ -103,6 +105,15 @@ When `PI_KNOWLEDGE_E2E_PDF` or `PI_KNOWLEDGE_E2E_DOCX` is unset, `npm run test:e
103
105
 
104
106
  `pi-knowledge` supports direct Pi extension loading through `pi -e ./extension.js` and OMP-compatible loading through the same packaged entry shim. The root `extension.js` and `index.ts` stay startup-light so install-time validation can inspect the extension without resolving native runtime dependencies such as `better-sqlite3` or `onnxruntime-node`.
105
107
 
108
+ Minimal Ubuntu/Debian servers need native build tools before `omp install npm:pi-knowledge`,
109
+ because OMP's Bun-based npm install compiles `better-sqlite3` from source when Bun cannot use
110
+ the package prebuild:
111
+
112
+ ```bash
113
+ sudo apt-get update && sudo apt-get install -y build-essential python3
114
+ omp install npm:pi-knowledge
115
+ ```
116
+
106
117
  Compatibility guarantees:
107
118
 
108
119
  - The packaged `extension.js` loads built `dist/index.js` when present and falls back to source `index.ts` for local development.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.10.0",
3
+ "version": "0.10.2",
4
4
  "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
@@ -54,22 +54,19 @@
54
54
  "better-sqlite3": "12.11.1",
55
55
  "ignore": "7.0.5",
56
56
  "mammoth": "1.12.0",
57
- "tree-sitter": "0.25.1",
58
- "tree-sitter-bash": "0.25.1",
59
- "tree-sitter-c": "0.24.1",
60
- "tree-sitter-cpp": "0.23.4",
61
- "tree-sitter-go": "0.25.0",
57
+ "tree-sitter": "0.21.1",
58
+ "tree-sitter-bash": "0.21.0",
59
+ "tree-sitter-c": "0.21.4",
60
+ "tree-sitter-cpp": "0.23.2",
61
+ "tree-sitter-go": "0.21.2",
62
62
  "tree-sitter-java": "0.23.5",
63
- "tree-sitter-javascript": "0.25.0",
64
- "tree-sitter-python": "0.25.0",
63
+ "tree-sitter-javascript": "0.23.1",
64
+ "tree-sitter-python": "0.21.0",
65
65
  "tree-sitter-qmljs": "0.3.1",
66
- "tree-sitter-rust": "0.24.0",
66
+ "tree-sitter-rust": "0.21.0",
67
67
  "tree-sitter-typescript": "0.23.2",
68
68
  "unpdf": "1.6.2"
69
69
  },
70
- "overrides": {
71
- "tree-sitter": "0.25.1"
72
- },
73
70
  "devDependencies": {
74
71
  "@biomejs/biome": "2.3.5",
75
72
  "@types/better-sqlite3": "7.6.13",