pi-knowledge 0.10.0 → 0.10.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +9 -2
- package/dist/src/embedding/provider.js +5 -4
- package/dist/src/engine.js +74 -11
- package/docs/configuration.md +13 -2
- package/package.json +9 -12
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.10.2] - 2026-09-19
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Allowed OpenAI-compatible embedding endpoints to run without an API key and added `PI_KNOWLEDGE_EMBEDDING_API_KEY` as an embedding-specific bearer token override (#15).
|
|
7
|
+
- Added `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` so self-hosted embedding servers can use request batch sizes below the default 64 chunk limit (#16).
|
|
8
|
+
|
|
9
|
+
## [0.10.1] - 2026-09-10
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
- Repaired `knowledge_update` recovery for knowledge bases whose SQLite chunks are ahead of a short or truncated vector file; update now rebuilds or re-embeds missing vectors instead of leaving the KB in `error` (#13).
|
|
13
|
+
- Aligned the published Tree-sitter runtime and grammar dependency graph so clean npm installs no longer emit peer override warnings for `tree-sitter@0.25.1` incompatibilities (#14).
|
|
14
|
+
|
|
3
15
|
## [0.10.0] - 2026-08-27
|
|
4
16
|
|
|
5
17
|
### Added
|
package/README.md
CHANGED
|
@@ -83,6 +83,11 @@ pi install npm:pi-knowledge
|
|
|
83
83
|
# Install for OMP
|
|
84
84
|
omp install npm:pi-knowledge
|
|
85
85
|
|
|
86
|
+
# On minimal Ubuntu/Debian servers, install native build tools first because
|
|
87
|
+
# OMP's Bun-based npm install compiles better-sqlite3 from source.
|
|
88
|
+
sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
89
|
+
omp install npm:pi-knowledge
|
|
90
|
+
|
|
86
91
|
# Or from source
|
|
87
92
|
pi install ./pi-knowledge
|
|
88
93
|
omp install ./pi-knowledge
|
|
@@ -161,13 +166,15 @@ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BA
|
|
|
161
166
|
```bash
|
|
162
167
|
export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
|
|
163
168
|
export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
164
|
-
|
|
169
|
+
# Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
|
|
165
170
|
```
|
|
166
171
|
|
|
167
172
|
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
|
|
168
173
|
|
|
169
174
|
API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
|
|
170
175
|
|
|
176
|
+
For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
|
|
177
|
+
|
|
171
178
|
Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
|
|
172
179
|
|
|
173
180
|
## Configuration
|
|
@@ -178,7 +185,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
178
185
|
|------|-----------------------|
|
|
179
186
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
180
187
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
181
|
-
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
188
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
182
189
|
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
183
190
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
184
191
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
@@ -127,15 +127,16 @@ export async function prepareForShutdown() {
|
|
|
127
127
|
async function embedViaAPI(texts, prefix, config, signal) {
|
|
128
128
|
const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
|
|
129
129
|
const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
|
|
130
|
-
const apiKey = process.env.OPENAI_API_KEY;
|
|
131
|
-
if (!apiKey)
|
|
132
|
-
throw new Error("OPENAI_API_KEY required for openai embedding");
|
|
130
|
+
const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
|
|
133
131
|
if (!config.baseUrl)
|
|
134
132
|
throw new Error("OpenAI embedding base URL is not configured");
|
|
135
133
|
const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
|
|
134
|
+
const headers = { "Content-Type": "application/json" };
|
|
135
|
+
if (apiKey)
|
|
136
|
+
headers.Authorization = `Bearer ${apiKey}`;
|
|
136
137
|
const res = await fetch(endpoint, {
|
|
137
138
|
method: "POST",
|
|
138
|
-
headers
|
|
139
|
+
headers,
|
|
139
140
|
body: JSON.stringify({ input: safeTexts, model: config.model }),
|
|
140
141
|
signal,
|
|
141
142
|
});
|
package/dist/src/engine.js
CHANGED
|
@@ -15,14 +15,29 @@ import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js"
|
|
|
15
15
|
import { searchVectorFile } from "./search/vector.js";
|
|
16
16
|
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
|
|
17
17
|
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
|
|
18
|
-
const
|
|
18
|
+
const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
|
|
19
19
|
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
20
|
+
function resolveIndexEmbedBatchSize(env = process.env) {
|
|
21
|
+
const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
|
|
22
|
+
if (!raw)
|
|
23
|
+
return DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
24
|
+
const value = Number(raw);
|
|
25
|
+
if (!Number.isFinite(value))
|
|
26
|
+
return DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
27
|
+
const batchSize = Math.trunc(value);
|
|
28
|
+
return batchSize > 0 ? batchSize : DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
29
|
+
}
|
|
20
30
|
function isCancellationError(error) {
|
|
21
31
|
return error instanceof Error && error.message === "Cancelled";
|
|
22
32
|
}
|
|
23
33
|
function tempVectorPath(vectorPath) {
|
|
24
34
|
return `${vectorPath}.${Date.now()}.${Math.random().toString(16).slice(2)}.tmp`;
|
|
25
35
|
}
|
|
36
|
+
function assertEmbeddingBatchSize(vectors, expected, operation) {
|
|
37
|
+
if (vectors.length !== expected) {
|
|
38
|
+
throw new Error(`Embedding provider returned ${vectors.length} vectors for ${expected} ${operation} chunks`);
|
|
39
|
+
}
|
|
40
|
+
}
|
|
26
41
|
function retrievalModeFor(mode) {
|
|
27
42
|
if (mode === "auto")
|
|
28
43
|
return "hybrid";
|
|
@@ -699,6 +714,7 @@ export class KnowledgeEngine {
|
|
|
699
714
|
throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
|
|
700
715
|
}
|
|
701
716
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
717
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
702
718
|
const kb = createKB(db, {
|
|
703
719
|
name,
|
|
704
720
|
source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
|
|
@@ -754,11 +770,12 @@ export class KnowledgeEngine {
|
|
|
754
770
|
return;
|
|
755
771
|
if (signal?.aborted)
|
|
756
772
|
throw new Error("Cancelled");
|
|
757
|
-
const batch = pendingChunks.splice(0,
|
|
773
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
758
774
|
reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
|
|
759
775
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
760
776
|
if (signal?.aborted)
|
|
761
777
|
throw new Error("Cancelled");
|
|
778
|
+
assertEmbeddingBatchSize(vectors, batch.length, "add");
|
|
762
779
|
persistEmbeddingMetadata(db, kb.id, embeddingConfig, vectors);
|
|
763
780
|
insertChunks(db, kb.id, batch);
|
|
764
781
|
writer.append(vectors);
|
|
@@ -768,7 +785,7 @@ export class KnowledgeEngine {
|
|
|
768
785
|
};
|
|
769
786
|
const addChunks = async (chunks, processedFiles, totalFiles) => {
|
|
770
787
|
pendingChunks.push(...chunks);
|
|
771
|
-
while (pendingChunks.length >=
|
|
788
|
+
while (pendingChunks.length >= embeddingBatchSize) {
|
|
772
789
|
await flushPending(processedFiles, totalFiles);
|
|
773
790
|
}
|
|
774
791
|
};
|
|
@@ -921,9 +938,10 @@ export class KnowledgeEngine {
|
|
|
921
938
|
startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
|
|
922
939
|
const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
|
|
923
940
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
941
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
924
942
|
const embeddingModel = embeddingConfigLabel(embeddingConfig);
|
|
925
943
|
const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
|
|
926
|
-
|
|
944
|
+
let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
|
|
927
945
|
let replacementVectorPath;
|
|
928
946
|
let addedVectorPath;
|
|
929
947
|
let addedVectorWriter;
|
|
@@ -943,6 +961,24 @@ export class KnowledgeEngine {
|
|
|
943
961
|
existingHashes.set(chunk.content_hash, entries);
|
|
944
962
|
existingIndex++;
|
|
945
963
|
}
|
|
964
|
+
if (canReuseExistingVectors) {
|
|
965
|
+
const oldVectorReader = openVectorReader(vectorPath);
|
|
966
|
+
try {
|
|
967
|
+
if (!oldVectorReader ||
|
|
968
|
+
oldVectorReader.count !== existingIndex ||
|
|
969
|
+
oldVectorReader.dim !== kb.embedding_dimension) {
|
|
970
|
+
canReuseExistingVectors = false;
|
|
971
|
+
const vectorCount = oldVectorReader?.count ?? 0;
|
|
972
|
+
const vectorDimension = oldVectorReader?.dim ?? 0;
|
|
973
|
+
const message = `Stored vector file for "${kb.name}" is incomplete or incompatible (${vectorCount} vectors/${vectorDimension}d for ${existingIndex} chunks/${kb.embedding_dimension}d); rebuilding all vectors`;
|
|
974
|
+
updateIndexingJob(this.db, kb.id, { phase: "embedding", message });
|
|
975
|
+
onProgress?.(message);
|
|
976
|
+
}
|
|
977
|
+
}
|
|
978
|
+
finally {
|
|
979
|
+
oldVectorReader?.close();
|
|
980
|
+
}
|
|
981
|
+
}
|
|
946
982
|
const reusableHashes = canReuseExistingVectors
|
|
947
983
|
? existingHashes
|
|
948
984
|
: new Map();
|
|
@@ -967,7 +1003,7 @@ export class KnowledgeEngine {
|
|
|
967
1003
|
return;
|
|
968
1004
|
if (signal?.aborted)
|
|
969
1005
|
throw new Error("Cancelled");
|
|
970
|
-
const batch = pendingChunks.splice(0,
|
|
1006
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
971
1007
|
const elapsed = Date.now() - startedAt;
|
|
972
1008
|
const message = `Embedding update batch: ${addedCount} new chunks stored, ${scannedFiles} files scanned, elapsed ${formatDuration(elapsed)}`;
|
|
973
1009
|
updateIndexingJob(this.db, kb.id, {
|
|
@@ -983,6 +1019,7 @@ export class KnowledgeEngine {
|
|
|
983
1019
|
const newVectors = await embedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
|
|
984
1020
|
if (signal?.aborted)
|
|
985
1021
|
throw new Error("Cancelled");
|
|
1022
|
+
assertEmbeddingBatchSize(newVectors, batch.length, "update");
|
|
986
1023
|
addedVectorWriter.append(newVectors);
|
|
987
1024
|
for (let i = 0; i < batch.length; i++) {
|
|
988
1025
|
const indexes = newVectorIndexByHash.get(batch[i].content_hash) ?? [];
|
|
@@ -1020,7 +1057,7 @@ export class KnowledgeEngine {
|
|
|
1020
1057
|
continue;
|
|
1021
1058
|
}
|
|
1022
1059
|
pendingChunks.push(chunk);
|
|
1023
|
-
if (pendingChunks.length >=
|
|
1060
|
+
if (pendingChunks.length >= embeddingBatchSize)
|
|
1024
1061
|
await flushPending();
|
|
1025
1062
|
}
|
|
1026
1063
|
};
|
|
@@ -1122,6 +1159,29 @@ export class KnowledgeEngine {
|
|
|
1122
1159
|
const indexes = indexesByHash.get(hash);
|
|
1123
1160
|
return indexes?.shift();
|
|
1124
1161
|
};
|
|
1162
|
+
const repairMissingVector = async (chunk) => {
|
|
1163
|
+
const message = `Re-embedding chunk with missing stored vector: ${chunk.id}`;
|
|
1164
|
+
if (!this.db)
|
|
1165
|
+
throw new Error("Engine not initialized");
|
|
1166
|
+
updateIndexingJob(this.db, kb.id, {
|
|
1167
|
+
phase: "embedding",
|
|
1168
|
+
message,
|
|
1169
|
+
processed_files: scannedFiles,
|
|
1170
|
+
processed_chunks: finalChunkCount,
|
|
1171
|
+
added_chunks: addedCount,
|
|
1172
|
+
removed_chunks: idsToRemove.length,
|
|
1173
|
+
unchanged_chunks: unchanged,
|
|
1174
|
+
});
|
|
1175
|
+
onProgress?.(message);
|
|
1176
|
+
const repairedVectors = await embedDocuments([buildChunkEmbeddingText(chunk)], signal);
|
|
1177
|
+
if (signal?.aborted)
|
|
1178
|
+
throw new Error("Cancelled");
|
|
1179
|
+
assertEmbeddingBatchSize(repairedVectors, 1, "repair");
|
|
1180
|
+
const [repairedVector] = repairedVectors;
|
|
1181
|
+
if (!repairedVector)
|
|
1182
|
+
throw new Error(`Embedding provider returned no repaired vector for chunk: ${chunk.id}`);
|
|
1183
|
+
return repairedVector;
|
|
1184
|
+
};
|
|
1125
1185
|
try {
|
|
1126
1186
|
for (const chunk of iterateChunksByKB(this.db, kb.id)) {
|
|
1127
1187
|
if (signal?.aborted)
|
|
@@ -1130,13 +1190,13 @@ export class KnowledgeEngine {
|
|
|
1130
1190
|
continue;
|
|
1131
1191
|
const oldVectorIndex = takeVectorIndex(oldVectorIndexByHash, chunk.content_hash);
|
|
1132
1192
|
const newVectorIndex = takeVectorIndex(newVectorIndexByHash, chunk.content_hash);
|
|
1133
|
-
|
|
1193
|
+
let vector = oldVectorReader && oldVectorIndex !== undefined
|
|
1134
1194
|
? oldVectorReader.read(oldVectorIndex)
|
|
1135
1195
|
: newVectorReader && newVectorIndex !== undefined
|
|
1136
1196
|
? newVectorReader.read(newVectorIndex)
|
|
1137
1197
|
: undefined;
|
|
1138
1198
|
if (!vector)
|
|
1139
|
-
|
|
1199
|
+
vector = await repairMissingVector(chunk);
|
|
1140
1200
|
finalEmbeddingDimension ??= vector.length;
|
|
1141
1201
|
vectorWriter.append([vector]);
|
|
1142
1202
|
finalChunkCount++;
|
|
@@ -1741,6 +1801,7 @@ export class KnowledgeEngine {
|
|
|
1741
1801
|
const { createWriteStream } = await import("node:fs");
|
|
1742
1802
|
const tempOutputPath = tempVectorPath(outputPath);
|
|
1743
1803
|
const stream = createWriteStream(tempOutputPath, { encoding: "utf-8" });
|
|
1804
|
+
const progressBatchSize = resolveIndexEmbedBatchSize();
|
|
1744
1805
|
let count = 0;
|
|
1745
1806
|
const header = JSON.stringify({
|
|
1746
1807
|
name: kb.name,
|
|
@@ -1761,7 +1822,7 @@ export class KnowledgeEngine {
|
|
|
1761
1822
|
metadata_json: chunk.metadata_json,
|
|
1762
1823
|
}));
|
|
1763
1824
|
count++;
|
|
1764
|
-
if (count %
|
|
1825
|
+
if (count % progressBatchSize === 0)
|
|
1765
1826
|
onProgress?.(`Exported ${count}/${kb.chunk_count} chunks...`);
|
|
1766
1827
|
}
|
|
1767
1828
|
await finishWriteStream(stream);
|
|
@@ -1790,6 +1851,7 @@ export class KnowledgeEngine {
|
|
|
1790
1851
|
const stream = createReadStream(inputPath, { encoding: "utf-8" });
|
|
1791
1852
|
const lines = createInterface({ input: stream, crlfDelay: Infinity });
|
|
1792
1853
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
1854
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
1793
1855
|
let header;
|
|
1794
1856
|
let kb;
|
|
1795
1857
|
let vectorWriter;
|
|
@@ -1802,7 +1864,7 @@ export class KnowledgeEngine {
|
|
|
1802
1864
|
if (!this.db || !kb || !vectorWriter || pendingChunks.length === 0)
|
|
1803
1865
|
return;
|
|
1804
1866
|
throwIfAborted(signal);
|
|
1805
|
-
const batch = pendingChunks.splice(0,
|
|
1867
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
1806
1868
|
const symbols = pendingSymbols.splice(0);
|
|
1807
1869
|
const total = header?.chunk_count;
|
|
1808
1870
|
const message = `Embedding import batch: ${inserted}/${total ?? "unknown"} chunks stored`;
|
|
@@ -1816,6 +1878,7 @@ export class KnowledgeEngine {
|
|
|
1816
1878
|
onProgress?.(message);
|
|
1817
1879
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
1818
1880
|
throwIfAborted(signal);
|
|
1881
|
+
assertEmbeddingBatchSize(vectors, batch.length, "import");
|
|
1819
1882
|
persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
|
|
1820
1883
|
insertChunks(this.db, kb.id, batch);
|
|
1821
1884
|
insertSymbols(this.db, kb.id, symbols);
|
|
@@ -1870,7 +1933,7 @@ export class KnowledgeEngine {
|
|
|
1870
1933
|
chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
|
|
1871
1934
|
pendingChunks.push(chunk);
|
|
1872
1935
|
pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
|
|
1873
|
-
if (pendingChunks.length >=
|
|
1936
|
+
if (pendingChunks.length >= embeddingBatchSize)
|
|
1874
1937
|
await flushPending();
|
|
1875
1938
|
}
|
|
1876
1939
|
if (!header)
|
package/docs/configuration.md
CHANGED
|
@@ -21,10 +21,12 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
21
21
|
| Variable | Default | Purpose |
|
|
22
22
|
|----------|---------|---------|
|
|
23
23
|
| `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
|
|
24
|
-
| `
|
|
24
|
+
| `PI_KNOWLEDGE_EMBEDDING_API_KEY` | unset | API key for OpenAI-compatible embedding APIs. Takes precedence over `OPENAI_API_KEY`; omit for trusted local endpoints that do not require auth. |
|
|
25
|
+
| `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
|
|
25
26
|
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
26
27
|
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
27
28
|
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
29
|
+
| `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
|
|
28
30
|
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
|
|
29
31
|
| `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
|
|
30
32
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
@@ -59,7 +61,7 @@ For Hugging Face Text Embeddings Inference rerankers, set `PI_KNOWLEDGE_RERANKER
|
|
|
59
61
|
|
|
60
62
|
Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
|
|
61
63
|
|
|
62
|
-
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names,
|
|
64
|
+
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, context-window errors, or server batch limits and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB. OpenAI-compatible embedding requests send `Authorization: Bearer <key>` only when `PI_KNOWLEDGE_EMBEDDING_API_KEY` or `OPENAI_API_KEY` is configured.
|
|
63
65
|
|
|
64
66
|
## Runtime Features and Diagnostics
|
|
65
67
|
|
|
@@ -103,6 +105,15 @@ When `PI_KNOWLEDGE_E2E_PDF` or `PI_KNOWLEDGE_E2E_DOCX` is unset, `npm run test:e
|
|
|
103
105
|
|
|
104
106
|
`pi-knowledge` supports direct Pi extension loading through `pi -e ./extension.js` and OMP-compatible loading through the same packaged entry shim. The root `extension.js` and `index.ts` stay startup-light so install-time validation can inspect the extension without resolving native runtime dependencies such as `better-sqlite3` or `onnxruntime-node`.
|
|
105
107
|
|
|
108
|
+
Minimal Ubuntu/Debian servers need native build tools before `omp install npm:pi-knowledge`,
|
|
109
|
+
because OMP's Bun-based npm install compiles `better-sqlite3` from source when Bun cannot use
|
|
110
|
+
the package prebuild:
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
114
|
+
omp install npm:pi-knowledge
|
|
115
|
+
```
|
|
116
|
+
|
|
106
117
|
Compatibility guarantees:
|
|
107
118
|
|
|
108
119
|
- The packaged `extension.js` loads built `dist/index.js` when present and falls back to source `index.ts` for local development.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.2",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -54,22 +54,19 @@
|
|
|
54
54
|
"better-sqlite3": "12.11.1",
|
|
55
55
|
"ignore": "7.0.5",
|
|
56
56
|
"mammoth": "1.12.0",
|
|
57
|
-
"tree-sitter": "0.
|
|
58
|
-
"tree-sitter-bash": "0.
|
|
59
|
-
"tree-sitter-c": "0.
|
|
60
|
-
"tree-sitter-cpp": "0.23.
|
|
61
|
-
"tree-sitter-go": "0.
|
|
57
|
+
"tree-sitter": "0.21.1",
|
|
58
|
+
"tree-sitter-bash": "0.21.0",
|
|
59
|
+
"tree-sitter-c": "0.21.4",
|
|
60
|
+
"tree-sitter-cpp": "0.23.2",
|
|
61
|
+
"tree-sitter-go": "0.21.2",
|
|
62
62
|
"tree-sitter-java": "0.23.5",
|
|
63
|
-
"tree-sitter-javascript": "0.
|
|
64
|
-
"tree-sitter-python": "0.
|
|
63
|
+
"tree-sitter-javascript": "0.23.1",
|
|
64
|
+
"tree-sitter-python": "0.21.0",
|
|
65
65
|
"tree-sitter-qmljs": "0.3.1",
|
|
66
|
-
"tree-sitter-rust": "0.
|
|
66
|
+
"tree-sitter-rust": "0.21.0",
|
|
67
67
|
"tree-sitter-typescript": "0.23.2",
|
|
68
68
|
"unpdf": "1.6.2"
|
|
69
69
|
},
|
|
70
|
-
"overrides": {
|
|
71
|
-
"tree-sitter": "0.25.1"
|
|
72
|
-
},
|
|
73
70
|
"devDependencies": {
|
|
74
71
|
"@biomejs/biome": "2.3.5",
|
|
75
72
|
"@types/better-sqlite3": "7.6.13",
|