pi-knowledge 0.10.1 → 0.10.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +4 -2
- package/dist/src/embedding/provider.js +5 -4
- package/dist/src/engine.js +22 -8
- package/docs/configuration.md +4 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.10.2] - 2026-09-19
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Allowed OpenAI-compatible embedding endpoints to run without an API key and added `PI_KNOWLEDGE_EMBEDDING_API_KEY` as an embedding-specific bearer token override (#15).
|
|
7
|
+
- Added `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` so self-hosted embedding servers can use request batch sizes below the default 64 chunk limit (#16).
|
|
8
|
+
|
|
3
9
|
## [0.10.1] - 2026-09-10
|
|
4
10
|
|
|
5
11
|
### Fixed
|
package/README.md
CHANGED
|
@@ -166,13 +166,15 @@ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BA
|
|
|
166
166
|
```bash
|
|
167
167
|
export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
|
|
168
168
|
export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
169
|
-
|
|
169
|
+
# Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
|
|
170
170
|
```
|
|
171
171
|
|
|
172
172
|
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
|
|
173
173
|
|
|
174
174
|
API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
|
|
175
175
|
|
|
176
|
+
For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
|
|
177
|
+
|
|
176
178
|
Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
|
|
177
179
|
|
|
178
180
|
## Configuration
|
|
@@ -183,7 +185,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
183
185
|
|------|-----------------------|
|
|
184
186
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
185
187
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
186
|
-
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
188
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
187
189
|
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
188
190
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
189
191
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
@@ -127,15 +127,16 @@ export async function prepareForShutdown() {
|
|
|
127
127
|
async function embedViaAPI(texts, prefix, config, signal) {
|
|
128
128
|
const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
|
|
129
129
|
const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
|
|
130
|
-
const apiKey = process.env.OPENAI_API_KEY;
|
|
131
|
-
if (!apiKey)
|
|
132
|
-
throw new Error("OPENAI_API_KEY required for openai embedding");
|
|
130
|
+
const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
|
|
133
131
|
if (!config.baseUrl)
|
|
134
132
|
throw new Error("OpenAI embedding base URL is not configured");
|
|
135
133
|
const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
|
|
134
|
+
const headers = { "Content-Type": "application/json" };
|
|
135
|
+
if (apiKey)
|
|
136
|
+
headers.Authorization = `Bearer ${apiKey}`;
|
|
136
137
|
const res = await fetch(endpoint, {
|
|
137
138
|
method: "POST",
|
|
138
|
-
headers
|
|
139
|
+
headers,
|
|
139
140
|
body: JSON.stringify({ input: safeTexts, model: config.model }),
|
|
140
141
|
signal,
|
|
141
142
|
});
|
package/dist/src/engine.js
CHANGED
|
@@ -15,8 +15,18 @@ import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js"
|
|
|
15
15
|
import { searchVectorFile } from "./search/vector.js";
|
|
16
16
|
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
|
|
17
17
|
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
|
|
18
|
-
const
|
|
18
|
+
const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
|
|
19
19
|
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
20
|
+
function resolveIndexEmbedBatchSize(env = process.env) {
|
|
21
|
+
const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
|
|
22
|
+
if (!raw)
|
|
23
|
+
return DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
24
|
+
const value = Number(raw);
|
|
25
|
+
if (!Number.isFinite(value))
|
|
26
|
+
return DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
27
|
+
const batchSize = Math.trunc(value);
|
|
28
|
+
return batchSize > 0 ? batchSize : DEFAULT_INDEX_EMBED_BATCH_SIZE;
|
|
29
|
+
}
|
|
20
30
|
function isCancellationError(error) {
|
|
21
31
|
return error instanceof Error && error.message === "Cancelled";
|
|
22
32
|
}
|
|
@@ -704,6 +714,7 @@ export class KnowledgeEngine {
|
|
|
704
714
|
throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
|
|
705
715
|
}
|
|
706
716
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
717
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
707
718
|
const kb = createKB(db, {
|
|
708
719
|
name,
|
|
709
720
|
source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
|
|
@@ -759,7 +770,7 @@ export class KnowledgeEngine {
|
|
|
759
770
|
return;
|
|
760
771
|
if (signal?.aborted)
|
|
761
772
|
throw new Error("Cancelled");
|
|
762
|
-
const batch = pendingChunks.splice(0,
|
|
773
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
763
774
|
reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
|
|
764
775
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
765
776
|
if (signal?.aborted)
|
|
@@ -774,7 +785,7 @@ export class KnowledgeEngine {
|
|
|
774
785
|
};
|
|
775
786
|
const addChunks = async (chunks, processedFiles, totalFiles) => {
|
|
776
787
|
pendingChunks.push(...chunks);
|
|
777
|
-
while (pendingChunks.length >=
|
|
788
|
+
while (pendingChunks.length >= embeddingBatchSize) {
|
|
778
789
|
await flushPending(processedFiles, totalFiles);
|
|
779
790
|
}
|
|
780
791
|
};
|
|
@@ -927,6 +938,7 @@ export class KnowledgeEngine {
|
|
|
927
938
|
startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
|
|
928
939
|
const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
|
|
929
940
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
941
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
930
942
|
const embeddingModel = embeddingConfigLabel(embeddingConfig);
|
|
931
943
|
const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
|
|
932
944
|
let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
|
|
@@ -991,7 +1003,7 @@ export class KnowledgeEngine {
|
|
|
991
1003
|
return;
|
|
992
1004
|
if (signal?.aborted)
|
|
993
1005
|
throw new Error("Cancelled");
|
|
994
|
-
const batch = pendingChunks.splice(0,
|
|
1006
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
995
1007
|
const elapsed = Date.now() - startedAt;
|
|
996
1008
|
const message = `Embedding update batch: ${addedCount} new chunks stored, ${scannedFiles} files scanned, elapsed ${formatDuration(elapsed)}`;
|
|
997
1009
|
updateIndexingJob(this.db, kb.id, {
|
|
@@ -1045,7 +1057,7 @@ export class KnowledgeEngine {
|
|
|
1045
1057
|
continue;
|
|
1046
1058
|
}
|
|
1047
1059
|
pendingChunks.push(chunk);
|
|
1048
|
-
if (pendingChunks.length >=
|
|
1060
|
+
if (pendingChunks.length >= embeddingBatchSize)
|
|
1049
1061
|
await flushPending();
|
|
1050
1062
|
}
|
|
1051
1063
|
};
|
|
@@ -1789,6 +1801,7 @@ export class KnowledgeEngine {
|
|
|
1789
1801
|
const { createWriteStream } = await import("node:fs");
|
|
1790
1802
|
const tempOutputPath = tempVectorPath(outputPath);
|
|
1791
1803
|
const stream = createWriteStream(tempOutputPath, { encoding: "utf-8" });
|
|
1804
|
+
const progressBatchSize = resolveIndexEmbedBatchSize();
|
|
1792
1805
|
let count = 0;
|
|
1793
1806
|
const header = JSON.stringify({
|
|
1794
1807
|
name: kb.name,
|
|
@@ -1809,7 +1822,7 @@ export class KnowledgeEngine {
|
|
|
1809
1822
|
metadata_json: chunk.metadata_json,
|
|
1810
1823
|
}));
|
|
1811
1824
|
count++;
|
|
1812
|
-
if (count %
|
|
1825
|
+
if (count % progressBatchSize === 0)
|
|
1813
1826
|
onProgress?.(`Exported ${count}/${kb.chunk_count} chunks...`);
|
|
1814
1827
|
}
|
|
1815
1828
|
await finishWriteStream(stream);
|
|
@@ -1838,6 +1851,7 @@ export class KnowledgeEngine {
|
|
|
1838
1851
|
const stream = createReadStream(inputPath, { encoding: "utf-8" });
|
|
1839
1852
|
const lines = createInterface({ input: stream, crlfDelay: Infinity });
|
|
1840
1853
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
1854
|
+
const embeddingBatchSize = resolveIndexEmbedBatchSize();
|
|
1841
1855
|
let header;
|
|
1842
1856
|
let kb;
|
|
1843
1857
|
let vectorWriter;
|
|
@@ -1850,7 +1864,7 @@ export class KnowledgeEngine {
|
|
|
1850
1864
|
if (!this.db || !kb || !vectorWriter || pendingChunks.length === 0)
|
|
1851
1865
|
return;
|
|
1852
1866
|
throwIfAborted(signal);
|
|
1853
|
-
const batch = pendingChunks.splice(0,
|
|
1867
|
+
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
1854
1868
|
const symbols = pendingSymbols.splice(0);
|
|
1855
1869
|
const total = header?.chunk_count;
|
|
1856
1870
|
const message = `Embedding import batch: ${inserted}/${total ?? "unknown"} chunks stored`;
|
|
@@ -1919,7 +1933,7 @@ export class KnowledgeEngine {
|
|
|
1919
1933
|
chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
|
|
1920
1934
|
pendingChunks.push(chunk);
|
|
1921
1935
|
pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
|
|
1922
|
-
if (pendingChunks.length >=
|
|
1936
|
+
if (pendingChunks.length >= embeddingBatchSize)
|
|
1923
1937
|
await flushPending();
|
|
1924
1938
|
}
|
|
1925
1939
|
if (!header)
|
package/docs/configuration.md
CHANGED
|
@@ -21,10 +21,12 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
21
21
|
| Variable | Default | Purpose |
|
|
22
22
|
|----------|---------|---------|
|
|
23
23
|
| `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
|
|
24
|
-
| `
|
|
24
|
+
| `PI_KNOWLEDGE_EMBEDDING_API_KEY` | unset | API key for OpenAI-compatible embedding APIs. Takes precedence over `OPENAI_API_KEY`; omit for trusted local endpoints that do not require auth. |
|
|
25
|
+
| `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
|
|
25
26
|
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
26
27
|
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
27
28
|
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
29
|
+
| `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
|
|
28
30
|
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
|
|
29
31
|
| `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
|
|
30
32
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
@@ -59,7 +61,7 @@ For Hugging Face Text Embeddings Inference rerankers, set `PI_KNOWLEDGE_RERANKER
|
|
|
59
61
|
|
|
60
62
|
Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
|
|
61
63
|
|
|
62
|
-
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names,
|
|
64
|
+
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, context-window errors, or server batch limits and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB. OpenAI-compatible embedding requests send `Authorization: Bearer <key>` only when `PI_KNOWLEDGE_EMBEDDING_API_KEY` or `OPENAI_API_KEY` is configured.
|
|
63
65
|
|
|
64
66
|
## Runtime Features and Diagnostics
|
|
65
67
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.2",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|