pi-knowledge 0.10.1 → 0.10.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,11 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.10.2] - 2026-09-19
4
+
5
+ ### Fixed
6
+ - Allowed OpenAI-compatible embedding endpoints to run without an API key and added `PI_KNOWLEDGE_EMBEDDING_API_KEY` as an embedding-specific bearer token override (#15).
7
+ - Added `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` so self-hosted embedding servers can use request batch sizes below the default 64 chunk limit (#16).
8
+
3
9
  ## [0.10.1] - 2026-09-10
4
10
 
5
11
  ### Fixed
package/README.md CHANGED
@@ -166,13 +166,15 @@ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BA
166
166
  ```bash
167
167
  export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
168
168
  export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
169
- export OPENAI_API_KEY=local-placeholder
169
+ # Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
170
170
  ```
171
171
 
172
172
  API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
173
173
 
174
174
  API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
175
175
 
176
+ For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
177
+
176
178
  Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
177
179
 
178
180
  ## Configuration
@@ -183,7 +185,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
183
185
  |------|-----------------------|
184
186
  | Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
185
187
  | Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
186
- | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
188
+ | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
187
189
  | Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
188
190
  | Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
189
191
  | Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
@@ -127,15 +127,16 @@ export async function prepareForShutdown() {
127
127
  async function embedViaAPI(texts, prefix, config, signal) {
128
128
  const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
129
129
  const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
130
- const apiKey = process.env.OPENAI_API_KEY;
131
- if (!apiKey)
132
- throw new Error("OPENAI_API_KEY required for openai embedding");
130
+ const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
133
131
  if (!config.baseUrl)
134
132
  throw new Error("OpenAI embedding base URL is not configured");
135
133
  const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
134
+ const headers = { "Content-Type": "application/json" };
135
+ if (apiKey)
136
+ headers.Authorization = `Bearer ${apiKey}`;
136
137
  const res = await fetch(endpoint, {
137
138
  method: "POST",
138
- headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
139
+ headers,
139
140
  body: JSON.stringify({ input: safeTexts, model: config.model }),
140
141
  signal,
141
142
  });
@@ -15,8 +15,18 @@ import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js"
15
15
  import { searchVectorFile } from "./search/vector.js";
16
16
  import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
17
17
  export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
18
- const INDEX_EMBED_BATCH_SIZE = 64;
18
+ const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
19
19
  const VECTOR_REDUNDANCY_WEIGHT = 0.35;
20
+ function resolveIndexEmbedBatchSize(env = process.env) {
21
+ const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
22
+ if (!raw)
23
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
24
+ const value = Number(raw);
25
+ if (!Number.isFinite(value))
26
+ return DEFAULT_INDEX_EMBED_BATCH_SIZE;
27
+ const batchSize = Math.trunc(value);
28
+ return batchSize > 0 ? batchSize : DEFAULT_INDEX_EMBED_BATCH_SIZE;
29
+ }
20
30
  function isCancellationError(error) {
21
31
  return error instanceof Error && error.message === "Cancelled";
22
32
  }
@@ -704,6 +714,7 @@ export class KnowledgeEngine {
704
714
  throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
705
715
  }
706
716
  const embeddingConfig = resolveEmbeddingConfig();
717
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
707
718
  const kb = createKB(db, {
708
719
  name,
709
720
  source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
@@ -759,7 +770,7 @@ export class KnowledgeEngine {
759
770
  return;
760
771
  if (signal?.aborted)
761
772
  throw new Error("Cancelled");
762
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
773
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
763
774
  reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
764
775
  const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
765
776
  if (signal?.aborted)
@@ -774,7 +785,7 @@ export class KnowledgeEngine {
774
785
  };
775
786
  const addChunks = async (chunks, processedFiles, totalFiles) => {
776
787
  pendingChunks.push(...chunks);
777
- while (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE) {
788
+ while (pendingChunks.length >= embeddingBatchSize) {
778
789
  await flushPending(processedFiles, totalFiles);
779
790
  }
780
791
  };
@@ -927,6 +938,7 @@ export class KnowledgeEngine {
927
938
  startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
928
939
  const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
929
940
  const embeddingConfig = resolveEmbeddingConfig();
941
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
930
942
  const embeddingModel = embeddingConfigLabel(embeddingConfig);
931
943
  const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
932
944
  let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
@@ -991,7 +1003,7 @@ export class KnowledgeEngine {
991
1003
  return;
992
1004
  if (signal?.aborted)
993
1005
  throw new Error("Cancelled");
994
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1006
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
995
1007
  const elapsed = Date.now() - startedAt;
996
1008
  const message = `Embedding update batch: ${addedCount} new chunks stored, ${scannedFiles} files scanned, elapsed ${formatDuration(elapsed)}`;
997
1009
  updateIndexingJob(this.db, kb.id, {
@@ -1045,7 +1057,7 @@ export class KnowledgeEngine {
1045
1057
  continue;
1046
1058
  }
1047
1059
  pendingChunks.push(chunk);
1048
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1060
+ if (pendingChunks.length >= embeddingBatchSize)
1049
1061
  await flushPending();
1050
1062
  }
1051
1063
  };
@@ -1789,6 +1801,7 @@ export class KnowledgeEngine {
1789
1801
  const { createWriteStream } = await import("node:fs");
1790
1802
  const tempOutputPath = tempVectorPath(outputPath);
1791
1803
  const stream = createWriteStream(tempOutputPath, { encoding: "utf-8" });
1804
+ const progressBatchSize = resolveIndexEmbedBatchSize();
1792
1805
  let count = 0;
1793
1806
  const header = JSON.stringify({
1794
1807
  name: kb.name,
@@ -1809,7 +1822,7 @@ export class KnowledgeEngine {
1809
1822
  metadata_json: chunk.metadata_json,
1810
1823
  }));
1811
1824
  count++;
1812
- if (count % INDEX_EMBED_BATCH_SIZE === 0)
1825
+ if (count % progressBatchSize === 0)
1813
1826
  onProgress?.(`Exported ${count}/${kb.chunk_count} chunks...`);
1814
1827
  }
1815
1828
  await finishWriteStream(stream);
@@ -1838,6 +1851,7 @@ export class KnowledgeEngine {
1838
1851
  const stream = createReadStream(inputPath, { encoding: "utf-8" });
1839
1852
  const lines = createInterface({ input: stream, crlfDelay: Infinity });
1840
1853
  const embeddingConfig = resolveEmbeddingConfig();
1854
+ const embeddingBatchSize = resolveIndexEmbedBatchSize();
1841
1855
  let header;
1842
1856
  let kb;
1843
1857
  let vectorWriter;
@@ -1850,7 +1864,7 @@ export class KnowledgeEngine {
1850
1864
  if (!this.db || !kb || !vectorWriter || pendingChunks.length === 0)
1851
1865
  return;
1852
1866
  throwIfAborted(signal);
1853
- const batch = pendingChunks.splice(0, INDEX_EMBED_BATCH_SIZE);
1867
+ const batch = pendingChunks.splice(0, embeddingBatchSize);
1854
1868
  const symbols = pendingSymbols.splice(0);
1855
1869
  const total = header?.chunk_count;
1856
1870
  const message = `Embedding import batch: ${inserted}/${total ?? "unknown"} chunks stored`;
@@ -1919,7 +1933,7 @@ export class KnowledgeEngine {
1919
1933
  chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
1920
1934
  pendingChunks.push(chunk);
1921
1935
  pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
1922
- if (pendingChunks.length >= INDEX_EMBED_BATCH_SIZE)
1936
+ if (pendingChunks.length >= embeddingBatchSize)
1923
1937
  await flushPending();
1924
1938
  }
1925
1939
  if (!header)
@@ -21,10 +21,12 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
21
21
  | Variable | Default | Purpose |
22
22
  |----------|---------|---------|
23
23
  | `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
24
- | `OPENAI_API_KEY` | unset | API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Some local OpenAI-compatible servers accept a placeholder value. |
24
+ | `PI_KNOWLEDGE_EMBEDDING_API_KEY` | unset | API key for OpenAI-compatible embedding APIs. Takes precedence over `OPENAI_API_KEY`; omit for trusted local endpoints that do not require auth. |
25
+ | `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
25
26
  | `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
26
27
  | `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
27
28
  | `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
29
+ | `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
28
30
  | `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
29
31
  | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
30
32
  | `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
@@ -59,7 +61,7 @@ For Hugging Face Text Embeddings Inference rerankers, set `PI_KNOWLEDGE_RERANKER
59
61
 
60
62
  Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
61
63
 
62
- API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB.
64
+ API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, context-window errors, or server batch limits and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB. OpenAI-compatible embedding requests send `Authorization: Bearer <key>` only when `PI_KNOWLEDGE_EMBEDDING_API_KEY` or `OPENAI_API_KEY` is configured.
63
65
 
64
66
  ## Runtime Features and Diagnostics
65
67
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.10.1",
3
+ "version": "0.10.2",
4
4
  "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",