pi-knowledge 0.4.5 → 0.4.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,29 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.4.7] - 2026-07-04
4
+
5
+ ### Fixed
6
+ - Avoided live SQLite statement iterators in chunk row scans so search, diagnostics, and vector rebuild paths cannot leave the database connection busy before `knowledge_update`.
7
+ - Coalesced overlapping `knowledge_update` calls per knowledge base so watcher-triggered updates, manual updates, retries, and shutdown do not re-enter the same update flow.
8
+
9
+ ### Changed
10
+ - Increased SQLite busy timeout to tolerate short-lived external contention during long indexing/update work.
11
+ - Updated npm-facing README and package metadata to show Pi and OMP (`omp.sh`) coding-agent support.
12
+
13
+ ## [0.4.6] - 2026-06-26
14
+
15
+ ### Fixed
16
+ - Split oversized Markdown paragraphs and code blocks into bounded chunks so external embedding servers do not receive single over-context Markdown chunks.
17
+ - Added OpenAI-compatible embedding base URL support via `PI_KNOWLEDGE_EMBEDDING_BASE_URL` / `OPENAI_BASE_URL`.
18
+ - Added configurable API embedding input truncation via `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` as a final context-window safety guard.
19
+ - Surfaced embedding API failures by default instead of silently falling back to local embeddings; explicit local fallback is available with `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`.
20
+ - Included model worker stderr in startup/exit failures to make `knowledge_add` worker crashes diagnosable.
21
+
22
+ ### Changed
23
+ - Documented all runtime environment variable overrides in `docs/configuration.md`.
24
+ - Documented Pi and OMP support boundaries, storage path resolution, and release validation expectations.
25
+ - Wired `PI_KNOWLEDGE_OFFLINE` to disable remote Transformers.js model downloads when using a pre-populated local model cache.
26
+
3
27
  ## [0.4.5] - 2026-06-24
4
28
 
5
29
  ### Fixed
package/README.md CHANGED
@@ -1,14 +1,14 @@
1
1
  # pi-knowledge
2
2
 
3
- Local-first RAG knowledge base for Pi agents.
3
+ Local-first RAG knowledge base for Pi and OMP coding agents.
4
4
 
5
- Index your codebase, docs, PDFs, URLs, and notes into persistent knowledge bases that Pi can search across sessions. `pi-knowledge` combines semantic embeddings, BM25 keyword search, code-aware chunking, reranking, diagnostics, and stable large-project indexing so agents can answer from your actual project knowledge instead of guessing.
5
+ Index your codebase, docs, PDFs, URLs, and notes into persistent knowledge bases that Pi and [OMP](https://omp.sh/) can search across sessions. `pi-knowledge` combines semantic embeddings, BM25 keyword search, code-aware chunking, reranking, diagnostics, and stable large-project indexing so agents can answer from your actual project knowledge instead of guessing.
6
6
 
7
- Built as a native [Pi extension](https://pi.dev/docs/latest/extensions), deeply integrated with the agent lifecycle. Designed for local-first project memory, agentic code search, and retrieval-augmented development workflows.
7
+ Built as a native [Pi extension](https://pi.dev/docs/latest/extensions) with verified [OMP](https://omp.sh/) compatibility through the packaged extension entry. Designed for local-first project memory, agentic code search, and retrieval-augmented development workflows.
8
8
 
9
9
  ## Why It Matters
10
10
 
11
- Agents lose project context between sessions and cannot fit large repositories into one prompt. `pi-knowledge` gives Pi durable, searchable project memory:
11
+ Agents lose project context between sessions and cannot fit large repositories into one prompt. `pi-knowledge` gives Pi and OMP durable, searchable project memory:
12
12
 
13
13
  - Search code and documentation by meaning, exact symbols, or both.
14
14
  - Keep private source data local by default.
@@ -74,11 +74,15 @@ In project-level dogfood, these changes improved a real codebase evaluation from
74
74
  ## Quick Start
75
75
 
76
76
  ```bash
77
- # Install
77
+ # Install for Pi
78
78
  pi install npm:pi-knowledge
79
79
 
80
+ # Install for OMP
81
+ omp install npm:pi-knowledge
82
+
80
83
  # Or from source
81
84
  pi install ./pi-knowledge
85
+ omp install ./pi-knowledge
82
86
 
83
87
  # Index a directory
84
88
  # (agent will call knowledge_add automatically, or you can ask it)
@@ -128,6 +132,54 @@ Search results use balanced diversity reranking by default so near-duplicate chu
128
132
 
129
133
  For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and code symbol names while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, or functions, and reduces duplicate-looking chunk hits.
130
134
 
135
+ ## Embedding Configuration
136
+
137
+ Local embeddings are the default and require no API key:
138
+
139
+ ```bash
140
+ PI_KNOWLEDGE_EMBEDDING=local:multilingual-e5-small
141
+ ```
142
+
143
+ OpenAI or OpenAI-compatible embedding APIs can be selected with `PI_KNOWLEDGE_EMBEDDING`:
144
+
145
+ ```bash
146
+ export PI_KNOWLEDGE_EMBEDDING=openai:text-embedding-3-small
147
+ export OPENAI_API_KEY=...
148
+ ```
149
+
150
+ For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BASE_URL` or `OPENAI_BASE_URL` to the API root that contains `/embeddings`:
151
+
152
+ ```bash
153
+ export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
154
+ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
155
+ export OPENAI_API_KEY=local-placeholder
156
+ ```
157
+
158
+ API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow a local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`.
159
+
160
+ API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window.
161
+
162
+ ## Configuration
163
+
164
+ Full configuration details are in [docs/configuration.md](docs/configuration.md).
165
+
166
+ | Area | Environment variables |
167
+ |------|-----------------------|
168
+ | Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
169
+ | Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
170
+ | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
171
+ | Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
172
+ | Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE` |
173
+ | Release fixtures | `PI_KNOWLEDGE_E2E_PDF`, `PI_KNOWLEDGE_E2E_DOCX` |
174
+
175
+ ## Pi and OMP Support
176
+
177
+ `pi-knowledge` supports Pi and [OMP](https://omp.sh/) extension loading through the packaged `extension.js` entry shim. The entry stays startup-light: install-time validation can inspect the extension without resolving native runtime dependencies, and runtime modules load lazily only when tools or lifecycle hooks need them.
178
+
179
+ Default storage is `~/.pi/knowledge` for Pi and `~/.omp/knowledge` for OMP. Explicit overrides are available with `PI_KNOWLEDGE_DIR` and `OMP_KNOWLEDGE_DIR`. Under the default home OMP root, existing legacy `~/.pi/knowledge` data remains visible when `~/.omp/knowledge` has not been created yet.
180
+
181
+ OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated model-worker startup, and idempotent shutdown. Compatibility-sensitive releases should validate both Pi and OMP install/runtime flows.
182
+
131
183
  ## Large Project Indexing
132
184
 
133
185
  Indexing is designed as a stable long-running operation, not a quick background trick. `knowledge_add`, `knowledge_update`, and `knowledge_import` scan directories incrementally, embed and store chunks in hard-capped batches, stream vector files to disk, and report progress with file/chunk counts, chunks/sec, skipped file counts, elapsed time, and file ETA where available. Directory indexing starts with a metadata-only planning scan so large repositories can show total scannable files and skipped counts before expensive embedding starts.
@@ -140,11 +192,11 @@ Update and diagnostics paths are also streaming-oriented: changed chunks are emb
140
192
 
141
193
  ## Architecture
142
194
 
143
- See [DESIGN.md](DESIGN.md) for the full technical design document.
195
+ See [DESIGN.md](DESIGN.md) for the full technical design document and [docs/configuration.md](docs/configuration.md) for runtime configuration.
144
196
 
145
197
  ## Data Storage
146
198
 
147
- All data is stored globally at `~/.pi/knowledge/` (never in your project directory):
199
+ All data is stored globally at `~/.pi/knowledge/` under Pi or `~/.omp/knowledge/` under OMP unless overridden (never in your project directory):
148
200
 
149
201
  ```
150
202
  ~/.pi/knowledge/
@@ -153,8 +205,9 @@ All data is stored globally at `~/.pi/knowledge/` (never in your project directo
153
205
  └── models/ ← Downloaded ONNX models (~32MB, cached)
154
206
  ```
155
207
 
156
- - **Backup**: copy `~/.pi/knowledge/` directory
157
- - **Reset**: delete `~/.pi/knowledge/` to start fresh
208
+ - **Backup**: copy the active knowledge directory, usually `~/.pi/knowledge/` or `~/.omp/knowledge/`
209
+ - **Reset**: delete the active knowledge directory to start fresh
210
+ - **Override**: set `PI_KNOWLEDGE_DIR` or `OMP_KNOWLEDGE_DIR`
158
211
  - **Project safety**: pi-knowledge is read-only on indexed directories — no files are created or modified in your project
159
212
  - **Updates**: extension updates do not affect existing indexed data. Schema migrations run automatically if needed.
160
213
 
@@ -166,6 +219,7 @@ npm test # Unit tests
166
219
  npm run test:e2e # Smoke integration tests; PDF/DOCX cases skip unless fixture env vars are set
167
220
  PI_KNOWLEDGE_E2E_PDF=/path/to/file.pdf PI_KNOWLEDGE_E2E_DOCX=/path/to/file.docx npm run test:e2e
168
221
  npm run bench # Indexing/search benchmarks
222
+ node --experimental-strip-types -e "import('./index.ts')" # Startup-light source import smoke
169
223
  ```
170
224
 
171
225
  PDF/DOCX fixtures should be real local files outside the repository. Do not commit private fixture files, extracted fixture text, snapshots, or machine-specific fixture paths. A release-grade e2e pass requires both fixture env vars; a run with skipped PDF/DOCX cases is only a smoke pass.
@@ -182,8 +236,10 @@ npm run build
182
236
  npm run test:e2e
183
237
  PI_KNOWLEDGE_E2E_PDF=/path/to/file.pdf PI_KNOWLEDGE_E2E_DOCX=/path/to/file.docx npm run test:e2e
184
238
  node -e "import('./extension.js')"
239
+ node --experimental-strip-types -e "import('./index.ts')"
185
240
  npm pack --dry-run
186
241
  pi -e ./extension.js
242
+ omp -e ./extension.js
187
243
  git push origin main
188
244
  gh release create vX.Y.Z --title "vX.Y.Z" --notes-file /path/to/release-notes.md
189
245
  npm publish
@@ -7,6 +7,9 @@ const idleWaiters = [];
7
7
  const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
8
8
  const EMBEDDING_CONFIG = process.env.PI_KNOWLEDGE_EMBEDDING ?? "local:multilingual-e5-small";
9
9
  const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
10
+ const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
11
+ const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
12
+ const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
10
13
  function clearIdleTimer() {
11
14
  if (disposeTimer)
12
15
  clearTimeout(disposeTimer);
@@ -56,18 +59,27 @@ export async function prepareForShutdown() {
56
59
  }
57
60
  async function embedViaAPI(texts, prefix) {
58
61
  const [provider, model] = EMBEDDING_CONFIG.split(":");
62
+ const configuredMaxChars = Number(process.env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS ?? DEFAULT_API_MAX_EMBED_CHARS);
63
+ const maxChars = Number.isFinite(configuredMaxChars) && configuredMaxChars > 0 ? configuredMaxChars : DEFAULT_API_MAX_EMBED_CHARS;
59
64
  const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
65
+ const safeTexts = prefixedTexts.map((text) => (text.length > maxChars ? text.slice(0, maxChars) : text));
60
66
  if (provider === "openai") {
61
67
  const apiKey = process.env.OPENAI_API_KEY;
62
68
  if (!apiKey)
63
69
  throw new Error("OPENAI_API_KEY required for openai embedding");
64
- const res = await fetch("https://api.openai.com/v1/embeddings", {
70
+ const baseUrl = process.env.PI_KNOWLEDGE_EMBEDDING_BASE_URL?.trim() ||
71
+ process.env.OPENAI_BASE_URL?.trim() ||
72
+ DEFAULT_OPENAI_BASE_URL;
73
+ const endpoint = new URL("embeddings", `${baseUrl.replace(/\/+$/, "")}/`);
74
+ const res = await fetch(endpoint, {
65
75
  method: "POST",
66
76
  headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
67
- body: JSON.stringify({ input: prefixedTexts, model: model || "text-embedding-3-small" }),
77
+ body: JSON.stringify({ input: safeTexts, model: model || "text-embedding-3-small" }),
68
78
  });
69
- if (!res.ok)
70
- throw new Error(`OpenAI API error: ${res.status}`);
79
+ if (!res.ok) {
80
+ const detail = await res.text().catch(() => "");
81
+ throw new Error(`OpenAI embedding API error: ${res.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
82
+ }
71
83
  const data = (await res.json());
72
84
  return data.data.map((d) => new Float32Array(d.embedding));
73
85
  }
@@ -78,8 +90,10 @@ export async function embedTexts(texts, prefix, signal) {
78
90
  try {
79
91
  return await embedViaAPI(texts, prefix);
80
92
  }
81
- catch {
82
- /* fallback to local */
93
+ catch (error) {
94
+ if (!API_FALLBACK_TO_LOCAL)
95
+ throw error;
96
+ console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
83
97
  }
84
98
  }
85
99
  beginRun();
@@ -62,6 +62,7 @@ export interface IndexPlan {
62
62
  export declare class KnowledgeEngine {
63
63
  private db;
64
64
  private knowledgeDir;
65
+ private activeUpdates;
65
66
  initialize(knowledgeDir: string): Promise<void>;
66
67
  plan(source: string, options?: AddOptions): IndexPlan;
67
68
  add(source: string, name: string, onProgress?: ProgressCallback, signal?: AbortSignal, options?: AddOptions): Promise<{
@@ -73,6 +74,7 @@ export declare class KnowledgeEngine {
73
74
  removed: number;
74
75
  unchanged: number;
75
76
  }>;
77
+ private runUpdate;
76
78
  search(query: string, options?: SearchOptions): Promise<SearchResponse>;
77
79
  remove(nameOrId: string): boolean;
78
80
  list(): KnowledgeBase[];
@@ -350,6 +350,7 @@ function normalizeExtractedText(text) {
350
350
  export class KnowledgeEngine {
351
351
  db = null;
352
352
  knowledgeDir = "";
353
+ activeUpdates = new Map();
353
354
  async initialize(knowledgeDir) {
354
355
  this.knowledgeDir = knowledgeDir;
355
356
  this.db = openDatabase(knowledgeDir);
@@ -614,6 +615,22 @@ export class KnowledgeEngine {
614
615
  if (!kb.source_path || (kb.source_type !== "url" && !existsSync(kb.source_path))) {
615
616
  throw new Error(`Source path not available or missing: ${kb.source_path}`);
616
617
  }
618
+ const updateableKB = { ...kb, source_path: kb.source_path };
619
+ const activeUpdate = this.activeUpdates.get(updateableKB.id);
620
+ if (activeUpdate) {
621
+ onProgress?.(`Update already running for "${updateableKB.name}"; waiting for the active update.`);
622
+ return activeUpdate;
623
+ }
624
+ const updateRun = this.runUpdate(updateableKB, onProgress, signal).finally(() => {
625
+ if (this.activeUpdates.get(updateableKB.id) === updateRun)
626
+ this.activeUpdates.delete(updateableKB.id);
627
+ });
628
+ this.activeUpdates.set(updateableKB.id, updateRun);
629
+ return updateRun;
630
+ }
631
+ async runUpdate(kb, onProgress, signal) {
632
+ if (!this.db)
633
+ throw new Error("Engine not initialized");
617
634
  updateKBStatus(this.db, kb.id, "indexing");
618
635
  startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
619
636
  const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
@@ -1354,6 +1371,9 @@ export class KnowledgeEngine {
1354
1371
  }
1355
1372
  }
1356
1373
  async dispose(options = {}) {
1374
+ const activeUpdates = [...this.activeUpdates.values()];
1375
+ if (activeUpdates.length > 0)
1376
+ await Promise.allSettled(activeUpdates);
1357
1377
  const disposeModels = options.disposeModels ?? true;
1358
1378
  if (disposeModels) {
1359
1379
  await disposeEmbedding();
@@ -381,6 +381,18 @@ export function chunkMarkdown(content, filePath) {
381
381
  breadcrumb: currentBreadcrumb(),
382
382
  }));
383
383
  }
384
+ function pushOversizedMarkdownParagraph(para, start) {
385
+ let offset = 0;
386
+ let sliceStart = start;
387
+ while (offset < para.length) {
388
+ const slice = para.slice(offset, offset + MAX_TEXT_CHUNK_CHARS).trim();
389
+ if (slice.length >= 50) {
390
+ pushMarkdownChunk(slice, sliceStart, sliceStart + slice.split("\n").length);
391
+ }
392
+ sliceStart += slice.split("\n").length;
393
+ offset += MAX_TEXT_CHUNK_CHARS;
394
+ }
395
+ }
384
396
  function flush(endLine) {
385
397
  const text = sectionLines.join("\n").trim();
386
398
  if (text.length < 50)
@@ -390,6 +402,16 @@ export function chunkMarkdown(content, filePath) {
390
402
  let buffer = [];
391
403
  let bufferStart = startLine;
392
404
  for (const para of paragraphs) {
405
+ if (para.length > MAX_TEXT_CHUNK_CHARS) {
406
+ const oversizedText = buffer.length > 0 ? [...buffer, para].join("\n\n") : para;
407
+ const oversizedStart = buffer.length > 0 ? bufferStart : endLine;
408
+ if (buffer.length > 0) {
409
+ buffer = [];
410
+ }
411
+ pushOversizedMarkdownParagraph(oversizedText, oversizedStart);
412
+ bufferStart = endLine;
413
+ continue;
414
+ }
393
415
  const next = [...buffer, para].join("\n\n");
394
416
  if (estimateTokens(next) > MARKDOWN_TARGET_TOKENS && buffer.length > 0) {
395
417
  pushMarkdownChunk(buffer.join("\n\n"), bufferStart, endLine);
@@ -4,6 +4,7 @@ import { fileURLToPath } from "node:url";
4
4
  let worker = null;
5
5
  let nextRequestId = 1;
6
6
  const pending = new Map();
7
+ const WORKER_STDERR_TAIL_CHARS = 4_000;
7
8
  function rejectPending(error) {
8
9
  for (const request of pending.values()) {
9
10
  request.reject(error);
@@ -24,6 +25,17 @@ function getNodeExecPath() {
24
25
  const execName = basename(process.execPath).toLowerCase();
25
26
  return execName === "node" || execName === "node.exe" ? process.execPath : "node";
26
27
  }
28
+ function appendWorkerStderr(current, chunk) {
29
+ const next = `${current}${chunk.toString("utf-8")}`;
30
+ return next.length > WORKER_STDERR_TAIL_CHARS ? next.slice(-WORKER_STDERR_TAIL_CHARS) : next;
31
+ }
32
+ function formatWorkerExitError(code, signal, stderrTail) {
33
+ const reason = `Model worker exited before responding (code ${code ?? "null"}, signal ${signal ?? "null"})`;
34
+ const stderr = stderrTail.trim();
35
+ if (!stderr)
36
+ return new Error(`${reason}. Set PI_KNOWLEDGE_NODE_PATH to a working Node binary if Pi is not running under Node.`);
37
+ return new Error(`${reason}. Worker stderr:\n${stderr}`);
38
+ }
27
39
  function getWorker() {
28
40
  if (worker?.connected)
29
41
  return worker;
@@ -31,10 +43,14 @@ function getWorker() {
31
43
  worker = fork(workerPath, {
32
44
  execPath: getNodeExecPath(),
33
45
  execArgv: getWorkerExecArgv(),
34
- stdio: ["ignore", "ignore", "ignore", "ipc"],
46
+ stdio: ["ignore", "ignore", "pipe", "ipc"],
35
47
  env: process.env,
36
48
  });
37
49
  const child = worker;
50
+ let stderrTail = "";
51
+ child.stderr?.on("data", (chunk) => {
52
+ stderrTail = appendWorkerStderr(stderrTail, chunk);
53
+ });
38
54
  worker.on("message", (message) => {
39
55
  const request = pending.get(message.id);
40
56
  if (!request)
@@ -52,7 +68,7 @@ function getWorker() {
52
68
  return;
53
69
  worker = null;
54
70
  if (pending.size > 0) {
55
- rejectPending(new Error(`Model worker exited before responding (code ${code ?? "null"}, signal ${signal ?? "null"})`));
71
+ rejectPending(formatWorkerExitError(code, signal, stderrTail));
56
72
  }
57
73
  });
58
74
  worker.on("error", (error) => {
@@ -5,11 +5,19 @@ let rerankerPipeline = null;
5
5
  function getModelCacheDir() {
6
6
  return process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? join(getDefaultKnowledgeDir(), "models");
7
7
  }
8
+ function configureTransformersEnv(env) {
9
+ const cacheDir = getModelCacheDir();
10
+ env.cacheDir = cacheDir;
11
+ if (process.env.PI_KNOWLEDGE_OFFLINE === "true") {
12
+ env.allowRemoteModels = false;
13
+ env.localModelPath = cacheDir;
14
+ }
15
+ }
8
16
  async function loadEmbeddingPipeline() {
9
17
  if (embeddingPipeline)
10
18
  return embeddingPipeline;
11
19
  const { pipeline, env } = await import("@huggingface/transformers");
12
- env.cacheDir = getModelCacheDir();
20
+ configureTransformersEnv(env);
13
21
  const createPipeline = pipeline;
14
22
  const loaded = (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
15
23
  quantized: true,
@@ -22,7 +30,7 @@ async function loadRerankerPipeline() {
22
30
  if (rerankerPipeline)
23
31
  return rerankerPipeline;
24
32
  const { pipeline, env } = await import("@huggingface/transformers");
25
- env.cacheDir = getModelCacheDir();
33
+ configureTransformersEnv(env);
26
34
  const createPipeline = pipeline;
27
35
  const loaded = (await createPipeline("text-classification", "Xenova/ms-marco-MiniLM-L-4-v2"));
28
36
  rerankerPipeline = loaded;
@@ -5,6 +5,7 @@ import { homedir } from "node:os";
5
5
  import { basename, dirname, join } from "node:path";
6
6
  import { fileURLToPath } from "node:url";
7
7
  const SCHEMA_VERSION = 3;
8
+ const ITERATION_BATCH_SIZE = 500;
8
9
  const SCHEMA_SQL = `
9
10
  CREATE TABLE IF NOT EXISTS schema_version (
10
11
  version INTEGER NOT NULL
@@ -189,7 +190,7 @@ export function openDatabase(knowledgeDir) {
189
190
  const nativeBinding = isBunRuntime() ? undefined : findBetterSqlite3NativeBinding();
190
191
  const db = nativeBinding ? new Database(dbPath, { nativeBinding }) : new Database(dbPath);
191
192
  applyPragma(db, "journal_mode = WAL");
192
- applyPragma(db, "busy_timeout = 5000");
193
+ applyPragma(db, "busy_timeout = 60000");
193
194
  applyPragma(db, "foreign_keys = ON");
194
195
  const hasVersion = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name='schema_version'").get();
195
196
  if (!hasVersion) {
@@ -345,14 +346,14 @@ export function getChunksByKB(db, kbId) {
345
346
  return db.prepare("SELECT * FROM chunks WHERE kb_id = ? ORDER BY rowid").all(kbId);
346
347
  }
347
348
  export function iterateChunksByKB(db, kbId) {
348
- return db.prepare("SELECT * FROM chunks WHERE kb_id = ? ORDER BY rowid").iterate(kbId);
349
+ return iterateChunkRowsByKB(db, kbId);
349
350
  }
350
351
  export function getChunkIdsByKB(db, kbId) {
351
352
  const rows = db.prepare("SELECT id FROM chunks WHERE kb_id = ? ORDER BY rowid").all(kbId);
352
353
  return rows.map((r) => r.id);
353
354
  }
354
355
  export function iterateChunkIdsByKB(db, kbId) {
355
- return db.prepare("SELECT id FROM chunks WHERE kb_id = ? ORDER BY rowid").iterate(kbId);
356
+ return iterateChunkIdRowsByKB(db, kbId);
356
357
  }
357
358
  export function getChunkById(db, id) {
358
359
  return db.prepare("SELECT * FROM chunks WHERE id = ?").get(id);
@@ -387,9 +388,56 @@ export function getChunkHashesByKB(db, kbId) {
387
388
  return new Map(rows.map((r) => [r.content_hash, r.id]));
388
389
  }
389
390
  export function iterateChunkHashesByKB(db, kbId) {
390
- return db
391
- .prepare("SELECT id, content_hash FROM chunks WHERE kb_id = ? ORDER BY rowid")
392
- .iterate(kbId);
391
+ return iterateChunkHashRowsByKB(db, kbId);
392
+ }
393
+ function* iterateChunkRowsByKB(db, kbId) {
394
+ let lastRowid = 0;
395
+ const statement = db.prepare(`SELECT rowid as rowid, * FROM chunks
396
+ WHERE kb_id = ? AND rowid > ?
397
+ ORDER BY rowid
398
+ LIMIT ?`);
399
+ while (true) {
400
+ const rows = statement.all(kbId, lastRowid, ITERATION_BATCH_SIZE);
401
+ if (rows.length === 0)
402
+ return;
403
+ for (const row of rows) {
404
+ lastRowid = row.rowid;
405
+ const { rowid: _rowid, ...chunk } = row;
406
+ yield chunk;
407
+ }
408
+ }
409
+ }
410
+ function* iterateChunkIdRowsByKB(db, kbId) {
411
+ let lastRowid = 0;
412
+ const statement = db.prepare(`SELECT rowid, id FROM chunks
413
+ WHERE kb_id = ? AND rowid > ?
414
+ ORDER BY rowid
415
+ LIMIT ?`);
416
+ while (true) {
417
+ const rows = statement.all(kbId, lastRowid, ITERATION_BATCH_SIZE);
418
+ if (rows.length === 0)
419
+ return;
420
+ for (const row of rows) {
421
+ lastRowid = row.rowid;
422
+ yield { id: row.id };
423
+ }
424
+ }
425
+ }
426
+ function* iterateChunkHashRowsByKB(db, kbId) {
427
+ let lastRowid = 0;
428
+ const statement = db.prepare(`SELECT rowid, id, content_hash FROM chunks
429
+ WHERE kb_id = ? AND rowid > ?
430
+ ORDER BY rowid
431
+ LIMIT ?`);
432
+ while (true) {
433
+ const rows = statement.all(kbId, lastRowid, ITERATION_BATCH_SIZE);
434
+ if (rows.length === 0)
435
+ return;
436
+ for (const row of rows) {
437
+ lastRowid = row.rowid;
438
+ yield { id: row.id, content_hash: row.content_hash };
439
+ }
440
+ }
393
441
  }
394
442
  export function getChunkCount(db, kbId) {
395
443
  const row = db.prepare("SELECT COUNT(*) as count FROM chunks WHERE kb_id = ?").get(kbId);
@@ -0,0 +1,65 @@
1
+ # Configuration
2
+
3
+ `pi-knowledge` is local-first by default, but several environment variables are available for runtime selection, compatibility, and release validation.
4
+
5
+ ## Storage and Host Runtime
6
+
7
+ | Variable | Default | Purpose |
8
+ |----------|---------|---------|
9
+ | `PI_KNOWLEDGE_DIR` | host-derived knowledge dir | Override the directory that stores `knowledge.db`, vectors, and model cache parent data for Pi and OMP. Takes precedence over `OMP_KNOWLEDGE_DIR`. |
10
+ | `OMP_KNOWLEDGE_DIR` | host-derived knowledge dir | Override the knowledge directory when running under OMP. Used when `PI_KNOWLEDGE_DIR` is not set. |
11
+ | `PI_CODING_AGENT_DIR` | unset | Derive the Pi host root from a configured agent directory. Useful for isolated installs and validation. |
12
+ | `OMP_CODING_AGENT_DIR` | unset | Derive the OMP host root from a configured agent directory. Used when `PI_CODING_AGENT_DIR` is not set. |
13
+ | `OMP_PROFILE` | unset | Treat the current process as OMP-hosted for default path selection. |
14
+ | `PI_KNOWLEDGE_MODEL_CACHE_DIR` | `<knowledge-dir>/models` | Override the local Transformers.js model cache directory. |
15
+ | `PI_KNOWLEDGE_NODE_PATH` | current Node when possible, otherwise `node` | Override the Node executable used to fork the isolated model worker. |
16
+
17
+ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under OMP. For the default home OMP root, `pi-knowledge` preserves an existing legacy `~/.pi/knowledge` directory when `~/.omp/knowledge` does not exist, so existing Pi knowledge bases remain visible during migration.
18
+
19
+ ## Embeddings
20
+
21
+ | Variable | Default | Purpose |
22
+ |----------|---------|---------|
23
+ | `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
24
+ | `OPENAI_API_KEY` | unset | API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Some local OpenAI-compatible servers accept a placeholder value. |
25
+ | `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
26
+ | `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
27
+ | `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
28
+ | `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings after API failures. Without this, API failures are surfaced. |
29
+ | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Set to `true` to opt into native ONNX idle disposal. Disabled by default for stable shutdown. |
30
+ | `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle-dispose timer used only when native idle disposal is enabled. Mainly for lifecycle stress tests. |
31
+ | `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
32
+
33
+ API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended.
34
+
35
+ ## Runtime Features and Diagnostics
36
+
37
+ | Variable | Default | Purpose |
38
+ |----------|---------|---------|
39
+ | `PI_KNOWLEDGE_WATCH` | unset | Set to `true` to start file watchers for directory KBs. The polling fallback remains available when native `fs.watch` fails. |
40
+ | `PI_KNOWLEDGE_AUTO_INJECT` | unset | Set to `true` to auto-search KB context before model calls. This is opt-in. |
41
+ | `PI_KNOWLEDGE_STALE_INDEXING_MS` | built-in stale threshold | Override the stale indexing threshold used by diagnostics and `knowledge_doctor`. |
42
+
43
+ ## Test and Release Fixtures
44
+
45
+ | Variable | Default | Purpose |
46
+ |----------|---------|---------|
47
+ | `PI_KNOWLEDGE_E2E_PDF` | unset | Path to an external PDF fixture for release-grade e2e extraction coverage. |
48
+ | `PI_KNOWLEDGE_E2E_DOCX` | unset | Path to an external DOCX fixture for release-grade e2e extraction coverage. |
49
+
50
+ When `PI_KNOWLEDGE_E2E_PDF` or `PI_KNOWLEDGE_E2E_DOCX` is unset, `npm run test:e2e` is only a smoke gate for those document formats. Do not commit fixture files, fixture paths, extracted text, or snapshots.
51
+
52
+ ## Pi and OMP Support
53
+
54
+ `pi-knowledge` supports direct Pi extension loading through `pi -e ./extension.js` and OMP-compatible loading through the same packaged entry shim. The root `extension.js` and `index.ts` stay startup-light so install-time validation can inspect the extension without resolving native runtime dependencies such as `better-sqlite3` or `onnxruntime-node`.
55
+
56
+ Compatibility guarantees:
57
+
58
+ - The packaged `extension.js` loads built `dist/index.js` when present and falls back to source `index.ts` for local development.
59
+ - Runtime modules are imported lazily, after the extension is actually used.
60
+ - Local embedding and reranking models run in an isolated model worker, not in the Pi or OMP TUI process.
61
+ - Native SQLite loading includes a fallback for hoisted plugin dependency layouts.
62
+ - OMP path resolution can use `OMP_KNOWLEDGE_DIR`, `OMP_CODING_AGENT_DIR`, and `OMP_PROFILE`.
63
+ - Existing default Pi knowledge data can remain visible under the default OMP home root through the legacy `~/.pi/knowledge` fallback described above.
64
+
65
+ Release validation should include both Pi and OMP when compatibility-sensitive code changes touch entry shims, native dependency loading, storage path resolution, model-worker startup, or lifecycle shutdown.
@@ -0,0 +1,58 @@
1
+ # 離線模式部署
2
+
3
+ ---
4
+
5
+ ## 1. 預下載 Model
6
+
7
+ ```bash
8
+ # 有網路時觸發一次 knowledge_add → model 自動下載到 ~/.pi/knowledge/models/
9
+ # 或手動從 https://huggingface.co/Xenova/multilingual-e5-small/tree/main/onnx 下載
10
+ ```
11
+
12
+ ---
13
+
14
+ ## 2. 配置
15
+
16
+ ```bash
17
+ export PI_KNOWLEDGE_OFFLINE=true
18
+ # Optional: point at a pre-populated model cache
19
+ export PI_KNOWLEDGE_MODEL_CACHE_DIR=/path/to/models
20
+ ```
21
+
22
+ Extension 行為:
23
+ ```typescript
24
+ import { env } from '@huggingface/transformers';
25
+ env.cacheDir = process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? '<knowledge-dir>/models';
26
+ env.allowRemoteModels = false;
27
+ env.localModelPath = env.cacheDir;
28
+ ```
29
+
30
+ ---
31
+
32
+ ## 3. Proxy 支援
33
+
34
+ @huggingface/transformers 用 Node `fetch()`,支持:
35
+ ```bash
36
+ export HTTPS_PROXY=http://proxy.corp.com:8080
37
+ export NODE_EXTRA_CA_CERTS=/path/to/corp-ca.pem
38
+ ```
39
+
40
+ ---
41
+
42
+ ## 4. Air-Gapped 部署
43
+
44
+ ```bash
45
+ # Build machine (有網路): 觸發下載,打包 ~/.pi/knowledge/models/
46
+ # Target machine: 解壓到同路徑,或設定 PI_KNOWLEDGE_MODEL_CACHE_DIR 指向解壓後路徑
47
+ export PI_KNOWLEDGE_OFFLINE=true
48
+ ```
49
+
50
+ ---
51
+
52
+ ## 5. 行為差異
53
+
54
+ | 功能 | Online | Offline (model cached) | Offline (no cache) |
55
+ |------|--------|----------------------|-------------------|
56
+ | knowledge_add | ✅ | ✅ | ❌ error |
57
+ | knowledge_search | ✅ | ✅ | ❌ error |
58
+ | Model update | auto-check | skip | skip |
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.4.5",
4
- "description": "Local-first RAG knowledge base for Pi agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
3
+ "version": "0.4.7",
4
+ "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
7
7
  "types": "dist/index.d.ts",
@@ -14,6 +14,8 @@
14
14
  "extension.js",
15
15
  "dist/",
16
16
  ".pi/",
17
+ "docs/configuration.md",
18
+ "docs/offline-mode.md",
17
19
  "README.md",
18
20
  "CHANGELOG.md",
19
21
  "LICENSE"
@@ -22,6 +24,8 @@
22
24
  "pi-package",
23
25
  "pi",
24
26
  "pi-extension",
27
+ "omp",
28
+ "omp-sh",
25
29
  "ai-agent",
26
30
  "coding-agent",
27
31
  "knowledge-base",