pi-knowledge 0.4.4 → 0.4.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +56 -4
- package/dist/index.d.ts +1 -1
- package/dist/index.js +121 -23
- package/dist/src/embedding/provider.js +20 -6
- package/dist/src/indexer/chunker.js +22 -0
- package/dist/src/model-worker-client.js +18 -2
- package/dist/src/model-worker.js +10 -2
- package/dist/src/storage/sqlite.d.ts +1 -1
- package/dist/src/storage/sqlite.js +76 -5
- package/docs/configuration.md +65 -0
- package/docs/offline-mode.md +58 -0
- package/extension.js +6 -2
- package/package.json +3 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.4.6] - 2026-06-26
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Split oversized Markdown paragraphs and code blocks into bounded chunks so external embedding servers do not receive single over-context Markdown chunks.
|
|
7
|
+
- Added OpenAI-compatible embedding base URL support via `PI_KNOWLEDGE_EMBEDDING_BASE_URL` / `OPENAI_BASE_URL`.
|
|
8
|
+
- Added configurable API embedding input truncation via `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` as a final context-window safety guard.
|
|
9
|
+
- Surfaced embedding API failures by default instead of silently falling back to local embeddings; explicit local fallback is available with `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`.
|
|
10
|
+
- Included model worker stderr in startup/exit failures to make `knowledge_add` worker crashes diagnosable.
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
- Documented all runtime environment variable overrides in `docs/configuration.md`.
|
|
14
|
+
- Documented Pi and OMP support boundaries, storage path resolution, and release validation expectations.
|
|
15
|
+
- Wired `PI_KNOWLEDGE_OFFLINE` to disable remote Transformers.js model downloads when using a pre-populated local model cache.
|
|
16
|
+
|
|
17
|
+
## [0.4.5] - 2026-06-24
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
- Deferred extension runtime imports so OMP plugin validation does not resolve native dependencies during install.
|
|
21
|
+
- Added Bun-binary-safe `better-sqlite3` loading fallback for hoisted plugin dependencies.
|
|
22
|
+
- Hardened lazy runtime shutdown so in-flight extension startup is disposed before Pi or OMP exits.
|
|
23
|
+
|
|
3
24
|
## [0.4.4] - 2026-06-23
|
|
4
25
|
|
|
5
26
|
### Fixed
|
package/README.md
CHANGED
|
@@ -128,6 +128,54 @@ Search results use balanced diversity reranking by default so near-duplicate chu
|
|
|
128
128
|
|
|
129
129
|
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and code symbol names while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, or functions, and reduces duplicate-looking chunk hits.
|
|
130
130
|
|
|
131
|
+
## Embedding Configuration
|
|
132
|
+
|
|
133
|
+
Local embeddings are the default and require no API key:
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
PI_KNOWLEDGE_EMBEDDING=local:multilingual-e5-small
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
OpenAI or OpenAI-compatible embedding APIs can be selected with `PI_KNOWLEDGE_EMBEDDING`:
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
export PI_KNOWLEDGE_EMBEDDING=openai:text-embedding-3-small
|
|
143
|
+
export OPENAI_API_KEY=...
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
For self-hosted OpenAI-compatible servers, set either `PI_KNOWLEDGE_EMBEDDING_BASE_URL` or `OPENAI_BASE_URL` to the API root that contains `/embeddings`:
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
export PI_KNOWLEDGE_EMBEDDING=openai:Qwen3-Embedding-8B
|
|
150
|
+
export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
151
|
+
export OPENAI_API_KEY=local-placeholder
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow a local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`.
|
|
155
|
+
|
|
156
|
+
API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window.
|
|
157
|
+
|
|
158
|
+
## Configuration
|
|
159
|
+
|
|
160
|
+
Full configuration details are in [docs/configuration.md](docs/configuration.md).
|
|
161
|
+
|
|
162
|
+
| Area | Environment variables |
|
|
163
|
+
|------|-----------------------|
|
|
164
|
+
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
165
|
+
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
166
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
167
|
+
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
168
|
+
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE` |
|
|
169
|
+
| Release fixtures | `PI_KNOWLEDGE_E2E_PDF`, `PI_KNOWLEDGE_E2E_DOCX` |
|
|
170
|
+
|
|
171
|
+
## Pi and OMP Support
|
|
172
|
+
|
|
173
|
+
`pi-knowledge` supports Pi and OMP-compatible extension loading through the packaged `extension.js` entry shim. The entry stays startup-light: install-time validation can inspect the extension without resolving native runtime dependencies, and runtime modules load lazily only when tools or lifecycle hooks need them.
|
|
174
|
+
|
|
175
|
+
Default storage is `~/.pi/knowledge` for Pi and `~/.omp/knowledge` for OMP. Explicit overrides are available with `PI_KNOWLEDGE_DIR` and `OMP_KNOWLEDGE_DIR`. Under the default home OMP root, existing legacy `~/.pi/knowledge` data remains visible when `~/.omp/knowledge` has not been created yet.
|
|
176
|
+
|
|
177
|
+
OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated model-worker startup, and idempotent shutdown. Compatibility-sensitive releases should validate both Pi and OMP install/runtime flows.
|
|
178
|
+
|
|
131
179
|
## Large Project Indexing
|
|
132
180
|
|
|
133
181
|
Indexing is designed as a stable long-running operation, not a quick background trick. `knowledge_add`, `knowledge_update`, and `knowledge_import` scan directories incrementally, embed and store chunks in hard-capped batches, stream vector files to disk, and report progress with file/chunk counts, chunks/sec, skipped file counts, elapsed time, and file ETA where available. Directory indexing starts with a metadata-only planning scan so large repositories can show total scannable files and skipped counts before expensive embedding starts.
|
|
@@ -140,11 +188,11 @@ Update and diagnostics paths are also streaming-oriented: changed chunks are emb
|
|
|
140
188
|
|
|
141
189
|
## Architecture
|
|
142
190
|
|
|
143
|
-
See [DESIGN.md](DESIGN.md) for the full technical design document.
|
|
191
|
+
See [DESIGN.md](DESIGN.md) for the full technical design document and [docs/configuration.md](docs/configuration.md) for runtime configuration.
|
|
144
192
|
|
|
145
193
|
## Data Storage
|
|
146
194
|
|
|
147
|
-
All data is stored globally at `~/.pi/knowledge/` (never in your project directory):
|
|
195
|
+
All data is stored globally at `~/.pi/knowledge/` under Pi or `~/.omp/knowledge/` under OMP unless overridden (never in your project directory):
|
|
148
196
|
|
|
149
197
|
```
|
|
150
198
|
~/.pi/knowledge/
|
|
@@ -153,8 +201,9 @@ All data is stored globally at `~/.pi/knowledge/` (never in your project directo
|
|
|
153
201
|
└── models/ ← Downloaded ONNX models (~32MB, cached)
|
|
154
202
|
```
|
|
155
203
|
|
|
156
|
-
- **Backup**: copy `~/.pi/knowledge/`
|
|
157
|
-
- **Reset**: delete
|
|
204
|
+
- **Backup**: copy the active knowledge directory, usually `~/.pi/knowledge/` or `~/.omp/knowledge/`
|
|
205
|
+
- **Reset**: delete the active knowledge directory to start fresh
|
|
206
|
+
- **Override**: set `PI_KNOWLEDGE_DIR` or `OMP_KNOWLEDGE_DIR`
|
|
158
207
|
- **Project safety**: pi-knowledge is read-only on indexed directories — no files are created or modified in your project
|
|
159
208
|
- **Updates**: extension updates do not affect existing indexed data. Schema migrations run automatically if needed.
|
|
160
209
|
|
|
@@ -166,6 +215,7 @@ npm test # Unit tests
|
|
|
166
215
|
npm run test:e2e # Smoke integration tests; PDF/DOCX cases skip unless fixture env vars are set
|
|
167
216
|
PI_KNOWLEDGE_E2E_PDF=/path/to/file.pdf PI_KNOWLEDGE_E2E_DOCX=/path/to/file.docx npm run test:e2e
|
|
168
217
|
npm run bench # Indexing/search benchmarks
|
|
218
|
+
node --experimental-strip-types -e "import('./index.ts')" # Startup-light source import smoke
|
|
169
219
|
```
|
|
170
220
|
|
|
171
221
|
PDF/DOCX fixtures should be real local files outside the repository. Do not commit private fixture files, extracted fixture text, snapshots, or machine-specific fixture paths. A release-grade e2e pass requires both fixture env vars; a run with skipped PDF/DOCX cases is only a smoke pass.
|
|
@@ -182,8 +232,10 @@ npm run build
|
|
|
182
232
|
npm run test:e2e
|
|
183
233
|
PI_KNOWLEDGE_E2E_PDF=/path/to/file.pdf PI_KNOWLEDGE_E2E_DOCX=/path/to/file.docx npm run test:e2e
|
|
184
234
|
node -e "import('./extension.js')"
|
|
235
|
+
node --experimental-strip-types -e "import('./index.ts')"
|
|
185
236
|
npm pack --dry-run
|
|
186
237
|
pi -e ./extension.js
|
|
238
|
+
omp -e ./extension.js
|
|
187
239
|
git push origin main
|
|
188
240
|
gh release create vX.Y.Z --title "vX.Y.Z" --notes-file /path/to/release-notes.md
|
|
189
241
|
npm publish
|
package/dist/index.d.ts
CHANGED
|
@@ -27,7 +27,7 @@ type BeforeAgentStartEvent = {
|
|
|
27
27
|
};
|
|
28
28
|
type ExtensionAPI = {
|
|
29
29
|
on(event: "context", handler: (event: ContextEvent, ctx: unknown) => unknown): void;
|
|
30
|
-
on(event: "before_agent_start", handler: (event: BeforeAgentStartEvent, ctx: unknown) => unknown): void;
|
|
30
|
+
on(event: "before_agent_start", handler: (event: BeforeAgentStartEvent, ctx: unknown) => unknown | Promise<unknown>): void;
|
|
31
31
|
on(event: "session_start" | "session_shutdown", handler: (event: unknown, ctx: unknown) => unknown): void;
|
|
32
32
|
registerTool(tool: ToolDefinition): void;
|
|
33
33
|
};
|
package/dist/index.js
CHANGED
|
@@ -1,6 +1,11 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
|
|
1
|
+
var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExtension) || function (path, preserveJsx) {
|
|
2
|
+
if (typeof path === "string" && /^\.\.?\//.test(path)) {
|
|
3
|
+
return path.replace(/\.(tsx)$|((?:\.d)?)((?:\.[^./]+?)?)\.([cm]?)ts$/i, function (m, tsx, d, ext, cm) {
|
|
4
|
+
return tsx ? preserveJsx ? ".jsx" : ".js" : d && (!ext || !cm) ? m : (d + ext + "." + cm.toLowerCase() + "js");
|
|
5
|
+
});
|
|
6
|
+
}
|
|
7
|
+
return path;
|
|
8
|
+
};
|
|
4
9
|
const Type = {
|
|
5
10
|
Object(properties) {
|
|
6
11
|
const required = Object.entries(properties)
|
|
@@ -34,16 +39,97 @@ const Type = {
|
|
|
34
39
|
return { ...schema, optional: true };
|
|
35
40
|
},
|
|
36
41
|
};
|
|
37
|
-
const
|
|
42
|
+
const RUNTIME_EXTENSION = import.meta.url.endsWith(".ts") ? ".ts" : ".js";
|
|
38
43
|
const WATCH_ENABLED = process.env.PI_KNOWLEDGE_WATCH === "true";
|
|
39
44
|
const AUTO_INJECT = process.env.PI_KNOWLEDGE_AUTO_INJECT === "true";
|
|
45
|
+
let runtime;
|
|
46
|
+
let runtimePromise;
|
|
47
|
+
let initializePromise;
|
|
48
|
+
let initialized = false;
|
|
49
|
+
let disposePromise;
|
|
50
|
+
function runtimeModule(modulePath) {
|
|
51
|
+
return `${modulePath}${RUNTIME_EXTENSION}`;
|
|
52
|
+
}
|
|
53
|
+
async function loadRuntime() {
|
|
54
|
+
if (disposePromise)
|
|
55
|
+
await disposePromise;
|
|
56
|
+
if (runtime)
|
|
57
|
+
return runtime;
|
|
58
|
+
runtimePromise ??= Promise.all([
|
|
59
|
+
import(__rewriteRelativeImportExtension(runtimeModule("./src/engine"))),
|
|
60
|
+
import(__rewriteRelativeImportExtension(runtimeModule("./src/storage/sqlite"))),
|
|
61
|
+
import(__rewriteRelativeImportExtension(runtimeModule("./src/watcher/file-watcher"))),
|
|
62
|
+
]).then(([engineModule, storage, watcher]) => {
|
|
63
|
+
runtime = { engine: new engineModule.KnowledgeEngine(), storage, watcher };
|
|
64
|
+
return runtime;
|
|
65
|
+
});
|
|
66
|
+
return runtimePromise;
|
|
67
|
+
}
|
|
68
|
+
async function ensureInitialized() {
|
|
69
|
+
if (disposePromise)
|
|
70
|
+
await disposePromise;
|
|
71
|
+
if (initialized && runtime)
|
|
72
|
+
return runtime;
|
|
73
|
+
initializePromise ??= (async () => {
|
|
74
|
+
const loaded = await loadRuntime();
|
|
75
|
+
await loaded.engine.initialize(loaded.storage.getDefaultKnowledgeDir());
|
|
76
|
+
initialized = true;
|
|
77
|
+
return loaded;
|
|
78
|
+
})();
|
|
79
|
+
try {
|
|
80
|
+
return await initializePromise;
|
|
81
|
+
}
|
|
82
|
+
catch (error) {
|
|
83
|
+
initializePromise = undefined;
|
|
84
|
+
throw error;
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
async function disposeRuntime() {
|
|
88
|
+
if (disposePromise)
|
|
89
|
+
return disposePromise;
|
|
90
|
+
const pendingInitialize = initializePromise;
|
|
91
|
+
const pendingRuntime = runtimePromise;
|
|
92
|
+
disposePromise = (async () => {
|
|
93
|
+
let loaded = runtime;
|
|
94
|
+
if (!loaded && pendingInitialize) {
|
|
95
|
+
try {
|
|
96
|
+
loaded = await pendingInitialize;
|
|
97
|
+
}
|
|
98
|
+
catch {
|
|
99
|
+
loaded = runtime;
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
if (!loaded && pendingRuntime) {
|
|
103
|
+
try {
|
|
104
|
+
loaded = await pendingRuntime;
|
|
105
|
+
}
|
|
106
|
+
catch {
|
|
107
|
+
loaded = runtime;
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
initialized = false;
|
|
111
|
+
initializePromise = undefined;
|
|
112
|
+
runtime = undefined;
|
|
113
|
+
runtimePromise = undefined;
|
|
114
|
+
if (!loaded)
|
|
115
|
+
return;
|
|
116
|
+
loaded.watcher.stopAllWatchers();
|
|
117
|
+
await loaded.engine.dispose({ disposeModels: false });
|
|
118
|
+
})();
|
|
119
|
+
try {
|
|
120
|
+
await disposePromise;
|
|
121
|
+
}
|
|
122
|
+
finally {
|
|
123
|
+
disposePromise = undefined;
|
|
124
|
+
}
|
|
125
|
+
}
|
|
40
126
|
export default function (pi) {
|
|
41
127
|
pi.on("session_start", async () => {
|
|
42
|
-
await
|
|
128
|
+
const { engine, watcher } = await ensureInitialized();
|
|
43
129
|
if (WATCH_ENABLED) {
|
|
44
130
|
for (const kb of engine.list()) {
|
|
45
131
|
if (kb.source_path && kb.source_type === "directory") {
|
|
46
|
-
startWatcher(kb.id, kb.source_path, (kbId) => {
|
|
132
|
+
watcher.startWatcher(kb.id, kb.source_path, (kbId) => {
|
|
47
133
|
engine.update(kbId).catch(() => { });
|
|
48
134
|
});
|
|
49
135
|
}
|
|
@@ -51,23 +137,23 @@ export default function (pi) {
|
|
|
51
137
|
}
|
|
52
138
|
});
|
|
53
139
|
pi.on("session_shutdown", async () => {
|
|
54
|
-
|
|
55
|
-
await engine.dispose({ disposeModels: false });
|
|
140
|
+
await disposeRuntime();
|
|
56
141
|
});
|
|
57
142
|
// Auto-inject: search KB for relevant context before each LLM call (opt-in)
|
|
58
143
|
if (AUTO_INJECT) {
|
|
59
144
|
pi.on("context", async (event) => {
|
|
60
|
-
const kbs = engine.list();
|
|
61
|
-
if (kbs.length === 0)
|
|
62
|
-
return;
|
|
63
|
-
// Find last user message
|
|
64
|
-
const lastUser = [...event.messages].reverse().find((m) => m.role === "user");
|
|
65
|
-
if (!lastUser)
|
|
66
|
-
return;
|
|
67
|
-
const text = "content" in lastUser && typeof lastUser.content === "string" ? lastUser.content : "";
|
|
68
|
-
if (!text || text.length < 5)
|
|
69
|
-
return;
|
|
70
145
|
try {
|
|
146
|
+
const { engine } = await ensureInitialized();
|
|
147
|
+
const kbs = engine.list();
|
|
148
|
+
if (kbs.length === 0)
|
|
149
|
+
return;
|
|
150
|
+
// Find last user message
|
|
151
|
+
const lastUser = [...event.messages].reverse().find((m) => m.role === "user");
|
|
152
|
+
if (!lastUser)
|
|
153
|
+
return;
|
|
154
|
+
const text = "content" in lastUser && typeof lastUser.content === "string" ? lastUser.content : "";
|
|
155
|
+
if (!text || text.length < 5)
|
|
156
|
+
return;
|
|
71
157
|
const results = await engine.search(text, { mode: "fast", limit: 3 });
|
|
72
158
|
if (results.results.length === 0)
|
|
73
159
|
return;
|
|
@@ -81,7 +167,8 @@ export default function (pi) {
|
|
|
81
167
|
}
|
|
82
168
|
});
|
|
83
169
|
}
|
|
84
|
-
pi.on("before_agent_start", (event) => {
|
|
170
|
+
pi.on("before_agent_start", async (event) => {
|
|
171
|
+
const { engine } = await ensureInitialized();
|
|
85
172
|
const kbs = engine.list();
|
|
86
173
|
if (kbs.length === 0)
|
|
87
174
|
return undefined;
|
|
@@ -112,7 +199,8 @@ export default function (pi) {
|
|
|
112
199
|
}))),
|
|
113
200
|
exclude_paths: Type.Optional(Type.Array(Type.String({ description: "Relative paths under a directory source to exclude from this plan" }))),
|
|
114
201
|
}),
|
|
115
|
-
execute(_id, params) {
|
|
202
|
+
async execute(_id, params) {
|
|
203
|
+
const { engine } = await ensureInitialized();
|
|
116
204
|
const { source, include_suggested_text, include_paths, exclude_paths } = params;
|
|
117
205
|
const plan = engine.plan(source, {
|
|
118
206
|
include_suggested_text: include_suggested_text === true,
|
|
@@ -170,6 +258,7 @@ export default function (pi) {
|
|
|
170
258
|
exclude_paths: Type.Optional(Type.Array(Type.String({ description: "Relative paths under a directory source to exclude from this KB" }))),
|
|
171
259
|
}),
|
|
172
260
|
async execute(_id, params, _signal, onUpdate) {
|
|
261
|
+
const { engine, watcher } = await ensureInitialized();
|
|
173
262
|
const { source, name, include_suggested_text, include_paths, exclude_paths } = params;
|
|
174
263
|
const { kb, chunkCount } = await engine.add(source, name, (msg) => {
|
|
175
264
|
onUpdate?.({ content: [{ type: "text", text: msg }] });
|
|
@@ -184,7 +273,7 @@ export default function (pi) {
|
|
|
184
273
|
});
|
|
185
274
|
// Start watcher for new directory KB
|
|
186
275
|
if (WATCH_ENABLED && kb.source_path && kb.source_type === "directory") {
|
|
187
|
-
startWatcher(kb.id, kb.source_path, (kbId) => {
|
|
276
|
+
watcher.startWatcher(kb.id, kb.source_path, (kbId) => {
|
|
188
277
|
engine.update(kbId).catch(() => { });
|
|
189
278
|
});
|
|
190
279
|
}
|
|
@@ -231,6 +320,7 @@ export default function (pi) {
|
|
|
231
320
|
diagnostics: Type.Optional(Type.Boolean({ description: "Include ranking diagnostics and mode/fallback details in the result" })),
|
|
232
321
|
}),
|
|
233
322
|
async execute(_id, params) {
|
|
323
|
+
const { engine } = await ensureInitialized();
|
|
234
324
|
const { query, mode, limit, kb_id, offset, file_type, diversity, diagnostics } = params;
|
|
235
325
|
const filters = file_type ? { file_type } : undefined;
|
|
236
326
|
const response = await engine.search(query, { mode, limit, kb_id, offset, filters, diversity });
|
|
@@ -275,6 +365,7 @@ export default function (pi) {
|
|
|
275
365
|
target: Type.String({ description: "KB name or ID to update" }),
|
|
276
366
|
}),
|
|
277
367
|
async execute(_id, params, _signal, onUpdate) {
|
|
368
|
+
const { engine } = await ensureInitialized();
|
|
278
369
|
const { added, removed, unchanged } = await engine.update(params.target, (msg) => {
|
|
279
370
|
onUpdate?.({ content: [{ type: "text", text: msg }] });
|
|
280
371
|
}, _signal);
|
|
@@ -289,11 +380,12 @@ export default function (pi) {
|
|
|
289
380
|
description: "Show knowledge engine status with health diagnostics: staleness, orphans, and coverage",
|
|
290
381
|
parameters: Type.Object({}),
|
|
291
382
|
async execute() {
|
|
383
|
+
const { engine, storage, watcher } = await ensureInitialized();
|
|
292
384
|
const kbs = engine.list();
|
|
293
|
-
const watchCount = getActiveWatcherCount();
|
|
385
|
+
const watchCount = watcher.getActiveWatcherCount();
|
|
294
386
|
const diagnostics = engine.diagnose();
|
|
295
387
|
const lines = [
|
|
296
|
-
`Storage: ${getDefaultKnowledgeDir()}`,
|
|
388
|
+
`Storage: ${storage.getDefaultKnowledgeDir()}`,
|
|
297
389
|
`Knowledge bases: ${kbs.length}`,
|
|
298
390
|
`Active watchers: ${watchCount}`,
|
|
299
391
|
"",
|
|
@@ -354,6 +446,7 @@ export default function (pi) {
|
|
|
354
446
|
promptSnippet: "Diagnose knowledge base health and recommended actions",
|
|
355
447
|
parameters: Type.Object({}),
|
|
356
448
|
async execute() {
|
|
449
|
+
const { engine } = await ensureInitialized();
|
|
357
450
|
const report = engine.doctor();
|
|
358
451
|
const lines = [`Health score: ${report.health_score}/100`, report.summary, ""];
|
|
359
452
|
if (report.issues.length === 0) {
|
|
@@ -375,6 +468,7 @@ export default function (pi) {
|
|
|
375
468
|
description: "List all indexed knowledge bases",
|
|
376
469
|
parameters: Type.Object({}),
|
|
377
470
|
async execute() {
|
|
471
|
+
const { engine } = await ensureInitialized();
|
|
378
472
|
const kbs = engine.list();
|
|
379
473
|
if (kbs.length === 0)
|
|
380
474
|
return { content: [{ type: "text", text: "No knowledge bases." }] };
|
|
@@ -390,6 +484,7 @@ export default function (pi) {
|
|
|
390
484
|
target: Type.String({ description: "KB name or ID to remove" }),
|
|
391
485
|
}),
|
|
392
486
|
async execute(_id, params) {
|
|
487
|
+
const { engine } = await ensureInitialized();
|
|
393
488
|
const ok = engine.remove(params.target);
|
|
394
489
|
return { content: [{ type: "text", text: ok ? "Removed." : "Not found." }] };
|
|
395
490
|
},
|
|
@@ -403,6 +498,7 @@ export default function (pi) {
|
|
|
403
498
|
output: Type.String({ description: "Output file path (.jsonl)" }),
|
|
404
499
|
}),
|
|
405
500
|
async execute(_id, params) {
|
|
501
|
+
const { engine } = await ensureInitialized();
|
|
406
502
|
const { target, output } = params;
|
|
407
503
|
const count = await engine.exportKB(target, output);
|
|
408
504
|
return { content: [{ type: "text", text: `Exported ${count} chunks to ${output}` }] };
|
|
@@ -416,6 +512,7 @@ export default function (pi) {
|
|
|
416
512
|
input: Type.String({ description: "Input JSONL file path" }),
|
|
417
513
|
}),
|
|
418
514
|
async execute(_id, params, _signal, onUpdate) {
|
|
515
|
+
const { engine } = await ensureInitialized();
|
|
419
516
|
const { kb, chunkCount } = await engine.importKB(params.input, (msg) => {
|
|
420
517
|
onUpdate?.({ content: [{ type: "text", text: msg }] });
|
|
421
518
|
}, _signal);
|
|
@@ -428,6 +525,7 @@ export default function (pi) {
|
|
|
428
525
|
description: "Remove all knowledge bases",
|
|
429
526
|
parameters: Type.Object({}),
|
|
430
527
|
async execute() {
|
|
528
|
+
const { engine } = await ensureInitialized();
|
|
431
529
|
engine.clear();
|
|
432
530
|
return { content: [{ type: "text", text: "All knowledge bases cleared." }] };
|
|
433
531
|
},
|
|
@@ -7,6 +7,9 @@ const idleWaiters = [];
|
|
|
7
7
|
const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
|
|
8
8
|
const EMBEDDING_CONFIG = process.env.PI_KNOWLEDGE_EMBEDDING ?? "local:multilingual-e5-small";
|
|
9
9
|
const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
|
|
10
|
+
const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
|
|
11
|
+
const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
12
|
+
const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
|
|
10
13
|
function clearIdleTimer() {
|
|
11
14
|
if (disposeTimer)
|
|
12
15
|
clearTimeout(disposeTimer);
|
|
@@ -56,18 +59,27 @@ export async function prepareForShutdown() {
|
|
|
56
59
|
}
|
|
57
60
|
async function embedViaAPI(texts, prefix) {
|
|
58
61
|
const [provider, model] = EMBEDDING_CONFIG.split(":");
|
|
62
|
+
const configuredMaxChars = Number(process.env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS ?? DEFAULT_API_MAX_EMBED_CHARS);
|
|
63
|
+
const maxChars = Number.isFinite(configuredMaxChars) && configuredMaxChars > 0 ? configuredMaxChars : DEFAULT_API_MAX_EMBED_CHARS;
|
|
59
64
|
const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
|
|
65
|
+
const safeTexts = prefixedTexts.map((text) => (text.length > maxChars ? text.slice(0, maxChars) : text));
|
|
60
66
|
if (provider === "openai") {
|
|
61
67
|
const apiKey = process.env.OPENAI_API_KEY;
|
|
62
68
|
if (!apiKey)
|
|
63
69
|
throw new Error("OPENAI_API_KEY required for openai embedding");
|
|
64
|
-
const
|
|
70
|
+
const baseUrl = process.env.PI_KNOWLEDGE_EMBEDDING_BASE_URL?.trim() ||
|
|
71
|
+
process.env.OPENAI_BASE_URL?.trim() ||
|
|
72
|
+
DEFAULT_OPENAI_BASE_URL;
|
|
73
|
+
const endpoint = new URL("embeddings", `${baseUrl.replace(/\/+$/, "")}/`);
|
|
74
|
+
const res = await fetch(endpoint, {
|
|
65
75
|
method: "POST",
|
|
66
76
|
headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
|
|
67
|
-
body: JSON.stringify({ input:
|
|
77
|
+
body: JSON.stringify({ input: safeTexts, model: model || "text-embedding-3-small" }),
|
|
68
78
|
});
|
|
69
|
-
if (!res.ok)
|
|
70
|
-
|
|
79
|
+
if (!res.ok) {
|
|
80
|
+
const detail = await res.text().catch(() => "");
|
|
81
|
+
throw new Error(`OpenAI embedding API error: ${res.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
|
|
82
|
+
}
|
|
71
83
|
const data = (await res.json());
|
|
72
84
|
return data.data.map((d) => new Float32Array(d.embedding));
|
|
73
85
|
}
|
|
@@ -78,8 +90,10 @@ export async function embedTexts(texts, prefix, signal) {
|
|
|
78
90
|
try {
|
|
79
91
|
return await embedViaAPI(texts, prefix);
|
|
80
92
|
}
|
|
81
|
-
catch {
|
|
82
|
-
|
|
93
|
+
catch (error) {
|
|
94
|
+
if (!API_FALLBACK_TO_LOCAL)
|
|
95
|
+
throw error;
|
|
96
|
+
console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
|
|
83
97
|
}
|
|
84
98
|
}
|
|
85
99
|
beginRun();
|
|
@@ -381,6 +381,18 @@ export function chunkMarkdown(content, filePath) {
|
|
|
381
381
|
breadcrumb: currentBreadcrumb(),
|
|
382
382
|
}));
|
|
383
383
|
}
|
|
384
|
+
function pushOversizedMarkdownParagraph(para, start) {
|
|
385
|
+
let offset = 0;
|
|
386
|
+
let sliceStart = start;
|
|
387
|
+
while (offset < para.length) {
|
|
388
|
+
const slice = para.slice(offset, offset + MAX_TEXT_CHUNK_CHARS).trim();
|
|
389
|
+
if (slice.length >= 50) {
|
|
390
|
+
pushMarkdownChunk(slice, sliceStart, sliceStart + slice.split("\n").length);
|
|
391
|
+
}
|
|
392
|
+
sliceStart += slice.split("\n").length;
|
|
393
|
+
offset += MAX_TEXT_CHUNK_CHARS;
|
|
394
|
+
}
|
|
395
|
+
}
|
|
384
396
|
function flush(endLine) {
|
|
385
397
|
const text = sectionLines.join("\n").trim();
|
|
386
398
|
if (text.length < 50)
|
|
@@ -390,6 +402,16 @@ export function chunkMarkdown(content, filePath) {
|
|
|
390
402
|
let buffer = [];
|
|
391
403
|
let bufferStart = startLine;
|
|
392
404
|
for (const para of paragraphs) {
|
|
405
|
+
if (para.length > MAX_TEXT_CHUNK_CHARS) {
|
|
406
|
+
const oversizedText = buffer.length > 0 ? [...buffer, para].join("\n\n") : para;
|
|
407
|
+
const oversizedStart = buffer.length > 0 ? bufferStart : endLine;
|
|
408
|
+
if (buffer.length > 0) {
|
|
409
|
+
buffer = [];
|
|
410
|
+
}
|
|
411
|
+
pushOversizedMarkdownParagraph(oversizedText, oversizedStart);
|
|
412
|
+
bufferStart = endLine;
|
|
413
|
+
continue;
|
|
414
|
+
}
|
|
393
415
|
const next = [...buffer, para].join("\n\n");
|
|
394
416
|
if (estimateTokens(next) > MARKDOWN_TARGET_TOKENS && buffer.length > 0) {
|
|
395
417
|
pushMarkdownChunk(buffer.join("\n\n"), bufferStart, endLine);
|
|
@@ -4,6 +4,7 @@ import { fileURLToPath } from "node:url";
|
|
|
4
4
|
let worker = null;
|
|
5
5
|
let nextRequestId = 1;
|
|
6
6
|
const pending = new Map();
|
|
7
|
+
const WORKER_STDERR_TAIL_CHARS = 4_000;
|
|
7
8
|
function rejectPending(error) {
|
|
8
9
|
for (const request of pending.values()) {
|
|
9
10
|
request.reject(error);
|
|
@@ -24,6 +25,17 @@ function getNodeExecPath() {
|
|
|
24
25
|
const execName = basename(process.execPath).toLowerCase();
|
|
25
26
|
return execName === "node" || execName === "node.exe" ? process.execPath : "node";
|
|
26
27
|
}
|
|
28
|
+
function appendWorkerStderr(current, chunk) {
|
|
29
|
+
const next = `${current}${chunk.toString("utf-8")}`;
|
|
30
|
+
return next.length > WORKER_STDERR_TAIL_CHARS ? next.slice(-WORKER_STDERR_TAIL_CHARS) : next;
|
|
31
|
+
}
|
|
32
|
+
function formatWorkerExitError(code, signal, stderrTail) {
|
|
33
|
+
const reason = `Model worker exited before responding (code ${code ?? "null"}, signal ${signal ?? "null"})`;
|
|
34
|
+
const stderr = stderrTail.trim();
|
|
35
|
+
if (!stderr)
|
|
36
|
+
return new Error(`${reason}. Set PI_KNOWLEDGE_NODE_PATH to a working Node binary if Pi is not running under Node.`);
|
|
37
|
+
return new Error(`${reason}. Worker stderr:\n${stderr}`);
|
|
38
|
+
}
|
|
27
39
|
function getWorker() {
|
|
28
40
|
if (worker?.connected)
|
|
29
41
|
return worker;
|
|
@@ -31,10 +43,14 @@ function getWorker() {
|
|
|
31
43
|
worker = fork(workerPath, {
|
|
32
44
|
execPath: getNodeExecPath(),
|
|
33
45
|
execArgv: getWorkerExecArgv(),
|
|
34
|
-
stdio: ["ignore", "ignore", "
|
|
46
|
+
stdio: ["ignore", "ignore", "pipe", "ipc"],
|
|
35
47
|
env: process.env,
|
|
36
48
|
});
|
|
37
49
|
const child = worker;
|
|
50
|
+
let stderrTail = "";
|
|
51
|
+
child.stderr?.on("data", (chunk) => {
|
|
52
|
+
stderrTail = appendWorkerStderr(stderrTail, chunk);
|
|
53
|
+
});
|
|
38
54
|
worker.on("message", (message) => {
|
|
39
55
|
const request = pending.get(message.id);
|
|
40
56
|
if (!request)
|
|
@@ -52,7 +68,7 @@ function getWorker() {
|
|
|
52
68
|
return;
|
|
53
69
|
worker = null;
|
|
54
70
|
if (pending.size > 0) {
|
|
55
|
-
rejectPending(
|
|
71
|
+
rejectPending(formatWorkerExitError(code, signal, stderrTail));
|
|
56
72
|
}
|
|
57
73
|
});
|
|
58
74
|
worker.on("error", (error) => {
|
package/dist/src/model-worker.js
CHANGED
|
@@ -5,11 +5,19 @@ let rerankerPipeline = null;
|
|
|
5
5
|
function getModelCacheDir() {
|
|
6
6
|
return process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? join(getDefaultKnowledgeDir(), "models");
|
|
7
7
|
}
|
|
8
|
+
function configureTransformersEnv(env) {
|
|
9
|
+
const cacheDir = getModelCacheDir();
|
|
10
|
+
env.cacheDir = cacheDir;
|
|
11
|
+
if (process.env.PI_KNOWLEDGE_OFFLINE === "true") {
|
|
12
|
+
env.allowRemoteModels = false;
|
|
13
|
+
env.localModelPath = cacheDir;
|
|
14
|
+
}
|
|
15
|
+
}
|
|
8
16
|
async function loadEmbeddingPipeline() {
|
|
9
17
|
if (embeddingPipeline)
|
|
10
18
|
return embeddingPipeline;
|
|
11
19
|
const { pipeline, env } = await import("@huggingface/transformers");
|
|
12
|
-
env
|
|
20
|
+
configureTransformersEnv(env);
|
|
13
21
|
const createPipeline = pipeline;
|
|
14
22
|
const loaded = (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
|
|
15
23
|
quantized: true,
|
|
@@ -22,7 +30,7 @@ async function loadRerankerPipeline() {
|
|
|
22
30
|
if (rerankerPipeline)
|
|
23
31
|
return rerankerPipeline;
|
|
24
32
|
const { pipeline, env } = await import("@huggingface/transformers");
|
|
25
|
-
env
|
|
33
|
+
configureTransformersEnv(env);
|
|
26
34
|
const createPipeline = pipeline;
|
|
27
35
|
const loaded = (await createPipeline("text-classification", "Xenova/ms-marco-MiniLM-L-4-v2"));
|
|
28
36
|
rerankerPipeline = loaded;
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
import { randomUUID } from "node:crypto";
|
|
2
2
|
import { existsSync, mkdirSync } from "node:fs";
|
|
3
|
+
import { createRequire } from "node:module";
|
|
3
4
|
import { homedir } from "node:os";
|
|
4
5
|
import { basename, dirname, join } from "node:path";
|
|
5
|
-
import
|
|
6
|
+
import { fileURLToPath } from "node:url";
|
|
6
7
|
const SCHEMA_VERSION = 3;
|
|
7
8
|
const SCHEMA_SQL = `
|
|
8
9
|
CREATE TABLE IF NOT EXISTS schema_version (
|
|
@@ -82,6 +83,74 @@ CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
|
|
|
82
83
|
INSERT INTO chunks_fts(rowid, content_tokenized) VALUES (new.rowid, new.content_tokenized);
|
|
83
84
|
END;
|
|
84
85
|
`;
|
|
86
|
+
const localRequire = createRequire(import.meta.url);
|
|
87
|
+
let databaseConstructor;
|
|
88
|
+
let betterSqlite3PackageRoot;
|
|
89
|
+
function findBetterSqlite3PackageRoot() {
|
|
90
|
+
if (betterSqlite3PackageRoot)
|
|
91
|
+
return betterSqlite3PackageRoot;
|
|
92
|
+
let current = dirname(fileURLToPath(import.meta.url));
|
|
93
|
+
while (true) {
|
|
94
|
+
const candidate = join(current, "node_modules", "better-sqlite3");
|
|
95
|
+
if (existsSync(join(candidate, "package.json"))) {
|
|
96
|
+
betterSqlite3PackageRoot = candidate;
|
|
97
|
+
return betterSqlite3PackageRoot;
|
|
98
|
+
}
|
|
99
|
+
const parent = dirname(current);
|
|
100
|
+
if (parent === current)
|
|
101
|
+
return undefined;
|
|
102
|
+
current = parent;
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
function findBetterSqlite3Entry() {
|
|
106
|
+
const packageRoot = findBetterSqlite3PackageRoot();
|
|
107
|
+
return packageRoot ? join(packageRoot, "lib", "index.js") : undefined;
|
|
108
|
+
}
|
|
109
|
+
function findBetterSqlite3NativeBinding() {
|
|
110
|
+
const packageRoot = findBetterSqlite3PackageRoot();
|
|
111
|
+
if (!packageRoot)
|
|
112
|
+
return undefined;
|
|
113
|
+
const candidate = join(packageRoot, "build", "Release", "better_sqlite3.node");
|
|
114
|
+
return existsSync(candidate) ? candidate : undefined;
|
|
115
|
+
}
|
|
116
|
+
function betterSqlite3PackageName() {
|
|
117
|
+
return ["better", "sqlite3"].join("-");
|
|
118
|
+
}
|
|
119
|
+
function bunSqliteModuleName() {
|
|
120
|
+
return ["bun", "sqlite"].join(":");
|
|
121
|
+
}
|
|
122
|
+
function isBunRuntime() {
|
|
123
|
+
return typeof process.versions.bun === "string";
|
|
124
|
+
}
|
|
125
|
+
function loadDatabaseConstructor() {
|
|
126
|
+
if (databaseConstructor)
|
|
127
|
+
return databaseConstructor;
|
|
128
|
+
if (isBunRuntime()) {
|
|
129
|
+
const bunSqlite = localRequire(bunSqliteModuleName());
|
|
130
|
+
databaseConstructor = bunSqlite.Database;
|
|
131
|
+
return databaseConstructor;
|
|
132
|
+
}
|
|
133
|
+
try {
|
|
134
|
+
databaseConstructor = localRequire(betterSqlite3PackageName());
|
|
135
|
+
return databaseConstructor;
|
|
136
|
+
}
|
|
137
|
+
catch (error) {
|
|
138
|
+
const entry = findBetterSqlite3Entry();
|
|
139
|
+
if (entry) {
|
|
140
|
+
databaseConstructor = localRequire(entry);
|
|
141
|
+
return databaseConstructor;
|
|
142
|
+
}
|
|
143
|
+
throw error;
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
function applyPragma(db, pragma) {
|
|
147
|
+
const sqlite = db;
|
|
148
|
+
if (typeof sqlite.pragma === "function") {
|
|
149
|
+
sqlite.pragma(pragma);
|
|
150
|
+
return;
|
|
151
|
+
}
|
|
152
|
+
db.exec(`PRAGMA ${pragma}`);
|
|
153
|
+
}
|
|
85
154
|
export function getDefaultKnowledgeDir() {
|
|
86
155
|
const explicit = process.env.PI_KNOWLEDGE_DIR?.trim() || process.env.OMP_KNOWLEDGE_DIR?.trim();
|
|
87
156
|
if (explicit)
|
|
@@ -116,10 +185,12 @@ export function openDatabase(knowledgeDir) {
|
|
|
116
185
|
mkdirSync(dir, { recursive: true });
|
|
117
186
|
}
|
|
118
187
|
const dbPath = join(dir, "knowledge.db");
|
|
119
|
-
const
|
|
120
|
-
|
|
121
|
-
db
|
|
122
|
-
db
|
|
188
|
+
const Database = loadDatabaseConstructor();
|
|
189
|
+
const nativeBinding = isBunRuntime() ? undefined : findBetterSqlite3NativeBinding();
|
|
190
|
+
const db = nativeBinding ? new Database(dbPath, { nativeBinding }) : new Database(dbPath);
|
|
191
|
+
applyPragma(db, "journal_mode = WAL");
|
|
192
|
+
applyPragma(db, "busy_timeout = 5000");
|
|
193
|
+
applyPragma(db, "foreign_keys = ON");
|
|
123
194
|
const hasVersion = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name='schema_version'").get();
|
|
124
195
|
if (!hasVersion) {
|
|
125
196
|
db.exec(SCHEMA_SQL);
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Configuration
|
|
2
|
+
|
|
3
|
+
`pi-knowledge` is local-first by default, but several environment variables are available for runtime selection, compatibility, and release validation.
|
|
4
|
+
|
|
5
|
+
## Storage and Host Runtime
|
|
6
|
+
|
|
7
|
+
| Variable | Default | Purpose |
|
|
8
|
+
|----------|---------|---------|
|
|
9
|
+
| `PI_KNOWLEDGE_DIR` | host-derived knowledge dir | Override the directory that stores `knowledge.db`, vectors, and model cache parent data for Pi and OMP. Takes precedence over `OMP_KNOWLEDGE_DIR`. |
|
|
10
|
+
| `OMP_KNOWLEDGE_DIR` | host-derived knowledge dir | Override the knowledge directory when running under OMP. Used when `PI_KNOWLEDGE_DIR` is not set. |
|
|
11
|
+
| `PI_CODING_AGENT_DIR` | unset | Derive the Pi host root from a configured agent directory. Useful for isolated installs and validation. |
|
|
12
|
+
| `OMP_CODING_AGENT_DIR` | unset | Derive the OMP host root from a configured agent directory. Used when `PI_CODING_AGENT_DIR` is not set. |
|
|
13
|
+
| `OMP_PROFILE` | unset | Treat the current process as OMP-hosted for default path selection. |
|
|
14
|
+
| `PI_KNOWLEDGE_MODEL_CACHE_DIR` | `<knowledge-dir>/models` | Override the local Transformers.js model cache directory. |
|
|
15
|
+
| `PI_KNOWLEDGE_NODE_PATH` | current Node when possible, otherwise `node` | Override the Node executable used to fork the isolated model worker. |
|
|
16
|
+
|
|
17
|
+
Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under OMP. For the default home OMP root, `pi-knowledge` preserves an existing legacy `~/.pi/knowledge` directory when `~/.omp/knowledge` does not exist, so existing Pi knowledge bases remain visible during migration.
|
|
18
|
+
|
|
19
|
+
## Embeddings
|
|
20
|
+
|
|
21
|
+
| Variable | Default | Purpose |
|
|
22
|
+
|----------|---------|---------|
|
|
23
|
+
| `PI_KNOWLEDGE_EMBEDDING` | `local:multilingual-e5-small` | Select the embedding provider and model. Supported values are local embeddings and `openai:<model>`. |
|
|
24
|
+
| `OPENAI_API_KEY` | unset | API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Some local OpenAI-compatible servers accept a placeholder value. |
|
|
25
|
+
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
26
|
+
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
27
|
+
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
28
|
+
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings after API failures. Without this, API failures are surfaced. |
|
|
29
|
+
| `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Set to `true` to opt into native ONNX idle disposal. Disabled by default for stable shutdown. |
|
|
30
|
+
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle-dispose timer used only when native idle disposal is enabled. Mainly for lifecycle stress tests. |
|
|
31
|
+
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
32
|
+
|
|
33
|
+
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended.
|
|
34
|
+
|
|
35
|
+
## Runtime Features and Diagnostics
|
|
36
|
+
|
|
37
|
+
| Variable | Default | Purpose |
|
|
38
|
+
|----------|---------|---------|
|
|
39
|
+
| `PI_KNOWLEDGE_WATCH` | unset | Set to `true` to start file watchers for directory KBs. The polling fallback remains available when native `fs.watch` fails. |
|
|
40
|
+
| `PI_KNOWLEDGE_AUTO_INJECT` | unset | Set to `true` to auto-search KB context before model calls. This is opt-in. |
|
|
41
|
+
| `PI_KNOWLEDGE_STALE_INDEXING_MS` | built-in stale threshold | Override the stale indexing threshold used by diagnostics and `knowledge_doctor`. |
|
|
42
|
+
|
|
43
|
+
## Test and Release Fixtures
|
|
44
|
+
|
|
45
|
+
| Variable | Default | Purpose |
|
|
46
|
+
|----------|---------|---------|
|
|
47
|
+
| `PI_KNOWLEDGE_E2E_PDF` | unset | Path to an external PDF fixture for release-grade e2e extraction coverage. |
|
|
48
|
+
| `PI_KNOWLEDGE_E2E_DOCX` | unset | Path to an external DOCX fixture for release-grade e2e extraction coverage. |
|
|
49
|
+
|
|
50
|
+
When `PI_KNOWLEDGE_E2E_PDF` or `PI_KNOWLEDGE_E2E_DOCX` is unset, `npm run test:e2e` is only a smoke gate for those document formats. Do not commit fixture files, fixture paths, extracted text, or snapshots.
|
|
51
|
+
|
|
52
|
+
## Pi and OMP Support
|
|
53
|
+
|
|
54
|
+
`pi-knowledge` supports direct Pi extension loading through `pi -e ./extension.js` and OMP-compatible loading through the same packaged entry shim. The root `extension.js` and `index.ts` stay startup-light so install-time validation can inspect the extension without resolving native runtime dependencies such as `better-sqlite3` or `onnxruntime-node`.
|
|
55
|
+
|
|
56
|
+
Compatibility guarantees:
|
|
57
|
+
|
|
58
|
+
- The packaged `extension.js` loads built `dist/index.js` when present and falls back to source `index.ts` for local development.
|
|
59
|
+
- Runtime modules are imported lazily, after the extension is actually used.
|
|
60
|
+
- Local embedding and reranking models run in an isolated model worker, not in the Pi or OMP TUI process.
|
|
61
|
+
- Native SQLite loading includes a fallback for hoisted plugin dependency layouts.
|
|
62
|
+
- OMP path resolution can use `OMP_KNOWLEDGE_DIR`, `OMP_CODING_AGENT_DIR`, and `OMP_PROFILE`.
|
|
63
|
+
- Existing default Pi knowledge data can remain visible under the default OMP home root through the legacy `~/.pi/knowledge` fallback described above.
|
|
64
|
+
|
|
65
|
+
Release validation should include both Pi and OMP when compatibility-sensitive code changes touch entry shims, native dependency loading, storage path resolution, model-worker startup, or lifecycle shutdown.
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# 離線模式部署
|
|
2
|
+
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## 1. 預下載 Model
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
# 有網路時觸發一次 knowledge_add → model 自動下載到 ~/.pi/knowledge/models/
|
|
9
|
+
# 或手動從 https://huggingface.co/Xenova/multilingual-e5-small/tree/main/onnx 下載
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 2. 配置
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
export PI_KNOWLEDGE_OFFLINE=true
|
|
18
|
+
# Optional: point at a pre-populated model cache
|
|
19
|
+
export PI_KNOWLEDGE_MODEL_CACHE_DIR=/path/to/models
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Extension 行為:
|
|
23
|
+
```typescript
|
|
24
|
+
import { env } from '@huggingface/transformers';
|
|
25
|
+
env.cacheDir = process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? '<knowledge-dir>/models';
|
|
26
|
+
env.allowRemoteModels = false;
|
|
27
|
+
env.localModelPath = env.cacheDir;
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## 3. Proxy 支援
|
|
33
|
+
|
|
34
|
+
@huggingface/transformers 用 Node `fetch()`,支持:
|
|
35
|
+
```bash
|
|
36
|
+
export HTTPS_PROXY=http://proxy.corp.com:8080
|
|
37
|
+
export NODE_EXTRA_CA_CERTS=/path/to/corp-ca.pem
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## 4. Air-Gapped 部署
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
# Build machine (有網路): 觸發下載,打包 ~/.pi/knowledge/models/
|
|
46
|
+
# Target machine: 解壓到同路徑,或設定 PI_KNOWLEDGE_MODEL_CACHE_DIR 指向解壓後路徑
|
|
47
|
+
export PI_KNOWLEDGE_OFFLINE=true
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## 5. 行為差異
|
|
53
|
+
|
|
54
|
+
| 功能 | Online | Offline (model cached) | Offline (no cache) |
|
|
55
|
+
|------|--------|----------------------|-------------------|
|
|
56
|
+
| knowledge_add | ✅ | ✅ | ❌ error |
|
|
57
|
+
| knowledge_search | ✅ | ✅ | ❌ error |
|
|
58
|
+
| Model update | auto-check | skip | skip |
|
package/extension.js
CHANGED
|
@@ -1,7 +1,11 @@
|
|
|
1
1
|
import { existsSync } from "node:fs";
|
|
2
2
|
|
|
3
|
+
function extensionModule(modulePath) {
|
|
4
|
+
return `./${modulePath}`;
|
|
5
|
+
}
|
|
6
|
+
|
|
3
7
|
const module = existsSync(new URL("./dist/index.js", import.meta.url))
|
|
4
|
-
? await import("
|
|
5
|
-
: await import("
|
|
8
|
+
? await import(extensionModule("dist/index.js"))
|
|
9
|
+
: await import(extensionModule("index.ts"));
|
|
6
10
|
|
|
7
11
|
export default module.default;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.6",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -14,6 +14,8 @@
|
|
|
14
14
|
"extension.js",
|
|
15
15
|
"dist/",
|
|
16
16
|
".pi/",
|
|
17
|
+
"docs/configuration.md",
|
|
18
|
+
"docs/offline-mode.md",
|
|
17
19
|
"README.md",
|
|
18
20
|
"CHANGELOG.md",
|
|
19
21
|
"LICENSE"
|