pi-knowledge 0.5.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +12 -12
- package/dist/index.js +60 -25
- package/dist/src/diagnostics/health.js +1 -11
- package/dist/src/engine.js +74 -23
- package/dist/src/indexer/chunker.d.ts +3 -1
- package/dist/src/indexer/chunker.js +57 -48
- package/docs/configuration.md +3 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.5.1] - 2026-07-07
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Indexed PDF/DOC/DOCX files discovered during directory add and update through the document extractors instead of treating them as binary skips.
|
|
7
|
+
- Skipped failed directory document extraction as `extraction_failed` without failing the whole knowledge base.
|
|
8
|
+
- Required explicit `confirm: true` for public destructive `knowledge_remove` and `knowledge_clear` tool calls.
|
|
9
|
+
- Aligned `path_pattern` documentation and tool guidance with the implemented substring filter behavior.
|
|
10
|
+
|
|
3
11
|
## [0.5.0] - 2026-07-07
|
|
4
12
|
|
|
5
13
|
### Added
|
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
22
22
|
## Highlights
|
|
23
23
|
|
|
24
24
|
- **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
|
|
25
|
-
- **Hybrid retrieval**:
|
|
25
|
+
- **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
|
|
26
26
|
- **Code-aware indexing**: AST-aware chunking for TypeScript/JavaScript, Python, Go, Rust, and Java.
|
|
27
27
|
- **Lightweight symbol lookup**: indexes functions, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
|
|
28
28
|
- **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
|
|
@@ -65,7 +65,7 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
|
|
|
65
65
|
- **RAG-native project memory**: follows the Retrieval-Augmented Generation pattern from Lewis et al. 2020: keep source truth outside the model, retrieve it at answer time, and inject only relevant context.
|
|
66
66
|
- **Dense semantic recall**: uses multilingual dense embeddings in the spirit of Dense Passage Retrieval (Karpukhin et al. 2020), so conceptual queries can find code/docs even when wording differs.
|
|
67
67
|
- **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and code symbols with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
|
|
68
|
-
- **Hybrid retrieval with diagnosable scores**: combines BM25 and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
|
|
68
|
+
- **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
|
|
69
69
|
- **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results.
|
|
70
70
|
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
71
71
|
- **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
|
|
@@ -101,16 +101,16 @@ omp install ./pi-knowledge
|
|
|
101
101
|
|
|
102
102
|
| Tool | Description |
|
|
103
103
|
|------|-------------|
|
|
104
|
-
| `knowledge_plan` | Inspect an indexing source before writing a KB; reports scannable
|
|
104
|
+
| `knowledge_plan` | Inspect an indexing source before writing a KB; reports scannable counts, suggested exclusions, and technical skips |
|
|
105
105
|
| `knowledge_add` | Index files, directories, URLs, PDFs, DOCX, or inline text |
|
|
106
|
-
| `knowledge_search` |
|
|
107
|
-
| `knowledge_symbol_search` |
|
|
108
|
-
| `knowledge_remove` | Remove a knowledge base by name or ID |
|
|
109
|
-
| `knowledge_update` | Incrementally re-index changed files in a knowledge base |
|
|
106
|
+
| `knowledge_search` | Fast, semantic, lexical-anchored hybrid, deep, or adaptive search across one or all knowledge bases; supports file-type and path filters |
|
|
107
|
+
| `knowledge_symbol_search` | Lightweight exact or substring lookup for code symbols, route-like handlers, Markdown headings, config keys, and env vars; fall back to `knowledge_search` for methods or uncommon syntax |
|
|
108
|
+
| `knowledge_remove` | Remove a knowledge base by name or ID after `confirm: true` |
|
|
109
|
+
| `knowledge_update` | Incrementally re-index changed files in a source-backed file, directory, or URL knowledge base |
|
|
110
110
|
| `knowledge_show` | List all knowledge bases with stats |
|
|
111
111
|
| `knowledge_status` | Show engine status with health diagnostics (stale, orphans, coverage) |
|
|
112
112
|
| `knowledge_doctor` | Diagnose health score, skipped files, stuck jobs, stale data, and recommended fixes |
|
|
113
|
-
| `knowledge_clear` | Remove all knowledge bases |
|
|
113
|
+
| `knowledge_clear` | Remove all knowledge bases after `confirm: true` |
|
|
114
114
|
| `knowledge_export` | Export a KB to shareable JSONL file |
|
|
115
115
|
| `knowledge_import` | Import a KB from JSONL (re-embeds content) |
|
|
116
116
|
|
|
@@ -118,17 +118,17 @@ omp install ./pi-knowledge
|
|
|
118
118
|
|
|
119
119
|
- `fast`: BM25 keyword search for exact symbols, commands, and identifiers.
|
|
120
120
|
- `semantic`: vector search for conceptual matches.
|
|
121
|
-
- `hybrid`: BM25 + vector search with normalized weighted score fusion.
|
|
121
|
+
- `hybrid`: lexical-anchored BM25 + vector search with normalized weighted score fusion. It requires keyword evidence to avoid low-confidence semantic false positives.
|
|
122
122
|
- `deep`: hybrid retrieval followed by cross-encoder reranking.
|
|
123
123
|
- `adaptive`: hybrid retrieval followed by query-time contextual window expansion around seed chunks. It keeps the matched seed, prefers nearby/query-relevant neighboring chunks, and collapses overlapping windows from the same file.
|
|
124
124
|
- `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
|
|
125
|
-
- `code`, `config`, `errors`, `docs`, `decision`: intent aliases for agents. `code`, `config`, and `errors` bias toward exact lexical/symbol evidence; `docs` and `decision` keep
|
|
125
|
+
- `code`, `config`, `errors`, `docs`, `decision`: intent aliases for agents. `code`, `config`, and `errors` bias toward exact lexical/symbol evidence; `docs` and `decision` keep the intended source type explicit.
|
|
126
126
|
|
|
127
127
|
Mode selection contract:
|
|
128
128
|
|
|
129
|
-
- Start with `hybrid` for most project questions.
|
|
129
|
+
- Start with `hybrid` for most project questions that contain useful lexical anchors.
|
|
130
130
|
- Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
|
|
131
|
-
- Use `semantic` when the query is conceptual
|
|
131
|
+
- Use `semantic` when the query is conceptual, exact terms may differ from indexed wording, or hybrid returns no lexical matches.
|
|
132
132
|
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, or enough context to make a safe edit.
|
|
133
133
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
134
134
|
- If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
|
package/dist/index.js
CHANGED
|
@@ -192,7 +192,7 @@ export default function (pi) {
|
|
|
192
192
|
if (kbs.length === 0)
|
|
193
193
|
return undefined;
|
|
194
194
|
const desc = kbs.map((kb) => `"${kb.name}" (${kb.chunk_count} chunks, ${kb.file_count} files)`).join(", ");
|
|
195
|
-
const guidance = `Available knowledge bases: ${desc}. Use knowledge_search
|
|
195
|
+
const guidance = `Available knowledge bases: ${desc}. Use knowledge_symbol_search for exact symbols/config keys/env vars/routes/headings, then use knowledge_search for broader domain or implementation context.`;
|
|
196
196
|
return {
|
|
197
197
|
systemPrompt: `${event.systemPrompt}\n\n${guidance}`,
|
|
198
198
|
};
|
|
@@ -200,11 +200,11 @@ export default function (pi) {
|
|
|
200
200
|
pi.registerTool({
|
|
201
201
|
name: "knowledge_plan",
|
|
202
202
|
label: "Knowledge Plan",
|
|
203
|
-
description: "Inspect an indexing source without writing a KB, showing scannable
|
|
203
|
+
description: "Inspect an indexing source without writing a KB, showing scannable counts, suggested exclusions, and technical skips",
|
|
204
204
|
promptSnippet: "Plan a knowledge-base indexing scope before calling knowledge_add",
|
|
205
205
|
promptGuidelines: [
|
|
206
206
|
"Use knowledge_plan before knowledge_add for broad directories, large repositories, or sources that may contain private or low-signal text",
|
|
207
|
-
"Show the user suggested exclusions
|
|
207
|
+
"Show the user suggested exclusions, technical skips, and scannable file counts before asking whether to include risky or low-signal text",
|
|
208
208
|
"After the user confirms scope, call knowledge_add with matching include_suggested_text, include_paths, or exclude_paths",
|
|
209
209
|
"Do not use knowledge_plan as a substitute for knowledge_search; it only plans indexing scope",
|
|
210
210
|
],
|
|
@@ -251,14 +251,16 @@ export default function (pi) {
|
|
|
251
251
|
pi.registerTool({
|
|
252
252
|
name: "knowledge_add",
|
|
253
253
|
label: "Knowledge Add",
|
|
254
|
-
description: "Index files, directories, or text into a named knowledge base for semantic search",
|
|
255
|
-
promptSnippet: "Index files/dirs/text into a searchable knowledge base",
|
|
254
|
+
description: "Index files, directories, URLs, PDFs, DOCX, or text into a named knowledge base for semantic search",
|
|
255
|
+
promptSnippet: "Index files/dirs/URLs/PDFs/DOCX/text into a searchable knowledge base",
|
|
256
256
|
promptGuidelines: [
|
|
257
257
|
"Use knowledge_plan first for broad directories, large repositories, or sources that may contain private or low-signal text",
|
|
258
|
-
"Use knowledge_add when the user asks to index, remember, or learn from files or documentation",
|
|
259
|
-
"
|
|
260
|
-
"
|
|
261
|
-
"
|
|
258
|
+
"Use knowledge_add when the user asks to index, remember, or learn from files, URLs, documents, or documentation",
|
|
259
|
+
"Pass http:// or https:// sources directly for URL indexing",
|
|
260
|
+
"Prefer one knowledge_add call for the project root or relevant directory with include_paths/exclude_paths; do not call it once per ordinary source file",
|
|
261
|
+
"If the source looks like a local path, verify it exists before calling knowledge_add; non-existing non-URL path-like sources are rejected instead of treated as inline text",
|
|
262
|
+
"If a knowledge base with the same name already exists, use knowledge_update only to refresh that KB's existing retained source; otherwise ask before removing/replacing it",
|
|
263
|
+
"Index source, documentation, supported document files, and ordinary configuration files that explain how the project works",
|
|
262
264
|
"Default directory indexing suggests excluding generated, vendor, browser runtime, obvious secret, and low-signal text artifacts; these are not permanent blocks",
|
|
263
265
|
"For ambiguous or risky text such as .env, private keys, certificates, credential-named files, settings.json, appsettings.json, cloud config, editor config, generated reports, lockfiles, or vendor text, explain the tradeoff and ask the user before including it",
|
|
264
266
|
"If the user confirms a suggested-excluded text file or directory should be indexed, call knowledge_add with include_suggested_text or a focused include_paths value so the tool follows the confirmed scope",
|
|
@@ -266,7 +268,7 @@ export default function (pi) {
|
|
|
266
268
|
"Provide a descriptive name for this single knowledge base",
|
|
267
269
|
],
|
|
268
270
|
parameters: Type.Object({
|
|
269
|
-
source: Type.String({ description: "File path, directory path, or inline text to index" }),
|
|
271
|
+
source: Type.String({ description: "File path, directory path, URL, PDF/DOCX path, or inline text to index" }),
|
|
270
272
|
name: Type.String({ description: "Display name for this knowledge base" }),
|
|
271
273
|
include_suggested_text: Type.Optional(Type.Boolean({
|
|
272
274
|
description: "Include text files that are normally suggested for exclusion, such as vendor/build/runtime/cache/secret-named text, after user confirmation",
|
|
@@ -317,13 +319,13 @@ export default function (pi) {
|
|
|
317
319
|
pi.registerTool({
|
|
318
320
|
name: "knowledge_search",
|
|
319
321
|
label: "Knowledge Search",
|
|
320
|
-
description: "Search indexed knowledge bases using
|
|
321
|
-
promptSnippet: "Search knowledge bases (hybrid BM25 + semantic
|
|
322
|
+
description: "Search indexed knowledge bases using lexical BM25, semantic vectors, reranking, and filters",
|
|
323
|
+
promptSnippet: "Search knowledge bases (hybrid is lexical-anchored BM25 + semantic fusion)",
|
|
322
324
|
promptGuidelines: [
|
|
323
325
|
"Use knowledge_search to find relevant context before answering domain questions",
|
|
324
|
-
"Default to mode 'hybrid' for most project questions
|
|
326
|
+
"Default to mode 'hybrid' for most project questions with useful lexical anchors; it fuses BM25 and semantic vectors but still requires keyword evidence",
|
|
325
327
|
"Use mode 'fast' for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings",
|
|
326
|
-
"Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording",
|
|
328
|
+
"Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording or hybrid returns no lexical matches",
|
|
327
329
|
"Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
|
|
328
330
|
"Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
|
|
329
331
|
"If a search returns no results or obviously weak results, retry once with a different mode before concluding the KB lacks the answer",
|
|
@@ -348,13 +350,14 @@ export default function (pi) {
|
|
|
348
350
|
kb_id: Type.Optional(Type.String({ description: "Limit search to a specific KB by ID or exact name" })),
|
|
349
351
|
offset: Type.Optional(Type.Number({ description: "Pagination offset" })),
|
|
350
352
|
file_type: Type.Optional(Type.String({ description: "Filter by file type (e.g. typescript, markdown, python)" })),
|
|
353
|
+
path_pattern: Type.Optional(Type.String({ description: "Filter by file path substring (for example src/engine.ts)" })),
|
|
351
354
|
diversity: Type.Optional(Type.Union([Type.Literal("off"), Type.Literal("balanced"), Type.Literal("strong")])),
|
|
352
355
|
diagnostics: Type.Optional(Type.Boolean({ description: "Include ranking diagnostics and mode/fallback details in the result" })),
|
|
353
356
|
}),
|
|
354
357
|
async execute(_id, params, _signal) {
|
|
355
358
|
const { engine } = await ensureInitialized();
|
|
356
|
-
const { query, mode, limit, kb_id, offset, file_type, diversity, diagnostics } = params;
|
|
357
|
-
const filters = file_type ? { file_type } : undefined;
|
|
359
|
+
const { query, mode, limit, kb_id, offset, file_type, path_pattern, diversity, diagnostics } = params;
|
|
360
|
+
const filters = file_type || path_pattern ? { file_type, path_pattern } : undefined;
|
|
358
361
|
const response = await engine.search(query, { mode, limit, kb_id, offset, filters, diversity }, _signal);
|
|
359
362
|
if (response.results.length === 0) {
|
|
360
363
|
const details = [
|
|
@@ -399,6 +402,7 @@ export default function (pi) {
|
|
|
399
402
|
promptGuidelines: [
|
|
400
403
|
"Use knowledge_symbol_search before knowledge_search when looking for a function, class, env var, config key, route, or Markdown heading",
|
|
401
404
|
"Use exact=true when the user provides a precise symbol or key",
|
|
405
|
+
"This is a lightweight declaration-pattern index, not full LSP; for methods, uncommon syntax, or missing symbols, fallback to knowledge_search mode 'fast' or 'adaptive'",
|
|
402
406
|
"After finding a symbol, use knowledge_search mode 'adaptive' with the file path or symbol name when surrounding implementation context is needed",
|
|
403
407
|
],
|
|
404
408
|
parameters: Type.Object({
|
|
@@ -434,8 +438,16 @@ export default function (pi) {
|
|
|
434
438
|
offset: params.offset,
|
|
435
439
|
exact: params.exact === true,
|
|
436
440
|
}, _signal);
|
|
437
|
-
if (response.results.length === 0)
|
|
438
|
-
return {
|
|
441
|
+
if (response.results.length === 0) {
|
|
442
|
+
return {
|
|
443
|
+
content: [
|
|
444
|
+
{
|
|
445
|
+
type: "text",
|
|
446
|
+
text: "No symbols found. This may mean the symbol is absent, symbol metadata is missing/stale, or the lightweight extractor does not cover this syntax. For important lookups, run knowledge_doctor or fallback to knowledge_search mode 'fast'/'adaptive'.",
|
|
447
|
+
},
|
|
448
|
+
],
|
|
449
|
+
};
|
|
450
|
+
}
|
|
439
451
|
const lines = response.results.map((result, index) => {
|
|
440
452
|
const signature = result.signature ? ` — ${result.signature}` : "";
|
|
441
453
|
return `[${index + 1}] ${result.name} (${result.kind}) ${result.file_path}:${result.start_line}-${result.end_line} [${result.kb_name}]${signature}`;
|
|
@@ -454,8 +466,8 @@ export default function (pi) {
|
|
|
454
466
|
pi.registerTool({
|
|
455
467
|
name: "knowledge_update",
|
|
456
468
|
label: "Knowledge Update",
|
|
457
|
-
description: "Incrementally re-index a knowledge base
|
|
458
|
-
promptSnippet: "
|
|
469
|
+
description: "Incrementally re-index a source-backed knowledge base with a retained file, directory, or URL source",
|
|
470
|
+
promptSnippet: "Refresh an existing source-backed KB",
|
|
459
471
|
parameters: Type.Object({
|
|
460
472
|
target: Type.String({ description: "KB name or ID to update" }),
|
|
461
473
|
}),
|
|
@@ -582,15 +594,26 @@ export default function (pi) {
|
|
|
582
594
|
pi.registerTool({
|
|
583
595
|
name: "knowledge_remove",
|
|
584
596
|
label: "Knowledge Remove",
|
|
585
|
-
description: "Remove a knowledge base by name or ID",
|
|
597
|
+
description: "Remove a knowledge base by name or ID after explicit confirmation",
|
|
598
|
+
promptGuidelines: [
|
|
599
|
+
"Destructive: only call after the user explicitly asks to remove knowledge data or confirms a proposed removal",
|
|
600
|
+
"Prefer knowledge_status or knowledge_doctor before removing data for troubleshooting",
|
|
601
|
+
"Use knowledge_export before removal if the user may need a backup",
|
|
602
|
+
],
|
|
586
603
|
parameters: Type.Object({
|
|
587
604
|
target: Type.String({ description: "KB name or ID to remove" }),
|
|
605
|
+
confirm: Type.Boolean({
|
|
606
|
+
description: "Must be true after explicit user confirmation for this destructive operation",
|
|
607
|
+
}),
|
|
588
608
|
}),
|
|
589
609
|
async execute(_id, params, _signal) {
|
|
590
610
|
if (_signal?.aborted)
|
|
591
611
|
throw new Error("Cancelled");
|
|
612
|
+
const { target, confirm } = params;
|
|
613
|
+
if (confirm !== true)
|
|
614
|
+
throw new Error("Confirmation required: pass confirm=true for destructive removal");
|
|
592
615
|
const { engine } = await ensureInitialized();
|
|
593
|
-
const ok = engine.remove(
|
|
616
|
+
const ok = engine.remove(target);
|
|
594
617
|
return { content: [{ type: "text", text: ok ? "Removed." : "Not found." }] };
|
|
595
618
|
},
|
|
596
619
|
});
|
|
@@ -629,11 +652,23 @@ export default function (pi) {
|
|
|
629
652
|
pi.registerTool({
|
|
630
653
|
name: "knowledge_clear",
|
|
631
654
|
label: "Knowledge Clear",
|
|
632
|
-
description: "Remove all knowledge bases",
|
|
633
|
-
|
|
634
|
-
|
|
655
|
+
description: "Remove all knowledge bases after explicit confirmation",
|
|
656
|
+
promptGuidelines: [
|
|
657
|
+
"Destructive: only call after the user explicitly asks to clear all knowledge data or confirms a proposed clear",
|
|
658
|
+
"Never use knowledge_clear as troubleshooting unless the user explicitly approves",
|
|
659
|
+
"Prefer knowledge_status, knowledge_doctor, or knowledge_export before clearing data",
|
|
660
|
+
],
|
|
661
|
+
parameters: Type.Object({
|
|
662
|
+
confirm: Type.Boolean({
|
|
663
|
+
description: "Must be true after explicit user confirmation for this destructive operation",
|
|
664
|
+
}),
|
|
665
|
+
}),
|
|
666
|
+
async execute(_id, params, _signal) {
|
|
635
667
|
if (_signal?.aborted)
|
|
636
668
|
throw new Error("Cancelled");
|
|
669
|
+
if (params.confirm !== true) {
|
|
670
|
+
throw new Error("Confirmation required: pass confirm=true for destructive clear");
|
|
671
|
+
}
|
|
637
672
|
const { engine } = await ensureInitialized();
|
|
638
673
|
engine.clear();
|
|
639
674
|
return { content: [{ type: "text", text: "All knowledge bases cleared." }] };
|
|
@@ -46,17 +46,7 @@ export function diagnoseKB(db, kb, signal) {
|
|
|
46
46
|
coverage_percent: 100,
|
|
47
47
|
total_source_files: 0,
|
|
48
48
|
indexed_files: kb.file_count,
|
|
49
|
-
skipped_files:
|
|
50
|
-
total: 0,
|
|
51
|
-
by_reason: {
|
|
52
|
-
suggested_excluded: 0,
|
|
53
|
-
oversized: 0,
|
|
54
|
-
binary: 0,
|
|
55
|
-
unreadable: 0,
|
|
56
|
-
inaccessible: 0,
|
|
57
|
-
},
|
|
58
|
-
samples: [],
|
|
59
|
-
},
|
|
49
|
+
skipped_files: createSkippedScanStats(),
|
|
60
50
|
job,
|
|
61
51
|
};
|
|
62
52
|
throwIfAborted(signal);
|
package/dist/src/engine.js
CHANGED
|
@@ -3,7 +3,7 @@ import { join, resolve } from "node:path";
|
|
|
3
3
|
import { diagnoseKB } from "./diagnostics/health.js";
|
|
4
4
|
import { dispose as disposeEmbedding, embedDocuments, embedQuery, prepareForShutdown as prepareEmbeddingForShutdown, } from "./embedding/provider.js";
|
|
5
5
|
import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
|
|
6
|
-
import { buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile,
|
|
6
|
+
import { addSkippedScanEntry, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
7
7
|
import { extractSymbols } from "./indexer/symbols.js";
|
|
8
8
|
import { shutdownModelWorker } from "./model-worker-client.js";
|
|
9
9
|
import { searchBM25 } from "./search/bm25.js";
|
|
@@ -416,6 +416,48 @@ async function extractSourceFileContent(filePath, signal) {
|
|
|
416
416
|
throwIfAborted(signal);
|
|
417
417
|
return { content: readFileSync(filePath, "utf-8"), fileType: "text" };
|
|
418
418
|
}
|
|
419
|
+
function looksLikeLocalPath(source) {
|
|
420
|
+
const trimmed = source.trim();
|
|
421
|
+
if (!trimmed || trimmed.includes("\n") || trimmed.includes("\r"))
|
|
422
|
+
return false;
|
|
423
|
+
if (/^(?:https?:)?\/\//.test(trimmed))
|
|
424
|
+
return false;
|
|
425
|
+
if (trimmed.startsWith(".") || trimmed.startsWith("/") || trimmed.startsWith("~"))
|
|
426
|
+
return true;
|
|
427
|
+
if (trimmed.includes("/") || trimmed.includes("\\"))
|
|
428
|
+
return true;
|
|
429
|
+
return /\.[A-Za-z0-9][A-Za-z0-9_-]{0,15}$/.test(trimmed);
|
|
430
|
+
}
|
|
431
|
+
function classifySource(source) {
|
|
432
|
+
const resolvedSource = resolve(source);
|
|
433
|
+
const isUrl = source.startsWith("http://") || source.startsWith("https://");
|
|
434
|
+
const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
|
|
435
|
+
const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
|
|
436
|
+
if (!isUrl && !isDir && !isFile && looksLikeLocalPath(source)) {
|
|
437
|
+
throw new Error(`Local path does not exist: ${source}. Pass inline text only when the source itself is text.`);
|
|
438
|
+
}
|
|
439
|
+
return {
|
|
440
|
+
resolvedSource,
|
|
441
|
+
isUrl,
|
|
442
|
+
isDir,
|
|
443
|
+
isFile,
|
|
444
|
+
sourceType: isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text",
|
|
445
|
+
};
|
|
446
|
+
}
|
|
447
|
+
async function extractScannableFileContent(file, signal) {
|
|
448
|
+
return extractSourceFileContent(file.path, signal);
|
|
449
|
+
}
|
|
450
|
+
async function extractScannableFileContentOrSkip(file, skipped, signal) {
|
|
451
|
+
try {
|
|
452
|
+
return await extractScannableFileContent(file, signal);
|
|
453
|
+
}
|
|
454
|
+
catch (error) {
|
|
455
|
+
if (signal?.aborted || isCancellationError(error))
|
|
456
|
+
throw error;
|
|
457
|
+
addSkippedScanEntry(skipped, { path: file.relPath, reason: "extraction_failed", size: file.size });
|
|
458
|
+
return undefined;
|
|
459
|
+
}
|
|
460
|
+
}
|
|
419
461
|
function throwIfAborted(signal) {
|
|
420
462
|
if (signal?.aborted)
|
|
421
463
|
throw new Error("Cancelled");
|
|
@@ -518,11 +560,7 @@ export class KnowledgeEngine {
|
|
|
518
560
|
}
|
|
519
561
|
plan(source, options = {}, signal) {
|
|
520
562
|
throwIfAborted(signal);
|
|
521
|
-
const resolvedSource =
|
|
522
|
-
const isUrl = source.startsWith("http://") || source.startsWith("https://");
|
|
523
|
-
const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
|
|
524
|
-
const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
|
|
525
|
-
const sourceType = isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text";
|
|
563
|
+
const { resolvedSource, isDir, isFile, sourceType } = classifySource(source);
|
|
526
564
|
if (isDir) {
|
|
527
565
|
const plan = planDirectoryScan(resolvedSource, toScanOptions(options), signal);
|
|
528
566
|
return {
|
|
@@ -535,7 +573,7 @@ export class KnowledgeEngine {
|
|
|
535
573
|
}
|
|
536
574
|
if (isFile) {
|
|
537
575
|
const skipped = createSkippedScanStats();
|
|
538
|
-
const supportedDocument =
|
|
576
|
+
const supportedDocument = isSupportedDocumentFile(resolvedSource);
|
|
539
577
|
if (!supportedDocument && !isReadableTextFile(resolvedSource)) {
|
|
540
578
|
skipped.total = 1;
|
|
541
579
|
skipped.by_reason.binary = 1;
|
|
@@ -575,11 +613,7 @@ export class KnowledgeEngine {
|
|
|
575
613
|
throw new Error("Engine not initialized");
|
|
576
614
|
throwIfAborted(signal);
|
|
577
615
|
const db = this.db;
|
|
578
|
-
const resolvedSource =
|
|
579
|
-
const isUrl = source.startsWith("http://") || source.startsWith("https://");
|
|
580
|
-
const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
|
|
581
|
-
const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
|
|
582
|
-
const sourceType = isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text";
|
|
616
|
+
const { resolvedSource, isUrl, isDir, isFile, sourceType } = classifySource(source);
|
|
583
617
|
const scanOptions = toScanOptions(options);
|
|
584
618
|
const existingKB = getKBByName(db, name);
|
|
585
619
|
if (existingKB) {
|
|
@@ -605,7 +639,8 @@ export class KnowledgeEngine {
|
|
|
605
639
|
const pendingChunks = [];
|
|
606
640
|
const startedAt = Date.now();
|
|
607
641
|
let latestSkippedTotal = 0;
|
|
608
|
-
|
|
642
|
+
let latestSkippedSummary = "none";
|
|
643
|
+
const reportProgress = (phase, processedFiles, totalFiles, skippedTotal = latestSkippedTotal) => {
|
|
609
644
|
latestSkippedTotal = skippedTotal;
|
|
610
645
|
const elapsed = Date.now() - startedAt;
|
|
611
646
|
const chunkRate = chunkCount / Math.max(1, elapsed / 1000);
|
|
@@ -693,20 +728,27 @@ export class KnowledgeEngine {
|
|
|
693
728
|
onProgress?.(scanningMessage);
|
|
694
729
|
const skipped = createSkippedScanStats();
|
|
695
730
|
let processedFiles = 0;
|
|
696
|
-
for (const file of
|
|
731
|
+
for (const file of iterateScannableFiles(resolvedSource, skipped, scanOptions)) {
|
|
697
732
|
if (signal?.aborted)
|
|
698
733
|
throw new Error("Cancelled");
|
|
699
|
-
const
|
|
734
|
+
const extracted = await extractScannableFileContentOrSkip(file, skipped, signal);
|
|
735
|
+
if (!extracted) {
|
|
736
|
+
latestSkippedTotal = skipped.total;
|
|
737
|
+
continue;
|
|
738
|
+
}
|
|
739
|
+
const chunks = await chunkFile(extracted.content, file.relPath);
|
|
700
740
|
processedFiles++;
|
|
701
741
|
latestSkippedTotal = skipped.total;
|
|
702
742
|
if (chunks.length > 0)
|
|
703
743
|
fileCount++;
|
|
704
|
-
addSymbols(
|
|
744
|
+
addSymbols(extracted.content, file.relPath, extracted.fileType);
|
|
705
745
|
await addChunks(chunks, processedFiles, plan.files);
|
|
706
746
|
if (processedFiles % 25 === 0)
|
|
707
747
|
reportProgress("Chunking", processedFiles, plan.files, skipped.total);
|
|
708
748
|
}
|
|
709
|
-
|
|
749
|
+
latestSkippedTotal = skipped.total;
|
|
750
|
+
latestSkippedSummary = summarizeSkippedScan(skipped);
|
|
751
|
+
const finalizingMessage = `Scanned ${processedFiles} files, skipped ${skipped.total} (${latestSkippedSummary}), finalizing...`;
|
|
710
752
|
updateIndexingJob(this.db, kb.id, {
|
|
711
753
|
phase: "finalizing",
|
|
712
754
|
message: finalizingMessage,
|
|
@@ -734,7 +776,8 @@ export class KnowledgeEngine {
|
|
|
734
776
|
const savedKB = getKB(this.db, kb.id);
|
|
735
777
|
if (!savedKB)
|
|
736
778
|
throw new Error(`Knowledge base disappeared after add: ${kb.id}`);
|
|
737
|
-
const
|
|
779
|
+
const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
|
|
780
|
+
const readyMessage = `Ready: ${chunkCount} chunks from ${savedFileCount} files in ${formatDuration(Date.now() - startedAt)}${skippedReadySuffix}`;
|
|
738
781
|
updateIndexingJob(this.db, kb.id, {
|
|
739
782
|
phase: "ready",
|
|
740
783
|
message: readyMessage,
|
|
@@ -800,6 +843,8 @@ export class KnowledgeEngine {
|
|
|
800
843
|
const vectorPath = this.vectorPathFor(kb.id);
|
|
801
844
|
addedVectorPath = tempVectorPath(`${vectorPath}.added`);
|
|
802
845
|
addedVectorWriter = openVectorWriter(addedVectorPath);
|
|
846
|
+
let latestSkippedTotal = 0;
|
|
847
|
+
let latestSkippedSummary = "none";
|
|
803
848
|
const existingHashes = new Map();
|
|
804
849
|
let existingIndex = 0;
|
|
805
850
|
for (const chunk of iterateChunksByKB(this.db, kb.id)) {
|
|
@@ -903,12 +948,15 @@ export class KnowledgeEngine {
|
|
|
903
948
|
});
|
|
904
949
|
onProgress?.(planningMessage);
|
|
905
950
|
const skipped = createSkippedScanStats();
|
|
906
|
-
for (const file of
|
|
951
|
+
for (const file of iterateScannableFiles(kb.source_path, skipped, scanOptions)) {
|
|
907
952
|
if (signal?.aborted)
|
|
908
953
|
throw new Error("Cancelled");
|
|
954
|
+
const extracted = await extractScannableFileContentOrSkip(file, skipped, signal);
|
|
955
|
+
if (!extracted)
|
|
956
|
+
continue;
|
|
909
957
|
scannedFiles++;
|
|
910
|
-
stagedSymbols.push(...extractSymbols(
|
|
911
|
-
await processChunks(await chunkFile(
|
|
958
|
+
stagedSymbols.push(...extractSymbols(extracted.content, file.relPath, extracted.fileType));
|
|
959
|
+
await processChunks(await chunkFile(extracted.content, file.relPath));
|
|
912
960
|
if (scannedFiles % 25 === 0) {
|
|
913
961
|
const message = `Scanned ${scannedFiles} files, ${scannedChunks} chunks, skipped ${skipped.total}, +${addedCount} =${unchanged}`;
|
|
914
962
|
updateIndexingJob(this.db, kb.id, {
|
|
@@ -924,7 +972,9 @@ export class KnowledgeEngine {
|
|
|
924
972
|
onProgress?.(message);
|
|
925
973
|
}
|
|
926
974
|
}
|
|
927
|
-
|
|
975
|
+
latestSkippedTotal = skipped.total;
|
|
976
|
+
latestSkippedSummary = summarizeSkippedScan(skipped);
|
|
977
|
+
const message = `Scanned ${scannedFiles} files, skipped ${skipped.total} (${latestSkippedSummary}), reconciling deletes...`;
|
|
928
978
|
updateIndexingJob(this.db, kb.id, {
|
|
929
979
|
phase: "reconciling",
|
|
930
980
|
message,
|
|
@@ -1019,7 +1069,8 @@ export class KnowledgeEngine {
|
|
|
1019
1069
|
insertSymbols(this.db, kb.id, stagedSymbols);
|
|
1020
1070
|
updateKBCounts(this.db, kb.id, finalChunkCount, getFileCount(this.db, kb.id));
|
|
1021
1071
|
updateKBStatus(this.db, kb.id, "ready");
|
|
1022
|
-
const
|
|
1072
|
+
const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
|
|
1073
|
+
const readyMessage = `Ready: +${addedCount} -${idsToRemove.length} =${unchanged}${skippedReadySuffix}`;
|
|
1023
1074
|
updateIndexingJob(this.db, kb.id, {
|
|
1024
1075
|
phase: "ready",
|
|
1025
1076
|
message: readyMessage,
|
|
@@ -18,7 +18,7 @@ export interface ScannableFile {
|
|
|
18
18
|
}
|
|
19
19
|
export interface SkippedScanEntry {
|
|
20
20
|
path: string;
|
|
21
|
-
reason: "suggested_excluded" | "oversized" | "binary" | "unreadable" | "inaccessible";
|
|
21
|
+
reason: "suggested_excluded" | "oversized" | "binary" | "unreadable" | "inaccessible" | "extraction_failed";
|
|
22
22
|
size?: number;
|
|
23
23
|
}
|
|
24
24
|
export interface ScanResult {
|
|
@@ -30,7 +30,9 @@ export interface ScanResult {
|
|
|
30
30
|
};
|
|
31
31
|
}
|
|
32
32
|
export declare function createSkippedScanStats(): ScanResult["skipped"];
|
|
33
|
+
export declare function addSkippedScanEntry(skipped: ScanResult["skipped"], entry: SkippedScanEntry): void;
|
|
33
34
|
export declare function summarizeSkippedScan(skipped: ScanResult["skipped"]): string;
|
|
35
|
+
export declare function isSupportedDocumentFile(filePath: string): boolean;
|
|
34
36
|
export declare function iterateScannableFiles(dirPath: string, skipped?: ScanResult["skipped"], options?: ScanOptions): Generator<ScannableFile>;
|
|
35
37
|
export declare function iterateScannedFiles(dirPath: string, skipped?: ScanResult["skipped"], options?: ScanOptions): Generator<ScannedFile>;
|
|
36
38
|
export declare function iterateScannedFilesAsync(dirPath: string, onFile: (file: ScannedFile) => Promise<void> | void, skipped?: ScanResult["skipped"], options?: ScanOptions): Promise<ScanResult["skipped"]>;
|
|
@@ -3,53 +3,50 @@ import { closeSync, existsSync, openSync, readdirSync, readFileSync, readSync, s
|
|
|
3
3
|
import { extname, join, relative, sep } from "node:path";
|
|
4
4
|
import ignore from "ignore";
|
|
5
5
|
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10 MB
|
|
6
|
-
const
|
|
7
|
-
|
|
8
|
-
".
|
|
9
|
-
".
|
|
10
|
-
".
|
|
11
|
-
".
|
|
12
|
-
".bmp",
|
|
13
|
-
".
|
|
14
|
-
".
|
|
15
|
-
".
|
|
16
|
-
".
|
|
17
|
-
".
|
|
18
|
-
".eot",
|
|
19
|
-
".
|
|
20
|
-
".
|
|
21
|
-
".gz",
|
|
22
|
-
".
|
|
23
|
-
".
|
|
24
|
-
".
|
|
25
|
-
".
|
|
26
|
-
".
|
|
27
|
-
".
|
|
28
|
-
".
|
|
29
|
-
".
|
|
30
|
-
".
|
|
31
|
-
".
|
|
32
|
-
".
|
|
33
|
-
".
|
|
34
|
-
".
|
|
35
|
-
".
|
|
36
|
-
".
|
|
37
|
-
".
|
|
38
|
-
".
|
|
39
|
-
".
|
|
40
|
-
".
|
|
41
|
-
".
|
|
42
|
-
".
|
|
43
|
-
".
|
|
44
|
-
".
|
|
45
|
-
".
|
|
46
|
-
".
|
|
47
|
-
".
|
|
48
|
-
".
|
|
49
|
-
|
|
50
|
-
".a",
|
|
51
|
-
".lib",
|
|
52
|
-
]);
|
|
6
|
+
const DOCUMENT_EXTENSIONS = { ".doc": true, ".docx": true, ".pdf": true };
|
|
7
|
+
const BINARY_EXTENSIONS = {
|
|
8
|
+
".7z": true,
|
|
9
|
+
".a": true,
|
|
10
|
+
".avi": true,
|
|
11
|
+
".bin": true,
|
|
12
|
+
".bmp": true,
|
|
13
|
+
".bz2": true,
|
|
14
|
+
".dat": true,
|
|
15
|
+
".db": true,
|
|
16
|
+
".dll": true,
|
|
17
|
+
".dylib": true,
|
|
18
|
+
".eot": true,
|
|
19
|
+
".exe": true,
|
|
20
|
+
".gif": true,
|
|
21
|
+
".gz": true,
|
|
22
|
+
".ico": true,
|
|
23
|
+
".jpg": true,
|
|
24
|
+
".jpeg": true,
|
|
25
|
+
".lib": true,
|
|
26
|
+
".lock": true,
|
|
27
|
+
".mov": true,
|
|
28
|
+
".mp3": true,
|
|
29
|
+
".mp4": true,
|
|
30
|
+
".node": true,
|
|
31
|
+
".o": true,
|
|
32
|
+
".otf": true,
|
|
33
|
+
".ppt": true,
|
|
34
|
+
".rar": true,
|
|
35
|
+
".so": true,
|
|
36
|
+
".sqlite": true,
|
|
37
|
+
".svg": true,
|
|
38
|
+
".tar": true,
|
|
39
|
+
".ttf": true,
|
|
40
|
+
".wasm": true,
|
|
41
|
+
".wav": true,
|
|
42
|
+
".webm": true,
|
|
43
|
+
".webp": true,
|
|
44
|
+
".woff": true,
|
|
45
|
+
".woff2": true,
|
|
46
|
+
".xls": true,
|
|
47
|
+
".xlsx": true,
|
|
48
|
+
".zip": true,
|
|
49
|
+
};
|
|
53
50
|
const DEFAULT_SUGGESTED_EXCLUDE = [
|
|
54
51
|
"node_modules",
|
|
55
52
|
".git",
|
|
@@ -111,6 +108,7 @@ export function createSkippedScanStats() {
|
|
|
111
108
|
binary: 0,
|
|
112
109
|
unreadable: 0,
|
|
113
110
|
inaccessible: 0,
|
|
111
|
+
extraction_failed: 0,
|
|
114
112
|
},
|
|
115
113
|
samples: [],
|
|
116
114
|
};
|
|
@@ -121,6 +119,9 @@ function addSkipped(skipped, entry) {
|
|
|
121
119
|
if (skipped.samples.length < MAX_SKIPPED_SAMPLES)
|
|
122
120
|
skipped.samples.push(entry);
|
|
123
121
|
}
|
|
122
|
+
export function addSkippedScanEntry(skipped, entry) {
|
|
123
|
+
addSkipped(skipped, entry);
|
|
124
|
+
}
|
|
124
125
|
export function summarizeSkippedScan(skipped) {
|
|
125
126
|
return (Object.entries(skipped.by_reason)
|
|
126
127
|
.filter(([, count]) => count > 0)
|
|
@@ -157,14 +158,22 @@ function detectFileType(filePath) {
|
|
|
157
158
|
".zsh": "shell",
|
|
158
159
|
".sql": "sql",
|
|
159
160
|
".graphql": "graphql",
|
|
161
|
+
".pdf": "pdf",
|
|
162
|
+
".doc": "docx",
|
|
163
|
+
".docx": "docx",
|
|
160
164
|
".txt": "text",
|
|
161
165
|
".csv": "text",
|
|
162
166
|
".log": "text",
|
|
163
167
|
};
|
|
164
168
|
return map[ext] ?? "text";
|
|
165
169
|
}
|
|
170
|
+
export function isSupportedDocumentFile(filePath) {
|
|
171
|
+
return DOCUMENT_EXTENSIONS[extname(filePath).toLowerCase()] === true;
|
|
172
|
+
}
|
|
166
173
|
function isBinaryFile(filePath) {
|
|
167
|
-
if (
|
|
174
|
+
if (isSupportedDocumentFile(filePath))
|
|
175
|
+
return false;
|
|
176
|
+
if (BINARY_EXTENSIONS[extname(filePath).toLowerCase()] === true)
|
|
168
177
|
return true;
|
|
169
178
|
let fd;
|
|
170
179
|
try {
|
package/docs/configuration.md
CHANGED
|
@@ -42,9 +42,11 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
42
42
|
|
|
43
43
|
Search diagnostics include result provenance in both text output and structured `details`: chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
44
44
|
|
|
45
|
+
`knowledge_search` supports `file_type` and `path_pattern` filters. `hybrid` mode is lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording.
|
|
46
|
+
|
|
45
47
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
46
48
|
|
|
47
|
-
`knowledge_symbol_search` uses a lightweight
|
|
49
|
+
`knowledge_symbol_search` uses a lightweight declaration-pattern index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find common code symbols, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; for methods, uncommon syntax, or empty important lookups, use `knowledge_search` mode `fast`/`adaptive` and check `knowledge_doctor` for stale or missing symbol metadata. Older file/directory/URL KBs without symbol metadata should be updated before relying on symbol lookup. Imported portable KBs and inline text KBs do not retain an active source path; rebuild their symbols by re-importing or re-adding the content.
|
|
48
50
|
|
|
49
51
|
## Test and Release Fixtures
|
|
50
52
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.1",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|