pi-knowledge 0.5.0 → 0.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,18 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.5.2] - 2026-07-19
4
+
5
+ ### Fixed
6
+ - Updated `better-sqlite3` to 12.11.1 so Node 26 installs and native SQLite loading work (#4).
7
+
8
+ ## [0.5.1] - 2026-07-07
9
+
10
+ ### Fixed
11
+ - Indexed PDF/DOC/DOCX files discovered during directory add and update through the document extractors instead of treating them as binary skips.
12
+ - Skipped failed directory document extraction as `extraction_failed` without failing the whole knowledge base.
13
+ - Required explicit `confirm: true` for public destructive `knowledge_remove` and `knowledge_clear` tool calls.
14
+ - Aligned `path_pattern` documentation and tool guidance with the implemented substring filter behavior.
15
+
3
16
  ## [0.5.0] - 2026-07-07
4
17
 
5
18
  ### Added
package/README.md CHANGED
@@ -22,7 +22,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
22
22
  ## Highlights
23
23
 
24
24
  - **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
25
- - **Hybrid retrieval**: semantic vectors + BM25 keyword search + normalized weighted score fusion.
25
+ - **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
26
26
  - **Code-aware indexing**: AST-aware chunking for TypeScript/JavaScript, Python, Go, Rust, and Java.
27
27
  - **Lightweight symbol lookup**: indexes functions, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
28
28
  - **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
@@ -65,7 +65,7 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
65
65
  - **RAG-native project memory**: follows the Retrieval-Augmented Generation pattern from Lewis et al. 2020: keep source truth outside the model, retrieve it at answer time, and inject only relevant context.
66
66
  - **Dense semantic recall**: uses multilingual dense embeddings in the spirit of Dense Passage Retrieval (Karpukhin et al. 2020), so conceptual queries can find code/docs even when wording differs.
67
67
  - **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and code symbols with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
68
- - **Hybrid retrieval with diagnosable scores**: combines BM25 and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
68
+ - **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
69
69
  - **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results.
70
70
  - **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
71
71
  - **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
@@ -101,16 +101,16 @@ omp install ./pi-knowledge
101
101
 
102
102
  | Tool | Description |
103
103
  |------|-------------|
104
- | `knowledge_plan` | Inspect an indexing source before writing a KB; reports scannable files, suggested exclusions, and technical skips |
104
+ | `knowledge_plan` | Inspect an indexing source before writing a KB; reports scannable counts, suggested exclusions, and technical skips |
105
105
  | `knowledge_add` | Index files, directories, URLs, PDFs, DOCX, or inline text |
106
- | `knowledge_search` | Hybrid, deep, or adaptive search across one or all knowledge bases |
107
- | `knowledge_symbol_search` | Exact or substring lookup for code symbols, route-like handlers, Markdown headings, config keys, and env vars |
108
- | `knowledge_remove` | Remove a knowledge base by name or ID |
109
- | `knowledge_update` | Incrementally re-index changed files in a knowledge base |
106
+ | `knowledge_search` | Fast, semantic, lexical-anchored hybrid, deep, or adaptive search across one or all knowledge bases; supports file-type and path filters |
107
+ | `knowledge_symbol_search` | Lightweight exact or substring lookup for code symbols, route-like handlers, Markdown headings, config keys, and env vars; fall back to `knowledge_search` for methods or uncommon syntax |
108
+ | `knowledge_remove` | Remove a knowledge base by name or ID after `confirm: true` |
109
+ | `knowledge_update` | Incrementally re-index changed files in a source-backed file, directory, or URL knowledge base |
110
110
  | `knowledge_show` | List all knowledge bases with stats |
111
111
  | `knowledge_status` | Show engine status with health diagnostics (stale, orphans, coverage) |
112
112
  | `knowledge_doctor` | Diagnose health score, skipped files, stuck jobs, stale data, and recommended fixes |
113
- | `knowledge_clear` | Remove all knowledge bases |
113
+ | `knowledge_clear` | Remove all knowledge bases after `confirm: true` |
114
114
  | `knowledge_export` | Export a KB to shareable JSONL file |
115
115
  | `knowledge_import` | Import a KB from JSONL (re-embeds content) |
116
116
 
@@ -118,17 +118,17 @@ omp install ./pi-knowledge
118
118
 
119
119
  - `fast`: BM25 keyword search for exact symbols, commands, and identifiers.
120
120
  - `semantic`: vector search for conceptual matches.
121
- - `hybrid`: BM25 + vector search with normalized weighted score fusion.
121
+ - `hybrid`: lexical-anchored BM25 + vector search with normalized weighted score fusion. It requires keyword evidence to avoid low-confidence semantic false positives.
122
122
  - `deep`: hybrid retrieval followed by cross-encoder reranking.
123
123
  - `adaptive`: hybrid retrieval followed by query-time contextual window expansion around seed chunks. It keeps the matched seed, prefers nearby/query-relevant neighboring chunks, and collapses overlapping windows from the same file.
124
124
  - `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
125
- - `code`, `config`, `errors`, `docs`, `decision`: intent aliases for agents. `code`, `config`, and `errors` bias toward exact lexical/symbol evidence; `docs` and `decision` keep hybrid recall while making the intended source type explicit.
125
+ - `code`, `config`, `errors`, `docs`, `decision`: intent aliases for agents. `code`, `config`, and `errors` bias toward exact lexical/symbol evidence; `docs` and `decision` keep the intended source type explicit.
126
126
 
127
127
  Mode selection contract:
128
128
 
129
- - Start with `hybrid` for most project questions.
129
+ - Start with `hybrid` for most project questions that contain useful lexical anchors.
130
130
  - Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
131
- - Use `semantic` when the query is conceptual and exact terms may differ from the indexed wording.
131
+ - Use `semantic` when the query is conceptual, exact terms may differ from indexed wording, or hybrid returns no lexical matches.
132
132
  - Use `adaptive` when the answer needs nearby code, neighboring documentation sections, or enough context to make a safe edit.
133
133
  - Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
134
134
  - If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
package/dist/index.js CHANGED
@@ -192,7 +192,7 @@ export default function (pi) {
192
192
  if (kbs.length === 0)
193
193
  return undefined;
194
194
  const desc = kbs.map((kb) => `"${kb.name}" (${kb.chunk_count} chunks, ${kb.file_count} files)`).join(", ");
195
- const guidance = `Available knowledge bases: ${desc}. Use knowledge_search before answering domain questions.`;
195
+ const guidance = `Available knowledge bases: ${desc}. Use knowledge_symbol_search for exact symbols/config keys/env vars/routes/headings, then use knowledge_search for broader domain or implementation context.`;
196
196
  return {
197
197
  systemPrompt: `${event.systemPrompt}\n\n${guidance}`,
198
198
  };
@@ -200,11 +200,11 @@ export default function (pi) {
200
200
  pi.registerTool({
201
201
  name: "knowledge_plan",
202
202
  label: "Knowledge Plan",
203
- description: "Inspect an indexing source without writing a KB, showing scannable files, suggested exclusions, and technical skips",
203
+ description: "Inspect an indexing source without writing a KB, showing scannable counts, suggested exclusions, and technical skips",
204
204
  promptSnippet: "Plan a knowledge-base indexing scope before calling knowledge_add",
205
205
  promptGuidelines: [
206
206
  "Use knowledge_plan before knowledge_add for broad directories, large repositories, or sources that may contain private or low-signal text",
207
- "Show the user suggested exclusions and technical skips before asking whether to include risky or low-signal text",
207
+ "Show the user suggested exclusions, technical skips, and scannable file counts before asking whether to include risky or low-signal text",
208
208
  "After the user confirms scope, call knowledge_add with matching include_suggested_text, include_paths, or exclude_paths",
209
209
  "Do not use knowledge_plan as a substitute for knowledge_search; it only plans indexing scope",
210
210
  ],
@@ -251,14 +251,16 @@ export default function (pi) {
251
251
  pi.registerTool({
252
252
  name: "knowledge_add",
253
253
  label: "Knowledge Add",
254
- description: "Index files, directories, or text into a named knowledge base for semantic search",
255
- promptSnippet: "Index files/dirs/text into a searchable knowledge base",
254
+ description: "Index files, directories, URLs, PDFs, DOCX, or text into a named knowledge base for semantic search",
255
+ promptSnippet: "Index files/dirs/URLs/PDFs/DOCX/text into a searchable knowledge base",
256
256
  promptGuidelines: [
257
257
  "Use knowledge_plan first for broad directories, large repositories, or sources that may contain private or low-signal text",
258
- "Use knowledge_add when the user asks to index, remember, or learn from files or documentation",
259
- "Prefer one knowledge_add call for the project root or relevant directory with include_paths/exclude_paths; do not call it once per file",
260
- "If a knowledge base with the same name already exists, use knowledge_update or ask before replacing it",
261
- "Index source, documentation, and ordinary configuration files that explain how the project works",
258
+ "Use knowledge_add when the user asks to index, remember, or learn from files, URLs, documents, or documentation",
259
+ "Pass http:// or https:// sources directly for URL indexing",
260
+ "Prefer one knowledge_add call for the project root or relevant directory with include_paths/exclude_paths; do not call it once per ordinary source file",
261
+ "If the source looks like a local path, verify it exists before calling knowledge_add; non-existing non-URL path-like sources are rejected instead of treated as inline text",
262
+ "If a knowledge base with the same name already exists, use knowledge_update only to refresh that KB's existing retained source; otherwise ask before removing/replacing it",
263
+ "Index source, documentation, supported document files, and ordinary configuration files that explain how the project works",
262
264
  "Default directory indexing suggests excluding generated, vendor, browser runtime, obvious secret, and low-signal text artifacts; these are not permanent blocks",
263
265
  "For ambiguous or risky text such as .env, private keys, certificates, credential-named files, settings.json, appsettings.json, cloud config, editor config, generated reports, lockfiles, or vendor text, explain the tradeoff and ask the user before including it",
264
266
  "If the user confirms a suggested-excluded text file or directory should be indexed, call knowledge_add with include_suggested_text or a focused include_paths value so the tool follows the confirmed scope",
@@ -266,7 +268,7 @@ export default function (pi) {
266
268
  "Provide a descriptive name for this single knowledge base",
267
269
  ],
268
270
  parameters: Type.Object({
269
- source: Type.String({ description: "File path, directory path, or inline text to index" }),
271
+ source: Type.String({ description: "File path, directory path, URL, PDF/DOCX path, or inline text to index" }),
270
272
  name: Type.String({ description: "Display name for this knowledge base" }),
271
273
  include_suggested_text: Type.Optional(Type.Boolean({
272
274
  description: "Include text files that are normally suggested for exclusion, such as vendor/build/runtime/cache/secret-named text, after user confirmation",
@@ -317,13 +319,13 @@ export default function (pi) {
317
319
  pi.registerTool({
318
320
  name: "knowledge_search",
319
321
  label: "Knowledge Search",
320
- description: "Search indexed knowledge bases using hybrid semantic + keyword search",
321
- promptSnippet: "Search knowledge bases (hybrid BM25 + semantic + weighted score fusion)",
322
+ description: "Search indexed knowledge bases using lexical BM25, semantic vectors, reranking, and filters",
323
+ promptSnippet: "Search knowledge bases (hybrid is lexical-anchored BM25 + semantic fusion)",
322
324
  promptGuidelines: [
323
325
  "Use knowledge_search to find relevant context before answering domain questions",
324
- "Default to mode 'hybrid' for most project questions because it combines lexical anchors with semantic recall",
326
+ "Default to mode 'hybrid' for most project questions with useful lexical anchors; it fuses BM25 and semantic vectors but still requires keyword evidence",
325
327
  "Use mode 'fast' for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings",
326
- "Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording",
328
+ "Use mode 'semantic' for broad conceptual questions when exact terms may differ from the indexed wording or hybrid returns no lexical matches",
327
329
  "Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
328
330
  "Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
329
331
  "If a search returns no results or obviously weak results, retry once with a different mode before concluding the KB lacks the answer",
@@ -348,13 +350,14 @@ export default function (pi) {
348
350
  kb_id: Type.Optional(Type.String({ description: "Limit search to a specific KB by ID or exact name" })),
349
351
  offset: Type.Optional(Type.Number({ description: "Pagination offset" })),
350
352
  file_type: Type.Optional(Type.String({ description: "Filter by file type (e.g. typescript, markdown, python)" })),
353
+ path_pattern: Type.Optional(Type.String({ description: "Filter by file path substring (for example src/engine.ts)" })),
351
354
  diversity: Type.Optional(Type.Union([Type.Literal("off"), Type.Literal("balanced"), Type.Literal("strong")])),
352
355
  diagnostics: Type.Optional(Type.Boolean({ description: "Include ranking diagnostics and mode/fallback details in the result" })),
353
356
  }),
354
357
  async execute(_id, params, _signal) {
355
358
  const { engine } = await ensureInitialized();
356
- const { query, mode, limit, kb_id, offset, file_type, diversity, diagnostics } = params;
357
- const filters = file_type ? { file_type } : undefined;
359
+ const { query, mode, limit, kb_id, offset, file_type, path_pattern, diversity, diagnostics } = params;
360
+ const filters = file_type || path_pattern ? { file_type, path_pattern } : undefined;
358
361
  const response = await engine.search(query, { mode, limit, kb_id, offset, filters, diversity }, _signal);
359
362
  if (response.results.length === 0) {
360
363
  const details = [
@@ -399,6 +402,7 @@ export default function (pi) {
399
402
  promptGuidelines: [
400
403
  "Use knowledge_symbol_search before knowledge_search when looking for a function, class, env var, config key, route, or Markdown heading",
401
404
  "Use exact=true when the user provides a precise symbol or key",
405
+ "This is a lightweight declaration-pattern index, not full LSP; for methods, uncommon syntax, or missing symbols, fallback to knowledge_search mode 'fast' or 'adaptive'",
402
406
  "After finding a symbol, use knowledge_search mode 'adaptive' with the file path or symbol name when surrounding implementation context is needed",
403
407
  ],
404
408
  parameters: Type.Object({
@@ -434,8 +438,16 @@ export default function (pi) {
434
438
  offset: params.offset,
435
439
  exact: params.exact === true,
436
440
  }, _signal);
437
- if (response.results.length === 0)
438
- return { content: [{ type: "text", text: "No symbols found." }] };
441
+ if (response.results.length === 0) {
442
+ return {
443
+ content: [
444
+ {
445
+ type: "text",
446
+ text: "No symbols found. This may mean the symbol is absent, symbol metadata is missing/stale, or the lightweight extractor does not cover this syntax. For important lookups, run knowledge_doctor or fallback to knowledge_search mode 'fast'/'adaptive'.",
447
+ },
448
+ ],
449
+ };
450
+ }
439
451
  const lines = response.results.map((result, index) => {
440
452
  const signature = result.signature ? ` — ${result.signature}` : "";
441
453
  return `[${index + 1}] ${result.name} (${result.kind}) ${result.file_path}:${result.start_line}-${result.end_line} [${result.kb_name}]${signature}`;
@@ -454,8 +466,8 @@ export default function (pi) {
454
466
  pi.registerTool({
455
467
  name: "knowledge_update",
456
468
  label: "Knowledge Update",
457
- description: "Incrementally re-index a knowledge base (only re-embeds changed content)",
458
- promptSnippet: "Incrementally update a knowledge base (only changed files re-embedded)",
469
+ description: "Incrementally re-index a source-backed knowledge base with a retained file, directory, or URL source",
470
+ promptSnippet: "Refresh an existing source-backed KB",
459
471
  parameters: Type.Object({
460
472
  target: Type.String({ description: "KB name or ID to update" }),
461
473
  }),
@@ -582,15 +594,26 @@ export default function (pi) {
582
594
  pi.registerTool({
583
595
  name: "knowledge_remove",
584
596
  label: "Knowledge Remove",
585
- description: "Remove a knowledge base by name or ID",
597
+ description: "Remove a knowledge base by name or ID after explicit confirmation",
598
+ promptGuidelines: [
599
+ "Destructive: only call after the user explicitly asks to remove knowledge data or confirms a proposed removal",
600
+ "Prefer knowledge_status or knowledge_doctor before removing data for troubleshooting",
601
+ "Use knowledge_export before removal if the user may need a backup",
602
+ ],
586
603
  parameters: Type.Object({
587
604
  target: Type.String({ description: "KB name or ID to remove" }),
605
+ confirm: Type.Boolean({
606
+ description: "Must be true after explicit user confirmation for this destructive operation",
607
+ }),
588
608
  }),
589
609
  async execute(_id, params, _signal) {
590
610
  if (_signal?.aborted)
591
611
  throw new Error("Cancelled");
612
+ const { target, confirm } = params;
613
+ if (confirm !== true)
614
+ throw new Error("Confirmation required: pass confirm=true for destructive removal");
592
615
  const { engine } = await ensureInitialized();
593
- const ok = engine.remove(params.target);
616
+ const ok = engine.remove(target);
594
617
  return { content: [{ type: "text", text: ok ? "Removed." : "Not found." }] };
595
618
  },
596
619
  });
@@ -629,11 +652,23 @@ export default function (pi) {
629
652
  pi.registerTool({
630
653
  name: "knowledge_clear",
631
654
  label: "Knowledge Clear",
632
- description: "Remove all knowledge bases",
633
- parameters: Type.Object({}),
634
- async execute(_id, _params, _signal) {
655
+ description: "Remove all knowledge bases after explicit confirmation",
656
+ promptGuidelines: [
657
+ "Destructive: only call after the user explicitly asks to clear all knowledge data or confirms a proposed clear",
658
+ "Never use knowledge_clear as troubleshooting unless the user explicitly approves",
659
+ "Prefer knowledge_status, knowledge_doctor, or knowledge_export before clearing data",
660
+ ],
661
+ parameters: Type.Object({
662
+ confirm: Type.Boolean({
663
+ description: "Must be true after explicit user confirmation for this destructive operation",
664
+ }),
665
+ }),
666
+ async execute(_id, params, _signal) {
635
667
  if (_signal?.aborted)
636
668
  throw new Error("Cancelled");
669
+ if (params.confirm !== true) {
670
+ throw new Error("Confirmation required: pass confirm=true for destructive clear");
671
+ }
637
672
  const { engine } = await ensureInitialized();
638
673
  engine.clear();
639
674
  return { content: [{ type: "text", text: "All knowledge bases cleared." }] };
@@ -46,17 +46,7 @@ export function diagnoseKB(db, kb, signal) {
46
46
  coverage_percent: 100,
47
47
  total_source_files: 0,
48
48
  indexed_files: kb.file_count,
49
- skipped_files: {
50
- total: 0,
51
- by_reason: {
52
- suggested_excluded: 0,
53
- oversized: 0,
54
- binary: 0,
55
- unreadable: 0,
56
- inaccessible: 0,
57
- },
58
- samples: [],
59
- },
49
+ skipped_files: createSkippedScanStats(),
60
50
  job,
61
51
  };
62
52
  throwIfAborted(signal);
@@ -3,7 +3,7 @@ import { join, resolve } from "node:path";
3
3
  import { diagnoseKB } from "./diagnostics/health.js";
4
4
  import { dispose as disposeEmbedding, embedDocuments, embedQuery, prepareForShutdown as prepareEmbeddingForShutdown, } from "./embedding/provider.js";
5
5
  import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
6
- import { buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, iterateScannableFiles, iterateScannedFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
6
+ import { addSkippedScanEntry, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
7
7
  import { extractSymbols } from "./indexer/symbols.js";
8
8
  import { shutdownModelWorker } from "./model-worker-client.js";
9
9
  import { searchBM25 } from "./search/bm25.js";
@@ -416,6 +416,48 @@ async function extractSourceFileContent(filePath, signal) {
416
416
  throwIfAborted(signal);
417
417
  return { content: readFileSync(filePath, "utf-8"), fileType: "text" };
418
418
  }
419
+ function looksLikeLocalPath(source) {
420
+ const trimmed = source.trim();
421
+ if (!trimmed || trimmed.includes("\n") || trimmed.includes("\r"))
422
+ return false;
423
+ if (/^(?:https?:)?\/\//.test(trimmed))
424
+ return false;
425
+ if (trimmed.startsWith(".") || trimmed.startsWith("/") || trimmed.startsWith("~"))
426
+ return true;
427
+ if (trimmed.includes("/") || trimmed.includes("\\"))
428
+ return true;
429
+ return /\.[A-Za-z0-9][A-Za-z0-9_-]{0,15}$/.test(trimmed);
430
+ }
431
+ function classifySource(source) {
432
+ const resolvedSource = resolve(source);
433
+ const isUrl = source.startsWith("http://") || source.startsWith("https://");
434
+ const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
435
+ const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
436
+ if (!isUrl && !isDir && !isFile && looksLikeLocalPath(source)) {
437
+ throw new Error(`Local path does not exist: ${source}. Pass inline text only when the source itself is text.`);
438
+ }
439
+ return {
440
+ resolvedSource,
441
+ isUrl,
442
+ isDir,
443
+ isFile,
444
+ sourceType: isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text",
445
+ };
446
+ }
447
+ async function extractScannableFileContent(file, signal) {
448
+ return extractSourceFileContent(file.path, signal);
449
+ }
450
+ async function extractScannableFileContentOrSkip(file, skipped, signal) {
451
+ try {
452
+ return await extractScannableFileContent(file, signal);
453
+ }
454
+ catch (error) {
455
+ if (signal?.aborted || isCancellationError(error))
456
+ throw error;
457
+ addSkippedScanEntry(skipped, { path: file.relPath, reason: "extraction_failed", size: file.size });
458
+ return undefined;
459
+ }
460
+ }
419
461
  function throwIfAborted(signal) {
420
462
  if (signal?.aborted)
421
463
  throw new Error("Cancelled");
@@ -518,11 +560,7 @@ export class KnowledgeEngine {
518
560
  }
519
561
  plan(source, options = {}, signal) {
520
562
  throwIfAborted(signal);
521
- const resolvedSource = resolve(source);
522
- const isUrl = source.startsWith("http://") || source.startsWith("https://");
523
- const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
524
- const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
525
- const sourceType = isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text";
563
+ const { resolvedSource, isDir, isFile, sourceType } = classifySource(source);
526
564
  if (isDir) {
527
565
  const plan = planDirectoryScan(resolvedSource, toScanOptions(options), signal);
528
566
  return {
@@ -535,7 +573,7 @@ export class KnowledgeEngine {
535
573
  }
536
574
  if (isFile) {
537
575
  const skipped = createSkippedScanStats();
538
- const supportedDocument = resolvedSource.endsWith(".pdf") || resolvedSource.endsWith(".docx") || resolvedSource.endsWith(".doc");
576
+ const supportedDocument = isSupportedDocumentFile(resolvedSource);
539
577
  if (!supportedDocument && !isReadableTextFile(resolvedSource)) {
540
578
  skipped.total = 1;
541
579
  skipped.by_reason.binary = 1;
@@ -575,11 +613,7 @@ export class KnowledgeEngine {
575
613
  throw new Error("Engine not initialized");
576
614
  throwIfAborted(signal);
577
615
  const db = this.db;
578
- const resolvedSource = resolve(source);
579
- const isUrl = source.startsWith("http://") || source.startsWith("https://");
580
- const isDir = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isDirectory();
581
- const isFile = !isUrl && existsSync(resolvedSource) && statSync(resolvedSource).isFile();
582
- const sourceType = isUrl ? "url" : isDir ? "directory" : isFile ? "file" : "text";
616
+ const { resolvedSource, isUrl, isDir, isFile, sourceType } = classifySource(source);
583
617
  const scanOptions = toScanOptions(options);
584
618
  const existingKB = getKBByName(db, name);
585
619
  if (existingKB) {
@@ -605,7 +639,8 @@ export class KnowledgeEngine {
605
639
  const pendingChunks = [];
606
640
  const startedAt = Date.now();
607
641
  let latestSkippedTotal = 0;
608
- const reportProgress = (phase, processedFiles, totalFiles, skippedTotal = 0) => {
642
+ let latestSkippedSummary = "none";
643
+ const reportProgress = (phase, processedFiles, totalFiles, skippedTotal = latestSkippedTotal) => {
609
644
  latestSkippedTotal = skippedTotal;
610
645
  const elapsed = Date.now() - startedAt;
611
646
  const chunkRate = chunkCount / Math.max(1, elapsed / 1000);
@@ -693,20 +728,27 @@ export class KnowledgeEngine {
693
728
  onProgress?.(scanningMessage);
694
729
  const skipped = createSkippedScanStats();
695
730
  let processedFiles = 0;
696
- for (const file of iterateScannedFiles(resolvedSource, skipped, scanOptions)) {
731
+ for (const file of iterateScannableFiles(resolvedSource, skipped, scanOptions)) {
697
732
  if (signal?.aborted)
698
733
  throw new Error("Cancelled");
699
- const chunks = await chunkFile(file.content, file.relPath);
734
+ const extracted = await extractScannableFileContentOrSkip(file, skipped, signal);
735
+ if (!extracted) {
736
+ latestSkippedTotal = skipped.total;
737
+ continue;
738
+ }
739
+ const chunks = await chunkFile(extracted.content, file.relPath);
700
740
  processedFiles++;
701
741
  latestSkippedTotal = skipped.total;
702
742
  if (chunks.length > 0)
703
743
  fileCount++;
704
- addSymbols(file.content, file.relPath, file.fileType);
744
+ addSymbols(extracted.content, file.relPath, extracted.fileType);
705
745
  await addChunks(chunks, processedFiles, plan.files);
706
746
  if (processedFiles % 25 === 0)
707
747
  reportProgress("Chunking", processedFiles, plan.files, skipped.total);
708
748
  }
709
- const finalizingMessage = `Scanned ${processedFiles} files, skipped ${skipped.total} (${summarizeSkippedScan(skipped)}), finalizing...`;
749
+ latestSkippedTotal = skipped.total;
750
+ latestSkippedSummary = summarizeSkippedScan(skipped);
751
+ const finalizingMessage = `Scanned ${processedFiles} files, skipped ${skipped.total} (${latestSkippedSummary}), finalizing...`;
710
752
  updateIndexingJob(this.db, kb.id, {
711
753
  phase: "finalizing",
712
754
  message: finalizingMessage,
@@ -734,7 +776,8 @@ export class KnowledgeEngine {
734
776
  const savedKB = getKB(this.db, kb.id);
735
777
  if (!savedKB)
736
778
  throw new Error(`Knowledge base disappeared after add: ${kb.id}`);
737
- const readyMessage = `Ready: ${chunkCount} chunks from ${savedFileCount} files in ${formatDuration(Date.now() - startedAt)}`;
779
+ const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
780
+ const readyMessage = `Ready: ${chunkCount} chunks from ${savedFileCount} files in ${formatDuration(Date.now() - startedAt)}${skippedReadySuffix}`;
738
781
  updateIndexingJob(this.db, kb.id, {
739
782
  phase: "ready",
740
783
  message: readyMessage,
@@ -800,6 +843,8 @@ export class KnowledgeEngine {
800
843
  const vectorPath = this.vectorPathFor(kb.id);
801
844
  addedVectorPath = tempVectorPath(`${vectorPath}.added`);
802
845
  addedVectorWriter = openVectorWriter(addedVectorPath);
846
+ let latestSkippedTotal = 0;
847
+ let latestSkippedSummary = "none";
803
848
  const existingHashes = new Map();
804
849
  let existingIndex = 0;
805
850
  for (const chunk of iterateChunksByKB(this.db, kb.id)) {
@@ -903,12 +948,15 @@ export class KnowledgeEngine {
903
948
  });
904
949
  onProgress?.(planningMessage);
905
950
  const skipped = createSkippedScanStats();
906
- for (const file of iterateScannedFiles(kb.source_path, skipped, scanOptions)) {
951
+ for (const file of iterateScannableFiles(kb.source_path, skipped, scanOptions)) {
907
952
  if (signal?.aborted)
908
953
  throw new Error("Cancelled");
954
+ const extracted = await extractScannableFileContentOrSkip(file, skipped, signal);
955
+ if (!extracted)
956
+ continue;
909
957
  scannedFiles++;
910
- stagedSymbols.push(...extractSymbols(file.content, file.relPath, file.fileType));
911
- await processChunks(await chunkFile(file.content, file.relPath));
958
+ stagedSymbols.push(...extractSymbols(extracted.content, file.relPath, extracted.fileType));
959
+ await processChunks(await chunkFile(extracted.content, file.relPath));
912
960
  if (scannedFiles % 25 === 0) {
913
961
  const message = `Scanned ${scannedFiles} files, ${scannedChunks} chunks, skipped ${skipped.total}, +${addedCount} =${unchanged}`;
914
962
  updateIndexingJob(this.db, kb.id, {
@@ -924,7 +972,9 @@ export class KnowledgeEngine {
924
972
  onProgress?.(message);
925
973
  }
926
974
  }
927
- const message = `Scanned ${scannedFiles} files, skipped ${skipped.total} (${summarizeSkippedScan(skipped)}), reconciling deletes...`;
975
+ latestSkippedTotal = skipped.total;
976
+ latestSkippedSummary = summarizeSkippedScan(skipped);
977
+ const message = `Scanned ${scannedFiles} files, skipped ${skipped.total} (${latestSkippedSummary}), reconciling deletes...`;
928
978
  updateIndexingJob(this.db, kb.id, {
929
979
  phase: "reconciling",
930
980
  message,
@@ -1019,7 +1069,8 @@ export class KnowledgeEngine {
1019
1069
  insertSymbols(this.db, kb.id, stagedSymbols);
1020
1070
  updateKBCounts(this.db, kb.id, finalChunkCount, getFileCount(this.db, kb.id));
1021
1071
  updateKBStatus(this.db, kb.id, "ready");
1022
- const readyMessage = `Ready: +${addedCount} -${idsToRemove.length} =${unchanged}`;
1072
+ const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
1073
+ const readyMessage = `Ready: +${addedCount} -${idsToRemove.length} =${unchanged}${skippedReadySuffix}`;
1023
1074
  updateIndexingJob(this.db, kb.id, {
1024
1075
  phase: "ready",
1025
1076
  message: readyMessage,
@@ -18,7 +18,7 @@ export interface ScannableFile {
18
18
  }
19
19
  export interface SkippedScanEntry {
20
20
  path: string;
21
- reason: "suggested_excluded" | "oversized" | "binary" | "unreadable" | "inaccessible";
21
+ reason: "suggested_excluded" | "oversized" | "binary" | "unreadable" | "inaccessible" | "extraction_failed";
22
22
  size?: number;
23
23
  }
24
24
  export interface ScanResult {
@@ -30,7 +30,9 @@ export interface ScanResult {
30
30
  };
31
31
  }
32
32
  export declare function createSkippedScanStats(): ScanResult["skipped"];
33
+ export declare function addSkippedScanEntry(skipped: ScanResult["skipped"], entry: SkippedScanEntry): void;
33
34
  export declare function summarizeSkippedScan(skipped: ScanResult["skipped"]): string;
35
+ export declare function isSupportedDocumentFile(filePath: string): boolean;
34
36
  export declare function iterateScannableFiles(dirPath: string, skipped?: ScanResult["skipped"], options?: ScanOptions): Generator<ScannableFile>;
35
37
  export declare function iterateScannedFiles(dirPath: string, skipped?: ScanResult["skipped"], options?: ScanOptions): Generator<ScannedFile>;
36
38
  export declare function iterateScannedFilesAsync(dirPath: string, onFile: (file: ScannedFile) => Promise<void> | void, skipped?: ScanResult["skipped"], options?: ScanOptions): Promise<ScanResult["skipped"]>;
@@ -3,53 +3,50 @@ import { closeSync, existsSync, openSync, readdirSync, readFileSync, readSync, s
3
3
  import { extname, join, relative, sep } from "node:path";
4
4
  import ignore from "ignore";
5
5
  const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10 MB
6
- const BINARY_EXTENSIONS = new Set([
7
- ".png",
8
- ".jpg",
9
- ".jpeg",
10
- ".gif",
11
- ".ico",
12
- ".bmp",
13
- ".webp",
14
- ".svg",
15
- ".woff",
16
- ".woff2",
17
- ".ttf",
18
- ".eot",
19
- ".otf",
20
- ".zip",
21
- ".gz",
22
- ".tar",
23
- ".bz2",
24
- ".7z",
25
- ".rar",
26
- ".pdf",
27
- ".doc",
28
- ".docx",
29
- ".xls",
30
- ".xlsx",
31
- ".ppt",
32
- ".exe",
33
- ".dll",
34
- ".so",
35
- ".dylib",
36
- ".node",
37
- ".db",
38
- ".sqlite",
39
- ".bin",
40
- ".dat",
41
- ".lock",
42
- ".mp3",
43
- ".mp4",
44
- ".wav",
45
- ".avi",
46
- ".mov",
47
- ".webm",
48
- ".wasm",
49
- ".o",
50
- ".a",
51
- ".lib",
52
- ]);
6
+ const DOCUMENT_EXTENSIONS = { ".doc": true, ".docx": true, ".pdf": true };
7
+ const BINARY_EXTENSIONS = {
8
+ ".7z": true,
9
+ ".a": true,
10
+ ".avi": true,
11
+ ".bin": true,
12
+ ".bmp": true,
13
+ ".bz2": true,
14
+ ".dat": true,
15
+ ".db": true,
16
+ ".dll": true,
17
+ ".dylib": true,
18
+ ".eot": true,
19
+ ".exe": true,
20
+ ".gif": true,
21
+ ".gz": true,
22
+ ".ico": true,
23
+ ".jpg": true,
24
+ ".jpeg": true,
25
+ ".lib": true,
26
+ ".lock": true,
27
+ ".mov": true,
28
+ ".mp3": true,
29
+ ".mp4": true,
30
+ ".node": true,
31
+ ".o": true,
32
+ ".otf": true,
33
+ ".ppt": true,
34
+ ".rar": true,
35
+ ".so": true,
36
+ ".sqlite": true,
37
+ ".svg": true,
38
+ ".tar": true,
39
+ ".ttf": true,
40
+ ".wasm": true,
41
+ ".wav": true,
42
+ ".webm": true,
43
+ ".webp": true,
44
+ ".woff": true,
45
+ ".woff2": true,
46
+ ".xls": true,
47
+ ".xlsx": true,
48
+ ".zip": true,
49
+ };
53
50
  const DEFAULT_SUGGESTED_EXCLUDE = [
54
51
  "node_modules",
55
52
  ".git",
@@ -111,6 +108,7 @@ export function createSkippedScanStats() {
111
108
  binary: 0,
112
109
  unreadable: 0,
113
110
  inaccessible: 0,
111
+ extraction_failed: 0,
114
112
  },
115
113
  samples: [],
116
114
  };
@@ -121,6 +119,9 @@ function addSkipped(skipped, entry) {
121
119
  if (skipped.samples.length < MAX_SKIPPED_SAMPLES)
122
120
  skipped.samples.push(entry);
123
121
  }
122
+ export function addSkippedScanEntry(skipped, entry) {
123
+ addSkipped(skipped, entry);
124
+ }
124
125
  export function summarizeSkippedScan(skipped) {
125
126
  return (Object.entries(skipped.by_reason)
126
127
  .filter(([, count]) => count > 0)
@@ -157,14 +158,22 @@ function detectFileType(filePath) {
157
158
  ".zsh": "shell",
158
159
  ".sql": "sql",
159
160
  ".graphql": "graphql",
161
+ ".pdf": "pdf",
162
+ ".doc": "docx",
163
+ ".docx": "docx",
160
164
  ".txt": "text",
161
165
  ".csv": "text",
162
166
  ".log": "text",
163
167
  };
164
168
  return map[ext] ?? "text";
165
169
  }
170
+ export function isSupportedDocumentFile(filePath) {
171
+ return DOCUMENT_EXTENSIONS[extname(filePath).toLowerCase()] === true;
172
+ }
166
173
  function isBinaryFile(filePath) {
167
- if (BINARY_EXTENSIONS.has(extname(filePath).toLowerCase()))
174
+ if (isSupportedDocumentFile(filePath))
175
+ return false;
176
+ if (BINARY_EXTENSIONS[extname(filePath).toLowerCase()] === true)
168
177
  return true;
169
178
  let fd;
170
179
  try {
@@ -42,9 +42,11 @@ API embedding failures intentionally surface by default. Silent fallback can hid
42
42
 
43
43
  Search diagnostics include result provenance in both text output and structured `details`: chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
44
44
 
45
+ `knowledge_search` supports `file_type` and `path_pattern` filters. `hybrid` mode is lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording.
46
+
45
47
  `knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
46
48
 
47
- `knowledge_symbol_search` uses a lightweight symbol index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find code symbols, route-like handlers, Markdown headings, config keys, and environment variables. Older file/directory/URL KBs without symbol metadata should be updated before relying on symbol lookup. Imported portable KBs and inline text KBs do not retain an active source path; rebuild their symbols by re-importing or re-adding the content.
49
+ `knowledge_symbol_search` uses a lightweight declaration-pattern index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find common code symbols, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; for methods, uncommon syntax, or empty important lookups, use `knowledge_search` mode `fast`/`adaptive` and check `knowledge_doctor` for stale or missing symbol metadata. Older file/directory/URL KBs without symbol metadata should be updated before relying on symbol lookup. Imported portable KBs and inline text KBs do not retain an active source path; rebuild their symbols by re-importing or re-adding the content.
48
50
 
49
51
  ## Test and Release Fixtures
50
52
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.5.0",
3
+ "version": "0.5.2",
4
4
  "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
@@ -51,7 +51,7 @@
51
51
  },
52
52
  "dependencies": {
53
53
  "@huggingface/transformers": "3.8.1",
54
- "better-sqlite3": "11.9.1",
54
+ "better-sqlite3": "12.11.1",
55
55
  "ignore": "7.0.5",
56
56
  "mammoth": "1.12.0",
57
57
  "tree-sitter": "0.22.4",