pi-knowledge 0.9.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/README.md +7 -2
- package/dist/src/engine.js +52 -3
- package/dist/src/indexer/chunker.js +9 -4
- package/dist/src/indexer/chunkers/code-ast.d.ts +3 -1
- package/dist/src/indexer/chunkers/code-ast.js +201 -22
- package/dist/src/search/ranking.js +9 -1
- package/docs/configuration.md +9 -0
- package/package.json +10 -5
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,23 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.10.1] - 2026-09-10
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Repaired `knowledge_update` recovery for knowledge bases whose SQLite chunks are ahead of a short or truncated vector file; update now rebuilds or re-embeds missing vectors instead of leaving the KB in `error` (#13).
|
|
7
|
+
- Aligned the published Tree-sitter runtime and grammar dependency graph so clean npm installs no longer emit peer override warnings for `tree-sitter@0.25.1` incompatibilities (#14).
|
|
8
|
+
|
|
9
|
+
## [0.10.0] - 2026-08-27
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- Added Bash AST indexing for `.sh` and `.bash` scripts, including both `name() {}` and `function name {}` forms with bounded fallback chunking (#12).
|
|
13
|
+
- Added GNU C AST indexing for `.c` source files, covering functions, prototypes, structs, unions, enums, typedefs, reliable preprocessor symbols, and `static` metadata (#12).
|
|
14
|
+
- Added C++ AST indexing for `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hh`, and `.hxx` files, covering namespaces, classes, methods, constructors, destructors, enums, templates, and access visibility where the parser reports it reliably (#12).
|
|
15
|
+
- Added QML AST indexing for `.qml` files, covering imports, component/object hierarchy, ids, properties, signals, handlers, bindings, and embedded JavaScript functions (#12).
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- Kept ambiguous `.h` headers classified as text by default; use typed C++ header extensions for C++ AST indexing (#12).
|
|
19
|
+
- Parser errors in Bash, C, C++, and QML now intentionally fall back to existing text chunking instead of indexing partial structural claims (#12).
|
|
20
|
+
|
|
3
21
|
## [0.9.0] - 2026-08-26
|
|
4
22
|
|
|
5
23
|
### Added
|
package/README.md
CHANGED
|
@@ -23,7 +23,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
23
23
|
|
|
24
24
|
- **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
|
|
25
25
|
- **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
|
|
26
|
-
- **Code-aware indexing**: recursive AST chunking for TypeScript/JavaScript, Python, Go, Rust, and
|
|
26
|
+
- **Code-aware indexing**: recursive AST chunking for TypeScript/JavaScript, Python, Go, Rust, Java, Bash, GNU C, C++, and QML, with bounded fallback chunks for oversized declarations.
|
|
27
27
|
- **Lightweight symbol lookup**: indexes functions, methods, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
|
|
28
28
|
- **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
|
|
29
29
|
- **Large-project stability**: persisted indexing progress, capped batches, streaming vector scans, and stuck-job detection.
|
|
@@ -43,7 +43,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
43
43
|
| **Diversity reranking** | ✅ | ❌ | ❌ |
|
|
44
44
|
| **Incremental re-indexing** | ✅ | ❌ | ❌ |
|
|
45
45
|
| **File watcher (auto-update)** | ✅ | ❌ | ❌ |
|
|
46
|
-
| **Code-aware chunking** | ✅ (TS/JS/Py/Go/Rust/Java) | ❌ | ❌ |
|
|
46
|
+
| **Code-aware chunking** | ✅ (TS/JS/Py/Go/Rust/Java/Bash/C/C++/QML) | ❌ | ❌ |
|
|
47
47
|
| **Symbol/config/heading lookup** | ✅ | ❌ | ❌ |
|
|
48
48
|
| **Local embeddings (zero API)** | ✅ | ❌ | ✅ (qmd) |
|
|
49
49
|
| **Index quality diagnostics** | ✅ | ❌ | ❌ |
|
|
@@ -83,6 +83,11 @@ pi install npm:pi-knowledge
|
|
|
83
83
|
# Install for OMP
|
|
84
84
|
omp install npm:pi-knowledge
|
|
85
85
|
|
|
86
|
+
# On minimal Ubuntu/Debian servers, install native build tools first because
|
|
87
|
+
# OMP's Bun-based npm install compiles better-sqlite3 from source.
|
|
88
|
+
sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
89
|
+
omp install npm:pi-knowledge
|
|
90
|
+
|
|
86
91
|
# Or from source
|
|
87
92
|
pi install ./pi-knowledge
|
|
88
93
|
omp install ./pi-knowledge
|
package/dist/src/engine.js
CHANGED
|
@@ -23,6 +23,11 @@ function isCancellationError(error) {
|
|
|
23
23
|
function tempVectorPath(vectorPath) {
|
|
24
24
|
return `${vectorPath}.${Date.now()}.${Math.random().toString(16).slice(2)}.tmp`;
|
|
25
25
|
}
|
|
26
|
+
function assertEmbeddingBatchSize(vectors, expected, operation) {
|
|
27
|
+
if (vectors.length !== expected) {
|
|
28
|
+
throw new Error(`Embedding provider returned ${vectors.length} vectors for ${expected} ${operation} chunks`);
|
|
29
|
+
}
|
|
30
|
+
}
|
|
26
31
|
function retrievalModeFor(mode) {
|
|
27
32
|
if (mode === "auto")
|
|
28
33
|
return "hybrid";
|
|
@@ -759,6 +764,7 @@ export class KnowledgeEngine {
|
|
|
759
764
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
760
765
|
if (signal?.aborted)
|
|
761
766
|
throw new Error("Cancelled");
|
|
767
|
+
assertEmbeddingBatchSize(vectors, batch.length, "add");
|
|
762
768
|
persistEmbeddingMetadata(db, kb.id, embeddingConfig, vectors);
|
|
763
769
|
insertChunks(db, kb.id, batch);
|
|
764
770
|
writer.append(vectors);
|
|
@@ -923,7 +929,7 @@ export class KnowledgeEngine {
|
|
|
923
929
|
const embeddingConfig = resolveEmbeddingConfig();
|
|
924
930
|
const embeddingModel = embeddingConfigLabel(embeddingConfig);
|
|
925
931
|
const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
|
|
926
|
-
|
|
932
|
+
let canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
|
|
927
933
|
let replacementVectorPath;
|
|
928
934
|
let addedVectorPath;
|
|
929
935
|
let addedVectorWriter;
|
|
@@ -943,6 +949,24 @@ export class KnowledgeEngine {
|
|
|
943
949
|
existingHashes.set(chunk.content_hash, entries);
|
|
944
950
|
existingIndex++;
|
|
945
951
|
}
|
|
952
|
+
if (canReuseExistingVectors) {
|
|
953
|
+
const oldVectorReader = openVectorReader(vectorPath);
|
|
954
|
+
try {
|
|
955
|
+
if (!oldVectorReader ||
|
|
956
|
+
oldVectorReader.count !== existingIndex ||
|
|
957
|
+
oldVectorReader.dim !== kb.embedding_dimension) {
|
|
958
|
+
canReuseExistingVectors = false;
|
|
959
|
+
const vectorCount = oldVectorReader?.count ?? 0;
|
|
960
|
+
const vectorDimension = oldVectorReader?.dim ?? 0;
|
|
961
|
+
const message = `Stored vector file for "${kb.name}" is incomplete or incompatible (${vectorCount} vectors/${vectorDimension}d for ${existingIndex} chunks/${kb.embedding_dimension}d); rebuilding all vectors`;
|
|
962
|
+
updateIndexingJob(this.db, kb.id, { phase: "embedding", message });
|
|
963
|
+
onProgress?.(message);
|
|
964
|
+
}
|
|
965
|
+
}
|
|
966
|
+
finally {
|
|
967
|
+
oldVectorReader?.close();
|
|
968
|
+
}
|
|
969
|
+
}
|
|
946
970
|
const reusableHashes = canReuseExistingVectors
|
|
947
971
|
? existingHashes
|
|
948
972
|
: new Map();
|
|
@@ -983,6 +1007,7 @@ export class KnowledgeEngine {
|
|
|
983
1007
|
const newVectors = await embedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
|
|
984
1008
|
if (signal?.aborted)
|
|
985
1009
|
throw new Error("Cancelled");
|
|
1010
|
+
assertEmbeddingBatchSize(newVectors, batch.length, "update");
|
|
986
1011
|
addedVectorWriter.append(newVectors);
|
|
987
1012
|
for (let i = 0; i < batch.length; i++) {
|
|
988
1013
|
const indexes = newVectorIndexByHash.get(batch[i].content_hash) ?? [];
|
|
@@ -1122,6 +1147,29 @@ export class KnowledgeEngine {
|
|
|
1122
1147
|
const indexes = indexesByHash.get(hash);
|
|
1123
1148
|
return indexes?.shift();
|
|
1124
1149
|
};
|
|
1150
|
+
const repairMissingVector = async (chunk) => {
|
|
1151
|
+
const message = `Re-embedding chunk with missing stored vector: ${chunk.id}`;
|
|
1152
|
+
if (!this.db)
|
|
1153
|
+
throw new Error("Engine not initialized");
|
|
1154
|
+
updateIndexingJob(this.db, kb.id, {
|
|
1155
|
+
phase: "embedding",
|
|
1156
|
+
message,
|
|
1157
|
+
processed_files: scannedFiles,
|
|
1158
|
+
processed_chunks: finalChunkCount,
|
|
1159
|
+
added_chunks: addedCount,
|
|
1160
|
+
removed_chunks: idsToRemove.length,
|
|
1161
|
+
unchanged_chunks: unchanged,
|
|
1162
|
+
});
|
|
1163
|
+
onProgress?.(message);
|
|
1164
|
+
const repairedVectors = await embedDocuments([buildChunkEmbeddingText(chunk)], signal);
|
|
1165
|
+
if (signal?.aborted)
|
|
1166
|
+
throw new Error("Cancelled");
|
|
1167
|
+
assertEmbeddingBatchSize(repairedVectors, 1, "repair");
|
|
1168
|
+
const [repairedVector] = repairedVectors;
|
|
1169
|
+
if (!repairedVector)
|
|
1170
|
+
throw new Error(`Embedding provider returned no repaired vector for chunk: ${chunk.id}`);
|
|
1171
|
+
return repairedVector;
|
|
1172
|
+
};
|
|
1125
1173
|
try {
|
|
1126
1174
|
for (const chunk of iterateChunksByKB(this.db, kb.id)) {
|
|
1127
1175
|
if (signal?.aborted)
|
|
@@ -1130,13 +1178,13 @@ export class KnowledgeEngine {
|
|
|
1130
1178
|
continue;
|
|
1131
1179
|
const oldVectorIndex = takeVectorIndex(oldVectorIndexByHash, chunk.content_hash);
|
|
1132
1180
|
const newVectorIndex = takeVectorIndex(newVectorIndexByHash, chunk.content_hash);
|
|
1133
|
-
|
|
1181
|
+
let vector = oldVectorReader && oldVectorIndex !== undefined
|
|
1134
1182
|
? oldVectorReader.read(oldVectorIndex)
|
|
1135
1183
|
: newVectorReader && newVectorIndex !== undefined
|
|
1136
1184
|
? newVectorReader.read(newVectorIndex)
|
|
1137
1185
|
: undefined;
|
|
1138
1186
|
if (!vector)
|
|
1139
|
-
|
|
1187
|
+
vector = await repairMissingVector(chunk);
|
|
1140
1188
|
finalEmbeddingDimension ??= vector.length;
|
|
1141
1189
|
vectorWriter.append([vector]);
|
|
1142
1190
|
finalChunkCount++;
|
|
@@ -1816,6 +1864,7 @@ export class KnowledgeEngine {
|
|
|
1816
1864
|
onProgress?.(message);
|
|
1817
1865
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
1818
1866
|
throwIfAborted(signal);
|
|
1867
|
+
assertEmbeddingBatchSize(vectors, batch.length, "import");
|
|
1819
1868
|
persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
|
|
1820
1869
|
insertChunks(this.db, kb.id, batch);
|
|
1821
1870
|
insertSymbols(this.db, kb.id, symbols);
|
|
@@ -143,8 +143,13 @@ function detectFileType(filePath) {
|
|
|
143
143
|
".java": "java",
|
|
144
144
|
".c": "c",
|
|
145
145
|
".cpp": "cpp",
|
|
146
|
-
".
|
|
146
|
+
".cc": "cpp",
|
|
147
|
+
".cxx": "cpp",
|
|
148
|
+
".h": "text",
|
|
147
149
|
".hpp": "cpp",
|
|
150
|
+
".hh": "cpp",
|
|
151
|
+
".hxx": "cpp",
|
|
152
|
+
".qml": "qml",
|
|
148
153
|
".md": "markdown",
|
|
149
154
|
".mdx": "markdown",
|
|
150
155
|
".json": "json",
|
|
@@ -154,8 +159,8 @@ function detectFileType(filePath) {
|
|
|
154
159
|
".html": "html",
|
|
155
160
|
".css": "css",
|
|
156
161
|
".scss": "css",
|
|
157
|
-
".sh": "
|
|
158
|
-
".bash": "
|
|
162
|
+
".sh": "bash",
|
|
163
|
+
".bash": "bash",
|
|
159
164
|
".zsh": "shell",
|
|
160
165
|
".sql": "sql",
|
|
161
166
|
".graphql": "graphql",
|
|
@@ -525,7 +530,7 @@ export function chunkText(content, filePath) {
|
|
|
525
530
|
return chunks;
|
|
526
531
|
}
|
|
527
532
|
export function isCodeFileType(fileType) {
|
|
528
|
-
return ["typescript", "javascript", "python", "go", "rust", "java"].includes(fileType);
|
|
533
|
+
return ["typescript", "javascript", "python", "go", "rust", "java", "bash", "c", "cpp", "qml"].includes(fileType);
|
|
529
534
|
}
|
|
530
535
|
export async function analyzeIndexableContent(content, filePath, fileType = detectFileType(filePath)) {
|
|
531
536
|
const detectedFileType = detectFileType(filePath);
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { ChunkInsert, KnowledgeSymbolInsert } from "../../storage/sqlite.ts";
|
|
2
|
-
type StructureKind = "module" | "class" | "interface" | "type" | "function" | "method" | "constructor" | "variable";
|
|
2
|
+
type StructureKind = "module" | "namespace" | "import" | "class" | "interface" | "type" | "function" | "method" | "constructor" | "destructor" | "property" | "signal" | "handler" | "id" | "binding" | "variable";
|
|
3
3
|
export interface CodeStructureNode {
|
|
4
4
|
kind: StructureKind;
|
|
5
5
|
astType: string;
|
|
@@ -12,6 +12,8 @@ export interface CodeStructureNode {
|
|
|
12
12
|
endByte: number;
|
|
13
13
|
exported: boolean;
|
|
14
14
|
decorators: string[];
|
|
15
|
+
modifiers: string[];
|
|
16
|
+
visibility?: string;
|
|
15
17
|
parentSymbol?: string;
|
|
16
18
|
scope: string[];
|
|
17
19
|
astPath: string[];
|
|
@@ -17,8 +17,8 @@ const LANGS = {
|
|
|
17
17
|
},
|
|
18
18
|
javascript: {
|
|
19
19
|
grammar: async () => {
|
|
20
|
-
const m = await import("tree-sitter-
|
|
21
|
-
return
|
|
20
|
+
const m = await import("tree-sitter-javascript");
|
|
21
|
+
return moduleDefault(m) ?? m;
|
|
22
22
|
},
|
|
23
23
|
fileType: "javascript",
|
|
24
24
|
},
|
|
@@ -50,23 +50,64 @@ const LANGS = {
|
|
|
50
50
|
},
|
|
51
51
|
fileType: "java",
|
|
52
52
|
},
|
|
53
|
+
bash: {
|
|
54
|
+
grammar: async () => {
|
|
55
|
+
const m = await import("tree-sitter-bash");
|
|
56
|
+
return moduleDefault(m) ?? m;
|
|
57
|
+
},
|
|
58
|
+
fileType: "bash",
|
|
59
|
+
fallbackOnParseError: true,
|
|
60
|
+
},
|
|
61
|
+
c: {
|
|
62
|
+
grammar: async () => {
|
|
63
|
+
const m = await import("tree-sitter-c");
|
|
64
|
+
return moduleDefault(m) ?? m;
|
|
65
|
+
},
|
|
66
|
+
fileType: "c",
|
|
67
|
+
fallbackOnParseError: true,
|
|
68
|
+
},
|
|
69
|
+
cpp: {
|
|
70
|
+
grammar: async () => {
|
|
71
|
+
const m = await import("tree-sitter-cpp");
|
|
72
|
+
return moduleDefault(m) ?? m;
|
|
73
|
+
},
|
|
74
|
+
fileType: "cpp",
|
|
75
|
+
fallbackOnParseError: true,
|
|
76
|
+
},
|
|
77
|
+
qml: {
|
|
78
|
+
grammar: async () => {
|
|
79
|
+
const m = await import("tree-sitter-qmljs");
|
|
80
|
+
return moduleDefault(m) ?? m;
|
|
81
|
+
},
|
|
82
|
+
fileType: "qml",
|
|
83
|
+
fallbackOnParseError: true,
|
|
84
|
+
},
|
|
53
85
|
};
|
|
86
|
+
const NAMESPACE_TYPES = new Set(["namespace_definition"]);
|
|
87
|
+
const QML_TYPES = new Set(["ui_binding", "ui_import", "ui_object_definition", "ui_property", "ui_signal"]);
|
|
54
88
|
const CLASS_TYPES = new Set([
|
|
55
89
|
"abstract_class_declaration",
|
|
56
90
|
"class_declaration",
|
|
57
91
|
"class_definition",
|
|
92
|
+
"class_specifier",
|
|
58
93
|
"enum_declaration",
|
|
59
94
|
]);
|
|
60
95
|
const INTERFACE_TYPES = new Set(["interface_declaration"]);
|
|
61
96
|
const TYPE_TYPES = new Set([
|
|
62
97
|
"annotation_type_declaration",
|
|
63
98
|
"enum_item",
|
|
99
|
+
"enum_specifier",
|
|
64
100
|
"struct_item",
|
|
101
|
+
"struct_specifier",
|
|
65
102
|
"trait_item",
|
|
66
103
|
"type_alias_declaration",
|
|
67
104
|
"type_declaration",
|
|
105
|
+
"type_definition",
|
|
106
|
+
"union_specifier",
|
|
68
107
|
]);
|
|
69
108
|
const FUNCTION_TYPES = new Set([
|
|
109
|
+
"declaration",
|
|
110
|
+
"field_declaration",
|
|
70
111
|
"function_declaration",
|
|
71
112
|
"function_definition",
|
|
72
113
|
"function_item",
|
|
@@ -75,7 +116,7 @@ const FUNCTION_TYPES = new Set([
|
|
|
75
116
|
]);
|
|
76
117
|
const METHOD_TYPES = new Set(["method_definition"]);
|
|
77
118
|
const CONSTRUCTOR_TYPES = new Set(["constructor_declaration"]);
|
|
78
|
-
const DECLARATION_WRAPPERS = new Set(["export_statement", "decorated_definition"]);
|
|
119
|
+
const DECLARATION_WRAPPERS = new Set(["export_statement", "decorated_definition", "template_declaration"]);
|
|
79
120
|
const VARIABLE_DECLARATOR_TYPES = new Set(["variable_declarator"]);
|
|
80
121
|
const FIELD_DEFINITION_TYPES = new Set(["public_field_definition", "field_definition", "property_declaration"]);
|
|
81
122
|
const FUNCTION_VALUE_TYPES = new Set([
|
|
@@ -127,13 +168,61 @@ function sliceByBytes(map, startByte, endByte) {
|
|
|
127
168
|
function nodeChildren(node) {
|
|
128
169
|
return node.namedChildren && node.namedChildren.length > 0 ? node.namedChildren : node.children;
|
|
129
170
|
}
|
|
171
|
+
function nameFromDeclarator(node) {
|
|
172
|
+
if (!node)
|
|
173
|
+
return undefined;
|
|
174
|
+
if (node.type === "qualified_identifier" || node.type === "destructor_name") {
|
|
175
|
+
const text = node.text.trim();
|
|
176
|
+
return text || undefined;
|
|
177
|
+
}
|
|
178
|
+
const direct = node.childForFieldName("name")?.text;
|
|
179
|
+
if (direct?.trim())
|
|
180
|
+
return direct.trim();
|
|
181
|
+
const nested = node.childForFieldName("declarator");
|
|
182
|
+
if (nested && nested !== node)
|
|
183
|
+
return nameFromDeclarator(nested);
|
|
184
|
+
if (["field_identifier", "identifier", "type_identifier", "variable_name", "word"].includes(node.type)) {
|
|
185
|
+
const text = node.text.trim();
|
|
186
|
+
return text || undefined;
|
|
187
|
+
}
|
|
188
|
+
return nodeChildren(node)
|
|
189
|
+
.map(nameFromDeclarator)
|
|
190
|
+
.find((name) => Boolean(name));
|
|
191
|
+
}
|
|
192
|
+
function firstNamedChildText(node, type) {
|
|
193
|
+
const match = nodeChildren(node).find((child) => child.type === type);
|
|
194
|
+
const text = match?.text.trim();
|
|
195
|
+
return text || undefined;
|
|
196
|
+
}
|
|
197
|
+
function qmlBindingName(node) {
|
|
198
|
+
const key = node.childForFieldName("name")?.text.trim() ?? firstNamedChildText(node, "identifier");
|
|
199
|
+
if (key !== "id")
|
|
200
|
+
return key || undefined;
|
|
201
|
+
const value = node.childForFieldName("value");
|
|
202
|
+
return value ? (firstNamedChildText(value, "identifier") ?? value.text.trim()) : key;
|
|
203
|
+
}
|
|
130
204
|
function getName(node) {
|
|
205
|
+
if (node.type === "constructor_declaration")
|
|
206
|
+
return "constructor";
|
|
207
|
+
if (node.type === "ui_object_definition")
|
|
208
|
+
return firstNamedChildText(node, "identifier");
|
|
209
|
+
if (node.type === "ui_import") {
|
|
210
|
+
return firstNamedChildText(node, "nested_identifier") ?? firstNamedChildText(node, "identifier");
|
|
211
|
+
}
|
|
212
|
+
if (node.type === "ui_binding")
|
|
213
|
+
return qmlBindingName(node);
|
|
214
|
+
if (node.type === "type_definition" ||
|
|
215
|
+
node.type === "declaration" ||
|
|
216
|
+
node.type === "field_declaration" ||
|
|
217
|
+
node.type === "function_definition") {
|
|
218
|
+
const declaratorName = nameFromDeclarator(node.childForFieldName("declarator"));
|
|
219
|
+
if (declaratorName)
|
|
220
|
+
return declaratorName;
|
|
221
|
+
}
|
|
131
222
|
const name = node.childForFieldName("name")?.text ?? node.childForFieldName("type")?.text;
|
|
132
223
|
if (name?.trim())
|
|
133
224
|
return name.trim();
|
|
134
|
-
|
|
135
|
-
return "constructor";
|
|
136
|
-
return undefined;
|
|
225
|
+
return nameFromDeclarator(node.childForFieldName("declarator"));
|
|
137
226
|
}
|
|
138
227
|
function nodeHasFunctionValue(node) {
|
|
139
228
|
return nodeChildren(node).some((child) => FUNCTION_VALUE_TYPES.has(child.type) || nodeHasFunctionValue(child));
|
|
@@ -145,15 +234,56 @@ function decoratorsFor(node, inherited = []) {
|
|
|
145
234
|
const decorators = node.children.filter((child) => child.type === "decorator").map((child) => child.text.trim());
|
|
146
235
|
return [...inherited, ...decorators].filter(Boolean);
|
|
147
236
|
}
|
|
237
|
+
function modifiersFor(node) {
|
|
238
|
+
return node.children
|
|
239
|
+
.filter((child) => child.type === "storage_class_specifier")
|
|
240
|
+
.map((child) => child.text.trim())
|
|
241
|
+
.filter(Boolean);
|
|
242
|
+
}
|
|
243
|
+
function leafName(name) {
|
|
244
|
+
if (!name)
|
|
245
|
+
return undefined;
|
|
246
|
+
return name.split("::").at(-1);
|
|
247
|
+
}
|
|
248
|
+
function parentFromQualifiedName(name) {
|
|
249
|
+
if (!name?.includes("::"))
|
|
250
|
+
return undefined;
|
|
251
|
+
const parts = name.split("::");
|
|
252
|
+
return parts.length > 1 ? parts.at(-2) : undefined;
|
|
253
|
+
}
|
|
148
254
|
function isStructuralCandidate(node) {
|
|
149
|
-
return (
|
|
255
|
+
return (QML_TYPES.has(node.type) ||
|
|
256
|
+
NAMESPACE_TYPES.has(node.type) ||
|
|
257
|
+
CLASS_TYPES.has(node.type) ||
|
|
150
258
|
INTERFACE_TYPES.has(node.type) ||
|
|
151
259
|
TYPE_TYPES.has(node.type) ||
|
|
152
260
|
FUNCTION_TYPES.has(node.type) ||
|
|
153
261
|
METHOD_TYPES.has(node.type) ||
|
|
154
262
|
CONSTRUCTOR_TYPES.has(node.type));
|
|
155
263
|
}
|
|
156
|
-
function kindForNode(node) {
|
|
264
|
+
function kindForNode(node, context) {
|
|
265
|
+
if (node.type === "ui_import")
|
|
266
|
+
return "import";
|
|
267
|
+
if (node.type === "ui_object_definition")
|
|
268
|
+
return "class";
|
|
269
|
+
if (node.type === "ui_property")
|
|
270
|
+
return "property";
|
|
271
|
+
if (node.type === "ui_signal")
|
|
272
|
+
return "signal";
|
|
273
|
+
if (node.type === "ui_binding") {
|
|
274
|
+
const key = node.childForFieldName("name")?.text.trim() ?? firstNamedChildText(node, "identifier");
|
|
275
|
+
if (key === "id")
|
|
276
|
+
return "id";
|
|
277
|
+
return key?.startsWith("on") ? "handler" : "binding";
|
|
278
|
+
}
|
|
279
|
+
if (node.type === "preproc_def" || node.type === "preproc_function_def")
|
|
280
|
+
return "variable";
|
|
281
|
+
if ((node.type === "declaration" || node.type === "field_declaration") &&
|
|
282
|
+
!nodeChildren(node).some((child) => child.type === "function_declarator")) {
|
|
283
|
+
return undefined;
|
|
284
|
+
}
|
|
285
|
+
if (NAMESPACE_TYPES.has(node.type))
|
|
286
|
+
return "namespace";
|
|
157
287
|
if (CLASS_TYPES.has(node.type))
|
|
158
288
|
return "class";
|
|
159
289
|
if (INTERFACE_TYPES.has(node.type))
|
|
@@ -164,15 +294,27 @@ function kindForNode(node) {
|
|
|
164
294
|
return "method";
|
|
165
295
|
if (CONSTRUCTOR_TYPES.has(node.type))
|
|
166
296
|
return "constructor";
|
|
167
|
-
if (FUNCTION_TYPES.has(node.type))
|
|
297
|
+
if (FUNCTION_TYPES.has(node.type)) {
|
|
298
|
+
const name = getName(node);
|
|
299
|
+
const leaf = leafName(name);
|
|
300
|
+
if ((context?.parentSymbol && leaf === context.parentSymbol) || parentFromQualifiedName(name) === leaf) {
|
|
301
|
+
return "constructor";
|
|
302
|
+
}
|
|
303
|
+
if (leaf?.startsWith("~"))
|
|
304
|
+
return "destructor";
|
|
305
|
+
if (node.type === "field_declaration" || (context?.parentSymbol && node.type === "declaration"))
|
|
306
|
+
return "method";
|
|
307
|
+
if (name?.includes("::"))
|
|
308
|
+
return leaf?.startsWith("~") ? "destructor" : "method";
|
|
168
309
|
return "function";
|
|
310
|
+
}
|
|
169
311
|
return undefined;
|
|
170
312
|
}
|
|
171
313
|
function signatureFromText(text, kind) {
|
|
172
314
|
const normalized = text.trim();
|
|
173
315
|
if (!normalized)
|
|
174
316
|
return undefined;
|
|
175
|
-
const bodyMarkers = kind === "function" || kind === "method" || kind === "constructor"
|
|
317
|
+
const bodyMarkers = kind === "function" || kind === "method" || kind === "constructor" || kind === "destructor"
|
|
176
318
|
? normalized.startsWith("def ") || normalized.startsWith("async def ")
|
|
177
319
|
? [":"]
|
|
178
320
|
: ["{", "=>"]
|
|
@@ -188,7 +330,10 @@ function signatureFromText(text, kind) {
|
|
|
188
330
|
}
|
|
189
331
|
function createStructureNode(node, kind, name, context, map) {
|
|
190
332
|
const span = context.spanNode ?? node;
|
|
191
|
-
const
|
|
333
|
+
const qualifiedParent = parentFromQualifiedName(name);
|
|
334
|
+
const parentSymbol = qualifiedParent ?? context.parentSymbol;
|
|
335
|
+
const baseScope = qualifiedParent && context.scope.at(-1) !== qualifiedParent ? [...context.scope, qualifiedParent] : context.scope;
|
|
336
|
+
const scope = name ? [...baseScope, name] : [...baseScope];
|
|
192
337
|
const text = sliceByBytes(map, span.startIndex, span.endIndex);
|
|
193
338
|
const astPath = [...context.astPath, node.type];
|
|
194
339
|
return {
|
|
@@ -203,7 +348,9 @@ function createStructureNode(node, kind, name, context, map) {
|
|
|
203
348
|
endByte: span.endIndex,
|
|
204
349
|
exported: context.exported,
|
|
205
350
|
decorators: decoratorsFor(span, context.decorators),
|
|
206
|
-
|
|
351
|
+
modifiers: modifiersFor(span),
|
|
352
|
+
visibility: context.visibility,
|
|
353
|
+
parentSymbol,
|
|
207
354
|
scope,
|
|
208
355
|
astPath,
|
|
209
356
|
children: [],
|
|
@@ -216,6 +363,7 @@ function childContext(parent, context) {
|
|
|
216
363
|
decorators: [],
|
|
217
364
|
scope: parent.scope,
|
|
218
365
|
parentSymbol: parent.name ?? context.parentSymbol,
|
|
366
|
+
visibility: undefined,
|
|
219
367
|
astPath: parent.astPath,
|
|
220
368
|
};
|
|
221
369
|
}
|
|
@@ -269,7 +417,7 @@ function collectStructureNodes(node, context, map) {
|
|
|
269
417
|
if (field)
|
|
270
418
|
return [field];
|
|
271
419
|
}
|
|
272
|
-
const kind = kindForNode(node);
|
|
420
|
+
const kind = kindForNode(node, context);
|
|
273
421
|
if (kind) {
|
|
274
422
|
const structural = createStructureNode(node, kind, getName(node), context, map);
|
|
275
423
|
const nested = scanChildren(node, childContext(structural, context), map);
|
|
@@ -280,16 +428,28 @@ function collectStructureNodes(node, context, map) {
|
|
|
280
428
|
}
|
|
281
429
|
function scanChildren(node, context, map) {
|
|
282
430
|
const nodes = [];
|
|
431
|
+
let visibility = context.visibility;
|
|
283
432
|
for (const child of nodeChildren(node)) {
|
|
433
|
+
if (child.type === "access_specifier") {
|
|
434
|
+
visibility = child.text.trim();
|
|
435
|
+
continue;
|
|
436
|
+
}
|
|
437
|
+
const nextContext = { ...context, visibility };
|
|
284
438
|
if (DECLARATION_WRAPPERS.has(child.type) ||
|
|
285
439
|
FIELD_DEFINITION_TYPES.has(child.type) ||
|
|
286
440
|
isStructuralCandidate(child) ||
|
|
441
|
+
child.type === "preproc_def" ||
|
|
442
|
+
child.type === "preproc_function_def" ||
|
|
287
443
|
child.type.includes("declaration")) {
|
|
288
|
-
nodes.push(...collectStructureNodes(child, { ...
|
|
444
|
+
nodes.push(...collectStructureNodes(child, { ...nextContext, spanNode: undefined }, map));
|
|
289
445
|
continue;
|
|
290
446
|
}
|
|
291
|
-
if (child.type === "class_body" ||
|
|
292
|
-
|
|
447
|
+
if (child.type === "class_body" ||
|
|
448
|
+
child.type === "block" ||
|
|
449
|
+
child.type === "declaration_list" ||
|
|
450
|
+
child.type === "field_declaration_list" ||
|
|
451
|
+
child.type === "ui_object_initializer") {
|
|
452
|
+
nodes.push(...scanChildren(child, nextContext, map));
|
|
293
453
|
}
|
|
294
454
|
}
|
|
295
455
|
return nodes.sort((a, b) => a.startByte - b.startByte || a.endByte - b.endByte);
|
|
@@ -308,6 +468,9 @@ function metadataForNode(node) {
|
|
|
308
468
|
ast_depth: Math.max(0, node.scope.length - 1),
|
|
309
469
|
ast_path: node.astPath.join("/"),
|
|
310
470
|
decorators: node.decorators.length > 0 ? node.decorators : undefined,
|
|
471
|
+
modifiers: node.modifiers.length > 0 ? node.modifiers : undefined,
|
|
472
|
+
visibility: node.visibility ?? undefined,
|
|
473
|
+
static: node.modifiers.includes("static") ? true : undefined,
|
|
311
474
|
start_line: node.startLine,
|
|
312
475
|
end_line: node.endLine,
|
|
313
476
|
};
|
|
@@ -335,7 +498,8 @@ function draftForNode(node, map) {
|
|
|
335
498
|
startByte: node.startByte,
|
|
336
499
|
endByte: node.endByte,
|
|
337
500
|
metadata: metadataForNode(node),
|
|
338
|
-
packable: ["function", "method", "constructor", "variable"].includes(node.kind) &&
|
|
501
|
+
packable: ["function", "method", "constructor", "destructor", "variable"].includes(node.kind) &&
|
|
502
|
+
estimateTokens(text) < AST_TARGET_TOKENS,
|
|
339
503
|
packKey: node.parentSymbol ?? "module",
|
|
340
504
|
symbols: node.name ? [node.name] : [],
|
|
341
505
|
};
|
|
@@ -454,6 +618,10 @@ function packDrafts(drafts, map) {
|
|
|
454
618
|
function buildDrafts(nodes, map) {
|
|
455
619
|
const drafts = [];
|
|
456
620
|
for (const node of nodes.sort((a, b) => a.startByte - b.startByte || a.endByte - b.endByte)) {
|
|
621
|
+
if (node.kind === "namespace" && node.children.length > 0) {
|
|
622
|
+
drafts.push(...buildDrafts(node.children, map));
|
|
623
|
+
continue;
|
|
624
|
+
}
|
|
457
625
|
const text = sliceByBytes(map, node.startByte, node.endByte).trim();
|
|
458
626
|
if (!text)
|
|
459
627
|
continue;
|
|
@@ -474,13 +642,16 @@ function symbolKindForNode(node) {
|
|
|
474
642
|
return undefined;
|
|
475
643
|
if (node.kind === "class")
|
|
476
644
|
return "class";
|
|
477
|
-
if (node.kind === "
|
|
478
|
-
return "interface";
|
|
479
|
-
if (node.kind === "type")
|
|
645
|
+
if (node.kind === "namespace" || node.kind === "import" || node.kind === "type")
|
|
480
646
|
return "type";
|
|
481
|
-
if (node.kind === "variable")
|
|
647
|
+
if (node.kind === "variable" || node.kind === "property" || node.kind === "id" || node.kind === "binding")
|
|
482
648
|
return "variable";
|
|
483
|
-
if (node.kind === "function" ||
|
|
649
|
+
if (node.kind === "function" ||
|
|
650
|
+
node.kind === "method" ||
|
|
651
|
+
node.kind === "constructor" ||
|
|
652
|
+
node.kind === "destructor" ||
|
|
653
|
+
node.kind === "signal" ||
|
|
654
|
+
node.kind === "handler")
|
|
484
655
|
return "function";
|
|
485
656
|
return undefined;
|
|
486
657
|
}
|
|
@@ -493,6 +664,9 @@ function symbolMetadataForNode(node) {
|
|
|
493
664
|
exported: node.exported,
|
|
494
665
|
ast_depth: Math.max(0, node.scope.length - 1),
|
|
495
666
|
ast_path: node.astPath.join("/"),
|
|
667
|
+
modifiers: node.modifiers.length > 0 ? node.modifiers : undefined,
|
|
668
|
+
static: node.modifiers.includes("static") ? true : undefined,
|
|
669
|
+
visibility: node.visibility ?? undefined,
|
|
496
670
|
decorators: node.decorators.length > 0 ? node.decorators : undefined,
|
|
497
671
|
};
|
|
498
672
|
}
|
|
@@ -533,6 +707,7 @@ function emptyStructure(language) {
|
|
|
533
707
|
endByte: 0,
|
|
534
708
|
exported: false,
|
|
535
709
|
decorators: [],
|
|
710
|
+
modifiers: [],
|
|
536
711
|
scope: [],
|
|
537
712
|
astPath: ["program"],
|
|
538
713
|
children: [],
|
|
@@ -548,6 +723,8 @@ async function parseStructure(content, language) {
|
|
|
548
723
|
parser.setLanguage(grammar);
|
|
549
724
|
const tree = parser.parse(content);
|
|
550
725
|
const rootNode = tree.rootNode;
|
|
726
|
+
if (config.fallbackOnParseError && rootNode.hasError)
|
|
727
|
+
throw new Error(`AST parse failed for ${language}`);
|
|
551
728
|
const map = buildByteIndexMap(content);
|
|
552
729
|
const root = {
|
|
553
730
|
kind: "module",
|
|
@@ -559,6 +736,8 @@ async function parseStructure(content, language) {
|
|
|
559
736
|
endByte: rootNode.endIndex,
|
|
560
737
|
exported: false,
|
|
561
738
|
decorators: [],
|
|
739
|
+
modifiers: [],
|
|
740
|
+
visibility: undefined,
|
|
562
741
|
scope: [],
|
|
563
742
|
astPath: [rootNode.type],
|
|
564
743
|
children: [],
|
|
@@ -4,21 +4,28 @@ export const MIN_HYBRID_SCORE = 0.18;
|
|
|
4
4
|
export const FILE_TYPE_ALIASES = {
|
|
5
5
|
csharp: "csharp",
|
|
6
6
|
cs: "csharp",
|
|
7
|
+
c: "c",
|
|
8
|
+
cc: "cpp",
|
|
9
|
+
cxx: "cpp",
|
|
7
10
|
cpp: "cpp",
|
|
8
11
|
hpp: "cpp",
|
|
12
|
+
hh: "cpp",
|
|
13
|
+
hxx: "cpp",
|
|
9
14
|
js: "javascript",
|
|
10
15
|
jsx: "javascript",
|
|
11
16
|
md: "markdown",
|
|
12
17
|
mdx: "markdown",
|
|
13
18
|
php: "php",
|
|
14
19
|
py: "python",
|
|
20
|
+
qml: "qml",
|
|
15
21
|
rb: "ruby",
|
|
16
|
-
sh: "
|
|
22
|
+
sh: "bash",
|
|
17
23
|
ts: "typescript",
|
|
18
24
|
tsx: "typescript",
|
|
19
25
|
yml: "yaml",
|
|
20
26
|
};
|
|
21
27
|
const SOURCE_FILE_TYPES = new Set([
|
|
28
|
+
"bash",
|
|
22
29
|
"c",
|
|
23
30
|
"cpp",
|
|
24
31
|
"csharp",
|
|
@@ -27,6 +34,7 @@ const SOURCE_FILE_TYPES = new Set([
|
|
|
27
34
|
"javascript",
|
|
28
35
|
"php",
|
|
29
36
|
"python",
|
|
37
|
+
"qml",
|
|
30
38
|
"ruby",
|
|
31
39
|
"rust",
|
|
32
40
|
"typescript",
|
package/docs/configuration.md
CHANGED
|
@@ -103,6 +103,15 @@ When `PI_KNOWLEDGE_E2E_PDF` or `PI_KNOWLEDGE_E2E_DOCX` is unset, `npm run test:e
|
|
|
103
103
|
|
|
104
104
|
`pi-knowledge` supports direct Pi extension loading through `pi -e ./extension.js` and OMP-compatible loading through the same packaged entry shim. The root `extension.js` and `index.ts` stay startup-light so install-time validation can inspect the extension without resolving native runtime dependencies such as `better-sqlite3` or `onnxruntime-node`.
|
|
105
105
|
|
|
106
|
+
Minimal Ubuntu/Debian servers need native build tools before `omp install npm:pi-knowledge`,
|
|
107
|
+
because OMP's Bun-based npm install compiles `better-sqlite3` from source when Bun cannot use
|
|
108
|
+
the package prebuild:
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
112
|
+
omp install npm:pi-knowledge
|
|
113
|
+
```
|
|
114
|
+
|
|
106
115
|
Compatibility guarantees:
|
|
107
116
|
|
|
108
117
|
- The packaged `extension.js` loads built `dist/index.js` when present and falls back to source `index.ts` for local development.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.1",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -54,11 +54,16 @@
|
|
|
54
54
|
"better-sqlite3": "12.11.1",
|
|
55
55
|
"ignore": "7.0.5",
|
|
56
56
|
"mammoth": "1.12.0",
|
|
57
|
-
"tree-sitter": "0.
|
|
58
|
-
"tree-sitter-
|
|
57
|
+
"tree-sitter": "0.21.1",
|
|
58
|
+
"tree-sitter-bash": "0.21.0",
|
|
59
|
+
"tree-sitter-c": "0.21.4",
|
|
60
|
+
"tree-sitter-cpp": "0.23.2",
|
|
61
|
+
"tree-sitter-go": "0.21.2",
|
|
59
62
|
"tree-sitter-java": "0.23.5",
|
|
60
|
-
"tree-sitter-
|
|
61
|
-
"tree-sitter-
|
|
63
|
+
"tree-sitter-javascript": "0.23.1",
|
|
64
|
+
"tree-sitter-python": "0.21.0",
|
|
65
|
+
"tree-sitter-qmljs": "0.3.1",
|
|
66
|
+
"tree-sitter-rust": "0.21.0",
|
|
62
67
|
"tree-sitter-typescript": "0.23.2",
|
|
63
68
|
"unpdf": "1.6.2"
|
|
64
69
|
},
|