pi-knowledge 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +9 -9
- package/dist/index.d.ts +2 -0
- package/dist/index.js +13 -0
- package/dist/src/engine.js +84 -15
- package/dist/src/indexer/chunker.d.ts +6 -1
- package/dist/src/indexer/chunker.js +60 -2
- package/dist/src/indexer/chunkers/code-ast.d.ts +27 -1
- package/dist/src/indexer/chunkers/code-ast.js +530 -121
- package/docs/configuration.md +2 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.9.0] - 2026-08-26
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Added recursive AST chunking for supported code files, including same-parent sibling packing, bounded oversized leaf fallback, and structural chunk metadata for language, symbol kind, scope, parent symbol, signature, export state, AST path, and source line range (#11).
|
|
7
|
+
- Added AST-backed symbol extraction for supported code files so method symbols remain searchable independently from retrieval chunk boundaries (#11).
|
|
8
|
+
- Added metadata-driven AST-aware adaptive context expansion so `adaptive` search can prefer same-file parent/sibling chunks while preserving bounded context and clean returned source.
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
- Code indexing now analyzes supported source files once for both retrieval chunks and symbols, while preserving regex/text fallbacks for unsupported or unparsable content.
|
|
12
|
+
- Searchable embedding/FTS text now includes richer deterministic code structure; existing KBs should be updated or rebuilt to benefit from the new metadata.
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
## [0.8.1] - 2026-08-08
|
|
16
|
+
|
|
17
|
+
### Fixed
|
|
18
|
+
- Declared host approval tiers for public knowledge tools so read-only search no longer defaults to exec approval in OMP headless sessions (#10).
|
|
3
19
|
|
|
4
20
|
## [0.8.0] - 2026-08-04
|
|
5
21
|
|
package/README.md
CHANGED
|
@@ -23,8 +23,8 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
23
23
|
|
|
24
24
|
- **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
|
|
25
25
|
- **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
|
|
26
|
-
- **Code-aware indexing**: AST
|
|
27
|
-
- **Lightweight symbol lookup**: indexes functions, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
|
|
26
|
+
- **Code-aware indexing**: recursive AST chunking for TypeScript/JavaScript, Python, Go, Rust, and Java, with bounded fallback chunks for oversized declarations.
|
|
27
|
+
- **Lightweight symbol lookup**: indexes functions, methods, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
|
|
28
28
|
- **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
|
|
29
29
|
- **Large-project stability**: persisted indexing progress, capped batches, streaming vector scans, and stuck-job detection.
|
|
30
30
|
- **Private by default**: local embeddings run without an API key.
|
|
@@ -64,9 +64,9 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
|
|
|
64
64
|
|
|
65
65
|
- **RAG-native project memory**: follows the Retrieval-Augmented Generation pattern from Lewis et al. 2020: keep source truth outside the model, retrieve it at answer time, and inject only relevant context.
|
|
66
66
|
- **Dense semantic recall**: uses multilingual dense embeddings in the spirit of Dense Passage Retrieval (Karpukhin et al. 2020), so conceptual queries can find code/docs even when wording differs.
|
|
67
|
-
- **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and code symbols with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
|
|
67
|
+
- **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and AST-derived code structure such as symbols, scope, parent symbol, and signatures with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
|
|
68
68
|
- **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
|
|
69
|
-
- **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results.
|
|
69
|
+
- **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and bounded adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results. Code KBs with AST metadata can also use same-parent/sibling structure during adaptive expansion.
|
|
70
70
|
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
71
71
|
- **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
|
|
72
72
|
|
|
@@ -132,14 +132,14 @@ Mode selection contract:
|
|
|
132
132
|
- Start with `hybrid` for most project questions that contain useful lexical anchors.
|
|
133
133
|
- Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
|
|
134
134
|
- Use `semantic` when the query is conceptual, exact terms may differ from indexed wording, or hybrid returns no lexical matches.
|
|
135
|
-
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, or enough context to make a safe edit.
|
|
135
|
+
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, AST-related sibling methods where indexed, or enough context to make a safe edit.
|
|
136
136
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
137
137
|
- If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
|
|
138
138
|
|
|
139
|
-
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag,
|
|
139
|
+
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. In `adaptive` mode, code chunks with AST metadata can prefer same-parent/sibling chunks inside the same bounded file window; this is metadata-driven context expansion, not caller/callee analysis. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, source freshness, and adaptive `source_chunk_ids` where available.
|
|
140
140
|
|
|
141
|
-
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and code symbol names while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, or functions, and reduces duplicate-looking chunk hits.
|
|
142
|
-
Symbol/config/heading metadata is also rebuilt during `knowledge_update`;
|
|
141
|
+
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, classes, methods, or functions, and reduces duplicate-looking chunk hits.
|
|
142
|
+
Symbol/config/heading metadata is also rebuilt during `knowledge_update`; code symbol metadata includes AST-backed methods for supported languages. Older KBs created before this feature may show a doctor action recommending update.
|
|
143
143
|
|
|
144
144
|
## Embedding Configuration
|
|
145
145
|
|
|
@@ -224,7 +224,7 @@ All data is stored globally at `~/.pi/knowledge/` under Pi or `~/.omp/knowledge/
|
|
|
224
224
|
- **Override**: set `PI_KNOWLEDGE_DIR` or `OMP_KNOWLEDGE_DIR`
|
|
225
225
|
- **Project safety**: pi-knowledge is read-only on indexed directories — no files are created or modified in your project
|
|
226
226
|
- **Updates**: extension updates do not affect existing indexed data. Schema migrations run automatically if needed.
|
|
227
|
-
- **Symbol index**: `knowledge.db` also stores lightweight symbol/config/heading metadata used by `knowledge_symbol_search`; it is derived from indexed source. File, directory, and URL KBs with retained source paths can rebuild it with `knowledge_update`; imported portable KBs and inline text KBs should be re-imported or re-added because they intentionally do not store an active local source manifest.
|
|
227
|
+
- **Symbol index**: `knowledge.db` also stores lightweight symbol/config/heading metadata used by `knowledge_symbol_search`; it is derived from indexed source. Supported code files use the same AST analysis as chunking, so method symbols can be looked up even when their parent class remains one retrieval chunk. File, directory, and URL KBs with retained source paths can rebuild it with `knowledge_update`; imported portable KBs and inline text KBs should be re-imported or re-added because they intentionally do not store an active local source manifest.
|
|
228
228
|
|
|
229
229
|
## Development
|
|
230
230
|
|
package/dist/index.d.ts
CHANGED
|
@@ -7,10 +7,12 @@ type ToolResult = {
|
|
|
7
7
|
isError?: boolean;
|
|
8
8
|
};
|
|
9
9
|
type ToolUpdate = (result: ToolResult) => void;
|
|
10
|
+
type ToolApproval = "read" | "write" | "exec";
|
|
10
11
|
type ToolDefinition = {
|
|
11
12
|
name: string;
|
|
12
13
|
label: string;
|
|
13
14
|
description: string;
|
|
15
|
+
approval?: ToolApproval;
|
|
14
16
|
promptSnippet?: string;
|
|
15
17
|
promptGuidelines?: string[];
|
|
16
18
|
parameters: Schema;
|
package/dist/index.js
CHANGED
|
@@ -201,6 +201,7 @@ export default function (pi) {
|
|
|
201
201
|
name: "knowledge_plan",
|
|
202
202
|
label: "Knowledge Plan",
|
|
203
203
|
description: "Inspect an indexing source without writing a KB, showing scannable counts, suggested exclusions, and technical skips",
|
|
204
|
+
approval: "read",
|
|
204
205
|
promptSnippet: "Plan a knowledge-base indexing scope before calling knowledge_add",
|
|
205
206
|
promptGuidelines: [
|
|
206
207
|
"Use knowledge_plan before knowledge_add for broad directories, large repositories, or sources that may contain private or low-signal text",
|
|
@@ -252,6 +253,7 @@ export default function (pi) {
|
|
|
252
253
|
name: "knowledge_configure",
|
|
253
254
|
label: "Knowledge Configure",
|
|
254
255
|
description: "Configure pi-knowledge runtime settings such as the local model worker Node executable",
|
|
256
|
+
approval: "write",
|
|
255
257
|
promptSnippet: "Configure local pi-knowledge runtime prerequisites before indexing",
|
|
256
258
|
promptGuidelines: [
|
|
257
259
|
"Use knowledge_configure when knowledge_doctor or knowledge_add reports that local embeddings cannot find a usable Node executable",
|
|
@@ -287,6 +289,7 @@ export default function (pi) {
|
|
|
287
289
|
name: "knowledge_add",
|
|
288
290
|
label: "Knowledge Add",
|
|
289
291
|
description: "Index files, directories, URLs, PDFs, DOCX, or text into a named knowledge base for semantic search",
|
|
292
|
+
approval: "write",
|
|
290
293
|
promptSnippet: "Index files/dirs/URLs/PDFs/DOCX/text into a searchable knowledge base",
|
|
291
294
|
promptGuidelines: [
|
|
292
295
|
"Use knowledge_plan first for broad directories, large repositories, or sources that may contain private or low-signal text",
|
|
@@ -355,6 +358,7 @@ export default function (pi) {
|
|
|
355
358
|
name: "knowledge_search",
|
|
356
359
|
label: "Knowledge Search",
|
|
357
360
|
description: "Search indexed knowledge bases using lexical BM25, semantic vectors, reranking, and filters",
|
|
361
|
+
approval: "read",
|
|
358
362
|
promptSnippet: "Search knowledge bases (hybrid is lexical-anchored BM25 + semantic fusion)",
|
|
359
363
|
promptGuidelines: [
|
|
360
364
|
"Use knowledge_search to find relevant context before answering domain questions",
|
|
@@ -453,6 +457,7 @@ export default function (pi) {
|
|
|
453
457
|
name: "knowledge_symbol_search",
|
|
454
458
|
label: "Knowledge Symbol Search",
|
|
455
459
|
description: "Search indexed code symbols, Markdown headings, config keys, and environment variables exactly or by substring",
|
|
460
|
+
approval: "read",
|
|
456
461
|
promptSnippet: "Find exact symbols/config keys/headings before broad semantic search",
|
|
457
462
|
promptGuidelines: [
|
|
458
463
|
"Use knowledge_symbol_search before knowledge_search when looking for a function, class, env var, config key, route, or Markdown heading",
|
|
@@ -522,6 +527,7 @@ export default function (pi) {
|
|
|
522
527
|
name: "knowledge_update",
|
|
523
528
|
label: "Knowledge Update",
|
|
524
529
|
description: "Incrementally re-index a source-backed knowledge base with a retained file, directory, or URL source",
|
|
530
|
+
approval: "write",
|
|
525
531
|
promptSnippet: "Refresh an existing source-backed KB",
|
|
526
532
|
parameters: Type.Object({
|
|
527
533
|
target: Type.String({ description: "KB name or ID to update" }),
|
|
@@ -540,6 +546,7 @@ export default function (pi) {
|
|
|
540
546
|
name: "knowledge_status",
|
|
541
547
|
label: "Knowledge Status",
|
|
542
548
|
description: "Show knowledge engine status with health diagnostics: staleness, orphans, and coverage",
|
|
549
|
+
approval: "read",
|
|
543
550
|
parameters: Type.Object({}),
|
|
544
551
|
async execute(_id, _params, _signal) {
|
|
545
552
|
if (_signal?.aborted)
|
|
@@ -607,6 +614,7 @@ export default function (pi) {
|
|
|
607
614
|
name: "knowledge_doctor",
|
|
608
615
|
label: "Knowledge Doctor",
|
|
609
616
|
description: "Diagnose knowledge base health, skipped files, stale indexes, stuck jobs, and recommended fixes",
|
|
617
|
+
approval: "read",
|
|
610
618
|
promptSnippet: "Diagnose knowledge base health and recommended actions",
|
|
611
619
|
parameters: Type.Object({}),
|
|
612
620
|
async execute(_id, _params, _signal) {
|
|
@@ -632,6 +640,7 @@ export default function (pi) {
|
|
|
632
640
|
name: "knowledge_show",
|
|
633
641
|
label: "Knowledge Show",
|
|
634
642
|
description: "List all indexed knowledge bases",
|
|
643
|
+
approval: "read",
|
|
635
644
|
parameters: Type.Object({}),
|
|
636
645
|
async execute(_id, _params, _signal) {
|
|
637
646
|
if (_signal?.aborted)
|
|
@@ -650,6 +659,7 @@ export default function (pi) {
|
|
|
650
659
|
name: "knowledge_remove",
|
|
651
660
|
label: "Knowledge Remove",
|
|
652
661
|
description: "Remove a knowledge base by name or ID after explicit confirmation",
|
|
662
|
+
approval: "write",
|
|
653
663
|
promptGuidelines: [
|
|
654
664
|
"Destructive: only call after the user explicitly asks to remove knowledge data or confirms a proposed removal",
|
|
655
665
|
"Prefer knowledge_status or knowledge_doctor before removing data for troubleshooting",
|
|
@@ -676,6 +686,7 @@ export default function (pi) {
|
|
|
676
686
|
name: "knowledge_export",
|
|
677
687
|
label: "Knowledge Export",
|
|
678
688
|
description: "Export a knowledge base to a JSONL file (shareable, git-friendly)",
|
|
689
|
+
approval: "write",
|
|
679
690
|
parameters: Type.Object({
|
|
680
691
|
target: Type.String({ description: "KB name or ID to export" }),
|
|
681
692
|
output: Type.String({ description: "Output file path (.jsonl)" }),
|
|
@@ -693,6 +704,7 @@ export default function (pi) {
|
|
|
693
704
|
name: "knowledge_import",
|
|
694
705
|
label: "Knowledge Import",
|
|
695
706
|
description: "Import a knowledge base from a JSONL file (re-embeds content)",
|
|
707
|
+
approval: "write",
|
|
696
708
|
parameters: Type.Object({
|
|
697
709
|
input: Type.String({ description: "Input JSONL file path" }),
|
|
698
710
|
}),
|
|
@@ -708,6 +720,7 @@ export default function (pi) {
|
|
|
708
720
|
name: "knowledge_clear",
|
|
709
721
|
label: "Knowledge Clear",
|
|
710
722
|
description: "Remove all knowledge bases after explicit confirmation",
|
|
723
|
+
approval: "write",
|
|
711
724
|
promptGuidelines: [
|
|
712
725
|
"Destructive: only call after the user explicitly asks to clear all knowledge data or confirms a proposed clear",
|
|
713
726
|
"Never use knowledge_clear as troubleshooting unless the user explicitly approves",
|
package/dist/src/engine.js
CHANGED
|
@@ -3,7 +3,7 @@ import { join, resolve } from "node:path";
|
|
|
3
3
|
import { diagnoseKB } from "./diagnostics/health.js";
|
|
4
4
|
import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
|
|
5
5
|
import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
|
|
6
|
-
import { addSkippedScanEntry, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
6
|
+
import { addSkippedScanEntry, analyzeIndexableContent, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
7
7
|
import { extractSymbols } from "./indexer/symbols.js";
|
|
8
8
|
import { shutdownModelWorker } from "./model-worker-client.js";
|
|
9
9
|
import { searchBM25 } from "./search/bm25.js";
|
|
@@ -254,7 +254,67 @@ function buildQuerySnippet(content, query, maxLength = 240) {
|
|
|
254
254
|
const suffix = end < content.length ? "..." : "";
|
|
255
255
|
return `${prefix}${content.slice(start, end)}${suffix}`;
|
|
256
256
|
}
|
|
257
|
+
function metadataString(record, key) {
|
|
258
|
+
const value = record[key];
|
|
259
|
+
return typeof value === "string" && value.trim() ? value.trim() : undefined;
|
|
260
|
+
}
|
|
261
|
+
function metadataNumber(record, key) {
|
|
262
|
+
const value = record[key];
|
|
263
|
+
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
|
|
264
|
+
}
|
|
265
|
+
function metadataStringArray(record, key) {
|
|
266
|
+
const value = record[key];
|
|
267
|
+
return Array.isArray(value)
|
|
268
|
+
? value.filter((item) => typeof item === "string" && Boolean(item.trim()))
|
|
269
|
+
: [];
|
|
270
|
+
}
|
|
271
|
+
function parseAdaptiveMetadata(chunk) {
|
|
272
|
+
try {
|
|
273
|
+
const parsed = JSON.parse(chunk.metadata_json);
|
|
274
|
+
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed))
|
|
275
|
+
return { scope: [] };
|
|
276
|
+
const record = parsed;
|
|
277
|
+
return {
|
|
278
|
+
symbol: metadataString(record, "symbol") ?? metadataString(record, "function_name"),
|
|
279
|
+
symbolKind: metadataString(record, "symbol_kind"),
|
|
280
|
+
parentSymbol: metadataString(record, "parent_symbol"),
|
|
281
|
+
scope: metadataStringArray(record, "scope"),
|
|
282
|
+
startLine: metadataNumber(record, "start_line"),
|
|
283
|
+
endLine: metadataNumber(record, "end_line"),
|
|
284
|
+
};
|
|
285
|
+
}
|
|
286
|
+
catch {
|
|
287
|
+
return { scope: [] };
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
function hasAdaptiveStructure(metadata) {
|
|
291
|
+
return Boolean(metadata.symbol || metadata.symbolKind || metadata.parentSymbol || metadata.scope.length > 0);
|
|
292
|
+
}
|
|
293
|
+
function parentScope(scope) {
|
|
294
|
+
return scope.length > 1 ? scope.slice(0, -1) : [];
|
|
295
|
+
}
|
|
296
|
+
function sameScope(left, right) {
|
|
297
|
+
return left.length > 0 && left.length === right.length && left.every((item, index) => item === right[index]);
|
|
298
|
+
}
|
|
299
|
+
function isScopePrefix(parent, child) {
|
|
300
|
+
return parent.length > 0 && parent.length < child.length && parent.every((item, index) => item === child[index]);
|
|
301
|
+
}
|
|
302
|
+
function adaptiveRelationBoost(seed, candidate) {
|
|
303
|
+
if (!hasAdaptiveStructure(seed) || !hasAdaptiveStructure(candidate))
|
|
304
|
+
return 0;
|
|
305
|
+
const seedParentScope = parentScope(seed.scope);
|
|
306
|
+
const candidateParentScope = parentScope(candidate.scope);
|
|
307
|
+
if (sameScope(seedParentScope, candidateParentScope))
|
|
308
|
+
return seed.symbol === candidate.symbol ? 0.35 : 1.35;
|
|
309
|
+
if (seed.parentSymbol && seed.parentSymbol === candidate.parentSymbol) {
|
|
310
|
+
return seed.symbol === candidate.symbol ? 0.35 : 1.2;
|
|
311
|
+
}
|
|
312
|
+
if (isScopePrefix(seed.scope, candidate.scope) || isScopePrefix(candidate.scope, seed.scope))
|
|
313
|
+
return 0.9;
|
|
314
|
+
return 0;
|
|
315
|
+
}
|
|
257
316
|
function buildAdaptiveContext(seed, chunks, queryTokens, options) {
|
|
317
|
+
const seedMetadata = parseAdaptiveMetadata(seed);
|
|
258
318
|
const scored = chunks
|
|
259
319
|
.map((chunk) => {
|
|
260
320
|
const distance = chunk.id === seed.id
|
|
@@ -263,9 +323,10 @@ function buildAdaptiveContext(seed, chunks, queryTokens, options) {
|
|
|
263
323
|
const proximity = 1 / (1 + distance / 20);
|
|
264
324
|
const coverage = queryCoverage(chunk.content, queryTokens);
|
|
265
325
|
const seedBoost = chunk.id === seed.id ? 2 : 0;
|
|
266
|
-
|
|
326
|
+
const relationBoost = chunk.id === seed.id ? 0 : adaptiveRelationBoost(seedMetadata, parseAdaptiveMetadata(chunk));
|
|
327
|
+
return { chunk, score: seedBoost + relationBoost + proximity + coverage };
|
|
267
328
|
})
|
|
268
|
-
.sort((a, b) => b.score - a.score)
|
|
329
|
+
.sort((a, b) => b.score - a.score || a.chunk.start_line - b.chunk.start_line || a.chunk.end_line - b.chunk.end_line)
|
|
269
330
|
.slice(0, options.neighborTarget);
|
|
270
331
|
if (!scored.some((item) => item.chunk.id === seed.id))
|
|
271
332
|
scored.push({ chunk: seed, score: Number.MAX_SAFE_INTEGER });
|
|
@@ -275,6 +336,7 @@ function buildAdaptiveContext(seed, chunks, queryTokens, options) {
|
|
|
275
336
|
.sort((a, b) => a.start_line - b.start_line || a.end_line - b.end_line);
|
|
276
337
|
const parts = [];
|
|
277
338
|
const sourceChunkIds = [];
|
|
339
|
+
const includedChunks = [];
|
|
278
340
|
let totalLength = 0;
|
|
279
341
|
for (const chunk of ordered) {
|
|
280
342
|
if (totalLength >= options.maxContextChars)
|
|
@@ -283,12 +345,13 @@ function buildAdaptiveContext(seed, chunks, queryTokens, options) {
|
|
|
283
345
|
const text = chunk.content.slice(0, remaining);
|
|
284
346
|
parts.push(text);
|
|
285
347
|
sourceChunkIds.push(chunk.id);
|
|
348
|
+
includedChunks.push(chunk);
|
|
286
349
|
totalLength += text.length + 2;
|
|
287
350
|
}
|
|
288
351
|
return {
|
|
289
352
|
content: parts.join("\n\n"),
|
|
290
|
-
startLine: Math.min(...
|
|
291
|
-
endLine: Math.max(...
|
|
353
|
+
startLine: Math.min(...includedChunks.map((chunk) => chunk.start_line)),
|
|
354
|
+
endLine: Math.max(...includedChunks.map((chunk) => chunk.end_line)),
|
|
292
355
|
sourceChunkIds,
|
|
293
356
|
};
|
|
294
357
|
}
|
|
@@ -712,6 +775,11 @@ export class KnowledgeEngine {
|
|
|
712
775
|
const addSymbols = (content, filePath, fileType) => {
|
|
713
776
|
insertSymbols(db, kb.id, extractSymbols(content, filePath, fileType));
|
|
714
777
|
};
|
|
778
|
+
const analyzeAndAddSymbols = async (content, filePath, fileType) => {
|
|
779
|
+
const analysis = await analyzeIndexableContent(content, filePath, fileType);
|
|
780
|
+
insertSymbols(db, kb.id, analysis.symbols);
|
|
781
|
+
return analysis.chunks;
|
|
782
|
+
};
|
|
715
783
|
if (isUrl) {
|
|
716
784
|
const message = `Fetching ${source}...`;
|
|
717
785
|
updateIndexingJob(this.db, kb.id, { phase: "fetching", message });
|
|
@@ -724,8 +792,8 @@ export class KnowledgeEngine {
|
|
|
724
792
|
else if (isFile) {
|
|
725
793
|
const extracted = await extractSourceFileContent(resolvedSource, signal);
|
|
726
794
|
fileCount = 1;
|
|
727
|
-
|
|
728
|
-
await addChunks(
|
|
795
|
+
const chunks = await analyzeAndAddSymbols(extracted.content, resolvedSource, extracted.fileType);
|
|
796
|
+
await addChunks(chunks);
|
|
729
797
|
}
|
|
730
798
|
else if (isDir) {
|
|
731
799
|
const plan = planDirectoryScan(resolvedSource, scanOptions, signal);
|
|
@@ -755,12 +823,11 @@ export class KnowledgeEngine {
|
|
|
755
823
|
latestSkippedTotal = skipped.total;
|
|
756
824
|
continue;
|
|
757
825
|
}
|
|
758
|
-
const chunks = await
|
|
826
|
+
const chunks = await analyzeAndAddSymbols(extracted.content, file.relPath, extracted.fileType);
|
|
759
827
|
processedFiles++;
|
|
760
828
|
latestSkippedTotal = skipped.total;
|
|
761
829
|
if (chunks.length > 0)
|
|
762
830
|
fileCount++;
|
|
763
|
-
addSymbols(extracted.content, file.relPath, extracted.fileType);
|
|
764
831
|
await addChunks(chunks, processedFiles, plan.files);
|
|
765
832
|
if (processedFiles % 25 === 0)
|
|
766
833
|
reportProgress("Chunking", processedFiles, plan.files, skipped.total);
|
|
@@ -781,8 +848,8 @@ export class KnowledgeEngine {
|
|
|
781
848
|
}
|
|
782
849
|
else {
|
|
783
850
|
fileCount = 1;
|
|
784
|
-
|
|
785
|
-
await addChunks(
|
|
851
|
+
const chunks = await analyzeAndAddSymbols(source, "inline-text", "text");
|
|
852
|
+
await addChunks(chunks);
|
|
786
853
|
}
|
|
787
854
|
await flushPending();
|
|
788
855
|
vectorWriter.close();
|
|
@@ -986,9 +1053,10 @@ export class KnowledgeEngine {
|
|
|
986
1053
|
const extracted = await extractScannableFileContentOrSkip(file, skipped, signal);
|
|
987
1054
|
if (!extracted)
|
|
988
1055
|
continue;
|
|
1056
|
+
const analysis = await analyzeIndexableContent(extracted.content, file.relPath, extracted.fileType);
|
|
989
1057
|
scannedFiles++;
|
|
990
|
-
stagedSymbols.push(...
|
|
991
|
-
await processChunks(
|
|
1058
|
+
stagedSymbols.push(...analysis.symbols);
|
|
1059
|
+
await processChunks(analysis.chunks);
|
|
992
1060
|
if (scannedFiles % 25 === 0) {
|
|
993
1061
|
const message = `Scanned ${scannedFiles} files, ${scannedChunks} chunks, skipped ${skipped.total}, +${addedCount} =${unchanged}`;
|
|
994
1062
|
updateIndexingJob(this.db, kb.id, {
|
|
@@ -1022,8 +1090,9 @@ export class KnowledgeEngine {
|
|
|
1022
1090
|
else {
|
|
1023
1091
|
const extracted = await extractSourceFileContent(kb.source_path, signal);
|
|
1024
1092
|
scannedFiles = 1;
|
|
1025
|
-
|
|
1026
|
-
|
|
1093
|
+
const analysis = await analyzeIndexableContent(extracted.content, kb.source_path, extracted.fileType);
|
|
1094
|
+
stagedSymbols.push(...analysis.symbols);
|
|
1095
|
+
await processChunks(analysis.chunks);
|
|
1027
1096
|
}
|
|
1028
1097
|
await flushPending();
|
|
1029
1098
|
addedVectorWriter.close();
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { ChunkInsert } from "../storage/sqlite.ts";
|
|
1
|
+
import type { ChunkInsert, KnowledgeSymbolInsert } from "../storage/sqlite.ts";
|
|
2
2
|
export interface ScanOptions {
|
|
3
3
|
includeSuggestedText?: boolean;
|
|
4
4
|
includePaths?: string[];
|
|
@@ -52,4 +52,9 @@ export declare function chunkIdentityHash(opts: {
|
|
|
52
52
|
export declare function buildChunkEmbeddingText(chunk: Pick<ChunkInsert, "content" | "file_path" | "file_type" | "metadata_json">): string;
|
|
53
53
|
export declare function chunkMarkdown(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
|
|
54
54
|
export declare function chunkText(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
|
|
55
|
+
export declare function isCodeFileType(fileType: string): boolean;
|
|
56
|
+
export declare function analyzeIndexableContent(content: string, filePath: string, fileType?: string): Promise<{
|
|
57
|
+
chunks: Omit<ChunkInsert, "kb_id">[];
|
|
58
|
+
symbols: KnowledgeSymbolInsert[];
|
|
59
|
+
}>;
|
|
55
60
|
export declare function chunkFile(content: string, filePath: string): Promise<Omit<ChunkInsert, "kb_id">[]>;
|
|
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
|
|
|
2
2
|
import { closeSync, existsSync, openSync, readdirSync, readFileSync, readSync, statSync } from "node:fs";
|
|
3
3
|
import { extname, join, relative, sep } from "node:path";
|
|
4
4
|
import ignore from "ignore";
|
|
5
|
+
import { extractSymbols } from "./symbols.js";
|
|
5
6
|
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10 MB
|
|
6
7
|
const DOCUMENT_EXTENSIONS = { ".doc": true, ".docx": true, ".pdf": true };
|
|
7
8
|
const BINARY_EXTENSIONS = {
|
|
@@ -339,13 +340,32 @@ function buildContextPrefix(filePath, fileType, metadata) {
|
|
|
339
340
|
const parts = [`File: ${filePath}`, `Type: ${fileType}`];
|
|
340
341
|
const heading = typeof metadata.heading === "string" ? metadata.heading : "";
|
|
341
342
|
const breadcrumb = typeof metadata.breadcrumb === "string" ? metadata.breadcrumb : "";
|
|
342
|
-
const
|
|
343
|
+
const language = typeof metadata.language === "string" ? metadata.language : "";
|
|
344
|
+
const symbol = typeof metadata.symbol === "string"
|
|
345
|
+
? metadata.symbol
|
|
346
|
+
: typeof metadata.function_name === "string"
|
|
347
|
+
? metadata.function_name
|
|
348
|
+
: "";
|
|
349
|
+
const symbolKind = typeof metadata.symbol_kind === "string" ? metadata.symbol_kind : "";
|
|
350
|
+
const parentSymbol = typeof metadata.parent_symbol === "string" ? metadata.parent_symbol : "";
|
|
351
|
+
const signature = typeof metadata.signature === "string" ? metadata.signature : "";
|
|
352
|
+
const scope = Array.isArray(metadata.scope) ? metadata.scope.filter((item) => typeof item === "string") : [];
|
|
343
353
|
if (breadcrumb)
|
|
344
354
|
parts.push(`Section: ${breadcrumb}`);
|
|
345
355
|
else if (heading)
|
|
346
356
|
parts.push(`Section: ${normalizeHeading(heading)}`);
|
|
357
|
+
if (language)
|
|
358
|
+
parts.push(`Language: ${language}`);
|
|
359
|
+
if (scope.length > 0)
|
|
360
|
+
parts.push(`Scope: ${scope.join(" > ")}`);
|
|
361
|
+
else if (parentSymbol)
|
|
362
|
+
parts.push(`Parent: ${parentSymbol}`);
|
|
363
|
+
if (symbolKind)
|
|
364
|
+
parts.push(`Kind: ${symbolKind}`);
|
|
347
365
|
if (symbol)
|
|
348
366
|
parts.push(`Symbol: ${symbol}`);
|
|
367
|
+
if (signature)
|
|
368
|
+
parts.push(`Signature: ${signature}`);
|
|
349
369
|
return parts.join("\n");
|
|
350
370
|
}
|
|
351
371
|
export function buildChunkEmbeddingText(chunk) {
|
|
@@ -504,13 +524,51 @@ export function chunkText(content, filePath) {
|
|
|
504
524
|
flush();
|
|
505
525
|
return chunks;
|
|
506
526
|
}
|
|
527
|
+
export function isCodeFileType(fileType) {
|
|
528
|
+
return ["typescript", "javascript", "python", "go", "rust", "java"].includes(fileType);
|
|
529
|
+
}
|
|
530
|
+
export async function analyzeIndexableContent(content, filePath, fileType = detectFileType(filePath)) {
|
|
531
|
+
const detectedFileType = detectFileType(filePath);
|
|
532
|
+
const analysisFileType = isCodeFileType(detectedFileType) ? detectedFileType : fileType;
|
|
533
|
+
if (isCodeFileType(analysisFileType)) {
|
|
534
|
+
try {
|
|
535
|
+
const { analyzeCodeWithAST } = await import("./chunkers/code-ast.js");
|
|
536
|
+
const analysis = await analyzeCodeWithAST(content, filePath, analysisFileType);
|
|
537
|
+
const chunks = analysis.chunks.length > 0 ? analysis.chunks : await chunkFile(content, filePath);
|
|
538
|
+
const symbols = dedupeSymbols([...analysis.symbols, ...extractSymbols(content, filePath, analysisFileType)]);
|
|
539
|
+
return { chunks, symbols };
|
|
540
|
+
}
|
|
541
|
+
catch {
|
|
542
|
+
/* fallback below */
|
|
543
|
+
}
|
|
544
|
+
}
|
|
545
|
+
return { chunks: await chunkFile(content, filePath), symbols: extractSymbols(content, filePath, analysisFileType) };
|
|
546
|
+
}
|
|
547
|
+
function dedupeSymbols(symbols) {
|
|
548
|
+
const seen = new Set();
|
|
549
|
+
const deduped = [];
|
|
550
|
+
for (const symbol of symbols) {
|
|
551
|
+
const key = [
|
|
552
|
+
symbol.kind,
|
|
553
|
+
symbol.name,
|
|
554
|
+
symbol.file_path,
|
|
555
|
+
String(symbol.start_line),
|
|
556
|
+
symbol.container_name ?? "",
|
|
557
|
+
].join("\0");
|
|
558
|
+
if (seen.has(key))
|
|
559
|
+
continue;
|
|
560
|
+
seen.add(key);
|
|
561
|
+
deduped.push(symbol);
|
|
562
|
+
}
|
|
563
|
+
return deduped;
|
|
564
|
+
}
|
|
507
565
|
export async function chunkFile(content, filePath) {
|
|
508
566
|
const fileType = detectFileType(filePath);
|
|
509
567
|
let chunks = [];
|
|
510
568
|
if (fileType === "markdown") {
|
|
511
569
|
chunks = chunkMarkdown(content, filePath);
|
|
512
570
|
}
|
|
513
|
-
else if (
|
|
571
|
+
else if (isCodeFileType(fileType)) {
|
|
514
572
|
try {
|
|
515
573
|
const { chunkWithAST } = await import("./chunkers/code-ast.js");
|
|
516
574
|
chunks = await chunkWithAST(content, filePath, fileType);
|
|
@@ -1,3 +1,29 @@
|
|
|
1
|
-
import type { ChunkInsert } from "../../storage/sqlite.ts";
|
|
1
|
+
import type { ChunkInsert, KnowledgeSymbolInsert } from "../../storage/sqlite.ts";
|
|
2
|
+
type StructureKind = "module" | "class" | "interface" | "type" | "function" | "method" | "constructor" | "variable";
|
|
3
|
+
export interface CodeStructureNode {
|
|
4
|
+
kind: StructureKind;
|
|
5
|
+
astType: string;
|
|
6
|
+
name?: string;
|
|
7
|
+
signature?: string;
|
|
8
|
+
language: string;
|
|
9
|
+
startLine: number;
|
|
10
|
+
endLine: number;
|
|
11
|
+
startByte: number;
|
|
12
|
+
endByte: number;
|
|
13
|
+
exported: boolean;
|
|
14
|
+
decorators: string[];
|
|
15
|
+
parentSymbol?: string;
|
|
16
|
+
scope: string[];
|
|
17
|
+
astPath: string[];
|
|
18
|
+
children: CodeStructureNode[];
|
|
19
|
+
}
|
|
20
|
+
export interface CodeAnalysisResult {
|
|
21
|
+
chunks: Omit<ChunkInsert, "kb_id">[];
|
|
22
|
+
symbols: KnowledgeSymbolInsert[];
|
|
23
|
+
structure: CodeStructureNode;
|
|
24
|
+
}
|
|
25
|
+
export declare function parseCodeStructure(content: string, language: string): Promise<CodeStructureNode>;
|
|
26
|
+
export declare function analyzeCodeWithAST(content: string, filePath: string, language: string): Promise<CodeAnalysisResult>;
|
|
2
27
|
export declare function chunkWithAST(content: string, filePath: string, language: string): Promise<Omit<ChunkInsert, "kb_id">[]>;
|
|
3
28
|
export declare const SUPPORTED_LANGUAGES: Set<string>;
|
|
29
|
+
export {};
|
|
@@ -1,179 +1,588 @@
|
|
|
1
|
-
import { buildChunkEmbeddingText,
|
|
1
|
+
import { buildChunkEmbeddingText, chunkIdentityHash, preTokenizeForFTS } from "../chunker.js";
|
|
2
|
+
function moduleField(moduleValue, field) {
|
|
3
|
+
if (typeof moduleValue !== "object" || moduleValue === null)
|
|
4
|
+
return undefined;
|
|
5
|
+
return moduleValue[field];
|
|
6
|
+
}
|
|
7
|
+
function moduleDefault(moduleValue) {
|
|
8
|
+
return moduleField(moduleValue, "default");
|
|
9
|
+
}
|
|
2
10
|
const LANGS = {
|
|
3
11
|
typescript: {
|
|
4
12
|
grammar: async () => {
|
|
5
13
|
const m = await import("tree-sitter-typescript");
|
|
6
|
-
return m
|
|
14
|
+
return moduleField(m, "typescript") ?? moduleField(moduleDefault(m), "typescript");
|
|
7
15
|
},
|
|
8
|
-
fnTypes: new Set(["function_declaration", "arrow_function", "generator_function_declaration"]),
|
|
9
|
-
classTypes: new Set(["class_declaration", "interface_declaration"]),
|
|
10
|
-
classBody: "class_body",
|
|
11
16
|
fileType: "typescript",
|
|
12
17
|
},
|
|
13
18
|
javascript: {
|
|
14
19
|
grammar: async () => {
|
|
15
20
|
const m = await import("tree-sitter-typescript");
|
|
16
|
-
return m
|
|
21
|
+
return moduleField(m, "typescript") ?? moduleField(moduleDefault(m), "typescript");
|
|
17
22
|
},
|
|
18
|
-
fnTypes: new Set(["function_declaration", "arrow_function", "generator_function_declaration"]),
|
|
19
|
-
classTypes: new Set(["class_declaration"]),
|
|
20
|
-
classBody: "class_body",
|
|
21
23
|
fileType: "javascript",
|
|
22
24
|
},
|
|
23
25
|
python: {
|
|
24
26
|
grammar: async () => {
|
|
25
27
|
const m = await import("tree-sitter-python");
|
|
26
|
-
return m
|
|
28
|
+
return moduleDefault(m) ?? m;
|
|
27
29
|
},
|
|
28
|
-
fnTypes: new Set(["function_definition"]),
|
|
29
|
-
classTypes: new Set(["class_definition"]),
|
|
30
|
-
classBody: "block",
|
|
31
30
|
fileType: "python",
|
|
32
31
|
},
|
|
33
32
|
go: {
|
|
34
33
|
grammar: async () => {
|
|
35
34
|
const m = await import("tree-sitter-go");
|
|
36
|
-
return m
|
|
35
|
+
return moduleDefault(m) ?? m;
|
|
37
36
|
},
|
|
38
|
-
fnTypes: new Set(["function_declaration", "method_declaration"]),
|
|
39
|
-
classTypes: new Set(["type_declaration"]),
|
|
40
37
|
fileType: "go",
|
|
41
38
|
},
|
|
42
39
|
rust: {
|
|
43
40
|
grammar: async () => {
|
|
44
41
|
const m = await import("tree-sitter-rust");
|
|
45
|
-
return m
|
|
42
|
+
return moduleDefault(m) ?? m;
|
|
46
43
|
},
|
|
47
|
-
fnTypes: new Set(["function_item"]),
|
|
48
|
-
classTypes: new Set(["struct_item", "enum_item"]),
|
|
49
|
-
methodContainer: "impl_item",
|
|
50
44
|
fileType: "rust",
|
|
51
45
|
},
|
|
52
46
|
java: {
|
|
53
47
|
grammar: async () => {
|
|
54
48
|
const m = await import("tree-sitter-java");
|
|
55
|
-
return m
|
|
49
|
+
return moduleDefault(m) ?? m;
|
|
56
50
|
},
|
|
57
|
-
fnTypes: new Set(["method_declaration", "constructor_declaration"]),
|
|
58
|
-
classTypes: new Set(["class_declaration", "interface_declaration", "enum_declaration"]),
|
|
59
|
-
classBody: "class_body",
|
|
60
51
|
fileType: "java",
|
|
61
52
|
},
|
|
62
53
|
};
|
|
54
|
+
const CLASS_TYPES = new Set([
|
|
55
|
+
"abstract_class_declaration",
|
|
56
|
+
"class_declaration",
|
|
57
|
+
"class_definition",
|
|
58
|
+
"enum_declaration",
|
|
59
|
+
]);
|
|
60
|
+
const INTERFACE_TYPES = new Set(["interface_declaration"]);
|
|
61
|
+
const TYPE_TYPES = new Set([
|
|
62
|
+
"annotation_type_declaration",
|
|
63
|
+
"enum_item",
|
|
64
|
+
"struct_item",
|
|
65
|
+
"trait_item",
|
|
66
|
+
"type_alias_declaration",
|
|
67
|
+
"type_declaration",
|
|
68
|
+
]);
|
|
69
|
+
const FUNCTION_TYPES = new Set([
|
|
70
|
+
"function_declaration",
|
|
71
|
+
"function_definition",
|
|
72
|
+
"function_item",
|
|
73
|
+
"generator_function_declaration",
|
|
74
|
+
"method_declaration",
|
|
75
|
+
]);
|
|
76
|
+
const METHOD_TYPES = new Set(["method_definition"]);
|
|
77
|
+
const CONSTRUCTOR_TYPES = new Set(["constructor_declaration"]);
|
|
78
|
+
const DECLARATION_WRAPPERS = new Set(["export_statement", "decorated_definition"]);
|
|
79
|
+
const VARIABLE_DECLARATOR_TYPES = new Set(["variable_declarator"]);
|
|
80
|
+
const FIELD_DEFINITION_TYPES = new Set(["public_field_definition", "field_definition", "property_declaration"]);
|
|
81
|
+
const FUNCTION_VALUE_TYPES = new Set([
|
|
82
|
+
"arrow_function",
|
|
83
|
+
"function",
|
|
84
|
+
"function_expression",
|
|
85
|
+
"function_declaration",
|
|
86
|
+
"generator_function",
|
|
87
|
+
"generator_function_declaration",
|
|
88
|
+
]);
|
|
89
|
+
const AST_TARGET_TOKENS = 900;
|
|
90
|
+
const AST_MAX_TOKENS = 1_800;
|
|
91
|
+
const MAX_AST_FALLBACK_CHARS = 6_000;
|
|
92
|
+
function estimateTokens(text) {
|
|
93
|
+
return Math.ceil(text.length / 3);
|
|
94
|
+
}
|
|
95
|
+
function buildByteIndexMap(content) {
|
|
96
|
+
const byByte = new Map();
|
|
97
|
+
let byteOffset = 0;
|
|
98
|
+
for (let index = 0; index < content.length;) {
|
|
99
|
+
byByte.set(byteOffset, index);
|
|
100
|
+
const codePoint = content.codePointAt(index);
|
|
101
|
+
if (codePoint === undefined)
|
|
102
|
+
break;
|
|
103
|
+
const char = String.fromCodePoint(codePoint);
|
|
104
|
+
byteOffset += Buffer.byteLength(char);
|
|
105
|
+
index += char.length;
|
|
106
|
+
}
|
|
107
|
+
byByte.set(byteOffset, content.length);
|
|
108
|
+
return { content, byByte };
|
|
109
|
+
}
|
|
110
|
+
function stringIndexForByte(map, byteOffset) {
|
|
111
|
+
const exact = map.byByte.get(byteOffset);
|
|
112
|
+
if (exact !== undefined)
|
|
113
|
+
return exact;
|
|
114
|
+
let closestByte = 0;
|
|
115
|
+
let closestIndex = 0;
|
|
116
|
+
for (const [candidateByte, candidateIndex] of map.byByte) {
|
|
117
|
+
if (candidateByte > byteOffset)
|
|
118
|
+
break;
|
|
119
|
+
closestByte = candidateByte;
|
|
120
|
+
closestIndex = candidateIndex;
|
|
121
|
+
}
|
|
122
|
+
return closestByte === byteOffset ? closestIndex : closestIndex;
|
|
123
|
+
}
|
|
124
|
+
function sliceByBytes(map, startByte, endByte) {
|
|
125
|
+
return map.content.slice(stringIndexForByte(map, startByte), stringIndexForByte(map, endByte));
|
|
126
|
+
}
|
|
127
|
+
function nodeChildren(node) {
|
|
128
|
+
return node.namedChildren && node.namedChildren.length > 0 ? node.namedChildren : node.children;
|
|
129
|
+
}
|
|
63
130
|
function getName(node) {
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
131
|
+
const name = node.childForFieldName("name")?.text ?? node.childForFieldName("type")?.text;
|
|
132
|
+
if (name?.trim())
|
|
133
|
+
return name.trim();
|
|
134
|
+
if (node.type === "constructor_declaration")
|
|
135
|
+
return "constructor";
|
|
136
|
+
return undefined;
|
|
137
|
+
}
|
|
138
|
+
function nodeHasFunctionValue(node) {
|
|
139
|
+
return nodeChildren(node).some((child) => FUNCTION_VALUE_TYPES.has(child.type) || nodeHasFunctionValue(child));
|
|
140
|
+
}
|
|
141
|
+
function firstStructuralChild(node) {
|
|
142
|
+
return nodeChildren(node).find((child) => isStructuralCandidate(child) || child.type === "lexical_declaration");
|
|
143
|
+
}
|
|
144
|
+
function decoratorsFor(node, inherited = []) {
|
|
145
|
+
const decorators = node.children.filter((child) => child.type === "decorator").map((child) => child.text.trim());
|
|
146
|
+
return [...inherited, ...decorators].filter(Boolean);
|
|
147
|
+
}
|
|
148
|
+
function isStructuralCandidate(node) {
|
|
149
|
+
return (CLASS_TYPES.has(node.type) ||
|
|
150
|
+
INTERFACE_TYPES.has(node.type) ||
|
|
151
|
+
TYPE_TYPES.has(node.type) ||
|
|
152
|
+
FUNCTION_TYPES.has(node.type) ||
|
|
153
|
+
METHOD_TYPES.has(node.type) ||
|
|
154
|
+
CONSTRUCTOR_TYPES.has(node.type));
|
|
155
|
+
}
|
|
156
|
+
function kindForNode(node) {
|
|
157
|
+
if (CLASS_TYPES.has(node.type))
|
|
158
|
+
return "class";
|
|
159
|
+
if (INTERFACE_TYPES.has(node.type))
|
|
160
|
+
return "interface";
|
|
161
|
+
if (TYPE_TYPES.has(node.type))
|
|
162
|
+
return "type";
|
|
163
|
+
if (METHOD_TYPES.has(node.type) || node.type === "method_declaration")
|
|
164
|
+
return "method";
|
|
165
|
+
if (CONSTRUCTOR_TYPES.has(node.type))
|
|
166
|
+
return "constructor";
|
|
167
|
+
if (FUNCTION_TYPES.has(node.type))
|
|
168
|
+
return "function";
|
|
169
|
+
return undefined;
|
|
170
|
+
}
|
|
171
|
+
function signatureFromText(text, kind) {
|
|
172
|
+
const normalized = text.trim();
|
|
173
|
+
if (!normalized)
|
|
174
|
+
return undefined;
|
|
175
|
+
const bodyMarkers = kind === "function" || kind === "method" || kind === "constructor"
|
|
176
|
+
? normalized.startsWith("def ") || normalized.startsWith("async def ")
|
|
177
|
+
? [":"]
|
|
178
|
+
: ["{", "=>"]
|
|
179
|
+
: ["{"];
|
|
180
|
+
let end = normalized.length;
|
|
181
|
+
for (const marker of bodyMarkers) {
|
|
182
|
+
const index = normalized.indexOf(marker);
|
|
183
|
+
if (index > 0)
|
|
184
|
+
end = Math.min(end, marker === "=>" ? index + marker.length : index);
|
|
185
|
+
}
|
|
186
|
+
const firstLine = normalized.slice(0, end).replace(/\s+/g, " ").trim();
|
|
187
|
+
return firstLine.length > 280 ? `${firstLine.slice(0, 277)}...` : firstLine;
|
|
188
|
+
}
|
|
189
|
+
function createStructureNode(node, kind, name, context, map) {
|
|
190
|
+
const span = context.spanNode ?? node;
|
|
191
|
+
const scope = name ? [...context.scope, name] : [...context.scope];
|
|
192
|
+
const text = sliceByBytes(map, span.startIndex, span.endIndex);
|
|
193
|
+
const astPath = [...context.astPath, node.type];
|
|
194
|
+
return {
|
|
195
|
+
kind,
|
|
196
|
+
astType: node.type,
|
|
197
|
+
name,
|
|
198
|
+
signature: signatureFromText(text, kind),
|
|
199
|
+
language: context.language,
|
|
200
|
+
startLine: span.startPosition.row + 1,
|
|
201
|
+
endLine: span.endPosition.row + 1,
|
|
202
|
+
startByte: span.startIndex,
|
|
203
|
+
endByte: span.endIndex,
|
|
204
|
+
exported: context.exported,
|
|
205
|
+
decorators: decoratorsFor(span, context.decorators),
|
|
206
|
+
parentSymbol: context.parentSymbol,
|
|
207
|
+
scope,
|
|
208
|
+
astPath,
|
|
209
|
+
children: [],
|
|
210
|
+
};
|
|
211
|
+
}
|
|
212
|
+
function childContext(parent, context) {
|
|
213
|
+
return {
|
|
214
|
+
language: context.language,
|
|
215
|
+
exported: false,
|
|
216
|
+
decorators: [],
|
|
217
|
+
scope: parent.scope,
|
|
218
|
+
parentSymbol: parent.name ?? context.parentSymbol,
|
|
219
|
+
astPath: parent.astPath,
|
|
220
|
+
};
|
|
221
|
+
}
|
|
222
|
+
function buildVariableFunctionNodes(node, context, map) {
|
|
223
|
+
const declarators = nodeChildren(node).filter((child) => VARIABLE_DECLARATOR_TYPES.has(child.type));
|
|
224
|
+
const singleDeclarator = declarators.length === 1;
|
|
225
|
+
const nodes = [];
|
|
226
|
+
for (const declarator of declarators) {
|
|
227
|
+
if (!nodeHasFunctionValue(declarator))
|
|
228
|
+
continue;
|
|
229
|
+
const name = getName(declarator);
|
|
230
|
+
const spanNode = context.spanNode ?? (singleDeclarator ? node : declarator);
|
|
231
|
+
const structural = createStructureNode(declarator, "function", name, { ...context, spanNode, astPath: [...context.astPath, node.type] }, map);
|
|
232
|
+
nodes.push(structural);
|
|
233
|
+
}
|
|
234
|
+
return nodes;
|
|
235
|
+
}
|
|
236
|
+
function buildFieldFunctionNode(node, context, map) {
|
|
237
|
+
if (!nodeHasFunctionValue(node))
|
|
238
|
+
return undefined;
|
|
239
|
+
const structural = createStructureNode(node, context.parentSymbol ? "method" : "function", getName(node), context, map);
|
|
240
|
+
return structural;
|
|
241
|
+
}
|
|
242
|
+
function collectStructureNodes(node, context, map) {
|
|
243
|
+
if (node.type === "export_statement") {
|
|
244
|
+
const structuralChild = firstStructuralChild(node);
|
|
245
|
+
if (structuralChild) {
|
|
246
|
+
return collectStructureNodes(structuralChild, { ...context, exported: true, spanNode: node }, map);
|
|
85
247
|
}
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
return;
|
|
96
|
-
}
|
|
97
|
-
}
|
|
248
|
+
}
|
|
249
|
+
if (node.type === "decorated_definition") {
|
|
250
|
+
const structuralChild = firstStructuralChild(node);
|
|
251
|
+
if (structuralChild) {
|
|
252
|
+
return collectStructureNodes(structuralChild, {
|
|
253
|
+
...context,
|
|
254
|
+
decorators: decoratorsFor(node, context.decorators),
|
|
255
|
+
spanNode: node,
|
|
256
|
+
}, map);
|
|
98
257
|
}
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
258
|
+
}
|
|
259
|
+
if (node.type === "lexical_declaration" || node.type === "variable_declaration") {
|
|
260
|
+
const variableNodes = buildVariableFunctionNodes(node, context, map);
|
|
261
|
+
if (variableNodes.length > 0)
|
|
262
|
+
return variableNodes;
|
|
263
|
+
}
|
|
264
|
+
if (VARIABLE_DECLARATOR_TYPES.has(node.type) && nodeHasFunctionValue(node)) {
|
|
265
|
+
return [createStructureNode(node, "function", getName(node), context, map)];
|
|
266
|
+
}
|
|
267
|
+
if (FIELD_DEFINITION_TYPES.has(node.type)) {
|
|
268
|
+
const field = buildFieldFunctionNode(node, context, map);
|
|
269
|
+
if (field)
|
|
270
|
+
return [field];
|
|
271
|
+
}
|
|
272
|
+
const kind = kindForNode(node);
|
|
273
|
+
if (kind) {
|
|
274
|
+
const structural = createStructureNode(node, kind, getName(node), context, map);
|
|
275
|
+
const nested = scanChildren(node, childContext(structural, context), map);
|
|
276
|
+
structural.children.push(...nested.filter((child) => child.startByte >= structural.startByte && child.endByte <= structural.endByte));
|
|
277
|
+
return [structural];
|
|
278
|
+
}
|
|
279
|
+
return scanChildren(node, context, map);
|
|
280
|
+
}
|
|
281
|
+
function scanChildren(node, context, map) {
|
|
282
|
+
const nodes = [];
|
|
283
|
+
for (const child of nodeChildren(node)) {
|
|
284
|
+
if (DECLARATION_WRAPPERS.has(child.type) ||
|
|
285
|
+
FIELD_DEFINITION_TYPES.has(child.type) ||
|
|
286
|
+
isStructuralCandidate(child) ||
|
|
287
|
+
child.type.includes("declaration")) {
|
|
288
|
+
nodes.push(...collectStructureNodes(child, { ...context, spanNode: undefined }, map));
|
|
289
|
+
continue;
|
|
107
290
|
}
|
|
108
|
-
if (
|
|
109
|
-
|
|
110
|
-
let hasChildren = false;
|
|
111
|
-
for (const child of node.children) {
|
|
112
|
-
if (config.classBody && child.type === config.classBody) {
|
|
113
|
-
for (const m of child.children) {
|
|
114
|
-
if (config.fnTypes.has(m.type) || m.type === "method_definition") {
|
|
115
|
-
hasChildren = true;
|
|
116
|
-
chunks.push({
|
|
117
|
-
name: `${name}.${getName(m)}`,
|
|
118
|
-
text: m.text,
|
|
119
|
-
start: m.startPosition.row + 1,
|
|
120
|
-
end: m.endPosition.row + 1,
|
|
121
|
-
});
|
|
122
|
-
}
|
|
123
|
-
}
|
|
124
|
-
}
|
|
125
|
-
}
|
|
126
|
-
if (!hasChildren)
|
|
127
|
-
chunks.push({ name, text: node.text, start: node.startPosition.row + 1, end: node.endPosition.row + 1 });
|
|
128
|
-
return;
|
|
291
|
+
if (child.type === "class_body" || child.type === "block" || child.type === "declaration_list") {
|
|
292
|
+
nodes.push(...scanChildren(child, context, map));
|
|
129
293
|
}
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
294
|
+
}
|
|
295
|
+
return nodes.sort((a, b) => a.startByte - b.startByte || a.endByte - b.endByte);
|
|
296
|
+
}
|
|
297
|
+
function metadataForNode(node) {
|
|
298
|
+
const symbol = node.name ?? node.kind;
|
|
299
|
+
return {
|
|
300
|
+
language: node.language,
|
|
301
|
+
symbol,
|
|
302
|
+
function_name: symbol,
|
|
303
|
+
symbol_kind: node.kind,
|
|
304
|
+
scope: node.scope,
|
|
305
|
+
parent_symbol: node.parentSymbol ?? null,
|
|
306
|
+
signature: node.signature ?? null,
|
|
307
|
+
exported: node.exported,
|
|
308
|
+
ast_depth: Math.max(0, node.scope.length - 1),
|
|
309
|
+
ast_path: node.astPath.join("/"),
|
|
310
|
+
decorators: node.decorators.length > 0 ? node.decorators : undefined,
|
|
311
|
+
start_line: node.startLine,
|
|
312
|
+
end_line: node.endLine,
|
|
313
|
+
};
|
|
314
|
+
}
|
|
315
|
+
function makeChunk(content, filePath, fileType, startLine, endLine, metadata) {
|
|
316
|
+
const metadata_json = JSON.stringify(metadata);
|
|
317
|
+
const chunk = {
|
|
318
|
+
content_hash: chunkIdentityHash({ content, filePath, fileType, startLine, endLine, metadataJson: metadata_json }),
|
|
319
|
+
content,
|
|
320
|
+
content_tokenized: "",
|
|
321
|
+
file_path: filePath,
|
|
322
|
+
file_type: fileType,
|
|
323
|
+
start_line: startLine,
|
|
324
|
+
end_line: endLine,
|
|
325
|
+
metadata_json,
|
|
326
|
+
};
|
|
327
|
+
return { ...chunk, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(chunk)) };
|
|
328
|
+
}
|
|
329
|
+
function draftForNode(node, map) {
|
|
330
|
+
const text = sliceByBytes(map, node.startByte, node.endByte).trim();
|
|
331
|
+
return {
|
|
332
|
+
content: text,
|
|
333
|
+
startLine: node.startLine,
|
|
334
|
+
endLine: node.endLine,
|
|
335
|
+
startByte: node.startByte,
|
|
336
|
+
endByte: node.endByte,
|
|
337
|
+
metadata: metadataForNode(node),
|
|
338
|
+
packable: ["function", "method", "constructor", "variable"].includes(node.kind) && estimateTokens(text) < AST_TARGET_TOKENS,
|
|
339
|
+
packKey: node.parentSymbol ?? "module",
|
|
340
|
+
symbols: node.name ? [node.name] : [],
|
|
341
|
+
};
|
|
342
|
+
}
|
|
343
|
+
function fallbackDrafts(node, map) {
|
|
344
|
+
const text = sliceByBytes(map, node.startByte, node.endByte);
|
|
345
|
+
const lines = text.split("\n");
|
|
346
|
+
const drafts = [];
|
|
347
|
+
let buffer = [];
|
|
348
|
+
let bufferChars = 0;
|
|
349
|
+
let startLine = node.startLine;
|
|
350
|
+
let part = 1;
|
|
351
|
+
function pushDraft(content, draftStartLine, draftEndLine) {
|
|
352
|
+
const trimmed = content.trim();
|
|
353
|
+
if (!trimmed)
|
|
354
|
+
return;
|
|
355
|
+
drafts.push({
|
|
356
|
+
content: trimmed,
|
|
357
|
+
startLine: draftStartLine,
|
|
358
|
+
endLine: draftEndLine,
|
|
359
|
+
startByte: node.startByte,
|
|
360
|
+
endByte: node.endByte,
|
|
361
|
+
metadata: { ...metadataForNode(node), chunk_part: part, start_line: draftStartLine, end_line: draftEndLine },
|
|
362
|
+
packable: false,
|
|
363
|
+
packKey: `${node.parentSymbol ?? "module"}:${node.name ?? node.kind}`,
|
|
364
|
+
symbols: node.name ? [node.name] : [],
|
|
365
|
+
});
|
|
366
|
+
part++;
|
|
367
|
+
}
|
|
368
|
+
function flush(endLine) {
|
|
369
|
+
pushDraft(buffer.join("\n"), startLine, endLine);
|
|
370
|
+
buffer = [];
|
|
371
|
+
bufferChars = 0;
|
|
372
|
+
}
|
|
373
|
+
for (let index = 0; index < lines.length; index++) {
|
|
374
|
+
const line = lines[index];
|
|
375
|
+
const absoluteLine = node.startLine + index;
|
|
376
|
+
if (line.length > MAX_AST_FALLBACK_CHARS) {
|
|
377
|
+
if (buffer.length > 0)
|
|
378
|
+
flush(absoluteLine - 1);
|
|
379
|
+
for (let offset = 0; offset < line.length; offset += MAX_AST_FALLBACK_CHARS) {
|
|
380
|
+
pushDraft(line.slice(offset, offset + MAX_AST_FALLBACK_CHARS), absoluteLine, absoluteLine);
|
|
144
381
|
}
|
|
382
|
+
startLine = absoluteLine + 1;
|
|
383
|
+
continue;
|
|
384
|
+
}
|
|
385
|
+
if (buffer.length > 0 && bufferChars + line.length + 1 > MAX_AST_FALLBACK_CHARS)
|
|
386
|
+
flush(absoluteLine - 1);
|
|
387
|
+
if (buffer.length === 0)
|
|
388
|
+
startLine = absoluteLine;
|
|
389
|
+
buffer.push(line);
|
|
390
|
+
bufferChars += line.length + 1;
|
|
391
|
+
}
|
|
392
|
+
if (buffer.length > 0)
|
|
393
|
+
flush(node.endLine);
|
|
394
|
+
return drafts;
|
|
395
|
+
}
|
|
396
|
+
function packDrafts(drafts, map) {
|
|
397
|
+
const packed = [];
|
|
398
|
+
let buffer = [];
|
|
399
|
+
let bufferTokens = 0;
|
|
400
|
+
function flush() {
|
|
401
|
+
if (buffer.length === 0)
|
|
145
402
|
return;
|
|
403
|
+
if (buffer.length === 1) {
|
|
404
|
+
packed.push(buffer[0]);
|
|
405
|
+
}
|
|
406
|
+
else {
|
|
407
|
+
const first = buffer[0];
|
|
408
|
+
const last = buffer[buffer.length - 1];
|
|
409
|
+
const symbols = buffer.flatMap((draft) => draft.symbols);
|
|
410
|
+
const content = sliceByBytes(map, first.startByte, last.endByte).trim();
|
|
411
|
+
packed.push({
|
|
412
|
+
content,
|
|
413
|
+
startLine: first.startLine,
|
|
414
|
+
endLine: last.endLine,
|
|
415
|
+
startByte: first.startByte,
|
|
416
|
+
endByte: last.endByte,
|
|
417
|
+
metadata: {
|
|
418
|
+
...first.metadata,
|
|
419
|
+
symbol: symbols.join(", "),
|
|
420
|
+
function_name: symbols[0] ?? first.metadata.function_name,
|
|
421
|
+
symbol_kind: "group",
|
|
422
|
+
symbols,
|
|
423
|
+
signature: null,
|
|
424
|
+
start_line: first.startLine,
|
|
425
|
+
end_line: last.endLine,
|
|
426
|
+
},
|
|
427
|
+
packable: false,
|
|
428
|
+
packKey: first.packKey,
|
|
429
|
+
symbols,
|
|
430
|
+
});
|
|
431
|
+
}
|
|
432
|
+
buffer = [];
|
|
433
|
+
bufferTokens = 0;
|
|
434
|
+
}
|
|
435
|
+
for (const draft of drafts) {
|
|
436
|
+
const tokens = estimateTokens(draft.content);
|
|
437
|
+
const compatible = draft.packable &&
|
|
438
|
+
buffer.length > 0 &&
|
|
439
|
+
buffer.every((item) => item.packable && item.packKey === draft.packKey) &&
|
|
440
|
+
bufferTokens + tokens <= AST_TARGET_TOKENS;
|
|
441
|
+
if (!draft.packable) {
|
|
442
|
+
flush();
|
|
443
|
+
packed.push(draft);
|
|
444
|
+
continue;
|
|
146
445
|
}
|
|
147
|
-
|
|
148
|
-
|
|
446
|
+
if (!compatible && buffer.length > 0)
|
|
447
|
+
flush();
|
|
448
|
+
buffer.push(draft);
|
|
449
|
+
bufferTokens += tokens;
|
|
149
450
|
}
|
|
150
|
-
|
|
151
|
-
return
|
|
451
|
+
flush();
|
|
452
|
+
return packed;
|
|
152
453
|
}
|
|
153
|
-
|
|
454
|
+
function buildDrafts(nodes, map) {
|
|
455
|
+
const drafts = [];
|
|
456
|
+
for (const node of nodes.sort((a, b) => a.startByte - b.startByte || a.endByte - b.endByte)) {
|
|
457
|
+
const text = sliceByBytes(map, node.startByte, node.endByte).trim();
|
|
458
|
+
if (!text)
|
|
459
|
+
continue;
|
|
460
|
+
if (estimateTokens(text) <= AST_MAX_TOKENS) {
|
|
461
|
+
drafts.push(draftForNode(node, map));
|
|
462
|
+
continue;
|
|
463
|
+
}
|
|
464
|
+
if (node.children.length > 0) {
|
|
465
|
+
drafts.push(...buildDrafts(node.children, map));
|
|
466
|
+
continue;
|
|
467
|
+
}
|
|
468
|
+
drafts.push(...fallbackDrafts(node, map));
|
|
469
|
+
}
|
|
470
|
+
return packDrafts(drafts, map);
|
|
471
|
+
}
|
|
472
|
+
function symbolKindForNode(node) {
|
|
473
|
+
if (!node.name)
|
|
474
|
+
return undefined;
|
|
475
|
+
if (node.kind === "class")
|
|
476
|
+
return "class";
|
|
477
|
+
if (node.kind === "interface")
|
|
478
|
+
return "interface";
|
|
479
|
+
if (node.kind === "type")
|
|
480
|
+
return "type";
|
|
481
|
+
if (node.kind === "variable")
|
|
482
|
+
return "variable";
|
|
483
|
+
if (node.kind === "function" || node.kind === "method" || node.kind === "constructor")
|
|
484
|
+
return "function";
|
|
485
|
+
return undefined;
|
|
486
|
+
}
|
|
487
|
+
function symbolMetadataForNode(node) {
|
|
488
|
+
return {
|
|
489
|
+
language: node.language,
|
|
490
|
+
symbol_kind: node.kind,
|
|
491
|
+
scope: node.scope,
|
|
492
|
+
parent_symbol: node.parentSymbol ?? null,
|
|
493
|
+
exported: node.exported,
|
|
494
|
+
ast_depth: Math.max(0, node.scope.length - 1),
|
|
495
|
+
ast_path: node.astPath.join("/"),
|
|
496
|
+
decorators: node.decorators.length > 0 ? node.decorators : undefined,
|
|
497
|
+
};
|
|
498
|
+
}
|
|
499
|
+
function collectSymbols(nodes, map) {
|
|
500
|
+
const symbols = [];
|
|
501
|
+
for (const node of nodes) {
|
|
502
|
+
const kind = symbolKindForNode(node);
|
|
503
|
+
if (kind && node.name) {
|
|
504
|
+
const source = sliceByBytes(map, node.startByte, node.endByte).trim();
|
|
505
|
+
symbols.push({
|
|
506
|
+
name: node.name,
|
|
507
|
+
kind,
|
|
508
|
+
file_path: "",
|
|
509
|
+
file_type: node.language,
|
|
510
|
+
start_line: node.startLine,
|
|
511
|
+
end_line: node.endLine,
|
|
512
|
+
container_name: node.parentSymbol ?? null,
|
|
513
|
+
signature: node.signature ?? null,
|
|
514
|
+
text: node.signature ?? source.split("\n")[0]?.trim() ?? node.name,
|
|
515
|
+
metadata_json: JSON.stringify(symbolMetadataForNode(node)),
|
|
516
|
+
});
|
|
517
|
+
}
|
|
518
|
+
symbols.push(...collectSymbols(node.children, map));
|
|
519
|
+
}
|
|
520
|
+
return symbols;
|
|
521
|
+
}
|
|
522
|
+
function withSymbolLocation(symbols, filePath, fileType) {
|
|
523
|
+
return symbols.map((symbol) => ({ ...symbol, file_path: filePath, file_type: fileType }));
|
|
524
|
+
}
|
|
525
|
+
function emptyStructure(language) {
|
|
526
|
+
return {
|
|
527
|
+
kind: "module",
|
|
528
|
+
astType: "program",
|
|
529
|
+
language,
|
|
530
|
+
startLine: 1,
|
|
531
|
+
endLine: 1,
|
|
532
|
+
startByte: 0,
|
|
533
|
+
endByte: 0,
|
|
534
|
+
exported: false,
|
|
535
|
+
decorators: [],
|
|
536
|
+
scope: [],
|
|
537
|
+
astPath: ["program"],
|
|
538
|
+
children: [],
|
|
539
|
+
};
|
|
540
|
+
}
|
|
541
|
+
async function parseStructure(content, language) {
|
|
154
542
|
const config = LANGS[language];
|
|
155
543
|
if (!config)
|
|
156
|
-
|
|
544
|
+
throw new Error(`Unsupported AST language: ${language}`);
|
|
157
545
|
const Parser = (await import("tree-sitter")).default;
|
|
158
546
|
const grammar = await config.grammar();
|
|
159
547
|
const parser = new Parser();
|
|
160
548
|
parser.setLanguage(grammar);
|
|
161
549
|
const tree = parser.parse(content);
|
|
162
|
-
const
|
|
163
|
-
|
|
550
|
+
const rootNode = tree.rootNode;
|
|
551
|
+
const map = buildByteIndexMap(content);
|
|
552
|
+
const root = {
|
|
553
|
+
kind: "module",
|
|
554
|
+
astType: rootNode.type,
|
|
555
|
+
language: config.fileType,
|
|
556
|
+
startLine: 1,
|
|
557
|
+
endLine: content.split("\n").length,
|
|
558
|
+
startByte: rootNode.startIndex,
|
|
559
|
+
endByte: rootNode.endIndex,
|
|
560
|
+
exported: false,
|
|
561
|
+
decorators: [],
|
|
562
|
+
scope: [],
|
|
563
|
+
astPath: [rootNode.type],
|
|
564
|
+
children: [],
|
|
565
|
+
};
|
|
566
|
+
root.children.push(...scanChildren(rootNode, { language: config.fileType, exported: false, decorators: [], scope: [], astPath: [rootNode.type] }, map));
|
|
567
|
+
return { root, map };
|
|
568
|
+
}
|
|
569
|
+
export async function parseCodeStructure(content, language) {
|
|
570
|
+
return (await parseStructure(content, language)).root;
|
|
571
|
+
}
|
|
572
|
+
export async function analyzeCodeWithAST(content, filePath, language) {
|
|
573
|
+
const config = LANGS[language];
|
|
574
|
+
if (!config)
|
|
575
|
+
return { chunks: [], symbols: [], structure: emptyStructure(language) };
|
|
576
|
+
const { root, map } = await parseStructure(content, language);
|
|
577
|
+
const drafts = buildDrafts(root.children, map);
|
|
578
|
+
const chunks = drafts.map((draft) => makeChunk(draft.content, filePath, config.fileType, draft.startLine, draft.endLine, draft.metadata));
|
|
579
|
+
const symbols = withSymbolLocation(collectSymbols(root.children, map), filePath, config.fileType);
|
|
580
|
+
return { chunks, symbols, structure: root };
|
|
581
|
+
}
|
|
582
|
+
export async function chunkWithAST(content, filePath, language) {
|
|
583
|
+
const config = LANGS[language];
|
|
584
|
+
if (!config)
|
|
164
585
|
return [];
|
|
165
|
-
return
|
|
166
|
-
const chunk = {
|
|
167
|
-
content_hash: contentHash(fn.text),
|
|
168
|
-
content: fn.text,
|
|
169
|
-
content_tokenized: "",
|
|
170
|
-
file_path: filePath,
|
|
171
|
-
file_type: config.fileType,
|
|
172
|
-
start_line: fn.start,
|
|
173
|
-
end_line: fn.end,
|
|
174
|
-
metadata_json: JSON.stringify({ function_name: fn.name }),
|
|
175
|
-
};
|
|
176
|
-
return { ...chunk, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(chunk)) };
|
|
177
|
-
});
|
|
586
|
+
return (await analyzeCodeWithAST(content, filePath, language)).chunks;
|
|
178
587
|
}
|
|
179
588
|
export const SUPPORTED_LANGUAGES = new Set(Object.keys(LANGS));
|
package/docs/configuration.md
CHANGED
|
@@ -82,13 +82,13 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
82
82
|
|
|
83
83
|
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool size, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
84
84
|
|
|
85
|
-
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording.
|
|
85
|
+
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` when the query is conceptual and exact terms may differ from indexed wording. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks, but it remains metadata-driven retrieval rather than LSP, caller/callee, or code graph analysis.
|
|
86
86
|
|
|
87
87
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
88
88
|
|
|
89
89
|
`knowledge_configure` writes runtime configuration to the knowledge directory's `config.json`. It validates Node 22+ before persisting `node_path`, so agents can configure a Codex- or user-installed `node.exe` once without relying on environment variables being injected into an already-running OMP process.
|
|
90
90
|
|
|
91
|
-
`knowledge_symbol_search` uses a lightweight
|
|
91
|
+
`knowledge_symbol_search` uses a lightweight symbol index stored in `knowledge.db`. It is rebuilt during `knowledge_add`, `knowledge_update`, and `knowledge_import`, and can find AST-backed methods/functions/classes/interfaces/types/variables for supported code files, route-like handlers, Markdown headings, config keys, and environment variables. It is not a full LSP or code graph; it does not answer caller/callee or rename-safety questions. For empty important lookups, use `knowledge_search` mode `fast`/`adaptive` and check `knowledge_doctor` for stale or missing symbol metadata. Older file/directory/URL KBs without symbol metadata should be updated before relying on symbol lookup. Imported portable KBs and inline text KBs do not retain an active source path; rebuild their symbols by re-importing or re-adding the content.
|
|
92
92
|
|
|
93
93
|
## Test and Release Fixtures
|
|
94
94
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.9.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|