pi-knowledge 0.10.2 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +12 -10
- package/dist/index.js +2 -1
- package/dist/src/embedding/provider.d.ts +14 -0
- package/dist/src/embedding/provider.js +69 -9
- package/dist/src/engine.d.ts +4 -2
- package/dist/src/engine.js +69 -22
- package/dist/src/indexer/chunker.d.ts +1 -1
- package/dist/src/indexer/chunker.js +2 -2
- package/dist/src/indexer/embedding-safe-chunks.d.ts +5 -0
- package/dist/src/indexer/embedding-safe-chunks.js +215 -0
- package/dist/src/indexer/html.d.ts +1 -0
- package/dist/src/indexer/html.js +49 -0
- package/dist/src/model-worker-client.d.ts +5 -1
- package/dist/src/model-worker-client.js +21 -3
- package/dist/src/model-worker.js +73 -15
- package/dist/src/search/tuning.d.ts +1 -1
- package/dist/src/search/tuning.js +1 -1
- package/dist/src/storage/sqlite.d.ts +1 -0
- package/dist/src/storage/sqlite.js +3 -0
- package/docs/configuration.md +3 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,22 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.12.0] - 2026-10-19
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Added embedding-safe chunk finalization for add, update, URL, inline-text, and JSONL import paths. Local inputs use the existing model worker tokenizer; OpenAI-compatible inputs use the configured complete-input character cap.
|
|
7
|
+
- Added `knowledge_doctor` action `reindex_for_embedding_budget` for KBs created before the current chunk input policy.
|
|
8
|
+
- Added dependency-free structured HTML normalization for URL sources, preserving headings, paragraphs, lists, tables, and entities for heading-aware chunking.
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
- Prevented local and OpenAI-compatible embedding inputs from silently discarding oversized multilingual or context-enriched chunk tails.
|
|
12
|
+
- Kept the model worker startup single-flight and Node strip-only compatible while reusing it for batched tokenizer measurement.
|
|
13
|
+
|
|
14
|
+
## [0.11.0] - 2026-09-23
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
- Added `PI_KNOWLEDGE_EMBEDDING_PREFIXES=on|off` so OpenAI-compatible and local embeddings can preserve or omit `query: ` / `passage: ` input prefixes while keeping default E5 behavior compatible with existing KB signatures (#18).
|
|
18
|
+
- Added `semantic_hybrid` search mode for vector-backed weighted fusion without the default hybrid lexical-evidence gate, giving semantic and multilingual queries an explicit recall-oriented path while preserving `hybrid` precision defaults (#17).
|
|
19
|
+
|
|
3
20
|
## [0.10.2] - 2026-09-19
|
|
4
21
|
|
|
5
22
|
### Fixed
|
package/README.md
CHANGED
|
@@ -23,7 +23,8 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
23
23
|
|
|
24
24
|
- **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
|
|
25
25
|
- **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
|
|
26
|
-
- **Code-aware indexing**: recursive AST chunking for
|
|
26
|
+
- **Code- and document-aware indexing**: recursive AST chunking for supported code, heading-aware Markdown chunks, and an embedding-safe finalizer that preserves oversized multilingual tails without adding a heavy document runtime.
|
|
27
|
+
- **Structured URL indexing**: preserves HTML headings, paragraphs, lists, tables, and entities as lightweight Markdown-like text without a browser, Docling runtime, or new dependency.
|
|
27
28
|
- **Lightweight symbol lookup**: indexes functions, methods, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
|
|
28
29
|
- **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
|
|
29
30
|
- **Large-project stability**: persisted indexing progress, capped batches, streaming vector scans, and stuck-job detection.
|
|
@@ -67,7 +68,7 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
|
|
|
67
68
|
- **Contextual Retrieval without remote chunk rewriting**: applies Anthropic's Contextual Retrieval insight locally by embedding file path, file type, Markdown breadcrumbs, and AST-derived code structure such as symbols, scope, parent symbol, and signatures with each chunk. This improves standalone chunk meaning without sending private source chunks to an LLM for context generation.
|
|
68
69
|
- **Hybrid retrieval with diagnosable scores**: combines lexical BM25 anchors and vectors with normalized weighted score fusion. RRF (Cormack et al. 2009) remains the baseline reference, but weighted fusion is used by default because project dogfood showed RRF compressed scores too much for ranking diagnostics.
|
|
69
70
|
- **MMR-style diversity**: uses Maximal Marginal Relevance ideas (Goldstein and Carbonell 1998), file interleaving, vector redundancy checks, and bounded adaptive-window overlap collapse so repeated README or same-file chunks do not dominate top results. Code KBs with AST metadata can also use same-parent/sibling structure during adaptive expansion.
|
|
70
|
-
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
71
|
+
- **Intent-aware and self-correcting agent UX**: mode selection (`auto`, `fast`, `semantic`, `semantic_hybrid`, `hybrid`, `adaptive`, `deep`), ranking diagnostics, and `knowledge_doctor` turn retrieval failures into concrete next actions instead of silent bad answers.
|
|
71
72
|
- **Confidence gating**: low-evidence hybrid matches can return zero results instead of unrelated chunks, reducing false confidence when the KB does not contain the answer.
|
|
72
73
|
|
|
73
74
|
This is intentionally not a heavy ColBERT-style late-interaction index yet (Khattab and Zaharia 2020). The current product chooses lightweight local embeddings, BM25, query-aware ranking, optional cross-encoder reranking, streamed vector scans, and health diagnostics for commercial usefulness with low setup cost.
|
|
@@ -125,6 +126,7 @@ omp install ./pi-knowledge
|
|
|
125
126
|
- `fast`: BM25 keyword search for exact symbols, commands, and identifiers.
|
|
126
127
|
- `semantic`: vector search for conceptual matches.
|
|
127
128
|
- `hybrid`: lexical-anchored BM25 + vector search with normalized weighted score fusion. It requires keyword evidence to avoid low-confidence semantic false positives.
|
|
129
|
+
- `semantic_hybrid`: vector search plus optional BM25 weighted fusion without the hybrid lexical-evidence gate; use explicitly for semantic/multilingual queries where lexical overlap is weak but fused scoring is still useful.
|
|
128
130
|
- `deep`: hybrid retrieval followed by cross-encoder reranking.
|
|
129
131
|
- `adaptive`: hybrid retrieval followed by query-time contextual window expansion around seed chunks. It keeps the matched seed, prefers nearby/query-relevant neighboring chunks, and collapses overlapping windows from the same file.
|
|
130
132
|
- `auto`: selects a primary mode from the query shape and retries alternate modes when results are empty or weak.
|
|
@@ -136,15 +138,15 @@ Mode selection contract:
|
|
|
136
138
|
|
|
137
139
|
- Start with `hybrid` for most project questions that contain useful lexical anchors.
|
|
138
140
|
- Use `fast` for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings.
|
|
139
|
-
- Use `semantic` when the query is conceptual
|
|
141
|
+
- Use `semantic` when the query is conceptual and exact terms may differ from indexed wording; use `semantic_hybrid` when weak lexical overlap should not block vector-backed fused results.
|
|
140
142
|
- Use `adaptive` when the answer needs nearby code, neighboring documentation sections, AST-related sibling methods where indexed, or enough context to make a safe edit.
|
|
141
143
|
- Use `deep` for high-stakes answers, ambiguous top results, or final verification when slower reranking is acceptable.
|
|
142
144
|
- If results are empty or weak but the KB should contain the answer, retry once with a different mode before concluding no answer exists.
|
|
143
145
|
|
|
144
146
|
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. In `adaptive` mode, code chunks with AST metadata can prefer same-parent/sibling chunks inside the same bounded file window; this is metadata-driven context expansion, not caller/callee analysis. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, source freshness, and adaptive `source_chunk_ids` where available.
|
|
145
147
|
|
|
146
|
-
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable.
|
|
147
|
-
Symbol/config/heading metadata
|
|
148
|
+
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. Before embedding, the complete contextual input is checked against the active provider budget; only oversized chunks are split again along structural boundaries.
|
|
149
|
+
Symbol/config/heading metadata and embedding-safe chunk boundaries are rebuilt during `knowledge_update`. `knowledge_doctor` reports `reindex_for_embedding_budget` for older KBs: file, directory, and URL sources can use `knowledge_update`; imported or inline-text sources should be re-imported or re-added from the original source.
|
|
148
150
|
|
|
149
151
|
## Embedding Configuration
|
|
150
152
|
|
|
@@ -169,9 +171,9 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
|
169
171
|
# Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
|
|
170
172
|
```
|
|
171
173
|
|
|
172
|
-
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
|
|
174
|
+
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Inputs are never silently truncated.
|
|
173
175
|
|
|
174
|
-
|
|
176
|
+
OpenAI-compatible embedding inputs use a 20000-character complete-input safety cap by default. Indexing splits oversized chunks before HTTP requests; direct oversized query/document calls fail explicitly. Adjust the cap with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when the remote model has a different context window. Changing the cap changes embedding compatibility and requires `knowledge_update`.
|
|
175
177
|
|
|
176
178
|
For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
|
|
177
179
|
|
|
@@ -185,7 +187,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
185
187
|
|------|-----------------------|
|
|
186
188
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
187
189
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
188
|
-
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
190
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
189
191
|
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
190
192
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
191
193
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
@@ -230,8 +232,8 @@ All data is stored globally at `~/.pi/knowledge/` under Pi or `~/.omp/knowledge/
|
|
|
230
232
|
- **Reset**: delete the active knowledge directory to start fresh
|
|
231
233
|
- **Override**: set `PI_KNOWLEDGE_DIR` or `OMP_KNOWLEDGE_DIR`
|
|
232
234
|
- **Project safety**: pi-knowledge is read-only on indexed directories — no files are created or modified in your project
|
|
233
|
-
- **Updates**:
|
|
234
|
-
- **Symbol
|
|
235
|
+
- **Updates**: schema migrations run automatically, but index-time behavior changes are not applied to existing vectors at startup. `knowledge_doctor` reports actionable reindex requirements without loading the embedding model.
|
|
236
|
+
- **Symbol and chunk-policy indexes**: `knowledge.db` stores lightweight symbol/config/heading metadata plus an embedding input policy version. File, directory, and URL KBs with retained source paths can rebuild them with `knowledge_update`; imported portable KBs and inline text KBs should be re-imported or re-added because they intentionally do not store an active local source manifest.
|
|
235
237
|
|
|
236
238
|
## Development
|
|
237
239
|
|
package/dist/index.js
CHANGED
|
@@ -364,7 +364,7 @@ export default function (pi) {
|
|
|
364
364
|
"Use knowledge_search to find relevant context before answering domain questions",
|
|
365
365
|
"Default to mode 'hybrid' for most project questions with useful lexical anchors; it fuses BM25 and semantic vectors but still requires keyword evidence",
|
|
366
366
|
"Use mode 'fast' for exact symbols, filenames, commands, error codes, API names, config keys, or quoted strings",
|
|
367
|
-
"Use mode 'semantic' for
|
|
367
|
+
"Use mode 'semantic' for pure vector search when exact terms may differ from indexed wording; use mode 'semantic_hybrid' when you want vector recall plus BM25/fusion scoring without the hybrid lexical evidence gate",
|
|
368
368
|
"Use mode 'adaptive' when the user needs surrounding implementation context, related nearby sections, or enough context to make a code change",
|
|
369
369
|
"Use mode 'deep' for high-stakes answers, ambiguous top results, or final verification where slower cross-encoder reranking is acceptable",
|
|
370
370
|
"Use profile 'low_token' for slow local models that need fewer, stricter, longer snippets; use profile 'precision' for identifiers/errors and 'recall' or 'long_context' for broad prose/document research",
|
|
@@ -378,6 +378,7 @@ export default function (pi) {
|
|
|
378
378
|
Type.Literal("fast"),
|
|
379
379
|
Type.Literal("semantic"),
|
|
380
380
|
Type.Literal("hybrid"),
|
|
381
|
+
Type.Literal("semantic_hybrid"),
|
|
381
382
|
Type.Literal("deep"),
|
|
382
383
|
Type.Literal("adaptive"),
|
|
383
384
|
Type.Literal("code"),
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
export type EmbeddingProvider = "local" | "openai";
|
|
2
2
|
export type EmbeddingPrefix = "query" | "passage";
|
|
3
|
+
export type EmbeddingPrefixStrategy = "on" | "off";
|
|
3
4
|
export interface EmbeddingConfig {
|
|
4
5
|
provider: EmbeddingProvider;
|
|
5
6
|
model: string;
|
|
@@ -7,27 +8,39 @@ export interface EmbeddingConfig {
|
|
|
7
8
|
maxChars: number;
|
|
8
9
|
queryPrefix: "query";
|
|
9
10
|
documentPrefix: "passage";
|
|
11
|
+
prefixes: EmbeddingPrefixStrategy;
|
|
10
12
|
pooling: "mean";
|
|
11
13
|
normalize: true;
|
|
12
14
|
}
|
|
15
|
+
export interface EmbeddingInputMeasurement {
|
|
16
|
+
counts: number[];
|
|
17
|
+
maximum: number;
|
|
18
|
+
preferredMaximum: number;
|
|
19
|
+
unit: "tokens" | "characters";
|
|
20
|
+
}
|
|
13
21
|
export declare const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
14
22
|
export declare const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
15
23
|
export declare const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
|
|
16
24
|
export declare const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
25
|
+
export declare const LOCAL_EMBEDDING_MAX_TOKENS = 512;
|
|
26
|
+
export declare const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
|
|
17
27
|
export declare const DEFAULT_API_MAX_EMBED_CHARS = 20000;
|
|
18
28
|
export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
|
|
19
29
|
export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
|
|
20
30
|
export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
|
|
21
31
|
export declare function dispose(): Promise<void>;
|
|
22
32
|
export declare function prepareForShutdown(): Promise<void>;
|
|
33
|
+
export declare function measureEmbeddingInputs(texts: string[], prefix: EmbeddingPrefix, config: EmbeddingConfig, signal?: AbortSignal): Promise<EmbeddingInputMeasurement>;
|
|
23
34
|
export declare function embedTextsWithConfig(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
|
|
24
35
|
allowApiFallback?: boolean;
|
|
36
|
+
inputValidated?: boolean;
|
|
25
37
|
}): Promise<{
|
|
26
38
|
vectors: Float32Array[];
|
|
27
39
|
config: EmbeddingConfig;
|
|
28
40
|
}>;
|
|
29
41
|
export declare function embedTexts(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
|
|
30
42
|
allowApiFallback?: boolean;
|
|
43
|
+
inputValidated?: boolean;
|
|
31
44
|
}): Promise<Float32Array[]>;
|
|
32
45
|
export declare function embedQueryWithConfig(text: string, signal?: AbortSignal): Promise<{
|
|
33
46
|
vector: Float32Array;
|
|
@@ -35,3 +48,4 @@ export declare function embedQueryWithConfig(text: string, signal?: AbortSignal)
|
|
|
35
48
|
}>;
|
|
36
49
|
export declare function embedQuery(text: string, signal?: AbortSignal): Promise<Float32Array>;
|
|
37
50
|
export declare function embedDocuments(texts: string[], signal?: AbortSignal): Promise<Float32Array[]>;
|
|
51
|
+
export declare function embedValidatedDocuments(texts: string[], signal?: AbortSignal): Promise<Float32Array[]>;
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { createHash } from "node:crypto";
|
|
2
|
-
import { embedInModelWorker } from "../model-worker-client.js";
|
|
2
|
+
import { embedInModelWorker, measureEmbeddingInputInModelWorker } from "../model-worker-client.js";
|
|
3
3
|
let disposeTimer = null;
|
|
4
4
|
let disposePromise = null;
|
|
5
5
|
let activeRuns = 0;
|
|
@@ -9,17 +9,20 @@ export const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
|
9
9
|
export const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
10
10
|
export const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
|
|
11
11
|
export const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
12
|
+
export const LOCAL_EMBEDDING_MAX_TOKENS = 512;
|
|
13
|
+
export const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
|
|
12
14
|
export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
|
|
13
15
|
const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
|
|
14
16
|
const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
|
|
15
17
|
const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
|
|
16
|
-
function localEmbeddingConfig(maxChars) {
|
|
18
|
+
function localEmbeddingConfig(maxChars, prefixes) {
|
|
17
19
|
return {
|
|
18
20
|
provider: "local",
|
|
19
21
|
model: DEFAULT_LOCAL_EMBEDDING_MODEL,
|
|
20
22
|
maxChars,
|
|
21
23
|
queryPrefix: "query",
|
|
22
24
|
documentPrefix: "passage",
|
|
25
|
+
prefixes,
|
|
23
26
|
pooling: "mean",
|
|
24
27
|
normalize: true,
|
|
25
28
|
};
|
|
@@ -65,6 +68,14 @@ function waitForNoActiveRuns() {
|
|
|
65
68
|
function providerInstanceHash(value) {
|
|
66
69
|
return createHash("sha256").update(value).digest("hex").slice(0, 12);
|
|
67
70
|
}
|
|
71
|
+
function resolveEmbeddingPrefixStrategy(env) {
|
|
72
|
+
const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_PREFIXES)?.toLowerCase();
|
|
73
|
+
if (!raw)
|
|
74
|
+
return "on";
|
|
75
|
+
if (raw === "on" || raw === "off")
|
|
76
|
+
return raw;
|
|
77
|
+
throw new Error(`Unsupported embedding prefix strategy: ${raw}. Use PI_KNOWLEDGE_EMBEDDING_PREFIXES=on or off.`);
|
|
78
|
+
}
|
|
68
79
|
export function resolveEmbeddingConfig(env = process.env) {
|
|
69
80
|
const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING) ?? `local:${DEFAULT_LOCAL_EMBEDDING_MODEL}`;
|
|
70
81
|
const separator = raw.indexOf(":");
|
|
@@ -72,12 +83,13 @@ export function resolveEmbeddingConfig(env = process.env) {
|
|
|
72
83
|
const model = separator >= 0 ? raw.slice(separator + 1).trim() : "";
|
|
73
84
|
const maxCharsValue = Number(cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS) ?? DEFAULT_API_MAX_EMBED_CHARS);
|
|
74
85
|
const maxChars = Number.isFinite(maxCharsValue) && maxCharsValue > 0 ? Math.trunc(maxCharsValue) : DEFAULT_API_MAX_EMBED_CHARS;
|
|
86
|
+
const prefixes = resolveEmbeddingPrefixStrategy(env);
|
|
75
87
|
if (provider === "local") {
|
|
76
88
|
const localModel = model || DEFAULT_LOCAL_EMBEDDING_MODEL;
|
|
77
89
|
if (localModel !== DEFAULT_LOCAL_EMBEDDING_MODEL) {
|
|
78
90
|
throw new Error(`Unsupported local embedding model: ${localModel}`);
|
|
79
91
|
}
|
|
80
|
-
return localEmbeddingConfig(maxChars);
|
|
92
|
+
return localEmbeddingConfig(maxChars, prefixes);
|
|
81
93
|
}
|
|
82
94
|
if (provider === "openai") {
|
|
83
95
|
return {
|
|
@@ -87,6 +99,7 @@ export function resolveEmbeddingConfig(env = process.env) {
|
|
|
87
99
|
maxChars,
|
|
88
100
|
queryPrefix: "query",
|
|
89
101
|
documentPrefix: "passage",
|
|
102
|
+
prefixes,
|
|
90
103
|
pooling: "mean",
|
|
91
104
|
normalize: true,
|
|
92
105
|
};
|
|
@@ -98,7 +111,7 @@ export function embeddingConfigLabel(config) {
|
|
|
98
111
|
}
|
|
99
112
|
export function embeddingSignature(config, dimension) {
|
|
100
113
|
const base = config.baseUrl ? `:base-sha256=${providerInstanceHash(config.baseUrl)}` : "";
|
|
101
|
-
|
|
114
|
+
const parts = [
|
|
102
115
|
`${config.provider}:${config.model}${base}`,
|
|
103
116
|
`dim=${dimension}`,
|
|
104
117
|
`apiMax=${config.provider === "openai" ? config.maxChars : "none"}`,
|
|
@@ -106,7 +119,10 @@ export function embeddingSignature(config, dimension) {
|
|
|
106
119
|
`normalize=${config.normalize}`,
|
|
107
120
|
`q=${config.queryPrefix}`,
|
|
108
121
|
`d=${config.documentPrefix}`,
|
|
109
|
-
]
|
|
122
|
+
];
|
|
123
|
+
if (config.prefixes === "off")
|
|
124
|
+
parts.push("prefixes=off");
|
|
125
|
+
return parts.join(":");
|
|
110
126
|
}
|
|
111
127
|
export async function dispose() {
|
|
112
128
|
clearIdleTimer();
|
|
@@ -124,9 +140,44 @@ export async function prepareForShutdown() {
|
|
|
124
140
|
clearIdleTimer();
|
|
125
141
|
await waitForNoActiveRuns();
|
|
126
142
|
}
|
|
143
|
+
function applyEmbeddingPrefix(text, prefix, config) {
|
|
144
|
+
return config.prefixes === "on" ? `${prefix}: ${text}` : text;
|
|
145
|
+
}
|
|
146
|
+
export async function measureEmbeddingInputs(texts, prefix, config, signal) {
|
|
147
|
+
if (signal?.aborted)
|
|
148
|
+
throw new Error("Cancelled");
|
|
149
|
+
if (config.provider === "openai") {
|
|
150
|
+
return {
|
|
151
|
+
counts: texts.map((text) => applyEmbeddingPrefix(text, prefix, config).length),
|
|
152
|
+
maximum: config.maxChars,
|
|
153
|
+
preferredMaximum: Math.max(1, Math.floor(config.maxChars * 0.875)),
|
|
154
|
+
unit: "characters",
|
|
155
|
+
};
|
|
156
|
+
}
|
|
157
|
+
beginRun();
|
|
158
|
+
try {
|
|
159
|
+
const measurement = await measureEmbeddingInputInModelWorker(texts, prefix, config.prefixes, signal);
|
|
160
|
+
return {
|
|
161
|
+
...measurement,
|
|
162
|
+
preferredMaximum: Math.max(1, Math.floor(measurement.maximum * 0.875)),
|
|
163
|
+
unit: "tokens",
|
|
164
|
+
};
|
|
165
|
+
}
|
|
166
|
+
finally {
|
|
167
|
+
endRun();
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
async function assertEmbeddingInputsWithinBudget(texts, prefix, config, signal) {
|
|
171
|
+
const measurement = await measureEmbeddingInputs(texts, prefix, config, signal);
|
|
172
|
+
for (let index = 0; index < measurement.counts.length; index++) {
|
|
173
|
+
const count = measurement.counts[index];
|
|
174
|
+
if (count !== undefined && count > measurement.maximum) {
|
|
175
|
+
throw new Error(`${config.provider} embedding ${prefix} input ${index + 1} is ${count} ${measurement.unit}; maximum is ${measurement.maximum}. Split the input before embedding.`);
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
}
|
|
127
179
|
async function embedViaAPI(texts, prefix, config, signal) {
|
|
128
|
-
const
|
|
129
|
-
const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
|
|
180
|
+
const safeTexts = texts.map((text) => applyEmbeddingPrefix(text, prefix, config));
|
|
130
181
|
const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
|
|
131
182
|
if (!config.baseUrl)
|
|
132
183
|
throw new Error("OpenAI embedding base URL is not configured");
|
|
@@ -150,7 +201,10 @@ async function embedViaAPI(texts, prefix, config, signal) {
|
|
|
150
201
|
export async function embedTextsWithConfig(texts, prefix, signal, options = {}) {
|
|
151
202
|
const config = resolveEmbeddingConfig();
|
|
152
203
|
let actualConfig = config;
|
|
204
|
+
let inputValidated = options.inputValidated === true;
|
|
153
205
|
if (config.provider === "openai") {
|
|
206
|
+
if (!inputValidated)
|
|
207
|
+
await assertEmbeddingInputsWithinBudget(texts, prefix, config, signal);
|
|
154
208
|
if (signal?.aborted)
|
|
155
209
|
throw new Error("Cancelled");
|
|
156
210
|
try {
|
|
@@ -162,12 +216,15 @@ export async function embedTextsWithConfig(texts, prefix, signal, options = {})
|
|
|
162
216
|
if (!API_FALLBACK_TO_LOCAL || !options.allowApiFallback)
|
|
163
217
|
throw error;
|
|
164
218
|
console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
|
|
165
|
-
actualConfig = localEmbeddingConfig(config.maxChars);
|
|
219
|
+
actualConfig = localEmbeddingConfig(config.maxChars, config.prefixes);
|
|
220
|
+
inputValidated = false;
|
|
166
221
|
}
|
|
167
222
|
}
|
|
223
|
+
if (!inputValidated)
|
|
224
|
+
await assertEmbeddingInputsWithinBudget(texts, prefix, actualConfig, signal);
|
|
168
225
|
beginRun();
|
|
169
226
|
try {
|
|
170
|
-
return { vectors: await embedInModelWorker(texts, prefix, signal), config: actualConfig };
|
|
227
|
+
return { vectors: await embedInModelWorker(texts, prefix, actualConfig.prefixes, signal), config: actualConfig };
|
|
171
228
|
}
|
|
172
229
|
finally {
|
|
173
230
|
endRun();
|
|
@@ -186,3 +243,6 @@ export async function embedQuery(text, signal) {
|
|
|
186
243
|
export async function embedDocuments(texts, signal) {
|
|
187
244
|
return embedTexts(texts, "passage", signal, { allowApiFallback: false });
|
|
188
245
|
}
|
|
246
|
+
export async function embedValidatedDocuments(texts, signal) {
|
|
247
|
+
return embedTexts(texts, "passage", signal, { allowApiFallback: false, inputValidated: true });
|
|
248
|
+
}
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { createSkippedScanStats } from "./indexer/chunker.ts";
|
|
|
3
3
|
import { type RankingDiagnostics } from "./search/ranking.ts";
|
|
4
4
|
import { type SearchProfile, type SearchTuningSummary } from "./search/tuning.ts";
|
|
5
5
|
import { type KnowledgeBase, type KnowledgeSymbol } from "./storage/sqlite.ts";
|
|
6
|
-
export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
6
|
+
export type SearchMode = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
7
7
|
export interface SearchOptions {
|
|
8
8
|
mode?: SearchMode;
|
|
9
9
|
profile?: SearchProfile;
|
|
@@ -58,7 +58,7 @@ export interface DoctorIssue {
|
|
|
58
58
|
kb_name?: string;
|
|
59
59
|
message: string;
|
|
60
60
|
action: string;
|
|
61
|
-
action_code?: "run_update" | "rebuild_kb" | "wait_for_indexing" | "review_skipped_scope" | "rebuild_vectors" | "check_source" | "none";
|
|
61
|
+
action_code?: "run_update" | "rebuild_kb" | "wait_for_indexing" | "review_skipped_scope" | "rebuild_vectors" | "reindex_for_embedding_budget" | "check_source" | "none";
|
|
62
62
|
}
|
|
63
63
|
export interface DoctorReport {
|
|
64
64
|
health_score: number;
|
|
@@ -97,6 +97,8 @@ export interface SymbolSearchResponse {
|
|
|
97
97
|
total_count: number;
|
|
98
98
|
has_more: boolean;
|
|
99
99
|
}
|
|
100
|
+
export declare const CURRENT_URL_STRUCTURE_POLICY = 1;
|
|
101
|
+
export declare const CURRENT_EMBEDDING_INPUT_POLICY = 2;
|
|
100
102
|
export interface IndexPlan {
|
|
101
103
|
source_type: "file" | "directory" | "text" | "url";
|
|
102
104
|
scannable_files: number;
|
package/dist/src/engine.js
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
import { existsSync, readFileSync, renameSync, rmSync, statSync } from "node:fs";
|
|
2
2
|
import { join, resolve } from "node:path";
|
|
3
3
|
import { diagnoseKB } from "./diagnostics/health.js";
|
|
4
|
-
import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
|
|
4
|
+
import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, embedValidatedDocuments, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
|
|
5
5
|
import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
|
|
6
|
-
import { addSkippedScanEntry, analyzeIndexableContent, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
6
|
+
import { addSkippedScanEntry, analyzeIndexableContent, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, chunkMarkdown, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
7
|
+
import { finalizeChunksForEmbedding } from "./indexer/embedding-safe-chunks.js";
|
|
8
|
+
import { htmlToStructuredMarkdown } from "./indexer/html.js";
|
|
7
9
|
import { extractSymbols } from "./indexer/symbols.js";
|
|
8
10
|
import { shutdownModelWorker } from "./model-worker-client.js";
|
|
9
11
|
import { searchBM25 } from "./search/bm25.js";
|
|
@@ -13,8 +15,10 @@ import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence, normalizeFileTypeFilte
|
|
|
13
15
|
import { disposeReranker, prepareRerankerForShutdown, rerank } from "./search/reranker.js";
|
|
14
16
|
import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js";
|
|
15
17
|
import { searchVectorFile } from "./search/vector.js";
|
|
16
|
-
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
|
|
18
|
+
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBSourceOptions, updateKBStatus, } from "./storage/sqlite.js";
|
|
17
19
|
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
|
|
20
|
+
export const CURRENT_URL_STRUCTURE_POLICY = 1;
|
|
21
|
+
export const CURRENT_EMBEDDING_INPUT_POLICY = 2;
|
|
18
22
|
const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
|
|
19
23
|
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
20
24
|
function resolveIndexEmbedBatchSize(env = process.env) {
|
|
@@ -66,14 +70,17 @@ function toScanOptions(options = {}) {
|
|
|
66
70
|
};
|
|
67
71
|
}
|
|
68
72
|
function serializeAddOptions(options = {}) {
|
|
69
|
-
const normalized = {
|
|
73
|
+
const normalized = {
|
|
74
|
+
embedding_input_policy: CURRENT_EMBEDDING_INPUT_POLICY,
|
|
75
|
+
url_structure_policy: CURRENT_URL_STRUCTURE_POLICY,
|
|
76
|
+
};
|
|
70
77
|
if (options.include_suggested_text === true)
|
|
71
78
|
normalized.include_suggested_text = true;
|
|
72
79
|
if (options.include_paths && options.include_paths.length > 0)
|
|
73
80
|
normalized.include_paths = options.include_paths;
|
|
74
81
|
if (options.exclude_paths && options.exclude_paths.length > 0)
|
|
75
82
|
normalized.exclude_paths = options.exclude_paths;
|
|
76
|
-
return
|
|
83
|
+
return JSON.stringify(normalized);
|
|
77
84
|
}
|
|
78
85
|
function parseAddOptions(raw) {
|
|
79
86
|
if (!raw)
|
|
@@ -94,6 +101,18 @@ function parseAddOptions(raw) {
|
|
|
94
101
|
return {};
|
|
95
102
|
}
|
|
96
103
|
}
|
|
104
|
+
function storedSourcePolicyVersion(raw, key) {
|
|
105
|
+
if (!raw)
|
|
106
|
+
return undefined;
|
|
107
|
+
try {
|
|
108
|
+
const parsed = JSON.parse(raw);
|
|
109
|
+
const version = parsed[key];
|
|
110
|
+
return typeof version === "number" && Number.isInteger(version) ? version : undefined;
|
|
111
|
+
}
|
|
112
|
+
catch {
|
|
113
|
+
return undefined;
|
|
114
|
+
}
|
|
115
|
+
}
|
|
97
116
|
function planDirectoryScan(dirPath, options = {}, signal) {
|
|
98
117
|
const skipped = createSkippedScanStats();
|
|
99
118
|
let files = 0;
|
|
@@ -452,14 +471,10 @@ async function chunkUrl(source, signal) {
|
|
|
452
471
|
if (!res.ok)
|
|
453
472
|
throw new Error(`Fetch failed: ${res.status}`);
|
|
454
473
|
const html = await res.text();
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
.replace(/&[a-z]+;/gi, " ")
|
|
460
|
-
.replace(/\s+/g, " ")
|
|
461
|
-
.trim();
|
|
462
|
-
return chunkFile(text, source);
|
|
474
|
+
throwIfAborted(signal);
|
|
475
|
+
const text = htmlToStructuredMarkdown(html);
|
|
476
|
+
const structuredChunks = chunkMarkdown(text, source, "html");
|
|
477
|
+
return structuredChunks.length > 0 ? structuredChunks : chunkFile(text, source);
|
|
463
478
|
}
|
|
464
479
|
function normalizeExtractedText(text) {
|
|
465
480
|
return Array.isArray(text) ? text.join("\n\n") : text;
|
|
@@ -772,7 +787,7 @@ export class KnowledgeEngine {
|
|
|
772
787
|
throw new Error("Cancelled");
|
|
773
788
|
const batch = pendingChunks.splice(0, embeddingBatchSize);
|
|
774
789
|
reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
|
|
775
|
-
const vectors = await
|
|
790
|
+
const vectors = await embedValidatedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
776
791
|
if (signal?.aborted)
|
|
777
792
|
throw new Error("Cancelled");
|
|
778
793
|
assertEmbeddingBatchSize(vectors, batch.length, "add");
|
|
@@ -784,7 +799,8 @@ export class KnowledgeEngine {
|
|
|
784
799
|
reportProgress("Stored batch", processedFiles, totalFiles);
|
|
785
800
|
};
|
|
786
801
|
const addChunks = async (chunks, processedFiles, totalFiles) => {
|
|
787
|
-
|
|
802
|
+
const finalizedChunks = await finalizeChunksForEmbedding(chunks, embeddingConfig, signal);
|
|
803
|
+
pendingChunks.push(...finalizedChunks);
|
|
788
804
|
while (pendingChunks.length >= embeddingBatchSize) {
|
|
789
805
|
await flushPending(processedFiles, totalFiles);
|
|
790
806
|
}
|
|
@@ -1016,7 +1032,7 @@ export class KnowledgeEngine {
|
|
|
1016
1032
|
unchanged_chunks: unchanged,
|
|
1017
1033
|
});
|
|
1018
1034
|
onProgress?.(message);
|
|
1019
|
-
const newVectors = await
|
|
1035
|
+
const newVectors = await embedValidatedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
|
|
1020
1036
|
if (signal?.aborted)
|
|
1021
1037
|
throw new Error("Cancelled");
|
|
1022
1038
|
assertEmbeddingBatchSize(newVectors, batch.length, "update");
|
|
@@ -1043,7 +1059,8 @@ export class KnowledgeEngine {
|
|
|
1043
1059
|
onProgress?.(storedMessage);
|
|
1044
1060
|
};
|
|
1045
1061
|
const processChunks = async (chunks) => {
|
|
1046
|
-
|
|
1062
|
+
const finalizedChunks = await finalizeChunksForEmbedding(chunks, embeddingConfig, signal);
|
|
1063
|
+
for (const chunk of finalizedChunks) {
|
|
1047
1064
|
scannedChunks++;
|
|
1048
1065
|
const existing = reusableHashes.get(chunk.content_hash);
|
|
1049
1066
|
if (existing && existing.length > 0) {
|
|
@@ -1231,6 +1248,7 @@ export class KnowledgeEngine {
|
|
|
1231
1248
|
insertSymbols(this.db, kb.id, stagedSymbols);
|
|
1232
1249
|
updateKBEmbeddingMetadata(this.db, kb.id, embeddingModel, finalEmbeddingDimension === undefined ? null : embeddingSignature(embeddingConfig, finalEmbeddingDimension), finalEmbeddingDimension ?? null);
|
|
1233
1250
|
updateKBCounts(this.db, kb.id, finalChunkCount, getFileCount(this.db, kb.id));
|
|
1251
|
+
updateKBSourceOptions(this.db, kb.id, serializeAddOptions(parseAddOptions(kb.source_options)));
|
|
1234
1252
|
updateKBStatus(this.db, kb.id, "ready");
|
|
1235
1253
|
const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
|
|
1236
1254
|
const readyMessage = `Ready: +${addedCount} -${idsToRemove.length} =${unchanged}${skippedReadySuffix}`;
|
|
@@ -1371,8 +1389,9 @@ export class KnowledgeEngine {
|
|
|
1371
1389
|
vectorsByChunkId.set(chunkId, vector);
|
|
1372
1390
|
}
|
|
1373
1391
|
else {
|
|
1392
|
+
const semanticHybrid = retrievalMode === "semantic_hybrid";
|
|
1374
1393
|
const bm25Results = searchBM25(db, normalizedQuery || query, candidateLimit, kb.id);
|
|
1375
|
-
if (bm25Results.length === 0)
|
|
1394
|
+
if (bm25Results.length === 0 && !semanticHybrid)
|
|
1376
1395
|
continue;
|
|
1377
1396
|
let vecResults = [];
|
|
1378
1397
|
const { vector: queryVec, config: queryEmbeddingConfig } = await embedQueryWithConfig(query, signal);
|
|
@@ -1422,6 +1441,8 @@ export class KnowledgeEngine {
|
|
|
1422
1441
|
const chunk = getChunkById(db, result.chunkId);
|
|
1423
1442
|
if (!chunk)
|
|
1424
1443
|
return false;
|
|
1444
|
+
if (retrievalMode === "semantic_hybrid")
|
|
1445
|
+
return result.score >= tuning.minHybridScore;
|
|
1425
1446
|
return result.score >= tuning.minHybridScore && hasEnoughLexicalEvidence(chunk, queryTokens);
|
|
1426
1447
|
});
|
|
1427
1448
|
}
|
|
@@ -1578,7 +1599,7 @@ export class KnowledgeEngine {
|
|
|
1578
1599
|
tuning: tuningSummary,
|
|
1579
1600
|
suggestions: results.length === 0
|
|
1580
1601
|
? [
|
|
1581
|
-
"Try mode 'fast' for exact symbols
|
|
1602
|
+
"Try mode 'fast' for exact symbols, mode 'semantic' for pure vector search, or mode 'semantic_hybrid' when weak lexical overlap should not block fused vector results.",
|
|
1582
1603
|
"Run knowledge_status if the KB should contain this answer.",
|
|
1583
1604
|
]
|
|
1584
1605
|
: undefined,
|
|
@@ -1749,6 +1770,30 @@ export class KnowledgeEngine {
|
|
|
1749
1770
|
if (this.db) {
|
|
1750
1771
|
for (const kb of kbs) {
|
|
1751
1772
|
throwIfAborted(signal);
|
|
1773
|
+
if ((kb.status === "ready" || kb.status === "stale") &&
|
|
1774
|
+
(storedSourcePolicyVersion(kb.source_options, "embedding_input_policy") ?? 0) < CURRENT_EMBEDDING_INPUT_POLICY) {
|
|
1775
|
+
const canUpdate = kb.source_path !== null && kb.source_type !== "text";
|
|
1776
|
+
issues.push({
|
|
1777
|
+
severity: "warning",
|
|
1778
|
+
kb_name: kb.name,
|
|
1779
|
+
message: "This KB predates embedding-safe chunk limits, so oversized chunk tails may be missing from vector retrieval.",
|
|
1780
|
+
action: canUpdate
|
|
1781
|
+
? "Run knowledge_update to re-chunk oversized inputs with the current embedding budget."
|
|
1782
|
+
: "Re-add inline text or re-import the original source to rebuild chunks with the current embedding budget.",
|
|
1783
|
+
action_code: "reindex_for_embedding_budget",
|
|
1784
|
+
});
|
|
1785
|
+
}
|
|
1786
|
+
if (kb.source_type === "url" &&
|
|
1787
|
+
(kb.status === "ready" || kb.status === "stale") &&
|
|
1788
|
+
(storedSourcePolicyVersion(kb.source_options, "url_structure_policy") ?? 0) < CURRENT_URL_STRUCTURE_POLICY) {
|
|
1789
|
+
issues.push({
|
|
1790
|
+
severity: "warning",
|
|
1791
|
+
kb_name: kb.name,
|
|
1792
|
+
message: "This URL KB predates structure-preserving HTML normalization.",
|
|
1793
|
+
action: "Run knowledge_update to rebuild URL headings, sections, lists, tables, and entities.",
|
|
1794
|
+
action_code: "run_update",
|
|
1795
|
+
});
|
|
1796
|
+
}
|
|
1752
1797
|
if (kb.status === "ready" && kb.chunk_count > 0 && getSymbolCount(this.db, kb.id) === 0) {
|
|
1753
1798
|
issues.push({
|
|
1754
1799
|
severity: "info",
|
|
@@ -1876,7 +1921,7 @@ export class KnowledgeEngine {
|
|
|
1876
1921
|
added_chunks: inserted,
|
|
1877
1922
|
});
|
|
1878
1923
|
onProgress?.(message);
|
|
1879
|
-
const vectors = await
|
|
1924
|
+
const vectors = await embedValidatedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
1880
1925
|
throwIfAborted(signal);
|
|
1881
1926
|
assertEmbeddingBatchSize(vectors, batch.length, "import");
|
|
1882
1927
|
persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
|
|
@@ -1912,6 +1957,7 @@ export class KnowledgeEngine {
|
|
|
1912
1957
|
name: header.name,
|
|
1913
1958
|
description: header.description,
|
|
1914
1959
|
source_type: "text",
|
|
1960
|
+
source_options: serializeAddOptions(),
|
|
1915
1961
|
embedding_model: embeddingConfigLabel(embeddingConfig),
|
|
1916
1962
|
});
|
|
1917
1963
|
updateKBStatus(this.db, kb.id, "indexing");
|
|
@@ -1931,9 +1977,10 @@ export class KnowledgeEngine {
|
|
|
1931
1977
|
const imported = JSON.parse(line);
|
|
1932
1978
|
const chunk = importedChunkToInsert(imported);
|
|
1933
1979
|
chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
|
|
1934
|
-
|
|
1980
|
+
const finalizedChunks = await finalizeChunksForEmbedding([chunk], embeddingConfig, signal);
|
|
1981
|
+
pendingChunks.push(...finalizedChunks);
|
|
1935
1982
|
pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
|
|
1936
|
-
|
|
1983
|
+
while (pendingChunks.length >= embeddingBatchSize)
|
|
1937
1984
|
await flushPending();
|
|
1938
1985
|
}
|
|
1939
1986
|
if (!header)
|
|
@@ -50,7 +50,7 @@ export declare function chunkIdentityHash(opts: {
|
|
|
50
50
|
metadataJson: string;
|
|
51
51
|
}): string;
|
|
52
52
|
export declare function buildChunkEmbeddingText(chunk: Pick<ChunkInsert, "content" | "file_path" | "file_type" | "metadata_json">): string;
|
|
53
|
-
export declare function chunkMarkdown(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
|
|
53
|
+
export declare function chunkMarkdown(content: string, filePath: string, fileType?: string): Omit<ChunkInsert, "kb_id">[];
|
|
54
54
|
export declare function chunkText(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
|
|
55
55
|
export declare function isCodeFileType(fileType: string): boolean;
|
|
56
56
|
export declare function analyzeIndexableContent(content: string, filePath: string, fileType?: string): Promise<{
|
|
@@ -397,7 +397,7 @@ function makeChunk(content, filePath, fileType, startLine, endLine, metadata = {
|
|
|
397
397
|
};
|
|
398
398
|
return { ...chunk, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(chunk)) };
|
|
399
399
|
}
|
|
400
|
-
export function chunkMarkdown(content, filePath) {
|
|
400
|
+
export function chunkMarkdown(content, filePath, fileType = "markdown") {
|
|
401
401
|
const lines = content.split("\n");
|
|
402
402
|
const chunks = [];
|
|
403
403
|
const headingStack = [];
|
|
@@ -410,7 +410,7 @@ export function chunkMarkdown(content, filePath) {
|
|
|
410
410
|
function pushMarkdownChunk(text, start, end) {
|
|
411
411
|
if (text.trim().length < 50)
|
|
412
412
|
return;
|
|
413
|
-
chunks.push(makeChunk(text.trim(), filePath,
|
|
413
|
+
chunks.push(makeChunk(text.trim(), filePath, fileType, start, end, {
|
|
414
414
|
heading: currentHeading,
|
|
415
415
|
breadcrumb: currentBreadcrumb(),
|
|
416
416
|
}));
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
import type { EmbeddingConfig } from "../embedding/provider.ts";
|
|
2
|
+
import type { ChunkInsert } from "../storage/sqlite.ts";
|
|
3
|
+
type IndexableChunk = Omit<ChunkInsert, "kb_id">;
|
|
4
|
+
export declare function finalizeChunksForEmbedding(chunks: IndexableChunk[], config: EmbeddingConfig, signal?: AbortSignal): Promise<IndexableChunk[]>;
|
|
5
|
+
export {};
|
|
@@ -0,0 +1,215 @@
|
|
|
1
|
+
import { LOCAL_EMBEDDING_PREFERRED_TOKENS, measureEmbeddingInputs } from "../embedding/provider.js";
|
|
2
|
+
import { buildChunkEmbeddingText, chunkIdentityHash, preTokenizeForFTS } from "./chunker.js";
|
|
3
|
+
const SENTENCE_BOUNDARIES = {
|
|
4
|
+
"。": true,
|
|
5
|
+
"!": true,
|
|
6
|
+
"?": true,
|
|
7
|
+
".": true,
|
|
8
|
+
"!": true,
|
|
9
|
+
"?": true,
|
|
10
|
+
";": true,
|
|
11
|
+
";": true,
|
|
12
|
+
};
|
|
13
|
+
function throwIfAborted(signal) {
|
|
14
|
+
if (signal?.aborted)
|
|
15
|
+
throw new Error("Cancelled");
|
|
16
|
+
}
|
|
17
|
+
function compareOrder(left, right) {
|
|
18
|
+
if (left.rootIndex !== right.rootIndex)
|
|
19
|
+
return left.rootIndex - right.rootIndex;
|
|
20
|
+
const length = Math.min(left.order.length, right.order.length);
|
|
21
|
+
for (let index = 0; index < length; index++) {
|
|
22
|
+
if (left.order[index] !== right.order[index])
|
|
23
|
+
return left.order[index] - right.order[index];
|
|
24
|
+
}
|
|
25
|
+
return left.order.length - right.order.length;
|
|
26
|
+
}
|
|
27
|
+
function nearestStringBoundary(content, desired, boundary, minimum) {
|
|
28
|
+
const before = content.lastIndexOf(boundary, desired);
|
|
29
|
+
if (before >= minimum)
|
|
30
|
+
return before + boundary.length;
|
|
31
|
+
const after = content.indexOf(boundary, desired);
|
|
32
|
+
if (after >= 0 && after <= content.length - minimum)
|
|
33
|
+
return after + boundary.length;
|
|
34
|
+
return undefined;
|
|
35
|
+
}
|
|
36
|
+
function nearestCharacterBoundary(content, desired, minimum, predicate) {
|
|
37
|
+
for (let offset = 0; offset < content.length; offset++) {
|
|
38
|
+
const before = desired - offset;
|
|
39
|
+
if (before >= minimum && predicate(content[before]))
|
|
40
|
+
return before + 1;
|
|
41
|
+
const after = desired + offset;
|
|
42
|
+
if (after < content.length - minimum && predicate(content[after]))
|
|
43
|
+
return after + 1;
|
|
44
|
+
if (before < minimum && after >= content.length - minimum)
|
|
45
|
+
break;
|
|
46
|
+
}
|
|
47
|
+
return undefined;
|
|
48
|
+
}
|
|
49
|
+
function safeCodeUnitBoundary(content, index) {
|
|
50
|
+
if (index <= 0 || index >= content.length)
|
|
51
|
+
return index;
|
|
52
|
+
const previous = content.charCodeAt(index - 1);
|
|
53
|
+
const next = content.charCodeAt(index);
|
|
54
|
+
return previous >= 0xd800 && previous <= 0xdbff && next >= 0xdc00 && next <= 0xdfff ? index - 1 : index;
|
|
55
|
+
}
|
|
56
|
+
function chooseSplitIndex(content, units, measurement) {
|
|
57
|
+
const estimatedParts = Math.max(2, Math.ceil(units / measurement.preferredMaximum));
|
|
58
|
+
const desired = Math.max(1, Math.floor(content.length / estimatedParts));
|
|
59
|
+
const minimum = Math.max(1, Math.min(64, Math.floor(content.length * 0.1)));
|
|
60
|
+
const structuralBoundary = nearestStringBoundary(content, desired, "\n\n", minimum) ??
|
|
61
|
+
nearestStringBoundary(content, desired, "\n", minimum) ??
|
|
62
|
+
nearestCharacterBoundary(content, desired, minimum, (character) => SENTENCE_BOUNDARIES[character] === true) ??
|
|
63
|
+
nearestCharacterBoundary(content, desired, minimum, (character) => /\s/u.test(character));
|
|
64
|
+
const splitIndex = safeCodeUnitBoundary(content, structuralBoundary ?? desired);
|
|
65
|
+
if (splitIndex <= 0 || splitIndex >= content.length) {
|
|
66
|
+
throw new Error(`Unable to split oversized embedding input containing ${content.length} characters`);
|
|
67
|
+
}
|
|
68
|
+
return splitIndex;
|
|
69
|
+
}
|
|
70
|
+
function lineCount(content) {
|
|
71
|
+
let count = 0;
|
|
72
|
+
for (const character of content) {
|
|
73
|
+
if (character === "\n")
|
|
74
|
+
count++;
|
|
75
|
+
}
|
|
76
|
+
return count;
|
|
77
|
+
}
|
|
78
|
+
function rebuildChunk(chunk, content, startLine, endLine) {
|
|
79
|
+
const rebuilt = {
|
|
80
|
+
...chunk,
|
|
81
|
+
content,
|
|
82
|
+
content_hash: chunkIdentityHash({
|
|
83
|
+
content,
|
|
84
|
+
filePath: chunk.file_path,
|
|
85
|
+
fileType: chunk.file_type,
|
|
86
|
+
startLine,
|
|
87
|
+
endLine,
|
|
88
|
+
metadataJson: chunk.metadata_json,
|
|
89
|
+
}),
|
|
90
|
+
content_tokenized: "",
|
|
91
|
+
start_line: startLine,
|
|
92
|
+
end_line: endLine,
|
|
93
|
+
};
|
|
94
|
+
return { ...rebuilt, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(rebuilt)) };
|
|
95
|
+
}
|
|
96
|
+
function unsplittableEmbeddingContext(piece, measurement) {
|
|
97
|
+
return new Error(`Embedding context for ${piece.chunk.file_path}:${piece.chunk.start_line}-${piece.chunk.end_line} exceeds ${measurement.maximum} ${measurement.unit} and cannot be split safely`);
|
|
98
|
+
}
|
|
99
|
+
function splitPiece(piece, units, measurement) {
|
|
100
|
+
const content = piece.chunk.content;
|
|
101
|
+
if ([...content].length < 2)
|
|
102
|
+
throw unsplittableEmbeddingContext(piece, measurement);
|
|
103
|
+
const splitIndex = chooseSplitIndex(content, units, measurement);
|
|
104
|
+
const leftRaw = content.slice(0, splitIndex);
|
|
105
|
+
const rightRaw = content.slice(splitIndex);
|
|
106
|
+
const left = leftRaw.trim();
|
|
107
|
+
const right = rightRaw.trim();
|
|
108
|
+
if (!left || !right || left === content || right === content) {
|
|
109
|
+
throw unsplittableEmbeddingContext(piece, measurement);
|
|
110
|
+
}
|
|
111
|
+
const leftLeadingLines = lineCount(leftRaw.slice(0, leftRaw.indexOf(left)));
|
|
112
|
+
const rightLeadingLines = lineCount(rightRaw.slice(0, rightRaw.indexOf(right)));
|
|
113
|
+
const leftStart = piece.chunk.start_line + leftLeadingLines;
|
|
114
|
+
const leftEnd = leftStart + lineCount(left);
|
|
115
|
+
const rightStart = piece.chunk.start_line + lineCount(leftRaw) + rightLeadingLines;
|
|
116
|
+
const rightEnd = rightStart + lineCount(right);
|
|
117
|
+
return [
|
|
118
|
+
{
|
|
119
|
+
chunk: rebuildChunk(piece.chunk, left, leftStart, leftEnd),
|
|
120
|
+
rootIndex: piece.rootIndex,
|
|
121
|
+
order: [...piece.order, 0],
|
|
122
|
+
},
|
|
123
|
+
{
|
|
124
|
+
chunk: rebuildChunk(piece.chunk, right, rightStart, rightEnd),
|
|
125
|
+
rootIndex: piece.rootIndex,
|
|
126
|
+
order: [...piece.order, 1],
|
|
127
|
+
},
|
|
128
|
+
];
|
|
129
|
+
}
|
|
130
|
+
function withPartMetadata(chunk, part, partCount) {
|
|
131
|
+
let metadata = {};
|
|
132
|
+
try {
|
|
133
|
+
const parsed = JSON.parse(chunk.metadata_json);
|
|
134
|
+
if (parsed && typeof parsed === "object" && !Array.isArray(parsed))
|
|
135
|
+
metadata = parsed;
|
|
136
|
+
}
|
|
137
|
+
catch {
|
|
138
|
+
metadata = {};
|
|
139
|
+
}
|
|
140
|
+
const metadataJson = JSON.stringify({ ...metadata, chunk_part: part, chunk_part_count: partCount });
|
|
141
|
+
const rebuilt = { ...chunk, metadata_json: metadataJson, content_tokenized: "" };
|
|
142
|
+
rebuilt.content_hash = chunkIdentityHash({
|
|
143
|
+
content: rebuilt.content,
|
|
144
|
+
filePath: rebuilt.file_path,
|
|
145
|
+
fileType: rebuilt.file_type,
|
|
146
|
+
startLine: rebuilt.start_line,
|
|
147
|
+
endLine: rebuilt.end_line,
|
|
148
|
+
metadataJson,
|
|
149
|
+
});
|
|
150
|
+
rebuilt.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(rebuilt));
|
|
151
|
+
return rebuilt;
|
|
152
|
+
}
|
|
153
|
+
function isProvablySafeLocalAsciiInput(text, config) {
|
|
154
|
+
if (config.provider !== "local")
|
|
155
|
+
return false;
|
|
156
|
+
const prefixLength = config.prefixes === "on" ? "passage: ".length : 0;
|
|
157
|
+
if (text.length + prefixLength + 2 > LOCAL_EMBEDDING_PREFERRED_TOKENS)
|
|
158
|
+
return false;
|
|
159
|
+
for (let index = 0; index < text.length; index++) {
|
|
160
|
+
if (text.charCodeAt(index) > 0x7f)
|
|
161
|
+
return false;
|
|
162
|
+
}
|
|
163
|
+
return true;
|
|
164
|
+
}
|
|
165
|
+
export async function finalizeChunksForEmbedding(chunks, config, signal) {
|
|
166
|
+
if (chunks.length === 0)
|
|
167
|
+
return [];
|
|
168
|
+
let pending = chunks.map((chunk, rootIndex) => ({ chunk, rootIndex, order: [] }));
|
|
169
|
+
const accepted = [];
|
|
170
|
+
while (pending.length > 0) {
|
|
171
|
+
throwIfAborted(signal);
|
|
172
|
+
const piecesToMeasure = [];
|
|
173
|
+
const textsToMeasure = [];
|
|
174
|
+
for (const piece of pending) {
|
|
175
|
+
const text = buildChunkEmbeddingText(piece.chunk);
|
|
176
|
+
if (isProvablySafeLocalAsciiInput(text, config))
|
|
177
|
+
accepted.push(piece);
|
|
178
|
+
else {
|
|
179
|
+
piecesToMeasure.push(piece);
|
|
180
|
+
textsToMeasure.push(text);
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
if (piecesToMeasure.length === 0)
|
|
184
|
+
break;
|
|
185
|
+
const measurement = await measureEmbeddingInputs(textsToMeasure, "passage", config, signal);
|
|
186
|
+
if (measurement.counts.length !== piecesToMeasure.length) {
|
|
187
|
+
throw new Error(`Embedding input measurement returned ${measurement.counts.length} counts for ${piecesToMeasure.length} chunks`);
|
|
188
|
+
}
|
|
189
|
+
const next = [];
|
|
190
|
+
for (let index = 0; index < piecesToMeasure.length; index++) {
|
|
191
|
+
const piece = piecesToMeasure[index];
|
|
192
|
+
const units = measurement.counts[index];
|
|
193
|
+
if (units === undefined)
|
|
194
|
+
throw new Error(`Missing embedding input measurement for chunk ${index}`);
|
|
195
|
+
if (units <= measurement.maximum)
|
|
196
|
+
accepted.push(piece);
|
|
197
|
+
else
|
|
198
|
+
next.push(...splitPiece(piece, units, measurement));
|
|
199
|
+
}
|
|
200
|
+
pending = next;
|
|
201
|
+
}
|
|
202
|
+
accepted.sort(compareOrder);
|
|
203
|
+
const partCounts = new Map();
|
|
204
|
+
for (const piece of accepted)
|
|
205
|
+
partCounts.set(piece.rootIndex, (partCounts.get(piece.rootIndex) ?? 0) + 1);
|
|
206
|
+
const partIndexes = new Map();
|
|
207
|
+
return accepted.map((piece) => {
|
|
208
|
+
const partCount = partCounts.get(piece.rootIndex) ?? 1;
|
|
209
|
+
if (partCount === 1)
|
|
210
|
+
return piece.chunk;
|
|
211
|
+
const part = (partIndexes.get(piece.rootIndex) ?? 0) + 1;
|
|
212
|
+
partIndexes.set(piece.rootIndex, part);
|
|
213
|
+
return withPartMetadata(piece.chunk, part, partCount);
|
|
214
|
+
});
|
|
215
|
+
}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export declare function htmlToStructuredMarkdown(html: string): string;
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
const HTML_ENTITIES = {
|
|
2
|
+
amp: "&",
|
|
3
|
+
apos: "'",
|
|
4
|
+
gt: ">",
|
|
5
|
+
lt: "<",
|
|
6
|
+
nbsp: " ",
|
|
7
|
+
quot: '"',
|
|
8
|
+
ndash: "–",
|
|
9
|
+
mdash: "—",
|
|
10
|
+
hellip: "…",
|
|
11
|
+
};
|
|
12
|
+
function decodeHtmlEntity(entity) {
|
|
13
|
+
if (!entity.startsWith("#"))
|
|
14
|
+
return HTML_ENTITIES[entity.toLowerCase()] ?? `&${entity};`;
|
|
15
|
+
const hexadecimal = entity[1]?.toLowerCase() === "x";
|
|
16
|
+
const digits = entity.slice(hexadecimal ? 2 : 1);
|
|
17
|
+
const codePoint = Number.parseInt(digits, hexadecimal ? 16 : 10);
|
|
18
|
+
if (!digits ||
|
|
19
|
+
!Number.isInteger(codePoint) ||
|
|
20
|
+
codePoint <= 0 ||
|
|
21
|
+
codePoint > 0x10ffff ||
|
|
22
|
+
(codePoint >= 0xd800 && codePoint <= 0xdfff)) {
|
|
23
|
+
return `&${entity};`;
|
|
24
|
+
}
|
|
25
|
+
return String.fromCodePoint(codePoint);
|
|
26
|
+
}
|
|
27
|
+
export function htmlToStructuredMarkdown(html) {
|
|
28
|
+
return html
|
|
29
|
+
.replace(/<!--[\s\S]*?-->/g, "")
|
|
30
|
+
.replace(/<(script|style|noscript|template|svg)\b[^>]*>[\s\S]*?(?:<\/\1\s*>|$)/gi, "")
|
|
31
|
+
.replace(/<h([1-6])\b[^>]*>/gi, (_tag, level) => `\n\n${"#".repeat(Number(level))} `)
|
|
32
|
+
.replace(/<\/h[1-6]\s*>/gi, "\n\n")
|
|
33
|
+
.replace(/<li\b[^>]*>/gi, "\n- ")
|
|
34
|
+
.replace(/<\/li\s*>/gi, "\n")
|
|
35
|
+
.replace(/<br\s*\/?\s*>/gi, "\n")
|
|
36
|
+
.replace(/<(th|td)\b[^>]*>/gi, " | ")
|
|
37
|
+
.replace(/<\/(th|td)\s*>/gi, "")
|
|
38
|
+
.replace(/<tr\b[^>]*>/gi, "\n")
|
|
39
|
+
.replace(/<\/tr\s*>/gi, "\n")
|
|
40
|
+
.replace(/<\/?(?:address|article|aside|blockquote|div|dl|dt|dd|fieldset|figcaption|figure|footer|form|header|hr|main|nav|ol|p|section|table|tbody|thead|tfoot|ul)\b[^>]*>/gi, "\n\n")
|
|
41
|
+
.replace(/<[^>]+>/g, "")
|
|
42
|
+
.replace(/&(#(?:x[0-9a-f]+|\d+)|[a-z][a-z0-9]+);/gi, (_match, entity) => decodeHtmlEntity(entity))
|
|
43
|
+
.replace(/\r\n?/g, "\n")
|
|
44
|
+
.split("\n")
|
|
45
|
+
.map((line) => line.replace(/[\t\f\v ]+/g, " ").trim())
|
|
46
|
+
.join("\n")
|
|
47
|
+
.replace(/\n{3,}/g, "\n\n")
|
|
48
|
+
.trim();
|
|
49
|
+
}
|
|
@@ -2,7 +2,11 @@ import type { HfRerankerConfig } from "./search/reranker-config.ts";
|
|
|
2
2
|
export declare function validateModelWorkerNodePath(nodePath: string): string;
|
|
3
3
|
export declare function resolveModelWorkerNodePath(): string;
|
|
4
4
|
export declare function configureModelWorkerNodePath(nodePath?: string): string;
|
|
5
|
-
export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", signal?: AbortSignal): Promise<Float32Array[]>;
|
|
5
|
+
export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", prefixesOrSignal?: "on" | "off" | AbortSignal, signal?: AbortSignal): Promise<Float32Array[]>;
|
|
6
|
+
export declare function measureEmbeddingInputInModelWorker(texts: string[], prefix: "query" | "passage", prefixes: "on" | "off", signal?: AbortSignal): Promise<{
|
|
7
|
+
counts: number[];
|
|
8
|
+
maximum: number;
|
|
9
|
+
}>;
|
|
6
10
|
export interface RerankWorkerCandidate {
|
|
7
11
|
chunkId: string;
|
|
8
12
|
content: string;
|
|
@@ -187,8 +187,8 @@ function isWorkerResponse(message) {
|
|
|
187
187
|
return !("error" in message) || message.error === undefined || typeof message.error === "string";
|
|
188
188
|
}
|
|
189
189
|
class ModelWorkerTransport {
|
|
190
|
-
child;
|
|
191
190
|
pending = new Map();
|
|
191
|
+
child;
|
|
192
192
|
stderrTail = "";
|
|
193
193
|
exited = false;
|
|
194
194
|
terminalError = null;
|
|
@@ -385,8 +385,10 @@ function getWorker() {
|
|
|
385
385
|
IpcModelWorkerTransport.create(workerPath, nodeExecPath) ?? new StdioModelWorkerTransport(workerPath, nodeExecPath);
|
|
386
386
|
return worker;
|
|
387
387
|
}
|
|
388
|
-
export async function embedInModelWorker(texts, prefix, signal) {
|
|
389
|
-
const
|
|
388
|
+
export async function embedInModelWorker(texts, prefix, prefixesOrSignal, signal) {
|
|
389
|
+
const prefixes = prefixesOrSignal === "off" ? "off" : "on";
|
|
390
|
+
const requestSignal = typeof prefixesOrSignal === "object" ? prefixesOrSignal : signal;
|
|
391
|
+
const result = await getWorker().request({ type: "embed", texts, prefix, prefixes }, requestSignal);
|
|
390
392
|
if (!Array.isArray(result))
|
|
391
393
|
throw new Error("Invalid embedding worker response");
|
|
392
394
|
return result.map((vector) => {
|
|
@@ -395,6 +397,22 @@ export async function embedInModelWorker(texts, prefix, signal) {
|
|
|
395
397
|
return new Float32Array(vector);
|
|
396
398
|
});
|
|
397
399
|
}
|
|
400
|
+
export async function measureEmbeddingInputInModelWorker(texts, prefix, prefixes, signal) {
|
|
401
|
+
const result = await getWorker().request({ type: "measure_embedding_input", texts, prefix, prefixes }, signal);
|
|
402
|
+
if (typeof result !== "object" ||
|
|
403
|
+
result === null ||
|
|
404
|
+
!("counts" in result) ||
|
|
405
|
+
!Array.isArray(result.counts) ||
|
|
406
|
+
result.counts.length !== texts.length ||
|
|
407
|
+
!result.counts.every((count) => typeof count === "number" && Number.isInteger(count) && count >= 0) ||
|
|
408
|
+
!("maximum" in result) ||
|
|
409
|
+
typeof result.maximum !== "number" ||
|
|
410
|
+
!Number.isInteger(result.maximum) ||
|
|
411
|
+
result.maximum <= 0) {
|
|
412
|
+
throw new Error("Invalid embedding input measurement from worker");
|
|
413
|
+
}
|
|
414
|
+
return { counts: result.counts, maximum: result.maximum };
|
|
415
|
+
}
|
|
398
416
|
export async function rerankInModelWorker(query, candidates, topK, reranker, signal) {
|
|
399
417
|
const result = await getWorker().request({ type: "rerank", query, candidates, topK, reranker }, signal);
|
|
400
418
|
if (!Array.isArray(result))
|
package/dist/src/model-worker.js
CHANGED
|
@@ -4,6 +4,7 @@ import { DEFAULT_RERANKER_REMOTE_HOST, DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE, is
|
|
|
4
4
|
import { rerankWithSingleRawLogit } from "./search/reranker-logits.js";
|
|
5
5
|
import { getDefaultKnowledgeDir } from "./storage/sqlite.js";
|
|
6
6
|
let embeddingPipeline = null;
|
|
7
|
+
let embeddingPipelinePromise = null;
|
|
7
8
|
let rerankerPipeline = null;
|
|
8
9
|
function getModelCacheDir() {
|
|
9
10
|
return process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? join(getDefaultKnowledgeDir(), "models");
|
|
@@ -24,15 +25,24 @@ function configureEmbeddingTransformersEnv(env) {
|
|
|
24
25
|
async function loadEmbeddingPipeline() {
|
|
25
26
|
if (embeddingPipeline)
|
|
26
27
|
return embeddingPipeline;
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
28
|
+
if (embeddingPipelinePromise)
|
|
29
|
+
return embeddingPipelinePromise;
|
|
30
|
+
embeddingPipelinePromise = (async () => {
|
|
31
|
+
const { pipeline, env } = await import("@huggingface/transformers");
|
|
32
|
+
configureEmbeddingTransformersEnv(env);
|
|
33
|
+
const createPipeline = pipeline;
|
|
34
|
+
return (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
|
|
35
|
+
quantized: true,
|
|
36
|
+
dtype: "fp32",
|
|
37
|
+
}));
|
|
38
|
+
})();
|
|
39
|
+
try {
|
|
40
|
+
embeddingPipeline = await embeddingPipelinePromise;
|
|
41
|
+
return embeddingPipeline;
|
|
42
|
+
}
|
|
43
|
+
finally {
|
|
44
|
+
embeddingPipelinePromise = null;
|
|
45
|
+
}
|
|
36
46
|
}
|
|
37
47
|
async function loadRerankerPipeline(config) {
|
|
38
48
|
const key = rerankerCacheKey(config);
|
|
@@ -66,15 +76,49 @@ async function loadRerankerPipeline(config) {
|
|
|
66
76
|
rerankerPipeline = { key, pipe: loaded };
|
|
67
77
|
return loaded;
|
|
68
78
|
}
|
|
79
|
+
function applyEmbeddingPrefix(text, prefix, prefixes) {
|
|
80
|
+
return prefixes === "on" ? `${prefix}: ${text}` : text;
|
|
81
|
+
}
|
|
69
82
|
async function handleEmbed(request) {
|
|
70
83
|
const pipe = await loadEmbeddingPipeline();
|
|
71
84
|
const vectors = [];
|
|
72
85
|
for (const text of request.texts) {
|
|
73
|
-
const output = await pipe(
|
|
86
|
+
const output = await pipe(applyEmbeddingPrefix(text, request.prefix, request.prefixes), {
|
|
87
|
+
pooling: "mean",
|
|
88
|
+
normalize: true,
|
|
89
|
+
});
|
|
74
90
|
vectors.push(Array.from(output.data));
|
|
75
91
|
}
|
|
76
92
|
return vectors;
|
|
77
93
|
}
|
|
94
|
+
function embeddingMaximumTokens(tokenizer) {
|
|
95
|
+
const configured = tokenizer.model_max_length;
|
|
96
|
+
return typeof configured === "number" && Number.isInteger(configured) && configured > 0 && configured <= 1_000_000
|
|
97
|
+
? configured
|
|
98
|
+
: 512;
|
|
99
|
+
}
|
|
100
|
+
async function handleMeasureEmbeddingInput(request) {
|
|
101
|
+
const pipe = await loadEmbeddingPipeline();
|
|
102
|
+
const texts = request.texts.map((text) => applyEmbeddingPrefix(text, request.prefix, request.prefixes));
|
|
103
|
+
const tokenized = pipe.tokenizer(texts, {
|
|
104
|
+
add_special_tokens: true,
|
|
105
|
+
truncation: false,
|
|
106
|
+
padding: false,
|
|
107
|
+
return_tensor: false,
|
|
108
|
+
});
|
|
109
|
+
const inputIds = tokenized.input_ids;
|
|
110
|
+
let counts;
|
|
111
|
+
if (texts.length === 1 && inputIds.every((token) => typeof token === "number")) {
|
|
112
|
+
counts = [inputIds.length];
|
|
113
|
+
}
|
|
114
|
+
else if (inputIds.every((tokens) => Array.isArray(tokens))) {
|
|
115
|
+
counts = inputIds.map((tokens) => tokens.length);
|
|
116
|
+
}
|
|
117
|
+
else {
|
|
118
|
+
counts = texts.map((text) => pipe.tokenizer.encode(text, { add_special_tokens: true }).length);
|
|
119
|
+
}
|
|
120
|
+
return { counts, maximum: embeddingMaximumTokens(pipe.tokenizer) };
|
|
121
|
+
}
|
|
78
122
|
async function handleRerank(request) {
|
|
79
123
|
const pipe = await loadRerankerPipeline(request.reranker);
|
|
80
124
|
const results = [];
|
|
@@ -104,13 +148,23 @@ function parseWorkerRequest(message) {
|
|
|
104
148
|
}
|
|
105
149
|
if (!("type" in message))
|
|
106
150
|
throw new Error("Invalid worker request type");
|
|
107
|
-
if (message.type === "embed") {
|
|
151
|
+
if (message.type === "embed" || message.type === "measure_embedding_input") {
|
|
108
152
|
if (!("texts" in message) || !isStringArray(message.texts))
|
|
109
|
-
throw new Error("Invalid
|
|
153
|
+
throw new Error("Invalid embedding texts");
|
|
110
154
|
if (!("prefix" in message) || (message.prefix !== "query" && message.prefix !== "passage")) {
|
|
111
|
-
throw new Error("Invalid
|
|
155
|
+
throw new Error("Invalid embedding prefix");
|
|
112
156
|
}
|
|
113
|
-
|
|
157
|
+
const prefixes = "prefixes" in message && (message.prefixes === "on" || message.prefixes === "off") ? message.prefixes : "on";
|
|
158
|
+
if ("prefixes" in message && message.prefixes !== "on" && message.prefixes !== "off") {
|
|
159
|
+
throw new Error("Invalid embedding prefix strategy");
|
|
160
|
+
}
|
|
161
|
+
return {
|
|
162
|
+
id: message.id,
|
|
163
|
+
type: message.type,
|
|
164
|
+
texts: message.texts,
|
|
165
|
+
prefix: message.prefix,
|
|
166
|
+
prefixes,
|
|
167
|
+
};
|
|
114
168
|
}
|
|
115
169
|
if (message.type === "rerank") {
|
|
116
170
|
if (!("query" in message) || typeof message.query !== "string")
|
|
@@ -141,7 +195,11 @@ async function handleRequestMessage(message) {
|
|
|
141
195
|
requestId = message.id;
|
|
142
196
|
}
|
|
143
197
|
const request = parseWorkerRequest(message);
|
|
144
|
-
const result = request.type === "embed"
|
|
198
|
+
const result = request.type === "embed"
|
|
199
|
+
? await handleEmbed(request)
|
|
200
|
+
: request.type === "measure_embedding_input"
|
|
201
|
+
? await handleMeasureEmbeddingInput(request)
|
|
202
|
+
: await handleRerank(request);
|
|
145
203
|
return { id: request.id, result };
|
|
146
204
|
}
|
|
147
205
|
catch (error) {
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
export type SearchProfile = "auto" | "balanced" | "low_token" | "precision" | "recall" | "long_context" | "code" | "docs";
|
|
2
2
|
export type RuntimeProfile = Exclude<SearchProfile, "auto">;
|
|
3
|
-
type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
3
|
+
type SearchModeForTuning = "auto" | "fast" | "semantic" | "hybrid" | "semantic_hybrid" | "deep" | "adaptive" | "code" | "config" | "docs" | "errors" | "decision";
|
|
4
4
|
export interface SearchTuning {
|
|
5
5
|
requestedProfile: SearchProfile;
|
|
6
6
|
selectedProfile: RuntimeProfile;
|
|
@@ -143,7 +143,7 @@ function chooseRuntimeProfile(input) {
|
|
|
143
143
|
}
|
|
144
144
|
if (input.mode === "docs" || input.mode === "decision" || input.mode === "adaptive")
|
|
145
145
|
return "long_context";
|
|
146
|
-
if (input.mode === "semantic")
|
|
146
|
+
if (input.mode === "semantic" || input.mode === "semantic_hybrid")
|
|
147
147
|
return "recall";
|
|
148
148
|
if (looksErrorQuery(input.query) || looksExactOrCodeQuery(input.query))
|
|
149
149
|
return "precision";
|
|
@@ -87,6 +87,7 @@ export declare function listKBs(db: Database.Database): KnowledgeBase[];
|
|
|
87
87
|
export declare function deleteKB(db: Database.Database, id: string): void;
|
|
88
88
|
export declare function updateKBStatus(db: Database.Database, id: string, status: KnowledgeBase["status"]): void;
|
|
89
89
|
export declare function updateKBCounts(db: Database.Database, id: string, chunkCount: number, fileCount: number): void;
|
|
90
|
+
export declare function updateKBSourceOptions(db: Database.Database, id: string, sourceOptions: string): void;
|
|
90
91
|
export declare function updateKBEmbeddingMetadata(db: Database.Database, id: string, embeddingModel: string, embeddingSignature: string | null, embeddingDimension: number | null): void;
|
|
91
92
|
export declare function startIndexingJob(db: Database.Database, kbId: string, operation: IndexingJob["operation"], message: string): void;
|
|
92
93
|
export declare function updateIndexingJob(db: Database.Database, kbId: string, progress: {
|
|
@@ -355,6 +355,9 @@ export function updateKBStatus(db, id, status) {
|
|
|
355
355
|
export function updateKBCounts(db, id, chunkCount, fileCount) {
|
|
356
356
|
db.prepare("UPDATE knowledge_bases SET chunk_count = ?, file_count = ?, updated_at = ? WHERE id = ?").run(chunkCount, fileCount, Date.now(), id);
|
|
357
357
|
}
|
|
358
|
+
export function updateKBSourceOptions(db, id, sourceOptions) {
|
|
359
|
+
db.prepare("UPDATE knowledge_bases SET source_options = ?, updated_at = ? WHERE id = ?").run(sourceOptions, Date.now(), id);
|
|
360
|
+
}
|
|
358
361
|
export function updateKBEmbeddingMetadata(db, id, embeddingModel, embeddingSignature, embeddingDimension) {
|
|
359
362
|
db.prepare(`UPDATE knowledge_bases
|
|
360
363
|
SET embedding_model = ?, embedding_signature = ?, embedding_dimension = ?, updated_at = ?
|
package/docs/configuration.md
CHANGED
|
@@ -25,6 +25,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
25
25
|
| `OPENAI_API_KEY` | unset | Fallback API key used when `PI_KNOWLEDGE_EMBEDDING=openai:<model>`. Required by OpenAI's hosted API; many local OpenAI-compatible servers do not need it. |
|
|
26
26
|
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
27
27
|
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
28
|
+
| `PI_KNOWLEDGE_EMBEDDING_PREFIXES` | `on` | Controls whether `query: ` and `passage: ` prefixes are prepended before embedding. Values: `on`, `off`. Keep `on` for the default E5 local model; use `off` only for models or self-hosted APIs that are evaluated without E5-style prefixes. |
|
|
28
29
|
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
29
30
|
| `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` | `64` | Maximum number of document chunks sent in one embedding request during add, update, and import. Lower this for self-hosted embedding servers with smaller request batch limits. |
|
|
30
31
|
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
|
|
@@ -32,7 +33,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
32
33
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
33
34
|
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
34
35
|
|
|
35
|
-
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character settings or after upgrading older KBs with no signature metadata.
|
|
36
|
+
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, non-default prefix strategy, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character/prefix settings or after upgrading older KBs with no signature metadata.
|
|
36
37
|
|
|
37
38
|
## Reranker
|
|
38
39
|
|
|
@@ -84,7 +85,7 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
84
85
|
|
|
85
86
|
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool size, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
86
87
|
|
|
87
|
-
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic`
|
|
88
|
+
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and explicit `limit` filters. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` for pure vector search, or `semantic_hybrid` when weak lexical overlap should not block vector-backed weighted fusion. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks, but it remains metadata-driven context expansion rather than caller/callee analysis.
|
|
88
89
|
|
|
89
90
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
90
91
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.12.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|