pi-knowledge 0.11.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,16 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.12.0] - 2026-10-19
4
+
5
+ ### Added
6
+ - Added embedding-safe chunk finalization for add, update, URL, inline-text, and JSONL import paths. Local inputs use the existing model worker tokenizer; OpenAI-compatible inputs use the configured complete-input character cap.
7
+ - Added `knowledge_doctor` action `reindex_for_embedding_budget` for KBs created before the current chunk input policy.
8
+ - Added dependency-free structured HTML normalization for URL sources, preserving headings, paragraphs, lists, tables, and entities for heading-aware chunking.
9
+
10
+ ### Fixed
11
+ - Prevented local and OpenAI-compatible embedding inputs from silently discarding oversized multilingual or context-enriched chunk tails.
12
+ - Kept the model worker startup single-flight and Node strip-only compatible while reusing it for batched tokenizer measurement.
13
+
3
14
  ## [0.11.0] - 2026-09-23
4
15
 
5
16
  ### Added
package/README.md CHANGED
@@ -23,7 +23,8 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
23
23
 
24
24
  - **Local-first project memory**: stores indexes under `~/.pi/knowledge/`; no project files are modified.
25
25
  - **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
26
- - **Code-aware indexing**: recursive AST chunking for TypeScript/JavaScript, Python, Go, Rust, Java, Bash, GNU C, C++, and QML, with bounded fallback chunks for oversized declarations.
26
+ - **Code- and document-aware indexing**: recursive AST chunking for supported code, heading-aware Markdown chunks, and an embedding-safe finalizer that preserves oversized multilingual tails without adding a heavy document runtime.
27
+ - **Structured URL indexing**: preserves HTML headings, paragraphs, lists, tables, and entities as lightweight Markdown-like text without a browser, Docling runtime, or new dependency.
27
28
  - **Lightweight symbol lookup**: indexes functions, methods, classes, interfaces, types, variables, route-like handlers, Markdown headings, config keys, and env vars for exact agent lookup.
28
29
  - **Better agent answers**: adaptive context windows, diversity reranking, optional cross-encoder reranking, and diagnostics.
29
30
  - **Large-project stability**: persisted indexing progress, capped batches, streaming vector scans, and stuck-job detection.
@@ -144,8 +145,8 @@ Mode selection contract:
144
145
 
145
146
  Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. In `adaptive` mode, code chunks with AST metadata can prefer same-parent/sibling chunks inside the same bounded file window; this is metadata-driven context expansion, not caller/callee analysis. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, source freshness, and adaptive `source_chunk_ids` where available.
146
147
 
147
- For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. This improves queries that mention project structure, filenames, sections, classes, methods, or functions, and reduces duplicate-looking chunk hits.
148
- Symbol/config/heading metadata is also rebuilt during `knowledge_update`; code symbol metadata includes AST-backed methods for supported languages. Older KBs created before this feature may show a doctor action recommending update.
148
+ For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. Before embedding, the complete contextual input is checked against the active provider budget; only oversized chunks are split again along structural boundaries.
149
+ Symbol/config/heading metadata and embedding-safe chunk boundaries are rebuilt during `knowledge_update`. `knowledge_doctor` reports `reindex_for_embedding_budget` for older KBs: file, directory, and URL sources can use `knowledge_update`; imported or inline-text sources should be re-imported or re-added from the original source.
149
150
 
150
151
  ## Embedding Configuration
151
152
 
@@ -170,9 +171,9 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
170
171
  # Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
171
172
  ```
172
173
 
173
- API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
174
+ API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Inputs are never silently truncated.
174
175
 
175
- API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
176
+ OpenAI-compatible embedding inputs use a 20000-character complete-input safety cap by default. Indexing splits oversized chunks before HTTP requests; direct oversized query/document calls fail explicitly. Adjust the cap with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when the remote model has a different context window. Changing the cap changes embedding compatibility and requires `knowledge_update`.
176
177
 
177
178
  For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
178
179
 
@@ -231,8 +232,8 @@ All data is stored globally at `~/.pi/knowledge/` under Pi or `~/.omp/knowledge/
231
232
  - **Reset**: delete the active knowledge directory to start fresh
232
233
  - **Override**: set `PI_KNOWLEDGE_DIR` or `OMP_KNOWLEDGE_DIR`
233
234
  - **Project safety**: pi-knowledge is read-only on indexed directories — no files are created or modified in your project
234
- - **Updates**: extension updates do not affect existing indexed data. Schema migrations run automatically if needed.
235
- - **Symbol index**: `knowledge.db` also stores lightweight symbol/config/heading metadata used by `knowledge_symbol_search`; it is derived from indexed source. Supported code files use the same AST analysis as chunking, so method symbols can be looked up even when their parent class remains one retrieval chunk. File, directory, and URL KBs with retained source paths can rebuild it with `knowledge_update`; imported portable KBs and inline text KBs should be re-imported or re-added because they intentionally do not store an active local source manifest.
235
+ - **Updates**: schema migrations run automatically, but index-time behavior changes are not applied to existing vectors at startup. `knowledge_doctor` reports actionable reindex requirements without loading the embedding model.
236
+ - **Symbol and chunk-policy indexes**: `knowledge.db` stores lightweight symbol/config/heading metadata plus an embedding input policy version. File, directory, and URL KBs with retained source paths can rebuild them with `knowledge_update`; imported portable KBs and inline text KBs should be re-imported or re-added because they intentionally do not store an active local source manifest.
236
237
 
237
238
  ## Development
238
239
 
@@ -12,24 +12,35 @@ export interface EmbeddingConfig {
12
12
  pooling: "mean";
13
13
  normalize: true;
14
14
  }
15
+ export interface EmbeddingInputMeasurement {
16
+ counts: number[];
17
+ maximum: number;
18
+ preferredMaximum: number;
19
+ unit: "tokens" | "characters";
20
+ }
15
21
  export declare const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
16
22
  export declare const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
17
23
  export declare const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
18
24
  export declare const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
25
+ export declare const LOCAL_EMBEDDING_MAX_TOKENS = 512;
26
+ export declare const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
19
27
  export declare const DEFAULT_API_MAX_EMBED_CHARS = 20000;
20
28
  export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
21
29
  export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
22
30
  export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
23
31
  export declare function dispose(): Promise<void>;
24
32
  export declare function prepareForShutdown(): Promise<void>;
33
+ export declare function measureEmbeddingInputs(texts: string[], prefix: EmbeddingPrefix, config: EmbeddingConfig, signal?: AbortSignal): Promise<EmbeddingInputMeasurement>;
25
34
  export declare function embedTextsWithConfig(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
26
35
  allowApiFallback?: boolean;
36
+ inputValidated?: boolean;
27
37
  }): Promise<{
28
38
  vectors: Float32Array[];
29
39
  config: EmbeddingConfig;
30
40
  }>;
31
41
  export declare function embedTexts(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
32
42
  allowApiFallback?: boolean;
43
+ inputValidated?: boolean;
33
44
  }): Promise<Float32Array[]>;
34
45
  export declare function embedQueryWithConfig(text: string, signal?: AbortSignal): Promise<{
35
46
  vector: Float32Array;
@@ -37,3 +48,4 @@ export declare function embedQueryWithConfig(text: string, signal?: AbortSignal)
37
48
  }>;
38
49
  export declare function embedQuery(text: string, signal?: AbortSignal): Promise<Float32Array>;
39
50
  export declare function embedDocuments(texts: string[], signal?: AbortSignal): Promise<Float32Array[]>;
51
+ export declare function embedValidatedDocuments(texts: string[], signal?: AbortSignal): Promise<Float32Array[]>;
@@ -1,5 +1,5 @@
1
1
  import { createHash } from "node:crypto";
2
- import { embedInModelWorker } from "../model-worker-client.js";
2
+ import { embedInModelWorker, measureEmbeddingInputInModelWorker } from "../model-worker-client.js";
3
3
  let disposeTimer = null;
4
4
  let disposePromise = null;
5
5
  let activeRuns = 0;
@@ -9,6 +9,8 @@ export const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
9
9
  export const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
10
10
  export const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
11
11
  export const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
12
+ export const LOCAL_EMBEDDING_MAX_TOKENS = 512;
13
+ export const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
12
14
  export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
13
15
  const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
14
16
  const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
@@ -141,11 +143,41 @@ export async function prepareForShutdown() {
141
143
  function applyEmbeddingPrefix(text, prefix, config) {
142
144
  return config.prefixes === "on" ? `${prefix}: ${text}` : text;
143
145
  }
146
+ export async function measureEmbeddingInputs(texts, prefix, config, signal) {
147
+ if (signal?.aborted)
148
+ throw new Error("Cancelled");
149
+ if (config.provider === "openai") {
150
+ return {
151
+ counts: texts.map((text) => applyEmbeddingPrefix(text, prefix, config).length),
152
+ maximum: config.maxChars,
153
+ preferredMaximum: Math.max(1, Math.floor(config.maxChars * 0.875)),
154
+ unit: "characters",
155
+ };
156
+ }
157
+ beginRun();
158
+ try {
159
+ const measurement = await measureEmbeddingInputInModelWorker(texts, prefix, config.prefixes, signal);
160
+ return {
161
+ ...measurement,
162
+ preferredMaximum: Math.max(1, Math.floor(measurement.maximum * 0.875)),
163
+ unit: "tokens",
164
+ };
165
+ }
166
+ finally {
167
+ endRun();
168
+ }
169
+ }
170
+ async function assertEmbeddingInputsWithinBudget(texts, prefix, config, signal) {
171
+ const measurement = await measureEmbeddingInputs(texts, prefix, config, signal);
172
+ for (let index = 0; index < measurement.counts.length; index++) {
173
+ const count = measurement.counts[index];
174
+ if (count !== undefined && count > measurement.maximum) {
175
+ throw new Error(`${config.provider} embedding ${prefix} input ${index + 1} is ${count} ${measurement.unit}; maximum is ${measurement.maximum}. Split the input before embedding.`);
176
+ }
177
+ }
178
+ }
144
179
  async function embedViaAPI(texts, prefix, config, signal) {
145
- const safeTexts = texts.map((text) => {
146
- const embeddingText = applyEmbeddingPrefix(text, prefix, config);
147
- return embeddingText.length > config.maxChars ? embeddingText.slice(0, config.maxChars) : embeddingText;
148
- });
180
+ const safeTexts = texts.map((text) => applyEmbeddingPrefix(text, prefix, config));
149
181
  const apiKey = cleanEnv(process.env.PI_KNOWLEDGE_EMBEDDING_API_KEY) ?? cleanEnv(process.env.OPENAI_API_KEY);
150
182
  if (!config.baseUrl)
151
183
  throw new Error("OpenAI embedding base URL is not configured");
@@ -169,7 +201,10 @@ async function embedViaAPI(texts, prefix, config, signal) {
169
201
  export async function embedTextsWithConfig(texts, prefix, signal, options = {}) {
170
202
  const config = resolveEmbeddingConfig();
171
203
  let actualConfig = config;
204
+ let inputValidated = options.inputValidated === true;
172
205
  if (config.provider === "openai") {
206
+ if (!inputValidated)
207
+ await assertEmbeddingInputsWithinBudget(texts, prefix, config, signal);
173
208
  if (signal?.aborted)
174
209
  throw new Error("Cancelled");
175
210
  try {
@@ -182,8 +217,11 @@ export async function embedTextsWithConfig(texts, prefix, signal, options = {})
182
217
  throw error;
183
218
  console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
184
219
  actualConfig = localEmbeddingConfig(config.maxChars, config.prefixes);
220
+ inputValidated = false;
185
221
  }
186
222
  }
223
+ if (!inputValidated)
224
+ await assertEmbeddingInputsWithinBudget(texts, prefix, actualConfig, signal);
187
225
  beginRun();
188
226
  try {
189
227
  return { vectors: await embedInModelWorker(texts, prefix, actualConfig.prefixes, signal), config: actualConfig };
@@ -205,3 +243,6 @@ export async function embedQuery(text, signal) {
205
243
  export async function embedDocuments(texts, signal) {
206
244
  return embedTexts(texts, "passage", signal, { allowApiFallback: false });
207
245
  }
246
+ export async function embedValidatedDocuments(texts, signal) {
247
+ return embedTexts(texts, "passage", signal, { allowApiFallback: false, inputValidated: true });
248
+ }
@@ -58,7 +58,7 @@ export interface DoctorIssue {
58
58
  kb_name?: string;
59
59
  message: string;
60
60
  action: string;
61
- action_code?: "run_update" | "rebuild_kb" | "wait_for_indexing" | "review_skipped_scope" | "rebuild_vectors" | "check_source" | "none";
61
+ action_code?: "run_update" | "rebuild_kb" | "wait_for_indexing" | "review_skipped_scope" | "rebuild_vectors" | "reindex_for_embedding_budget" | "check_source" | "none";
62
62
  }
63
63
  export interface DoctorReport {
64
64
  health_score: number;
@@ -97,6 +97,8 @@ export interface SymbolSearchResponse {
97
97
  total_count: number;
98
98
  has_more: boolean;
99
99
  }
100
+ export declare const CURRENT_URL_STRUCTURE_POLICY = 1;
101
+ export declare const CURRENT_EMBEDDING_INPUT_POLICY = 2;
100
102
  export interface IndexPlan {
101
103
  source_type: "file" | "directory" | "text" | "url";
102
104
  scannable_files: number;
@@ -1,9 +1,11 @@
1
1
  import { existsSync, readFileSync, renameSync, rmSync, statSync } from "node:fs";
2
2
  import { join, resolve } from "node:path";
3
3
  import { diagnoseKB } from "./diagnostics/health.js";
4
- import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
4
+ import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, embedValidatedDocuments, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
5
5
  import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
6
- import { addSkippedScanEntry, analyzeIndexableContent, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
6
+ import { addSkippedScanEntry, analyzeIndexableContent, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, chunkMarkdown, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
7
+ import { finalizeChunksForEmbedding } from "./indexer/embedding-safe-chunks.js";
8
+ import { htmlToStructuredMarkdown } from "./indexer/html.js";
7
9
  import { extractSymbols } from "./indexer/symbols.js";
8
10
  import { shutdownModelWorker } from "./model-worker-client.js";
9
11
  import { searchBM25 } from "./search/bm25.js";
@@ -13,8 +15,10 @@ import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence, normalizeFileTypeFilte
13
15
  import { disposeReranker, prepareRerankerForShutdown, rerank } from "./search/reranker.js";
14
16
  import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js";
15
17
  import { searchVectorFile } from "./search/vector.js";
16
- import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
18
+ import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBSourceOptions, updateKBStatus, } from "./storage/sqlite.js";
17
19
  export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
20
+ export const CURRENT_URL_STRUCTURE_POLICY = 1;
21
+ export const CURRENT_EMBEDDING_INPUT_POLICY = 2;
18
22
  const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
19
23
  const VECTOR_REDUNDANCY_WEIGHT = 0.35;
20
24
  function resolveIndexEmbedBatchSize(env = process.env) {
@@ -66,14 +70,17 @@ function toScanOptions(options = {}) {
66
70
  };
67
71
  }
68
72
  function serializeAddOptions(options = {}) {
69
- const normalized = {};
73
+ const normalized = {
74
+ embedding_input_policy: CURRENT_EMBEDDING_INPUT_POLICY,
75
+ url_structure_policy: CURRENT_URL_STRUCTURE_POLICY,
76
+ };
70
77
  if (options.include_suggested_text === true)
71
78
  normalized.include_suggested_text = true;
72
79
  if (options.include_paths && options.include_paths.length > 0)
73
80
  normalized.include_paths = options.include_paths;
74
81
  if (options.exclude_paths && options.exclude_paths.length > 0)
75
82
  normalized.exclude_paths = options.exclude_paths;
76
- return Object.keys(normalized).length > 0 ? JSON.stringify(normalized) : undefined;
83
+ return JSON.stringify(normalized);
77
84
  }
78
85
  function parseAddOptions(raw) {
79
86
  if (!raw)
@@ -94,6 +101,18 @@ function parseAddOptions(raw) {
94
101
  return {};
95
102
  }
96
103
  }
104
+ function storedSourcePolicyVersion(raw, key) {
105
+ if (!raw)
106
+ return undefined;
107
+ try {
108
+ const parsed = JSON.parse(raw);
109
+ const version = parsed[key];
110
+ return typeof version === "number" && Number.isInteger(version) ? version : undefined;
111
+ }
112
+ catch {
113
+ return undefined;
114
+ }
115
+ }
97
116
  function planDirectoryScan(dirPath, options = {}, signal) {
98
117
  const skipped = createSkippedScanStats();
99
118
  let files = 0;
@@ -452,14 +471,10 @@ async function chunkUrl(source, signal) {
452
471
  if (!res.ok)
453
472
  throw new Error(`Fetch failed: ${res.status}`);
454
473
  const html = await res.text();
455
- const text = html
456
- .replace(/<script[^>]*>[\s\S]*?<\/script>/gi, "")
457
- .replace(/<style[^>]*>[\s\S]*?<\/style>/gi, "")
458
- .replace(/<[^>]+>/g, " ")
459
- .replace(/&[a-z]+;/gi, " ")
460
- .replace(/\s+/g, " ")
461
- .trim();
462
- return chunkFile(text, source);
474
+ throwIfAborted(signal);
475
+ const text = htmlToStructuredMarkdown(html);
476
+ const structuredChunks = chunkMarkdown(text, source, "html");
477
+ return structuredChunks.length > 0 ? structuredChunks : chunkFile(text, source);
463
478
  }
464
479
  function normalizeExtractedText(text) {
465
480
  return Array.isArray(text) ? text.join("\n\n") : text;
@@ -772,7 +787,7 @@ export class KnowledgeEngine {
772
787
  throw new Error("Cancelled");
773
788
  const batch = pendingChunks.splice(0, embeddingBatchSize);
774
789
  reportProgress(`Embedding batch of ${batch.length}`, processedFiles, totalFiles);
775
- const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
790
+ const vectors = await embedValidatedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
776
791
  if (signal?.aborted)
777
792
  throw new Error("Cancelled");
778
793
  assertEmbeddingBatchSize(vectors, batch.length, "add");
@@ -784,7 +799,8 @@ export class KnowledgeEngine {
784
799
  reportProgress("Stored batch", processedFiles, totalFiles);
785
800
  };
786
801
  const addChunks = async (chunks, processedFiles, totalFiles) => {
787
- pendingChunks.push(...chunks);
802
+ const finalizedChunks = await finalizeChunksForEmbedding(chunks, embeddingConfig, signal);
803
+ pendingChunks.push(...finalizedChunks);
788
804
  while (pendingChunks.length >= embeddingBatchSize) {
789
805
  await flushPending(processedFiles, totalFiles);
790
806
  }
@@ -1016,7 +1032,7 @@ export class KnowledgeEngine {
1016
1032
  unchanged_chunks: unchanged,
1017
1033
  });
1018
1034
  onProgress?.(message);
1019
- const newVectors = await embedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
1035
+ const newVectors = await embedValidatedDocuments(batch.map((c) => buildChunkEmbeddingText(c)), signal);
1020
1036
  if (signal?.aborted)
1021
1037
  throw new Error("Cancelled");
1022
1038
  assertEmbeddingBatchSize(newVectors, batch.length, "update");
@@ -1043,7 +1059,8 @@ export class KnowledgeEngine {
1043
1059
  onProgress?.(storedMessage);
1044
1060
  };
1045
1061
  const processChunks = async (chunks) => {
1046
- for (const chunk of chunks) {
1062
+ const finalizedChunks = await finalizeChunksForEmbedding(chunks, embeddingConfig, signal);
1063
+ for (const chunk of finalizedChunks) {
1047
1064
  scannedChunks++;
1048
1065
  const existing = reusableHashes.get(chunk.content_hash);
1049
1066
  if (existing && existing.length > 0) {
@@ -1231,6 +1248,7 @@ export class KnowledgeEngine {
1231
1248
  insertSymbols(this.db, kb.id, stagedSymbols);
1232
1249
  updateKBEmbeddingMetadata(this.db, kb.id, embeddingModel, finalEmbeddingDimension === undefined ? null : embeddingSignature(embeddingConfig, finalEmbeddingDimension), finalEmbeddingDimension ?? null);
1233
1250
  updateKBCounts(this.db, kb.id, finalChunkCount, getFileCount(this.db, kb.id));
1251
+ updateKBSourceOptions(this.db, kb.id, serializeAddOptions(parseAddOptions(kb.source_options)));
1234
1252
  updateKBStatus(this.db, kb.id, "ready");
1235
1253
  const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
1236
1254
  const readyMessage = `Ready: +${addedCount} -${idsToRemove.length} =${unchanged}${skippedReadySuffix}`;
@@ -1752,6 +1770,30 @@ export class KnowledgeEngine {
1752
1770
  if (this.db) {
1753
1771
  for (const kb of kbs) {
1754
1772
  throwIfAborted(signal);
1773
+ if ((kb.status === "ready" || kb.status === "stale") &&
1774
+ (storedSourcePolicyVersion(kb.source_options, "embedding_input_policy") ?? 0) < CURRENT_EMBEDDING_INPUT_POLICY) {
1775
+ const canUpdate = kb.source_path !== null && kb.source_type !== "text";
1776
+ issues.push({
1777
+ severity: "warning",
1778
+ kb_name: kb.name,
1779
+ message: "This KB predates embedding-safe chunk limits, so oversized chunk tails may be missing from vector retrieval.",
1780
+ action: canUpdate
1781
+ ? "Run knowledge_update to re-chunk oversized inputs with the current embedding budget."
1782
+ : "Re-add inline text or re-import the original source to rebuild chunks with the current embedding budget.",
1783
+ action_code: "reindex_for_embedding_budget",
1784
+ });
1785
+ }
1786
+ if (kb.source_type === "url" &&
1787
+ (kb.status === "ready" || kb.status === "stale") &&
1788
+ (storedSourcePolicyVersion(kb.source_options, "url_structure_policy") ?? 0) < CURRENT_URL_STRUCTURE_POLICY) {
1789
+ issues.push({
1790
+ severity: "warning",
1791
+ kb_name: kb.name,
1792
+ message: "This URL KB predates structure-preserving HTML normalization.",
1793
+ action: "Run knowledge_update to rebuild URL headings, sections, lists, tables, and entities.",
1794
+ action_code: "run_update",
1795
+ });
1796
+ }
1755
1797
  if (kb.status === "ready" && kb.chunk_count > 0 && getSymbolCount(this.db, kb.id) === 0) {
1756
1798
  issues.push({
1757
1799
  severity: "info",
@@ -1879,7 +1921,7 @@ export class KnowledgeEngine {
1879
1921
  added_chunks: inserted,
1880
1922
  });
1881
1923
  onProgress?.(message);
1882
- const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
1924
+ const vectors = await embedValidatedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
1883
1925
  throwIfAborted(signal);
1884
1926
  assertEmbeddingBatchSize(vectors, batch.length, "import");
1885
1927
  persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
@@ -1915,6 +1957,7 @@ export class KnowledgeEngine {
1915
1957
  name: header.name,
1916
1958
  description: header.description,
1917
1959
  source_type: "text",
1960
+ source_options: serializeAddOptions(),
1918
1961
  embedding_model: embeddingConfigLabel(embeddingConfig),
1919
1962
  });
1920
1963
  updateKBStatus(this.db, kb.id, "indexing");
@@ -1934,9 +1977,10 @@ export class KnowledgeEngine {
1934
1977
  const imported = JSON.parse(line);
1935
1978
  const chunk = importedChunkToInsert(imported);
1936
1979
  chunk.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(chunk));
1937
- pendingChunks.push(chunk);
1980
+ const finalizedChunks = await finalizeChunksForEmbedding([chunk], embeddingConfig, signal);
1981
+ pendingChunks.push(...finalizedChunks);
1938
1982
  pendingSymbols.push(...extractSymbols(imported.content, imported.file_path, imported.file_type));
1939
- if (pendingChunks.length >= embeddingBatchSize)
1983
+ while (pendingChunks.length >= embeddingBatchSize)
1940
1984
  await flushPending();
1941
1985
  }
1942
1986
  if (!header)
@@ -50,7 +50,7 @@ export declare function chunkIdentityHash(opts: {
50
50
  metadataJson: string;
51
51
  }): string;
52
52
  export declare function buildChunkEmbeddingText(chunk: Pick<ChunkInsert, "content" | "file_path" | "file_type" | "metadata_json">): string;
53
- export declare function chunkMarkdown(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
53
+ export declare function chunkMarkdown(content: string, filePath: string, fileType?: string): Omit<ChunkInsert, "kb_id">[];
54
54
  export declare function chunkText(content: string, filePath: string): Omit<ChunkInsert, "kb_id">[];
55
55
  export declare function isCodeFileType(fileType: string): boolean;
56
56
  export declare function analyzeIndexableContent(content: string, filePath: string, fileType?: string): Promise<{
@@ -397,7 +397,7 @@ function makeChunk(content, filePath, fileType, startLine, endLine, metadata = {
397
397
  };
398
398
  return { ...chunk, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(chunk)) };
399
399
  }
400
- export function chunkMarkdown(content, filePath) {
400
+ export function chunkMarkdown(content, filePath, fileType = "markdown") {
401
401
  const lines = content.split("\n");
402
402
  const chunks = [];
403
403
  const headingStack = [];
@@ -410,7 +410,7 @@ export function chunkMarkdown(content, filePath) {
410
410
  function pushMarkdownChunk(text, start, end) {
411
411
  if (text.trim().length < 50)
412
412
  return;
413
- chunks.push(makeChunk(text.trim(), filePath, "markdown", start, end, {
413
+ chunks.push(makeChunk(text.trim(), filePath, fileType, start, end, {
414
414
  heading: currentHeading,
415
415
  breadcrumb: currentBreadcrumb(),
416
416
  }));
@@ -0,0 +1,5 @@
1
+ import type { EmbeddingConfig } from "../embedding/provider.ts";
2
+ import type { ChunkInsert } from "../storage/sqlite.ts";
3
+ type IndexableChunk = Omit<ChunkInsert, "kb_id">;
4
+ export declare function finalizeChunksForEmbedding(chunks: IndexableChunk[], config: EmbeddingConfig, signal?: AbortSignal): Promise<IndexableChunk[]>;
5
+ export {};
@@ -0,0 +1,215 @@
1
+ import { LOCAL_EMBEDDING_PREFERRED_TOKENS, measureEmbeddingInputs } from "../embedding/provider.js";
2
+ import { buildChunkEmbeddingText, chunkIdentityHash, preTokenizeForFTS } from "./chunker.js";
3
+ const SENTENCE_BOUNDARIES = {
4
+ "。": true,
5
+ "!": true,
6
+ "?": true,
7
+ ".": true,
8
+ "!": true,
9
+ "?": true,
10
+ ";": true,
11
+ ";": true,
12
+ };
13
+ function throwIfAborted(signal) {
14
+ if (signal?.aborted)
15
+ throw new Error("Cancelled");
16
+ }
17
+ function compareOrder(left, right) {
18
+ if (left.rootIndex !== right.rootIndex)
19
+ return left.rootIndex - right.rootIndex;
20
+ const length = Math.min(left.order.length, right.order.length);
21
+ for (let index = 0; index < length; index++) {
22
+ if (left.order[index] !== right.order[index])
23
+ return left.order[index] - right.order[index];
24
+ }
25
+ return left.order.length - right.order.length;
26
+ }
27
+ function nearestStringBoundary(content, desired, boundary, minimum) {
28
+ const before = content.lastIndexOf(boundary, desired);
29
+ if (before >= minimum)
30
+ return before + boundary.length;
31
+ const after = content.indexOf(boundary, desired);
32
+ if (after >= 0 && after <= content.length - minimum)
33
+ return after + boundary.length;
34
+ return undefined;
35
+ }
36
+ function nearestCharacterBoundary(content, desired, minimum, predicate) {
37
+ for (let offset = 0; offset < content.length; offset++) {
38
+ const before = desired - offset;
39
+ if (before >= minimum && predicate(content[before]))
40
+ return before + 1;
41
+ const after = desired + offset;
42
+ if (after < content.length - minimum && predicate(content[after]))
43
+ return after + 1;
44
+ if (before < minimum && after >= content.length - minimum)
45
+ break;
46
+ }
47
+ return undefined;
48
+ }
49
+ function safeCodeUnitBoundary(content, index) {
50
+ if (index <= 0 || index >= content.length)
51
+ return index;
52
+ const previous = content.charCodeAt(index - 1);
53
+ const next = content.charCodeAt(index);
54
+ return previous >= 0xd800 && previous <= 0xdbff && next >= 0xdc00 && next <= 0xdfff ? index - 1 : index;
55
+ }
56
+ function chooseSplitIndex(content, units, measurement) {
57
+ const estimatedParts = Math.max(2, Math.ceil(units / measurement.preferredMaximum));
58
+ const desired = Math.max(1, Math.floor(content.length / estimatedParts));
59
+ const minimum = Math.max(1, Math.min(64, Math.floor(content.length * 0.1)));
60
+ const structuralBoundary = nearestStringBoundary(content, desired, "\n\n", minimum) ??
61
+ nearestStringBoundary(content, desired, "\n", minimum) ??
62
+ nearestCharacterBoundary(content, desired, minimum, (character) => SENTENCE_BOUNDARIES[character] === true) ??
63
+ nearestCharacterBoundary(content, desired, minimum, (character) => /\s/u.test(character));
64
+ const splitIndex = safeCodeUnitBoundary(content, structuralBoundary ?? desired);
65
+ if (splitIndex <= 0 || splitIndex >= content.length) {
66
+ throw new Error(`Unable to split oversized embedding input containing ${content.length} characters`);
67
+ }
68
+ return splitIndex;
69
+ }
70
+ function lineCount(content) {
71
+ let count = 0;
72
+ for (const character of content) {
73
+ if (character === "\n")
74
+ count++;
75
+ }
76
+ return count;
77
+ }
78
+ function rebuildChunk(chunk, content, startLine, endLine) {
79
+ const rebuilt = {
80
+ ...chunk,
81
+ content,
82
+ content_hash: chunkIdentityHash({
83
+ content,
84
+ filePath: chunk.file_path,
85
+ fileType: chunk.file_type,
86
+ startLine,
87
+ endLine,
88
+ metadataJson: chunk.metadata_json,
89
+ }),
90
+ content_tokenized: "",
91
+ start_line: startLine,
92
+ end_line: endLine,
93
+ };
94
+ return { ...rebuilt, content_tokenized: preTokenizeForFTS(buildChunkEmbeddingText(rebuilt)) };
95
+ }
96
+ function unsplittableEmbeddingContext(piece, measurement) {
97
+ return new Error(`Embedding context for ${piece.chunk.file_path}:${piece.chunk.start_line}-${piece.chunk.end_line} exceeds ${measurement.maximum} ${measurement.unit} and cannot be split safely`);
98
+ }
99
+ function splitPiece(piece, units, measurement) {
100
+ const content = piece.chunk.content;
101
+ if ([...content].length < 2)
102
+ throw unsplittableEmbeddingContext(piece, measurement);
103
+ const splitIndex = chooseSplitIndex(content, units, measurement);
104
+ const leftRaw = content.slice(0, splitIndex);
105
+ const rightRaw = content.slice(splitIndex);
106
+ const left = leftRaw.trim();
107
+ const right = rightRaw.trim();
108
+ if (!left || !right || left === content || right === content) {
109
+ throw unsplittableEmbeddingContext(piece, measurement);
110
+ }
111
+ const leftLeadingLines = lineCount(leftRaw.slice(0, leftRaw.indexOf(left)));
112
+ const rightLeadingLines = lineCount(rightRaw.slice(0, rightRaw.indexOf(right)));
113
+ const leftStart = piece.chunk.start_line + leftLeadingLines;
114
+ const leftEnd = leftStart + lineCount(left);
115
+ const rightStart = piece.chunk.start_line + lineCount(leftRaw) + rightLeadingLines;
116
+ const rightEnd = rightStart + lineCount(right);
117
+ return [
118
+ {
119
+ chunk: rebuildChunk(piece.chunk, left, leftStart, leftEnd),
120
+ rootIndex: piece.rootIndex,
121
+ order: [...piece.order, 0],
122
+ },
123
+ {
124
+ chunk: rebuildChunk(piece.chunk, right, rightStart, rightEnd),
125
+ rootIndex: piece.rootIndex,
126
+ order: [...piece.order, 1],
127
+ },
128
+ ];
129
+ }
130
+ function withPartMetadata(chunk, part, partCount) {
131
+ let metadata = {};
132
+ try {
133
+ const parsed = JSON.parse(chunk.metadata_json);
134
+ if (parsed && typeof parsed === "object" && !Array.isArray(parsed))
135
+ metadata = parsed;
136
+ }
137
+ catch {
138
+ metadata = {};
139
+ }
140
+ const metadataJson = JSON.stringify({ ...metadata, chunk_part: part, chunk_part_count: partCount });
141
+ const rebuilt = { ...chunk, metadata_json: metadataJson, content_tokenized: "" };
142
+ rebuilt.content_hash = chunkIdentityHash({
143
+ content: rebuilt.content,
144
+ filePath: rebuilt.file_path,
145
+ fileType: rebuilt.file_type,
146
+ startLine: rebuilt.start_line,
147
+ endLine: rebuilt.end_line,
148
+ metadataJson,
149
+ });
150
+ rebuilt.content_tokenized = preTokenizeForFTS(buildChunkEmbeddingText(rebuilt));
151
+ return rebuilt;
152
+ }
153
+ function isProvablySafeLocalAsciiInput(text, config) {
154
+ if (config.provider !== "local")
155
+ return false;
156
+ const prefixLength = config.prefixes === "on" ? "passage: ".length : 0;
157
+ if (text.length + prefixLength + 2 > LOCAL_EMBEDDING_PREFERRED_TOKENS)
158
+ return false;
159
+ for (let index = 0; index < text.length; index++) {
160
+ if (text.charCodeAt(index) > 0x7f)
161
+ return false;
162
+ }
163
+ return true;
164
+ }
165
+ export async function finalizeChunksForEmbedding(chunks, config, signal) {
166
+ if (chunks.length === 0)
167
+ return [];
168
+ let pending = chunks.map((chunk, rootIndex) => ({ chunk, rootIndex, order: [] }));
169
+ const accepted = [];
170
+ while (pending.length > 0) {
171
+ throwIfAborted(signal);
172
+ const piecesToMeasure = [];
173
+ const textsToMeasure = [];
174
+ for (const piece of pending) {
175
+ const text = buildChunkEmbeddingText(piece.chunk);
176
+ if (isProvablySafeLocalAsciiInput(text, config))
177
+ accepted.push(piece);
178
+ else {
179
+ piecesToMeasure.push(piece);
180
+ textsToMeasure.push(text);
181
+ }
182
+ }
183
+ if (piecesToMeasure.length === 0)
184
+ break;
185
+ const measurement = await measureEmbeddingInputs(textsToMeasure, "passage", config, signal);
186
+ if (measurement.counts.length !== piecesToMeasure.length) {
187
+ throw new Error(`Embedding input measurement returned ${measurement.counts.length} counts for ${piecesToMeasure.length} chunks`);
188
+ }
189
+ const next = [];
190
+ for (let index = 0; index < piecesToMeasure.length; index++) {
191
+ const piece = piecesToMeasure[index];
192
+ const units = measurement.counts[index];
193
+ if (units === undefined)
194
+ throw new Error(`Missing embedding input measurement for chunk ${index}`);
195
+ if (units <= measurement.maximum)
196
+ accepted.push(piece);
197
+ else
198
+ next.push(...splitPiece(piece, units, measurement));
199
+ }
200
+ pending = next;
201
+ }
202
+ accepted.sort(compareOrder);
203
+ const partCounts = new Map();
204
+ for (const piece of accepted)
205
+ partCounts.set(piece.rootIndex, (partCounts.get(piece.rootIndex) ?? 0) + 1);
206
+ const partIndexes = new Map();
207
+ return accepted.map((piece) => {
208
+ const partCount = partCounts.get(piece.rootIndex) ?? 1;
209
+ if (partCount === 1)
210
+ return piece.chunk;
211
+ const part = (partIndexes.get(piece.rootIndex) ?? 0) + 1;
212
+ partIndexes.set(piece.rootIndex, part);
213
+ return withPartMetadata(piece.chunk, part, partCount);
214
+ });
215
+ }
@@ -0,0 +1 @@
1
+ export declare function htmlToStructuredMarkdown(html: string): string;
@@ -0,0 +1,49 @@
1
+ const HTML_ENTITIES = {
2
+ amp: "&",
3
+ apos: "'",
4
+ gt: ">",
5
+ lt: "<",
6
+ nbsp: " ",
7
+ quot: '"',
8
+ ndash: "–",
9
+ mdash: "—",
10
+ hellip: "…",
11
+ };
12
+ function decodeHtmlEntity(entity) {
13
+ if (!entity.startsWith("#"))
14
+ return HTML_ENTITIES[entity.toLowerCase()] ?? `&${entity};`;
15
+ const hexadecimal = entity[1]?.toLowerCase() === "x";
16
+ const digits = entity.slice(hexadecimal ? 2 : 1);
17
+ const codePoint = Number.parseInt(digits, hexadecimal ? 16 : 10);
18
+ if (!digits ||
19
+ !Number.isInteger(codePoint) ||
20
+ codePoint <= 0 ||
21
+ codePoint > 0x10ffff ||
22
+ (codePoint >= 0xd800 && codePoint <= 0xdfff)) {
23
+ return `&${entity};`;
24
+ }
25
+ return String.fromCodePoint(codePoint);
26
+ }
27
+ export function htmlToStructuredMarkdown(html) {
28
+ return html
29
+ .replace(/<!--[\s\S]*?-->/g, "")
30
+ .replace(/<(script|style|noscript|template|svg)\b[^>]*>[\s\S]*?(?:<\/\1\s*>|$)/gi, "")
31
+ .replace(/<h([1-6])\b[^>]*>/gi, (_tag, level) => `\n\n${"#".repeat(Number(level))} `)
32
+ .replace(/<\/h[1-6]\s*>/gi, "\n\n")
33
+ .replace(/<li\b[^>]*>/gi, "\n- ")
34
+ .replace(/<\/li\s*>/gi, "\n")
35
+ .replace(/<br\s*\/?\s*>/gi, "\n")
36
+ .replace(/<(th|td)\b[^>]*>/gi, " | ")
37
+ .replace(/<\/(th|td)\s*>/gi, "")
38
+ .replace(/<tr\b[^>]*>/gi, "\n")
39
+ .replace(/<\/tr\s*>/gi, "\n")
40
+ .replace(/<\/?(?:address|article|aside|blockquote|div|dl|dt|dd|fieldset|figcaption|figure|footer|form|header|hr|main|nav|ol|p|section|table|tbody|thead|tfoot|ul)\b[^>]*>/gi, "\n\n")
41
+ .replace(/<[^>]+>/g, "")
42
+ .replace(/&(#(?:x[0-9a-f]+|\d+)|[a-z][a-z0-9]+);/gi, (_match, entity) => decodeHtmlEntity(entity))
43
+ .replace(/\r\n?/g, "\n")
44
+ .split("\n")
45
+ .map((line) => line.replace(/[\t\f\v ]+/g, " ").trim())
46
+ .join("\n")
47
+ .replace(/\n{3,}/g, "\n\n")
48
+ .trim();
49
+ }
@@ -3,6 +3,10 @@ export declare function validateModelWorkerNodePath(nodePath: string): string;
3
3
  export declare function resolveModelWorkerNodePath(): string;
4
4
  export declare function configureModelWorkerNodePath(nodePath?: string): string;
5
5
  export declare function embedInModelWorker(texts: string[], prefix: "query" | "passage", prefixesOrSignal?: "on" | "off" | AbortSignal, signal?: AbortSignal): Promise<Float32Array[]>;
6
+ export declare function measureEmbeddingInputInModelWorker(texts: string[], prefix: "query" | "passage", prefixes: "on" | "off", signal?: AbortSignal): Promise<{
7
+ counts: number[];
8
+ maximum: number;
9
+ }>;
6
10
  export interface RerankWorkerCandidate {
7
11
  chunkId: string;
8
12
  content: string;
@@ -187,8 +187,8 @@ function isWorkerResponse(message) {
187
187
  return !("error" in message) || message.error === undefined || typeof message.error === "string";
188
188
  }
189
189
  class ModelWorkerTransport {
190
- child;
191
190
  pending = new Map();
191
+ child;
192
192
  stderrTail = "";
193
193
  exited = false;
194
194
  terminalError = null;
@@ -397,6 +397,22 @@ export async function embedInModelWorker(texts, prefix, prefixesOrSignal, signal
397
397
  return new Float32Array(vector);
398
398
  });
399
399
  }
400
+ export async function measureEmbeddingInputInModelWorker(texts, prefix, prefixes, signal) {
401
+ const result = await getWorker().request({ type: "measure_embedding_input", texts, prefix, prefixes }, signal);
402
+ if (typeof result !== "object" ||
403
+ result === null ||
404
+ !("counts" in result) ||
405
+ !Array.isArray(result.counts) ||
406
+ result.counts.length !== texts.length ||
407
+ !result.counts.every((count) => typeof count === "number" && Number.isInteger(count) && count >= 0) ||
408
+ !("maximum" in result) ||
409
+ typeof result.maximum !== "number" ||
410
+ !Number.isInteger(result.maximum) ||
411
+ result.maximum <= 0) {
412
+ throw new Error("Invalid embedding input measurement from worker");
413
+ }
414
+ return { counts: result.counts, maximum: result.maximum };
415
+ }
400
416
  export async function rerankInModelWorker(query, candidates, topK, reranker, signal) {
401
417
  const result = await getWorker().request({ type: "rerank", query, candidates, topK, reranker }, signal);
402
418
  if (!Array.isArray(result))
@@ -4,6 +4,7 @@ import { DEFAULT_RERANKER_REMOTE_HOST, DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE, is
4
4
  import { rerankWithSingleRawLogit } from "./search/reranker-logits.js";
5
5
  import { getDefaultKnowledgeDir } from "./storage/sqlite.js";
6
6
  let embeddingPipeline = null;
7
+ let embeddingPipelinePromise = null;
7
8
  let rerankerPipeline = null;
8
9
  function getModelCacheDir() {
9
10
  return process.env.PI_KNOWLEDGE_MODEL_CACHE_DIR ?? join(getDefaultKnowledgeDir(), "models");
@@ -24,15 +25,24 @@ function configureEmbeddingTransformersEnv(env) {
24
25
  async function loadEmbeddingPipeline() {
25
26
  if (embeddingPipeline)
26
27
  return embeddingPipeline;
27
- const { pipeline, env } = await import("@huggingface/transformers");
28
- configureEmbeddingTransformersEnv(env);
29
- const createPipeline = pipeline;
30
- const loaded = (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
31
- quantized: true,
32
- dtype: "fp32",
33
- }));
34
- embeddingPipeline = loaded;
35
- return loaded;
28
+ if (embeddingPipelinePromise)
29
+ return embeddingPipelinePromise;
30
+ embeddingPipelinePromise = (async () => {
31
+ const { pipeline, env } = await import("@huggingface/transformers");
32
+ configureEmbeddingTransformersEnv(env);
33
+ const createPipeline = pipeline;
34
+ return (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
35
+ quantized: true,
36
+ dtype: "fp32",
37
+ }));
38
+ })();
39
+ try {
40
+ embeddingPipeline = await embeddingPipelinePromise;
41
+ return embeddingPipeline;
42
+ }
43
+ finally {
44
+ embeddingPipelinePromise = null;
45
+ }
36
46
  }
37
47
  async function loadRerankerPipeline(config) {
38
48
  const key = rerankerCacheKey(config);
@@ -81,6 +91,34 @@ async function handleEmbed(request) {
81
91
  }
82
92
  return vectors;
83
93
  }
94
+ function embeddingMaximumTokens(tokenizer) {
95
+ const configured = tokenizer.model_max_length;
96
+ return typeof configured === "number" && Number.isInteger(configured) && configured > 0 && configured <= 1_000_000
97
+ ? configured
98
+ : 512;
99
+ }
100
+ async function handleMeasureEmbeddingInput(request) {
101
+ const pipe = await loadEmbeddingPipeline();
102
+ const texts = request.texts.map((text) => applyEmbeddingPrefix(text, request.prefix, request.prefixes));
103
+ const tokenized = pipe.tokenizer(texts, {
104
+ add_special_tokens: true,
105
+ truncation: false,
106
+ padding: false,
107
+ return_tensor: false,
108
+ });
109
+ const inputIds = tokenized.input_ids;
110
+ let counts;
111
+ if (texts.length === 1 && inputIds.every((token) => typeof token === "number")) {
112
+ counts = [inputIds.length];
113
+ }
114
+ else if (inputIds.every((tokens) => Array.isArray(tokens))) {
115
+ counts = inputIds.map((tokens) => tokens.length);
116
+ }
117
+ else {
118
+ counts = texts.map((text) => pipe.tokenizer.encode(text, { add_special_tokens: true }).length);
119
+ }
120
+ return { counts, maximum: embeddingMaximumTokens(pipe.tokenizer) };
121
+ }
84
122
  async function handleRerank(request) {
85
123
  const pipe = await loadRerankerPipeline(request.reranker);
86
124
  const results = [];
@@ -110,17 +148,23 @@ function parseWorkerRequest(message) {
110
148
  }
111
149
  if (!("type" in message))
112
150
  throw new Error("Invalid worker request type");
113
- if (message.type === "embed") {
151
+ if (message.type === "embed" || message.type === "measure_embedding_input") {
114
152
  if (!("texts" in message) || !isStringArray(message.texts))
115
- throw new Error("Invalid embed texts");
153
+ throw new Error("Invalid embedding texts");
116
154
  if (!("prefix" in message) || (message.prefix !== "query" && message.prefix !== "passage")) {
117
- throw new Error("Invalid embed prefix");
155
+ throw new Error("Invalid embedding prefix");
118
156
  }
119
157
  const prefixes = "prefixes" in message && (message.prefixes === "on" || message.prefixes === "off") ? message.prefixes : "on";
120
158
  if ("prefixes" in message && message.prefixes !== "on" && message.prefixes !== "off") {
121
- throw new Error("Invalid embed prefix strategy");
159
+ throw new Error("Invalid embedding prefix strategy");
122
160
  }
123
- return { id: message.id, type: "embed", texts: message.texts, prefix: message.prefix, prefixes };
161
+ return {
162
+ id: message.id,
163
+ type: message.type,
164
+ texts: message.texts,
165
+ prefix: message.prefix,
166
+ prefixes,
167
+ };
124
168
  }
125
169
  if (message.type === "rerank") {
126
170
  if (!("query" in message) || typeof message.query !== "string")
@@ -151,7 +195,11 @@ async function handleRequestMessage(message) {
151
195
  requestId = message.id;
152
196
  }
153
197
  const request = parseWorkerRequest(message);
154
- const result = request.type === "embed" ? await handleEmbed(request) : await handleRerank(request);
198
+ const result = request.type === "embed"
199
+ ? await handleEmbed(request)
200
+ : request.type === "measure_embedding_input"
201
+ ? await handleMeasureEmbeddingInput(request)
202
+ : await handleRerank(request);
155
203
  return { id: request.id, result };
156
204
  }
157
205
  catch (error) {
@@ -87,6 +87,7 @@ export declare function listKBs(db: Database.Database): KnowledgeBase[];
87
87
  export declare function deleteKB(db: Database.Database, id: string): void;
88
88
  export declare function updateKBStatus(db: Database.Database, id: string, status: KnowledgeBase["status"]): void;
89
89
  export declare function updateKBCounts(db: Database.Database, id: string, chunkCount: number, fileCount: number): void;
90
+ export declare function updateKBSourceOptions(db: Database.Database, id: string, sourceOptions: string): void;
90
91
  export declare function updateKBEmbeddingMetadata(db: Database.Database, id: string, embeddingModel: string, embeddingSignature: string | null, embeddingDimension: number | null): void;
91
92
  export declare function startIndexingJob(db: Database.Database, kbId: string, operation: IndexingJob["operation"], message: string): void;
92
93
  export declare function updateIndexingJob(db: Database.Database, kbId: string, progress: {
@@ -355,6 +355,9 @@ export function updateKBStatus(db, id, status) {
355
355
  export function updateKBCounts(db, id, chunkCount, fileCount) {
356
356
  db.prepare("UPDATE knowledge_bases SET chunk_count = ?, file_count = ?, updated_at = ? WHERE id = ?").run(chunkCount, fileCount, Date.now(), id);
357
357
  }
358
+ export function updateKBSourceOptions(db, id, sourceOptions) {
359
+ db.prepare("UPDATE knowledge_bases SET source_options = ?, updated_at = ? WHERE id = ?").run(sourceOptions, Date.now(), id);
360
+ }
358
361
  export function updateKBEmbeddingMetadata(db, id, embeddingModel, embeddingSignature, embeddingDimension) {
359
362
  db.prepare(`UPDATE knowledge_bases
360
363
  SET embedding_model = ?, embedding_signature = ?, embedding_dimension = ?, updated_at = ?
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-knowledge",
3
- "version": "0.11.0",
3
+ "version": "0.12.0",
4
4
  "description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",