pi-knowledge 0.5.6 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +5 -2
- package/dist/src/embedding/provider.d.ts +33 -1
- package/dist/src/embedding/provider.js +103 -37
- package/dist/src/engine.d.ts +1 -1
- package/dist/src/engine.js +63 -15
- package/dist/src/model-worker-client.d.ts +2 -1
- package/dist/src/model-worker-client.js +2 -2
- package/dist/src/model-worker.js +41 -8
- package/dist/src/search/reranker-api.d.ts +7 -0
- package/dist/src/search/reranker-api.js +89 -0
- package/dist/src/search/reranker-config.d.ts +37 -0
- package/dist/src/search/reranker-config.js +155 -0
- package/dist/src/search/reranker-logits.d.ts +20 -0
- package/dist/src/search/reranker-logits.js +16 -0
- package/dist/src/search/reranker.js +7 -1
- package/dist/src/storage/sqlite.d.ts +6 -0
- package/dist/src/storage/sqlite.js +23 -3
- package/docs/configuration.md +27 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,30 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
|
|
4
|
+
## [0.7.0] - 2026-07-28
|
|
5
|
+
|
|
6
|
+
### Added
|
|
7
|
+
- Added `PI_KNOWLEDGE_RERANKER_RAW_LOGITS` for trusted single-logit local/Hugging Face cross-encoder rerankers whose sigmoid scores flatten near 1.0.
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
- Preserved the default reranker config shape when raw logits are unset.
|
|
11
|
+
- Rejected multi-logit and non-finite raw reranker outputs instead of guessing a relevance score.
|
|
12
|
+
|
|
13
|
+
## [0.6.0] - 2026-07-25
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
- Added configurable local/Hugging Face reranker selection through `PI_KNOWLEDGE_RERANKER`, including revision, dtype, and trusted mirror settings.
|
|
17
|
+
- Added API reranker support for `deep` search with Cohere/Jina-compatible requests and declarative custom JSON response mapping.
|
|
18
|
+
- Added embedding metadata signatures and dimensions so knowledge bases can detect incompatible vector spaces after provider, API base URL, preprocessing, or dimension changes.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
- Document embedding batches now surface OpenAI-compatible API failures instead of falling back to local embeddings during add, update, or import, preventing mixed-provider vector files.
|
|
22
|
+
- `knowledge_search` skips incompatible or legacy unsigned vectors with warnings, while hybrid search can fall back to BM25-only results until `knowledge_update` rebuilds vectors.
|
|
23
|
+
|
|
24
|
+
### Fixed
|
|
25
|
+
- Rebuilds unchanged chunks during `knowledge_update` when embedding metadata is missing or incompatible, rather than reusing stale vectors.
|
|
26
|
+
- Resets Transformers.js remote model settings before embedding loads so custom reranker mirrors do not affect the embedding pipeline.
|
|
27
|
+
|
|
3
28
|
## [0.5.6] - 2026-07-23
|
|
4
29
|
|
|
5
30
|
### Added
|
package/README.md
CHANGED
|
@@ -164,9 +164,11 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
|
164
164
|
export OPENAI_API_KEY=local-placeholder
|
|
165
165
|
```
|
|
166
166
|
|
|
167
|
-
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow
|
|
167
|
+
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files.
|
|
168
168
|
|
|
169
|
-
API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window
|
|
169
|
+
API embedding requests are capped at 20000 characters per input by default as a final context-window safety guard for OpenAI-compatible servers. Adjust this with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when your embedding model has a different context window; changing the cap changes embedding compatibility and requires `knowledge_update`.
|
|
170
|
+
|
|
171
|
+
Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
|
|
170
172
|
|
|
171
173
|
## Configuration
|
|
172
174
|
|
|
@@ -177,6 +179,7 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
177
179
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
178
180
|
| Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
179
181
|
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
182
|
+
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
180
183
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
181
184
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
182
185
|
| Release fixtures | `PI_KNOWLEDGE_E2E_PDF`, `PI_KNOWLEDGE_E2E_DOCX` |
|
|
@@ -1,5 +1,37 @@
|
|
|
1
|
+
export type EmbeddingProvider = "local" | "openai";
|
|
2
|
+
export type EmbeddingPrefix = "query" | "passage";
|
|
3
|
+
export interface EmbeddingConfig {
|
|
4
|
+
provider: EmbeddingProvider;
|
|
5
|
+
model: string;
|
|
6
|
+
baseUrl?: string;
|
|
7
|
+
maxChars: number;
|
|
8
|
+
queryPrefix: "query";
|
|
9
|
+
documentPrefix: "passage";
|
|
10
|
+
pooling: "mean";
|
|
11
|
+
normalize: true;
|
|
12
|
+
}
|
|
13
|
+
export declare const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
14
|
+
export declare const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
15
|
+
export declare const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
|
|
16
|
+
export declare const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
17
|
+
export declare const DEFAULT_API_MAX_EMBED_CHARS = 20000;
|
|
18
|
+
export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
|
|
19
|
+
export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
|
|
20
|
+
export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
|
|
1
21
|
export declare function dispose(): Promise<void>;
|
|
2
22
|
export declare function prepareForShutdown(): Promise<void>;
|
|
3
|
-
export declare function
|
|
23
|
+
export declare function embedTextsWithConfig(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
|
|
24
|
+
allowApiFallback?: boolean;
|
|
25
|
+
}): Promise<{
|
|
26
|
+
vectors: Float32Array[];
|
|
27
|
+
config: EmbeddingConfig;
|
|
28
|
+
}>;
|
|
29
|
+
export declare function embedTexts(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
|
|
30
|
+
allowApiFallback?: boolean;
|
|
31
|
+
}): Promise<Float32Array[]>;
|
|
32
|
+
export declare function embedQueryWithConfig(text: string, signal?: AbortSignal): Promise<{
|
|
33
|
+
vector: Float32Array;
|
|
34
|
+
config: EmbeddingConfig;
|
|
35
|
+
}>;
|
|
4
36
|
export declare function embedQuery(text: string, signal?: AbortSignal): Promise<Float32Array>;
|
|
5
37
|
export declare function embedDocuments(texts: string[], signal?: AbortSignal): Promise<Float32Array[]>;
|
|
@@ -1,15 +1,33 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
1
2
|
import { embedInModelWorker } from "../model-worker-client.js";
|
|
2
3
|
let disposeTimer = null;
|
|
3
4
|
let disposePromise = null;
|
|
4
5
|
let activeRuns = 0;
|
|
5
6
|
let disposeRequested = false;
|
|
6
7
|
const idleWaiters = [];
|
|
8
|
+
export const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
9
|
+
export const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
10
|
+
export const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
|
|
11
|
+
export const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
12
|
+
export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
|
|
7
13
|
const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
|
|
8
|
-
const EMBEDDING_CONFIG = process.env.PI_KNOWLEDGE_EMBEDDING ?? "local:multilingual-e5-small";
|
|
9
14
|
const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
|
|
10
15
|
const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
|
|
11
|
-
|
|
12
|
-
|
|
16
|
+
function localEmbeddingConfig(maxChars) {
|
|
17
|
+
return {
|
|
18
|
+
provider: "local",
|
|
19
|
+
model: DEFAULT_LOCAL_EMBEDDING_MODEL,
|
|
20
|
+
maxChars,
|
|
21
|
+
queryPrefix: "query",
|
|
22
|
+
documentPrefix: "passage",
|
|
23
|
+
pooling: "mean",
|
|
24
|
+
normalize: true,
|
|
25
|
+
};
|
|
26
|
+
}
|
|
27
|
+
function cleanEnv(value) {
|
|
28
|
+
const trimmed = value?.trim();
|
|
29
|
+
return trimmed ? trimmed : undefined;
|
|
30
|
+
}
|
|
13
31
|
function isAbortError(error) {
|
|
14
32
|
return typeof error === "object" && error !== null && "name" in error && error.name === "AbortError";
|
|
15
33
|
}
|
|
@@ -44,6 +62,52 @@ function waitForNoActiveRuns() {
|
|
|
44
62
|
return Promise.resolve();
|
|
45
63
|
return new Promise((resolve) => idleWaiters.push(resolve));
|
|
46
64
|
}
|
|
65
|
+
function providerInstanceHash(value) {
|
|
66
|
+
return createHash("sha256").update(value).digest("hex").slice(0, 12);
|
|
67
|
+
}
|
|
68
|
+
export function resolveEmbeddingConfig(env = process.env) {
|
|
69
|
+
const raw = cleanEnv(env.PI_KNOWLEDGE_EMBEDDING) ?? `local:${DEFAULT_LOCAL_EMBEDDING_MODEL}`;
|
|
70
|
+
const separator = raw.indexOf(":");
|
|
71
|
+
const provider = separator >= 0 ? raw.slice(0, separator).toLowerCase() : raw.toLowerCase();
|
|
72
|
+
const model = separator >= 0 ? raw.slice(separator + 1).trim() : "";
|
|
73
|
+
const maxCharsValue = Number(cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS) ?? DEFAULT_API_MAX_EMBED_CHARS);
|
|
74
|
+
const maxChars = Number.isFinite(maxCharsValue) && maxCharsValue > 0 ? Math.trunc(maxCharsValue) : DEFAULT_API_MAX_EMBED_CHARS;
|
|
75
|
+
if (provider === "local") {
|
|
76
|
+
const localModel = model || DEFAULT_LOCAL_EMBEDDING_MODEL;
|
|
77
|
+
if (localModel !== DEFAULT_LOCAL_EMBEDDING_MODEL) {
|
|
78
|
+
throw new Error(`Unsupported local embedding model: ${localModel}`);
|
|
79
|
+
}
|
|
80
|
+
return localEmbeddingConfig(maxChars);
|
|
81
|
+
}
|
|
82
|
+
if (provider === "openai") {
|
|
83
|
+
return {
|
|
84
|
+
provider: "openai",
|
|
85
|
+
model: model || DEFAULT_OPENAI_EMBEDDING_MODEL,
|
|
86
|
+
baseUrl: cleanEnv(env.PI_KNOWLEDGE_EMBEDDING_BASE_URL) ?? cleanEnv(env.OPENAI_BASE_URL) ?? DEFAULT_OPENAI_BASE_URL,
|
|
87
|
+
maxChars,
|
|
88
|
+
queryPrefix: "query",
|
|
89
|
+
documentPrefix: "passage",
|
|
90
|
+
pooling: "mean",
|
|
91
|
+
normalize: true,
|
|
92
|
+
};
|
|
93
|
+
}
|
|
94
|
+
throw new Error(`Unsupported embedding provider: ${provider}`);
|
|
95
|
+
}
|
|
96
|
+
export function embeddingConfigLabel(config) {
|
|
97
|
+
return config.provider === "local" ? config.model : `${config.provider}:${config.model}`;
|
|
98
|
+
}
|
|
99
|
+
export function embeddingSignature(config, dimension) {
|
|
100
|
+
const base = config.baseUrl ? `:base-sha256=${providerInstanceHash(config.baseUrl)}` : "";
|
|
101
|
+
return [
|
|
102
|
+
`${config.provider}:${config.model}${base}`,
|
|
103
|
+
`dim=${dimension}`,
|
|
104
|
+
`apiMax=${config.provider === "openai" ? config.maxChars : "none"}`,
|
|
105
|
+
`pooling=${config.pooling}`,
|
|
106
|
+
`normalize=${config.normalize}`,
|
|
107
|
+
`q=${config.queryPrefix}`,
|
|
108
|
+
`d=${config.documentPrefix}`,
|
|
109
|
+
].join(":");
|
|
110
|
+
}
|
|
47
111
|
export async function dispose() {
|
|
48
112
|
clearIdleTimer();
|
|
49
113
|
if (disposePromise)
|
|
@@ -60,62 +124,64 @@ export async function prepareForShutdown() {
|
|
|
60
124
|
clearIdleTimer();
|
|
61
125
|
await waitForNoActiveRuns();
|
|
62
126
|
}
|
|
63
|
-
async function embedViaAPI(texts, prefix, signal) {
|
|
64
|
-
const [provider, model] = EMBEDDING_CONFIG.split(":");
|
|
65
|
-
const configuredMaxChars = Number(process.env.PI_KNOWLEDGE_EMBEDDING_MAX_CHARS ?? DEFAULT_API_MAX_EMBED_CHARS);
|
|
66
|
-
const maxChars = Number.isFinite(configuredMaxChars) && configuredMaxChars > 0 ? configuredMaxChars : DEFAULT_API_MAX_EMBED_CHARS;
|
|
127
|
+
async function embedViaAPI(texts, prefix, config, signal) {
|
|
67
128
|
const prefixedTexts = texts.map((t) => `${prefix}: ${t}`);
|
|
68
|
-
const safeTexts = prefixedTexts.map((text) =>
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
const detail = await res.text().catch(() => "");
|
|
85
|
-
throw new Error(`OpenAI embedding API error: ${res.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
|
|
86
|
-
}
|
|
87
|
-
const data = (await res.json());
|
|
88
|
-
return data.data.map((d) => new Float32Array(d.embedding));
|
|
129
|
+
const safeTexts = prefixedTexts.map((text) => text.length > config.maxChars ? text.slice(0, config.maxChars) : text);
|
|
130
|
+
const apiKey = process.env.OPENAI_API_KEY;
|
|
131
|
+
if (!apiKey)
|
|
132
|
+
throw new Error("OPENAI_API_KEY required for openai embedding");
|
|
133
|
+
if (!config.baseUrl)
|
|
134
|
+
throw new Error("OpenAI embedding base URL is not configured");
|
|
135
|
+
const endpoint = new URL("embeddings", `${config.baseUrl.replace(/\/+$/, "")}/`);
|
|
136
|
+
const res = await fetch(endpoint, {
|
|
137
|
+
method: "POST",
|
|
138
|
+
headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
|
|
139
|
+
body: JSON.stringify({ input: safeTexts, model: config.model }),
|
|
140
|
+
signal,
|
|
141
|
+
});
|
|
142
|
+
if (!res.ok) {
|
|
143
|
+
const detail = await res.text().catch(() => "");
|
|
144
|
+
throw new Error(`OpenAI embedding API error: ${res.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
|
|
89
145
|
}
|
|
90
|
-
|
|
146
|
+
const data = (await res.json());
|
|
147
|
+
return data.data.map((d) => new Float32Array(d.embedding));
|
|
91
148
|
}
|
|
92
|
-
export async function
|
|
93
|
-
|
|
149
|
+
export async function embedTextsWithConfig(texts, prefix, signal, options = {}) {
|
|
150
|
+
const config = resolveEmbeddingConfig();
|
|
151
|
+
let actualConfig = config;
|
|
152
|
+
if (config.provider === "openai") {
|
|
94
153
|
if (signal?.aborted)
|
|
95
154
|
throw new Error("Cancelled");
|
|
96
155
|
try {
|
|
97
|
-
return await embedViaAPI(texts, prefix, signal);
|
|
156
|
+
return { vectors: await embedViaAPI(texts, prefix, config, signal), config };
|
|
98
157
|
}
|
|
99
158
|
catch (error) {
|
|
100
159
|
if (signal?.aborted || isAbortError(error))
|
|
101
160
|
throw new Error("Cancelled");
|
|
102
|
-
if (!API_FALLBACK_TO_LOCAL)
|
|
161
|
+
if (!API_FALLBACK_TO_LOCAL || !options.allowApiFallback)
|
|
103
162
|
throw error;
|
|
104
163
|
console.warn(`pi-knowledge: embedding API failed; falling back to local model because PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local (${error instanceof Error ? error.message : String(error)})`);
|
|
164
|
+
actualConfig = localEmbeddingConfig(config.maxChars);
|
|
105
165
|
}
|
|
106
166
|
}
|
|
107
167
|
beginRun();
|
|
108
168
|
try {
|
|
109
|
-
return await embedInModelWorker(texts, prefix, signal);
|
|
169
|
+
return { vectors: await embedInModelWorker(texts, prefix, signal), config: actualConfig };
|
|
110
170
|
}
|
|
111
171
|
finally {
|
|
112
172
|
endRun();
|
|
113
173
|
}
|
|
114
174
|
}
|
|
175
|
+
export async function embedTexts(texts, prefix, signal, options = {}) {
|
|
176
|
+
return (await embedTextsWithConfig(texts, prefix, signal, options)).vectors;
|
|
177
|
+
}
|
|
178
|
+
export async function embedQueryWithConfig(text, signal) {
|
|
179
|
+
const { vectors, config } = await embedTextsWithConfig([text], "query", signal, { allowApiFallback: true });
|
|
180
|
+
return { vector: vectors[0], config };
|
|
181
|
+
}
|
|
115
182
|
export async function embedQuery(text, signal) {
|
|
116
|
-
|
|
117
|
-
return vec;
|
|
183
|
+
return (await embedQueryWithConfig(text, signal)).vector;
|
|
118
184
|
}
|
|
119
185
|
export async function embedDocuments(texts, signal) {
|
|
120
|
-
return embedTexts(texts, "passage", signal);
|
|
186
|
+
return embedTexts(texts, "passage", signal, { allowApiFallback: false });
|
|
121
187
|
}
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -36,7 +36,7 @@ export interface SearchResult {
|
|
|
36
36
|
source_chunk_ids?: string[];
|
|
37
37
|
};
|
|
38
38
|
}
|
|
39
|
-
export
|
|
39
|
+
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.ts";
|
|
40
40
|
export interface SearchResponse {
|
|
41
41
|
results: SearchResult[];
|
|
42
42
|
total_count: number;
|
package/dist/src/engine.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { existsSync, readFileSync, renameSync, rmSync, statSync } from "node:fs";
|
|
2
2
|
import { join, resolve } from "node:path";
|
|
3
3
|
import { diagnoseKB } from "./diagnostics/health.js";
|
|
4
|
-
import { dispose as disposeEmbedding, embedDocuments,
|
|
4
|
+
import { dispose as disposeEmbedding, embedDocuments, embeddingConfigLabel, embeddingSignature, embedQueryWithConfig, prepareForShutdown as prepareEmbeddingForShutdown, resolveEmbeddingConfig, } from "./embedding/provider.js";
|
|
5
5
|
import { openVectorReader, openVectorWriter } from "./embedding/vectors.js";
|
|
6
6
|
import { addSkippedScanEntry, buildChunkEmbeddingText, chunkFile, chunkIdentityHash, createSkippedScanStats, isReadableTextFile, isSupportedDocumentFile, iterateScannableFiles, preTokenizeForFTS, summarizeSkippedScan, } from "./indexer/chunker.js";
|
|
7
7
|
import { extractSymbols } from "./indexer/symbols.js";
|
|
@@ -13,8 +13,8 @@ import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence, normalizeFileTypeFilte
|
|
|
13
13
|
import { disposeReranker, prepareRerankerForShutdown, rerank } from "./search/reranker.js";
|
|
14
14
|
import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js";
|
|
15
15
|
import { searchVectorFile } from "./search/vector.js";
|
|
16
|
-
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBStatus, } from "./storage/sqlite.js";
|
|
17
|
-
export
|
|
16
|
+
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBStatus, } from "./storage/sqlite.js";
|
|
17
|
+
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
|
|
18
18
|
const INDEX_EMBED_BATCH_SIZE = 64;
|
|
19
19
|
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
20
20
|
function isCancellationError(error) {
|
|
@@ -456,6 +456,24 @@ async function extractScannableFileContentOrSkip(file, skipped, signal) {
|
|
|
456
456
|
return undefined;
|
|
457
457
|
}
|
|
458
458
|
}
|
|
459
|
+
function persistEmbeddingMetadata(db, kbId, config, vectors) {
|
|
460
|
+
const firstVector = vectors[0];
|
|
461
|
+
if (!firstVector)
|
|
462
|
+
return undefined;
|
|
463
|
+
const dimension = firstVector.length;
|
|
464
|
+
updateKBEmbeddingMetadata(db, kbId, embeddingConfigLabel(config), embeddingSignature(config, dimension), dimension);
|
|
465
|
+
return dimension;
|
|
466
|
+
}
|
|
467
|
+
function embeddingMismatchWarning(kb) {
|
|
468
|
+
return `"${kb.name}" has incompatible embedding metadata; vector retrieval was skipped and knowledge_update should rebuild it`;
|
|
469
|
+
}
|
|
470
|
+
function canSearchVectors(kb, config, queryDimension) {
|
|
471
|
+
if (kb.embedding_signature === null || kb.embedding_dimension === null)
|
|
472
|
+
return false;
|
|
473
|
+
if (kb.embedding_dimension !== queryDimension)
|
|
474
|
+
return false;
|
|
475
|
+
return kb.embedding_signature === embeddingSignature(config, queryDimension);
|
|
476
|
+
}
|
|
459
477
|
function throwIfAborted(signal) {
|
|
460
478
|
if (signal?.aborted)
|
|
461
479
|
throw new Error("Cancelled");
|
|
@@ -617,11 +635,13 @@ export class KnowledgeEngine {
|
|
|
617
635
|
if (existingKB) {
|
|
618
636
|
throw new Error(`Knowledge base "${name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before adding a replacement.`);
|
|
619
637
|
}
|
|
638
|
+
const embeddingConfig = resolveEmbeddingConfig();
|
|
620
639
|
const kb = createKB(db, {
|
|
621
640
|
name,
|
|
622
641
|
source_path: isDir || isFile ? resolvedSource : isUrl ? source : undefined,
|
|
623
642
|
source_type: sourceType,
|
|
624
643
|
source_options: serializeAddOptions(options),
|
|
644
|
+
embedding_model: embeddingConfigLabel(embeddingConfig),
|
|
625
645
|
});
|
|
626
646
|
updateKBStatus(db, kb.id, "indexing");
|
|
627
647
|
startIndexingJob(db, kb.id, "add", `Starting indexing for "${name}"`);
|
|
@@ -676,6 +696,7 @@ export class KnowledgeEngine {
|
|
|
676
696
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
677
697
|
if (signal?.aborted)
|
|
678
698
|
throw new Error("Cancelled");
|
|
699
|
+
persistEmbeddingMetadata(db, kb.id, embeddingConfig, vectors);
|
|
679
700
|
insertChunks(db, kb.id, batch);
|
|
680
701
|
writer.append(vectors);
|
|
681
702
|
chunkCount += batch.length;
|
|
@@ -832,6 +853,10 @@ export class KnowledgeEngine {
|
|
|
832
853
|
updateKBStatus(this.db, kb.id, "indexing");
|
|
833
854
|
startIndexingJob(this.db, kb.id, "update", `Starting update for "${kb.name}"`);
|
|
834
855
|
const scanOptions = toScanOptions(parseAddOptions(kb.source_options));
|
|
856
|
+
const embeddingConfig = resolveEmbeddingConfig();
|
|
857
|
+
const embeddingModel = embeddingConfigLabel(embeddingConfig);
|
|
858
|
+
const currentSignature = kb.embedding_dimension === null ? undefined : embeddingSignature(embeddingConfig, kb.embedding_dimension);
|
|
859
|
+
const canReuseExistingVectors = kb.embedding_signature !== null && kb.embedding_dimension !== null && kb.embedding_signature === currentSignature;
|
|
835
860
|
let replacementVectorPath;
|
|
836
861
|
let addedVectorPath;
|
|
837
862
|
let addedVectorWriter;
|
|
@@ -851,6 +876,14 @@ export class KnowledgeEngine {
|
|
|
851
876
|
existingHashes.set(chunk.content_hash, entries);
|
|
852
877
|
existingIndex++;
|
|
853
878
|
}
|
|
879
|
+
const reusableHashes = canReuseExistingVectors
|
|
880
|
+
? existingHashes
|
|
881
|
+
: new Map();
|
|
882
|
+
if (!canReuseExistingVectors) {
|
|
883
|
+
const message = `Embedding metadata changed or missing for "${kb.name}"; rebuilding all vectors`;
|
|
884
|
+
updateIndexingJob(this.db, kb.id, { phase: "embedding", message });
|
|
885
|
+
onProgress?.(message);
|
|
886
|
+
}
|
|
854
887
|
const oldVectorIndexByHash = new Map();
|
|
855
888
|
const newVectorIndexByHash = new Map();
|
|
856
889
|
const pendingChunks = [];
|
|
@@ -860,6 +893,7 @@ export class KnowledgeEngine {
|
|
|
860
893
|
let scannedFiles = 0;
|
|
861
894
|
let scannedChunks = 0;
|
|
862
895
|
let plannedTotalFiles;
|
|
896
|
+
let finalEmbeddingDimension;
|
|
863
897
|
const startedAt = Date.now();
|
|
864
898
|
const flushPending = async () => {
|
|
865
899
|
if (!this.db || !addedVectorWriter || pendingChunks.length === 0)
|
|
@@ -907,7 +941,7 @@ export class KnowledgeEngine {
|
|
|
907
941
|
const processChunks = async (chunks) => {
|
|
908
942
|
for (const chunk of chunks) {
|
|
909
943
|
scannedChunks++;
|
|
910
|
-
const existing =
|
|
944
|
+
const existing = reusableHashes.get(chunk.content_hash);
|
|
911
945
|
if (existing && existing.length > 0) {
|
|
912
946
|
const retained = existing.shift();
|
|
913
947
|
if (retained) {
|
|
@@ -1034,6 +1068,7 @@ export class KnowledgeEngine {
|
|
|
1034
1068
|
: undefined;
|
|
1035
1069
|
if (!vector)
|
|
1036
1070
|
throw new Error(`Missing vector while rebuilding knowledge base: ${chunk.id}`);
|
|
1071
|
+
finalEmbeddingDimension ??= vector.length;
|
|
1037
1072
|
vectorWriter.append([vector]);
|
|
1038
1073
|
finalChunkCount++;
|
|
1039
1074
|
if (finalChunkCount % 1_000 === 0) {
|
|
@@ -1065,6 +1100,7 @@ export class KnowledgeEngine {
|
|
|
1065
1100
|
deleteChunksByIds(this.db, idsToRemove);
|
|
1066
1101
|
deleteSymbolsByKB(this.db, kb.id);
|
|
1067
1102
|
insertSymbols(this.db, kb.id, stagedSymbols);
|
|
1103
|
+
updateKBEmbeddingMetadata(this.db, kb.id, embeddingModel, finalEmbeddingDimension === undefined ? null : embeddingSignature(embeddingConfig, finalEmbeddingDimension), finalEmbeddingDimension ?? null);
|
|
1068
1104
|
updateKBCounts(this.db, kb.id, finalChunkCount, getFileCount(this.db, kb.id));
|
|
1069
1105
|
updateKBStatus(this.db, kb.id, "ready");
|
|
1070
1106
|
const skippedReadySuffix = latestSkippedTotal > 0 ? `, skipped ${latestSkippedTotal} (${latestSkippedSummary})` : "";
|
|
@@ -1184,9 +1220,6 @@ export class KnowledgeEngine {
|
|
|
1184
1220
|
const vectorsByChunkId = new Map();
|
|
1185
1221
|
for (const kb of kbs) {
|
|
1186
1222
|
throwIfAborted(signal);
|
|
1187
|
-
if (kb.embedding_model !== CURRENT_EMBEDDING_MODEL) {
|
|
1188
|
-
warnings.push(`"${kb.name}" was indexed with ${kb.embedding_model} (current: ${CURRENT_EMBEDDING_MODEL}) — run knowledge_update for best results`);
|
|
1189
|
-
}
|
|
1190
1223
|
if (kb.chunk_count === 0)
|
|
1191
1224
|
continue;
|
|
1192
1225
|
const vectorPath = this.vectorPathFor(kb.id);
|
|
@@ -1197,25 +1230,33 @@ export class KnowledgeEngine {
|
|
|
1197
1230
|
})));
|
|
1198
1231
|
}
|
|
1199
1232
|
else if (retrievalMode === "semantic") {
|
|
1200
|
-
const queryVec = await
|
|
1233
|
+
const { vector: queryVec, config: queryEmbeddingConfig } = await embedQueryWithConfig(query, signal);
|
|
1201
1234
|
throwIfAborted(signal);
|
|
1235
|
+
if (!canSearchVectors(kb, queryEmbeddingConfig, queryVec.length)) {
|
|
1236
|
+
warnings.push(embeddingMismatchWarning(kb));
|
|
1237
|
+
continue;
|
|
1238
|
+
}
|
|
1202
1239
|
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id), candidateLimit, signal);
|
|
1203
1240
|
allResults.push(...vectorResults.results.map((result) => ({ ...result, score: result.score * kbTrustMultiplier(kb) })));
|
|
1204
1241
|
for (const [chunkId, vector] of vectorResults.vectorsByChunkId)
|
|
1205
1242
|
vectorsByChunkId.set(chunkId, vector);
|
|
1206
1243
|
}
|
|
1207
1244
|
else {
|
|
1208
|
-
// hybrid: BM25 + vector weighted fusion (both scoped to this KB)
|
|
1209
1245
|
const bm25Results = searchBM25(db, normalizedQuery || query, candidateLimit, kb.id);
|
|
1210
1246
|
if (bm25Results.length === 0)
|
|
1211
1247
|
continue;
|
|
1212
1248
|
let vecResults = [];
|
|
1213
|
-
const queryVec = await
|
|
1249
|
+
const { vector: queryVec, config: queryEmbeddingConfig } = await embedQueryWithConfig(query, signal);
|
|
1214
1250
|
throwIfAborted(signal);
|
|
1215
|
-
|
|
1216
|
-
|
|
1217
|
-
|
|
1218
|
-
|
|
1251
|
+
if (canSearchVectors(kb, queryEmbeddingConfig, queryVec.length)) {
|
|
1252
|
+
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id), candidateLimit, signal);
|
|
1253
|
+
vecResults = vectorResults.results;
|
|
1254
|
+
for (const [chunkId, vector] of vectorResults.vectorsByChunkId)
|
|
1255
|
+
vectorsByChunkId.set(chunkId, vector);
|
|
1256
|
+
}
|
|
1257
|
+
else {
|
|
1258
|
+
warnings.push(embeddingMismatchWarning(kb));
|
|
1259
|
+
}
|
|
1219
1260
|
const fused = weightedScoreFusion(bm25Results, vecResults);
|
|
1220
1261
|
allResults.push(...fused.map((result) => ({ ...result, score: result.score * kbTrustMultiplier(kb) })));
|
|
1221
1262
|
}
|
|
@@ -1679,6 +1720,7 @@ export class KnowledgeEngine {
|
|
|
1679
1720
|
const { createInterface } = await import("node:readline");
|
|
1680
1721
|
const stream = createReadStream(inputPath, { encoding: "utf-8" });
|
|
1681
1722
|
const lines = createInterface({ input: stream, crlfDelay: Infinity });
|
|
1723
|
+
const embeddingConfig = resolveEmbeddingConfig();
|
|
1682
1724
|
let header;
|
|
1683
1725
|
let kb;
|
|
1684
1726
|
let vectorWriter;
|
|
@@ -1705,6 +1747,7 @@ export class KnowledgeEngine {
|
|
|
1705
1747
|
onProgress?.(message);
|
|
1706
1748
|
const vectors = await embedDocuments(batch.map((chunk) => buildChunkEmbeddingText(chunk)), signal);
|
|
1707
1749
|
throwIfAborted(signal);
|
|
1750
|
+
persistEmbeddingMetadata(this.db, kb.id, embeddingConfig, vectors);
|
|
1708
1751
|
insertChunks(this.db, kb.id, batch);
|
|
1709
1752
|
insertSymbols(this.db, kb.id, symbols);
|
|
1710
1753
|
vectorWriter.append(vectors);
|
|
@@ -1733,7 +1776,12 @@ export class KnowledgeEngine {
|
|
|
1733
1776
|
if (existingKB) {
|
|
1734
1777
|
throw new Error(`Knowledge base "${header.name}" already exists. Use knowledge_update to refresh it, or knowledge_remove before importing a replacement.`);
|
|
1735
1778
|
}
|
|
1736
|
-
kb = createKB(this.db, {
|
|
1779
|
+
kb = createKB(this.db, {
|
|
1780
|
+
name: header.name,
|
|
1781
|
+
description: header.description,
|
|
1782
|
+
source_type: "text",
|
|
1783
|
+
embedding_model: embeddingConfigLabel(embeddingConfig),
|
|
1784
|
+
});
|
|
1737
1785
|
updateKBStatus(this.db, kb.id, "indexing");
|
|
1738
1786
|
startIndexingJob(this.db, kb.id, "import", `Starting import for "${header.name}"`);
|
|
1739
1787
|
const importMessage = `Importing ${header.chunk_count ?? "unknown"} chunks...`;
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import type { HfRerankerConfig } from "./search/reranker-config.ts";
|
|
1
2
|
export declare function validateModelWorkerNodePath(nodePath: string): string;
|
|
2
3
|
export declare function resolveModelWorkerNodePath(): string;
|
|
3
4
|
export declare function configureModelWorkerNodePath(nodePath?: string): string;
|
|
@@ -6,7 +7,7 @@ export interface RerankWorkerCandidate {
|
|
|
6
7
|
chunkId: string;
|
|
7
8
|
content: string;
|
|
8
9
|
}
|
|
9
|
-
export declare function rerankInModelWorker(query: string, candidates: RerankWorkerCandidate[], topK: number, signal?: AbortSignal): Promise<Array<{
|
|
10
|
+
export declare function rerankInModelWorker(query: string, candidates: RerankWorkerCandidate[], topK: number, reranker: HfRerankerConfig, signal?: AbortSignal): Promise<Array<{
|
|
10
11
|
chunkId: string;
|
|
11
12
|
score: number;
|
|
12
13
|
}>>;
|
|
@@ -395,8 +395,8 @@ export async function embedInModelWorker(texts, prefix, signal) {
|
|
|
395
395
|
return new Float32Array(vector);
|
|
396
396
|
});
|
|
397
397
|
}
|
|
398
|
-
export async function rerankInModelWorker(query, candidates, topK, signal) {
|
|
399
|
-
const result = await getWorker().request({ type: "rerank", query, candidates, topK }, signal);
|
|
398
|
+
export async function rerankInModelWorker(query, candidates, topK, reranker, signal) {
|
|
399
|
+
const result = await getWorker().request({ type: "rerank", query, candidates, topK, reranker }, signal);
|
|
400
400
|
if (!Array.isArray(result))
|
|
401
401
|
throw new Error("Invalid reranker worker response");
|
|
402
402
|
return result.map((item) => {
|
package/dist/src/model-worker.js
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
import { join } from "node:path";
|
|
2
2
|
import { createInterface } from "node:readline";
|
|
3
|
+
import { DEFAULT_RERANKER_REMOTE_HOST, DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE, isHfRerankerConfig, rerankerCacheKey, resolveRerankerConfig, } from "./search/reranker-config.js";
|
|
4
|
+
import { rerankWithSingleRawLogit } from "./search/reranker-logits.js";
|
|
3
5
|
import { getDefaultKnowledgeDir } from "./storage/sqlite.js";
|
|
4
6
|
let embeddingPipeline = null;
|
|
5
7
|
let rerankerPipeline = null;
|
|
@@ -14,11 +16,16 @@ function configureTransformersEnv(env) {
|
|
|
14
16
|
env.localModelPath = cacheDir;
|
|
15
17
|
}
|
|
16
18
|
}
|
|
19
|
+
function configureEmbeddingTransformersEnv(env) {
|
|
20
|
+
configureTransformersEnv(env);
|
|
21
|
+
env.remoteHost = DEFAULT_RERANKER_REMOTE_HOST;
|
|
22
|
+
env.remotePathTemplate = DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE;
|
|
23
|
+
}
|
|
17
24
|
async function loadEmbeddingPipeline() {
|
|
18
25
|
if (embeddingPipeline)
|
|
19
26
|
return embeddingPipeline;
|
|
20
27
|
const { pipeline, env } = await import("@huggingface/transformers");
|
|
21
|
-
|
|
28
|
+
configureEmbeddingTransformersEnv(env);
|
|
22
29
|
const createPipeline = pipeline;
|
|
23
30
|
const loaded = (await createPipeline("feature-extraction", "Xenova/multilingual-e5-small", {
|
|
24
31
|
quantized: true,
|
|
@@ -27,14 +34,36 @@ async function loadEmbeddingPipeline() {
|
|
|
27
34
|
embeddingPipeline = loaded;
|
|
28
35
|
return loaded;
|
|
29
36
|
}
|
|
30
|
-
async function loadRerankerPipeline() {
|
|
31
|
-
|
|
32
|
-
|
|
37
|
+
async function loadRerankerPipeline(config) {
|
|
38
|
+
const key = rerankerCacheKey(config);
|
|
39
|
+
if (rerankerPipeline?.key === key)
|
|
40
|
+
return rerankerPipeline.pipe;
|
|
41
|
+
if (config.rawLogits) {
|
|
42
|
+
const { AutoTokenizer, AutoModelForSequenceClassification, env } = await import("@huggingface/transformers");
|
|
43
|
+
const transformersEnv = env;
|
|
44
|
+
configureTransformersEnv(transformersEnv);
|
|
45
|
+
transformersEnv.remoteHost = config.remoteHost ?? DEFAULT_RERANKER_REMOTE_HOST;
|
|
46
|
+
transformersEnv.remotePathTemplate = config.remotePathTemplate ?? DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE;
|
|
47
|
+
const loadOpts = { revision: config.revision };
|
|
48
|
+
if (config.dtype)
|
|
49
|
+
loadOpts.dtype = config.dtype;
|
|
50
|
+
const tokenizer = (await AutoTokenizer.from_pretrained(config.model, loadOpts));
|
|
51
|
+
const model = (await AutoModelForSequenceClassification.from_pretrained(config.model, loadOpts));
|
|
52
|
+
const pipe = (input) => rerankWithSingleRawLogit(input, tokenizer, model, config.model);
|
|
53
|
+
rerankerPipeline = { key, pipe };
|
|
54
|
+
return pipe;
|
|
55
|
+
}
|
|
33
56
|
const { pipeline, env } = await import("@huggingface/transformers");
|
|
34
|
-
|
|
57
|
+
const transformersEnv = env;
|
|
58
|
+
configureTransformersEnv(transformersEnv);
|
|
59
|
+
transformersEnv.remoteHost = config.remoteHost ?? DEFAULT_RERANKER_REMOTE_HOST;
|
|
60
|
+
transformersEnv.remotePathTemplate = config.remotePathTemplate ?? DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE;
|
|
35
61
|
const createPipeline = pipeline;
|
|
36
|
-
const
|
|
37
|
-
|
|
62
|
+
const options = { revision: config.revision };
|
|
63
|
+
if (config.dtype)
|
|
64
|
+
options.dtype = config.dtype;
|
|
65
|
+
const loaded = (await createPipeline("text-classification", config.model, options));
|
|
66
|
+
rerankerPipeline = { key, pipe: loaded };
|
|
38
67
|
return loaded;
|
|
39
68
|
}
|
|
40
69
|
async function handleEmbed(request) {
|
|
@@ -47,7 +76,7 @@ async function handleEmbed(request) {
|
|
|
47
76
|
return vectors;
|
|
48
77
|
}
|
|
49
78
|
async function handleRerank(request) {
|
|
50
|
-
const pipe = await loadRerankerPipeline();
|
|
79
|
+
const pipe = await loadRerankerPipeline(request.reranker);
|
|
51
80
|
const results = [];
|
|
52
81
|
for (const candidate of request.candidates) {
|
|
53
82
|
const output = await pipe({ text: request.query, text_pair: candidate.content });
|
|
@@ -91,12 +120,16 @@ function parseWorkerRequest(message) {
|
|
|
91
120
|
if (!("candidates" in message) || !isRerankCandidateArray(message.candidates)) {
|
|
92
121
|
throw new Error("Invalid rerank candidates");
|
|
93
122
|
}
|
|
123
|
+
const reranker = "reranker" in message && isHfRerankerConfig(message.reranker) ? message.reranker : resolveRerankerConfig();
|
|
124
|
+
if (reranker.provider !== "hf")
|
|
125
|
+
throw new Error("Reranker API mode is not supported by the model worker");
|
|
94
126
|
return {
|
|
95
127
|
id: message.id,
|
|
96
128
|
type: "rerank",
|
|
97
129
|
query: message.query,
|
|
98
130
|
candidates: message.candidates,
|
|
99
131
|
topK: message.topK,
|
|
132
|
+
reranker,
|
|
100
133
|
};
|
|
101
134
|
}
|
|
102
135
|
throw new Error("Unsupported worker request type");
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
import type { RerankCandidate } from "./reranker.ts";
|
|
2
|
+
import type { ApiRerankerConfig } from "./reranker-config.ts";
|
|
3
|
+
export interface ApiRerankResult {
|
|
4
|
+
chunkId: string;
|
|
5
|
+
score: number;
|
|
6
|
+
}
|
|
7
|
+
export declare function rerankViaApi(query: string, candidates: RerankCandidate[], topK: number, config: ApiRerankerConfig, signal?: AbortSignal): Promise<ApiRerankResult[]>;
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
function valueAtPath(value, path) {
|
|
2
|
+
let current = value;
|
|
3
|
+
for (const segment of path.split(".")) {
|
|
4
|
+
if (segment.length === 0)
|
|
5
|
+
continue;
|
|
6
|
+
if (typeof current !== "object" || current === null)
|
|
7
|
+
return undefined;
|
|
8
|
+
current = current[segment];
|
|
9
|
+
}
|
|
10
|
+
return current;
|
|
11
|
+
}
|
|
12
|
+
function createAbortSignal(timeoutMs, signal) {
|
|
13
|
+
const controller = new AbortController();
|
|
14
|
+
const abortFromParent = () => controller.abort(signal?.reason);
|
|
15
|
+
if (signal?.aborted)
|
|
16
|
+
abortFromParent();
|
|
17
|
+
else
|
|
18
|
+
signal?.addEventListener("abort", abortFromParent, { once: true });
|
|
19
|
+
const timeout = setTimeout(() => controller.abort(new Error(`Reranker API timed out after ${timeoutMs}ms`)), timeoutMs);
|
|
20
|
+
return {
|
|
21
|
+
signal: controller.signal,
|
|
22
|
+
cleanup() {
|
|
23
|
+
clearTimeout(timeout);
|
|
24
|
+
signal?.removeEventListener("abort", abortFromParent);
|
|
25
|
+
},
|
|
26
|
+
};
|
|
27
|
+
}
|
|
28
|
+
function mapApiResult(result, candidates, config) {
|
|
29
|
+
const index = valueAtPath(result, config.indexField);
|
|
30
|
+
const score = valueAtPath(result, config.scoreField);
|
|
31
|
+
if (typeof index !== "number" || !Number.isInteger(index) || index < 0 || index >= candidates.length) {
|
|
32
|
+
throw new Error("Invalid reranker API response: result index is out of range");
|
|
33
|
+
}
|
|
34
|
+
if (typeof score !== "number" || !Number.isFinite(score)) {
|
|
35
|
+
throw new Error("Invalid reranker API response: result score is not a number");
|
|
36
|
+
}
|
|
37
|
+
return { chunkId: candidates[index].chunkId, score };
|
|
38
|
+
}
|
|
39
|
+
function requestBody(query, candidates, topK, config) {
|
|
40
|
+
const documents = candidates.map((candidate) => {
|
|
41
|
+
const text = candidate.content.length > config.maxDocumentChars
|
|
42
|
+
? candidate.content.slice(0, config.maxDocumentChars)
|
|
43
|
+
: candidate.content;
|
|
44
|
+
return config.format === "custom-json" ? { id: candidate.chunkId, text } : text;
|
|
45
|
+
});
|
|
46
|
+
return JSON.stringify({
|
|
47
|
+
model: config.model,
|
|
48
|
+
query,
|
|
49
|
+
documents,
|
|
50
|
+
top_n: topK,
|
|
51
|
+
return_documents: false,
|
|
52
|
+
});
|
|
53
|
+
}
|
|
54
|
+
export async function rerankViaApi(query, candidates, topK, config, signal) {
|
|
55
|
+
const abort = createAbortSignal(config.timeoutMs, signal);
|
|
56
|
+
try {
|
|
57
|
+
const headers = { "Content-Type": "application/json" };
|
|
58
|
+
if (config.apiKey)
|
|
59
|
+
headers.Authorization = `Bearer ${config.apiKey}`;
|
|
60
|
+
const response = await fetch(config.endpoint, {
|
|
61
|
+
method: "POST",
|
|
62
|
+
headers,
|
|
63
|
+
body: requestBody(query, candidates, topK, config),
|
|
64
|
+
signal: abort.signal,
|
|
65
|
+
});
|
|
66
|
+
if (!response.ok) {
|
|
67
|
+
const detail = await response.text().catch(() => "");
|
|
68
|
+
throw new Error(`Reranker API error: ${response.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
|
|
69
|
+
}
|
|
70
|
+
const data = (await response.json());
|
|
71
|
+
const results = valueAtPath(data, config.resultsPath);
|
|
72
|
+
if (!Array.isArray(results))
|
|
73
|
+
throw new Error("Invalid reranker API response: results field is not an array");
|
|
74
|
+
const ranked = results.map((result) => mapApiResult(result, candidates, config));
|
|
75
|
+
if (config.scoreDirection === "asc")
|
|
76
|
+
ranked.sort((a, b) => a.score - b.score);
|
|
77
|
+
else
|
|
78
|
+
ranked.sort((a, b) => b.score - a.score);
|
|
79
|
+
return ranked.slice(0, topK);
|
|
80
|
+
}
|
|
81
|
+
catch (error) {
|
|
82
|
+
if (signal?.aborted || (error instanceof DOMException && error.name === "AbortError"))
|
|
83
|
+
throw new Error("Cancelled");
|
|
84
|
+
throw error;
|
|
85
|
+
}
|
|
86
|
+
finally {
|
|
87
|
+
abort.cleanup();
|
|
88
|
+
}
|
|
89
|
+
}
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
export type RerankerProvider = "hf" | "api";
|
|
2
|
+
export type ApiRerankerFormat = "cohere" | "jina" | "custom-json";
|
|
3
|
+
export type ApiScoreDirection = "desc" | "asc";
|
|
4
|
+
export interface HfRerankerConfig {
|
|
5
|
+
provider: "hf";
|
|
6
|
+
model: string;
|
|
7
|
+
revision: string;
|
|
8
|
+
dtype?: string;
|
|
9
|
+
remoteHost?: string;
|
|
10
|
+
remotePathTemplate?: string;
|
|
11
|
+
/** When true, use raw logits from the model instead of pipeline sigmoid scores.
|
|
12
|
+
* Essential for cross-encoder models (e.g. bge-reranker) whose logits saturate through sigmoid. */
|
|
13
|
+
rawLogits?: boolean;
|
|
14
|
+
}
|
|
15
|
+
export interface ApiRerankerConfig {
|
|
16
|
+
provider: "api";
|
|
17
|
+
model: string;
|
|
18
|
+
endpoint: string;
|
|
19
|
+
apiKey?: string;
|
|
20
|
+
format: ApiRerankerFormat;
|
|
21
|
+
timeoutMs: number;
|
|
22
|
+
maxDocumentChars: number;
|
|
23
|
+
resultsPath: string;
|
|
24
|
+
indexField: string;
|
|
25
|
+
scoreField: string;
|
|
26
|
+
scoreDirection: ApiScoreDirection;
|
|
27
|
+
}
|
|
28
|
+
export type RerankerConfig = HfRerankerConfig | ApiRerankerConfig;
|
|
29
|
+
export declare const DEFAULT_RERANKER_MODEL = "Xenova/ms-marco-MiniLM-L-4-v2";
|
|
30
|
+
export declare const DEFAULT_RERANKER_REVISION = "main";
|
|
31
|
+
export declare const DEFAULT_RERANKER_REMOTE_HOST = "https://huggingface.co/";
|
|
32
|
+
export declare const DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE = "{model}/resolve/{revision}/";
|
|
33
|
+
export declare const DEFAULT_RERANKER_API_TIMEOUT_MS = 30000;
|
|
34
|
+
export declare const DEFAULT_RERANKER_MAX_DOC_CHARS = 12000;
|
|
35
|
+
export declare function resolveRerankerConfig(env?: NodeJS.ProcessEnv): RerankerConfig;
|
|
36
|
+
export declare function isHfRerankerConfig(value: unknown): value is HfRerankerConfig;
|
|
37
|
+
export declare function rerankerCacheKey(config: HfRerankerConfig): string;
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
export const DEFAULT_RERANKER_MODEL = "Xenova/ms-marco-MiniLM-L-4-v2";
|
|
2
|
+
export const DEFAULT_RERANKER_REVISION = "main";
|
|
3
|
+
export const DEFAULT_RERANKER_REMOTE_HOST = "https://huggingface.co/";
|
|
4
|
+
export const DEFAULT_RERANKER_REMOTE_PATH_TEMPLATE = "{model}/resolve/{revision}/";
|
|
5
|
+
export const DEFAULT_RERANKER_API_TIMEOUT_MS = 30_000;
|
|
6
|
+
export const DEFAULT_RERANKER_MAX_DOC_CHARS = 12_000;
|
|
7
|
+
function cleanEnv(value) {
|
|
8
|
+
const trimmed = value?.trim();
|
|
9
|
+
return trimmed ? trimmed : undefined;
|
|
10
|
+
}
|
|
11
|
+
function stripProviderPrefix(raw) {
|
|
12
|
+
const separator = raw.indexOf(":");
|
|
13
|
+
if (separator <= 0)
|
|
14
|
+
return { provider: undefined, value: raw };
|
|
15
|
+
const prefix = raw.slice(0, separator).toLowerCase();
|
|
16
|
+
if (prefix !== "hf" && prefix !== "api")
|
|
17
|
+
return { provider: undefined, value: raw };
|
|
18
|
+
return { provider: prefix, value: raw.slice(separator + 1) };
|
|
19
|
+
}
|
|
20
|
+
function parseHuggingFaceUrl(raw) {
|
|
21
|
+
let url;
|
|
22
|
+
try {
|
|
23
|
+
url = new URL(raw);
|
|
24
|
+
}
|
|
25
|
+
catch {
|
|
26
|
+
return undefined;
|
|
27
|
+
}
|
|
28
|
+
if (url.protocol !== "https:" && url.protocol !== "http:") {
|
|
29
|
+
throw new Error(`Unsupported reranker model URL protocol: ${url.protocol}`);
|
|
30
|
+
}
|
|
31
|
+
const parts = url.pathname.split("/").filter(Boolean);
|
|
32
|
+
if (parts.length < 2)
|
|
33
|
+
throw new Error(`Invalid Hugging Face reranker URL: ${raw}`);
|
|
34
|
+
const model = `${decodeURIComponent(parts[0])}/${decodeURIComponent(parts[1])}`;
|
|
35
|
+
let revision;
|
|
36
|
+
const treeIndex = parts.indexOf("tree");
|
|
37
|
+
const resolveIndex = parts.indexOf("resolve");
|
|
38
|
+
if (treeIndex >= 0 && parts[treeIndex + 1])
|
|
39
|
+
revision = decodeURIComponent(parts[treeIndex + 1]);
|
|
40
|
+
if (resolveIndex >= 0 && parts[resolveIndex + 1])
|
|
41
|
+
revision = decodeURIComponent(parts[resolveIndex + 1]);
|
|
42
|
+
return { model, revision, remoteHost: `${url.origin}/` };
|
|
43
|
+
}
|
|
44
|
+
function resolveHfModel(raw) {
|
|
45
|
+
const parsedUrl = parseHuggingFaceUrl(raw);
|
|
46
|
+
if (parsedUrl)
|
|
47
|
+
return parsedUrl;
|
|
48
|
+
if (!raw.includes("/") || raw.startsWith("/") || raw.endsWith("/")) {
|
|
49
|
+
throw new Error(`Invalid reranker model id: ${raw}`);
|
|
50
|
+
}
|
|
51
|
+
return { model: raw };
|
|
52
|
+
}
|
|
53
|
+
function parseNumber(value, fallback, min, max) {
|
|
54
|
+
const parsed = Number(value);
|
|
55
|
+
if (!Number.isFinite(parsed))
|
|
56
|
+
return fallback;
|
|
57
|
+
return Math.min(Math.max(Math.trunc(parsed), min), max);
|
|
58
|
+
}
|
|
59
|
+
function normalizeApiFormat(value) {
|
|
60
|
+
if (value === undefined)
|
|
61
|
+
return "cohere";
|
|
62
|
+
const normalized = value.toLowerCase();
|
|
63
|
+
if (normalized === "cohere" || normalized === "jina" || normalized === "custom-json")
|
|
64
|
+
return normalized;
|
|
65
|
+
throw new Error(`Unsupported reranker API format: ${value}`);
|
|
66
|
+
}
|
|
67
|
+
function normalizeScoreDirection(value) {
|
|
68
|
+
if (value === undefined)
|
|
69
|
+
return "desc";
|
|
70
|
+
const normalized = value.toLowerCase();
|
|
71
|
+
if (normalized === "desc" || normalized === "asc")
|
|
72
|
+
return normalized;
|
|
73
|
+
throw new Error(`Unsupported reranker score direction: ${value}`);
|
|
74
|
+
}
|
|
75
|
+
function endpointFromBaseUrl(baseUrl) {
|
|
76
|
+
return new URL("rerank", `${baseUrl.replace(/\/+$/, "")}/`).toString();
|
|
77
|
+
}
|
|
78
|
+
function normalizeHttpUrl(raw) {
|
|
79
|
+
const url = new URL(raw);
|
|
80
|
+
if (url.protocol !== "http:" && url.protocol !== "https:") {
|
|
81
|
+
throw new Error(`Unsupported reranker API endpoint protocol: ${url.protocol}`);
|
|
82
|
+
}
|
|
83
|
+
return url.toString();
|
|
84
|
+
}
|
|
85
|
+
function resolveApiRerankerConfig(model, env) {
|
|
86
|
+
const endpoint = normalizeHttpUrl(cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_ENDPOINT) ??
|
|
87
|
+
endpointFromBaseUrl(cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_BASE_URL) ??
|
|
88
|
+
cleanEnv(env.OPENAI_BASE_URL) ??
|
|
89
|
+
"https://api.cohere.com/v2"));
|
|
90
|
+
return {
|
|
91
|
+
provider: "api",
|
|
92
|
+
model,
|
|
93
|
+
endpoint,
|
|
94
|
+
apiKey: cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_KEY) ?? cleanEnv(env.OPENAI_API_KEY),
|
|
95
|
+
format: normalizeApiFormat(cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_FORMAT)),
|
|
96
|
+
timeoutMs: parseNumber(cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS), DEFAULT_RERANKER_API_TIMEOUT_MS, 1_000, 300_000),
|
|
97
|
+
maxDocumentChars: parseNumber(cleanEnv(env.PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS), DEFAULT_RERANKER_MAX_DOC_CHARS, 100, 200_000),
|
|
98
|
+
resultsPath: cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH) ?? "results",
|
|
99
|
+
indexField: cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD) ?? "index",
|
|
100
|
+
scoreField: cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD) ?? "relevance_score",
|
|
101
|
+
scoreDirection: normalizeScoreDirection(cleanEnv(env.PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION)),
|
|
102
|
+
};
|
|
103
|
+
}
|
|
104
|
+
export function resolveRerankerConfig(env = process.env) {
|
|
105
|
+
const raw = cleanEnv(env.PI_KNOWLEDGE_RERANKER) ?? `hf:${DEFAULT_RERANKER_MODEL}`;
|
|
106
|
+
const { provider, value } = stripProviderPrefix(raw);
|
|
107
|
+
if (provider === "api") {
|
|
108
|
+
const model = value.trim();
|
|
109
|
+
if (!model)
|
|
110
|
+
throw new Error("PI_KNOWLEDGE_RERANKER=api:<model> requires a model name");
|
|
111
|
+
return resolveApiRerankerConfig(model, env);
|
|
112
|
+
}
|
|
113
|
+
if (provider !== undefined && provider !== "hf")
|
|
114
|
+
throw new Error(`Unsupported reranker provider: ${provider}`);
|
|
115
|
+
const modelSource = value.trim();
|
|
116
|
+
if (!modelSource)
|
|
117
|
+
throw new Error("PI_KNOWLEDGE_RERANKER requires a model id or URL");
|
|
118
|
+
const resolved = resolveHfModel(modelSource);
|
|
119
|
+
const revision = cleanEnv(env.PI_KNOWLEDGE_RERANKER_REVISION) ?? resolved.revision ?? DEFAULT_RERANKER_REVISION;
|
|
120
|
+
const rawLogits = cleanEnv(env.PI_KNOWLEDGE_RERANKER_RAW_LOGITS) === "true";
|
|
121
|
+
return {
|
|
122
|
+
provider: "hf",
|
|
123
|
+
model: resolved.model,
|
|
124
|
+
revision,
|
|
125
|
+
dtype: cleanEnv(env.PI_KNOWLEDGE_RERANKER_DTYPE),
|
|
126
|
+
remoteHost: cleanEnv(env.PI_KNOWLEDGE_RERANKER_REMOTE_HOST) ?? resolved.remoteHost,
|
|
127
|
+
remotePathTemplate: cleanEnv(env.PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE),
|
|
128
|
+
...(rawLogits ? { rawLogits: true } : {}),
|
|
129
|
+
};
|
|
130
|
+
}
|
|
131
|
+
export function isHfRerankerConfig(value) {
|
|
132
|
+
if (typeof value !== "object" || value === null)
|
|
133
|
+
return false;
|
|
134
|
+
const config = value;
|
|
135
|
+
return (config.provider === "hf" &&
|
|
136
|
+
typeof config.model === "string" &&
|
|
137
|
+
config.model.length > 0 &&
|
|
138
|
+
typeof config.revision === "string" &&
|
|
139
|
+
config.revision.length > 0 &&
|
|
140
|
+
(config.dtype === undefined || typeof config.dtype === "string") &&
|
|
141
|
+
(config.remoteHost === undefined || typeof config.remoteHost === "string") &&
|
|
142
|
+
(config.remotePathTemplate === undefined || typeof config.remotePathTemplate === "string") &&
|
|
143
|
+
(config.rawLogits === undefined || typeof config.rawLogits === "boolean"));
|
|
144
|
+
}
|
|
145
|
+
export function rerankerCacheKey(config) {
|
|
146
|
+
return [
|
|
147
|
+
config.provider,
|
|
148
|
+
config.model,
|
|
149
|
+
config.revision,
|
|
150
|
+
config.dtype ?? "",
|
|
151
|
+
config.remoteHost ?? "",
|
|
152
|
+
config.remotePathTemplate ?? "",
|
|
153
|
+
config.rawLogits ? "raw" : "sigmoid",
|
|
154
|
+
].join("\u0000");
|
|
155
|
+
}
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
export type RawLogitRerankInput = {
|
|
2
|
+
text: string;
|
|
3
|
+
text_pair: string;
|
|
4
|
+
};
|
|
5
|
+
export type RawLogitTokenizer = (text: string, options: {
|
|
6
|
+
text_pair: string;
|
|
7
|
+
padding: true;
|
|
8
|
+
truncation: true;
|
|
9
|
+
}) => Record<string, unknown>;
|
|
10
|
+
export type RawLogitModel = (inputs: Record<string, unknown>) => Promise<{
|
|
11
|
+
logits: {
|
|
12
|
+
data: ArrayLike<number>;
|
|
13
|
+
};
|
|
14
|
+
}>;
|
|
15
|
+
export declare function scoreSingleRawLogit(logits: {
|
|
16
|
+
data: ArrayLike<number>;
|
|
17
|
+
}, model: string): number;
|
|
18
|
+
export declare function rerankWithSingleRawLogit(input: RawLogitRerankInput, tokenizer: RawLogitTokenizer, model: RawLogitModel, modelName: string): Promise<Array<{
|
|
19
|
+
score: number;
|
|
20
|
+
}>>;
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
export function scoreSingleRawLogit(logits, model) {
|
|
2
|
+
const scores = Array.from(logits.data);
|
|
3
|
+
if (scores.length !== 1) {
|
|
4
|
+
throw new Error(`Raw-logit reranker expected a single-logit sequence classifier for ${model}, got ${scores.length} logits. ` +
|
|
5
|
+
"Use a single-logit reranker model or disable PI_KNOWLEDGE_RERANKER_RAW_LOGITS.");
|
|
6
|
+
}
|
|
7
|
+
const score = scores[0];
|
|
8
|
+
if (!Number.isFinite(score))
|
|
9
|
+
throw new Error(`Raw-logit reranker returned a non-finite score for ${model}`);
|
|
10
|
+
return score;
|
|
11
|
+
}
|
|
12
|
+
export async function rerankWithSingleRawLogit(input, tokenizer, model, modelName) {
|
|
13
|
+
const inputs = tokenizer(input.text, { text_pair: input.text_pair, padding: true, truncation: true });
|
|
14
|
+
const { logits } = await model(inputs);
|
|
15
|
+
return [{ score: scoreSingleRawLogit(logits, modelName) }];
|
|
16
|
+
}
|
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
import { rerankInModelWorker } from "../model-worker-client.js";
|
|
2
|
+
import { rerankViaApi } from "./reranker-api.js";
|
|
3
|
+
import { resolveRerankerConfig } from "./reranker-config.js";
|
|
2
4
|
let disposeTimer = null;
|
|
3
5
|
let disposePromise = null;
|
|
4
6
|
let activeRuns = 0;
|
|
@@ -56,9 +58,13 @@ export async function prepareRerankerForShutdown() {
|
|
|
56
58
|
export async function rerank(query, candidates, topK, signal) {
|
|
57
59
|
if (candidates.length === 0)
|
|
58
60
|
return [];
|
|
61
|
+
const rerankerConfig = resolveRerankerConfig();
|
|
59
62
|
beginRun();
|
|
60
63
|
try {
|
|
61
|
-
|
|
64
|
+
if (rerankerConfig.provider === "api") {
|
|
65
|
+
return await rerankViaApi(query, candidates, topK, rerankerConfig, signal);
|
|
66
|
+
}
|
|
67
|
+
return await rerankInModelWorker(query, candidates, topK, rerankerConfig, signal);
|
|
62
68
|
}
|
|
63
69
|
finally {
|
|
64
70
|
endRun();
|
|
@@ -11,6 +11,8 @@ export interface KnowledgeBase {
|
|
|
11
11
|
chunk_count: number;
|
|
12
12
|
file_count: number;
|
|
13
13
|
embedding_model: string;
|
|
14
|
+
embedding_signature: string | null;
|
|
15
|
+
embedding_dimension: number | null;
|
|
14
16
|
status: "ready" | "indexing" | "error" | "stale";
|
|
15
17
|
}
|
|
16
18
|
export interface IndexingJob {
|
|
@@ -75,6 +77,9 @@ export declare function createKB(db: Database.Database, opts: {
|
|
|
75
77
|
source_path?: string;
|
|
76
78
|
source_type: KnowledgeBase["source_type"];
|
|
77
79
|
source_options?: string;
|
|
80
|
+
embedding_model?: string;
|
|
81
|
+
embedding_signature?: string;
|
|
82
|
+
embedding_dimension?: number;
|
|
78
83
|
}): KnowledgeBase;
|
|
79
84
|
export declare function getKB(db: Database.Database, id: string): KnowledgeBase | undefined;
|
|
80
85
|
export declare function getKBByName(db: Database.Database, name: string): KnowledgeBase | undefined;
|
|
@@ -82,6 +87,7 @@ export declare function listKBs(db: Database.Database): KnowledgeBase[];
|
|
|
82
87
|
export declare function deleteKB(db: Database.Database, id: string): void;
|
|
83
88
|
export declare function updateKBStatus(db: Database.Database, id: string, status: KnowledgeBase["status"]): void;
|
|
84
89
|
export declare function updateKBCounts(db: Database.Database, id: string, chunkCount: number, fileCount: number): void;
|
|
90
|
+
export declare function updateKBEmbeddingMetadata(db: Database.Database, id: string, embeddingModel: string, embeddingSignature: string | null, embeddingDimension: number | null): void;
|
|
85
91
|
export declare function startIndexingJob(db: Database.Database, kbId: string, operation: IndexingJob["operation"], message: string): void;
|
|
86
92
|
export declare function updateIndexingJob(db: Database.Database, kbId: string, progress: {
|
|
87
93
|
phase?: string;
|
|
@@ -4,7 +4,7 @@ import { createRequire } from "node:module";
|
|
|
4
4
|
import { homedir } from "node:os";
|
|
5
5
|
import { dirname, join } from "node:path";
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
|
-
const SCHEMA_VERSION =
|
|
7
|
+
const SCHEMA_VERSION = 5;
|
|
8
8
|
const ITERATION_BATCH_SIZE = 500;
|
|
9
9
|
const SCHEMA_SQL = `
|
|
10
10
|
CREATE TABLE IF NOT EXISTS schema_version (
|
|
@@ -23,6 +23,8 @@ CREATE TABLE IF NOT EXISTS knowledge_bases (
|
|
|
23
23
|
chunk_count INTEGER NOT NULL DEFAULT 0,
|
|
24
24
|
file_count INTEGER NOT NULL DEFAULT 0,
|
|
25
25
|
embedding_model TEXT NOT NULL DEFAULT 'multilingual-e5-small',
|
|
26
|
+
embedding_signature TEXT,
|
|
27
|
+
embedding_dimension INTEGER,
|
|
26
28
|
status TEXT NOT NULL DEFAULT 'ready'
|
|
27
29
|
);
|
|
28
30
|
|
|
@@ -304,6 +306,16 @@ CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(kb_id, file_path);
|
|
|
304
306
|
`);
|
|
305
307
|
continue;
|
|
306
308
|
}
|
|
309
|
+
if (v === 5) {
|
|
310
|
+
const columns = db.prepare("PRAGMA table_info(knowledge_bases)").all();
|
|
311
|
+
if (!columns.some((column) => column.name === "embedding_signature")) {
|
|
312
|
+
db.exec("ALTER TABLE knowledge_bases ADD COLUMN embedding_signature TEXT");
|
|
313
|
+
}
|
|
314
|
+
if (!columns.some((column) => column.name === "embedding_dimension")) {
|
|
315
|
+
db.exec("ALTER TABLE knowledge_bases ADD COLUMN embedding_dimension INTEGER");
|
|
316
|
+
}
|
|
317
|
+
continue;
|
|
318
|
+
}
|
|
307
319
|
if (migrations[v])
|
|
308
320
|
db.exec(migrations[v]);
|
|
309
321
|
}
|
|
@@ -312,8 +324,11 @@ CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(kb_id, file_path);
|
|
|
312
324
|
export function createKB(db, opts) {
|
|
313
325
|
const id = randomUUID();
|
|
314
326
|
const now = Date.now();
|
|
315
|
-
db.prepare(`INSERT INTO knowledge_bases (
|
|
316
|
-
|
|
327
|
+
db.prepare(`INSERT INTO knowledge_bases (
|
|
328
|
+
id, name, description, source_path, source_type, source_options, created_at, updated_at,
|
|
329
|
+
embedding_model, embedding_signature, embedding_dimension
|
|
330
|
+
)
|
|
331
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(id, opts.name, opts.description ?? null, opts.source_path ?? null, opts.source_type, opts.source_options ?? null, now, now, opts.embedding_model ?? "multilingual-e5-small", opts.embedding_signature ?? null, opts.embedding_dimension ?? null);
|
|
317
332
|
const kb = getKB(db, id);
|
|
318
333
|
if (!kb)
|
|
319
334
|
throw new Error(`Failed to create knowledge base: ${id}`);
|
|
@@ -340,6 +355,11 @@ export function updateKBStatus(db, id, status) {
|
|
|
340
355
|
export function updateKBCounts(db, id, chunkCount, fileCount) {
|
|
341
356
|
db.prepare("UPDATE knowledge_bases SET chunk_count = ?, file_count = ?, updated_at = ? WHERE id = ?").run(chunkCount, fileCount, Date.now(), id);
|
|
342
357
|
}
|
|
358
|
+
export function updateKBEmbeddingMetadata(db, id, embeddingModel, embeddingSignature, embeddingDimension) {
|
|
359
|
+
db.prepare(`UPDATE knowledge_bases
|
|
360
|
+
SET embedding_model = ?, embedding_signature = ?, embedding_dimension = ?, updated_at = ?
|
|
361
|
+
WHERE id = ?`).run(embeddingModel, embeddingSignature, embeddingDimension, Date.now(), id);
|
|
362
|
+
}
|
|
343
363
|
// --- Indexing Jobs ---
|
|
344
364
|
export function startIndexingJob(db, kbId, operation, message) {
|
|
345
365
|
const now = Date.now();
|
package/docs/configuration.md
CHANGED
|
@@ -25,14 +25,39 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
25
25
|
| `PI_KNOWLEDGE_EMBEDDING_BASE_URL` | unset | OpenAI-compatible embedding API root, such as `http://127.0.0.1:8080/v1`. Takes precedence over `OPENAI_BASE_URL`. |
|
|
26
26
|
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Common OpenAI-compatible API root fallback. |
|
|
27
27
|
| `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` | `20000` | Final per-input API embedding safety cap for OpenAI-compatible servers with smaller context windows. This does not replace chunker bounds. |
|
|
28
|
-
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings after API failures.
|
|
28
|
+
| `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` | unset | Set to `local` to explicitly fall back to local embeddings for query embeddings after API failures. Indexing/import/update document batches do not fall back because mixed-provider vector files are unsafe. |
|
|
29
29
|
| `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE` | unset | Enables idle timers for embedding/reranker run coordination. It does not guarantee immediate worker memory release; model-worker shutdown remains tied to engine/session shutdown. Disabled by default for stable shutdown. |
|
|
30
30
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
31
31
|
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
32
32
|
|
|
33
|
+
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character settings or after upgrading older KBs with no signature metadata.
|
|
34
|
+
|
|
35
|
+
## Reranker
|
|
36
|
+
|
|
37
|
+
| Variable | Default | Purpose |
|
|
38
|
+
|----------|---------|---------|
|
|
39
|
+
| `PI_KNOWLEDGE_RERANKER` | `Xenova/ms-marco-MiniLM-L-4-v2` | Reranker used by `deep` mode. Accepts local/HF forms (`<model>`, `hf:<model>`, Hugging Face model URLs) or `api:<model>` for an external HTTP rerank service. |
|
|
40
|
+
| `PI_KNOWLEDGE_RERANKER_REVISION` | `main` | Hugging Face revision passed to Transformers.js for local/HF rerankers. Explicit env value overrides a revision parsed from a model URL. |
|
|
41
|
+
| `PI_KNOWLEDGE_RERANKER_DTYPE` | unset | Optional Transformers.js dtype, such as `fp32`, for local/HF rerankers. |
|
|
42
|
+
| `PI_KNOWLEDGE_RERANKER_REMOTE_HOST` | Hugging Face host or URL-derived host | Remote artifact host for HF-compatible model repositories. Use with trusted mirrors only. |
|
|
43
|
+
| `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE` | `{model}/resolve/{revision}/` | Transformers.js remote path template for HF-compatible mirrors. |
|
|
44
|
+
| `PI_KNOWLEDGE_RERANKER_RAW_LOGITS` | unset | Set to `true` for single-logit cross-encoder rerankers whose raw scores are flattened by sigmoid scoring. Multi-logit classifiers are rejected unless this option is disabled. |
|
|
45
|
+
| `PI_KNOWLEDGE_RERANKER_API_ENDPOINT` | derived from base URL | Full HTTP rerank endpoint. Takes precedence over `PI_KNOWLEDGE_RERANKER_API_BASE_URL` and `OPENAI_BASE_URL`. |
|
|
46
|
+
| `PI_KNOWLEDGE_RERANKER_API_BASE_URL` | `https://api.cohere.com/v2` | Base URL for Cohere/Jina-compatible `/rerank` APIs. `OPENAI_BASE_URL` is accepted only as a convenience fallback for gateways exposing the same schema. |
|
|
47
|
+
| `PI_KNOWLEDGE_RERANKER_API_KEY` | `OPENAI_API_KEY` fallback | Bearer token for API rerankers. Omit only for trusted local services that do not require auth. |
|
|
48
|
+
| `PI_KNOWLEDGE_RERANKER_API_FORMAT` | `cohere` | Request/response preset: `cohere`, `jina`, or `custom-json`. All presets send `model`, `query`, `documents`, `top_n`, and `return_documents:false`. |
|
|
49
|
+
| `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS` | `30000` | Per-request timeout for API reranking. |
|
|
50
|
+
| `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS` | `12000` | Per-candidate document safety cap before sending content to the API. |
|
|
51
|
+
| `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH` | `results` | Dot path to the result array for `custom-json` or compatible responses. |
|
|
52
|
+
| `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD` | `index` | Field name or dot path holding the original candidate index. |
|
|
53
|
+
| `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD` | `relevance_score` | Field name or dot path holding the rerank score. |
|
|
54
|
+
| `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` | `desc` | Sort direction for returned scores: `desc` when higher is better, `asc` when lower is better. |
|
|
55
|
+
|
|
56
|
+
Reranker settings affect query-time `deep` search only; they do not change indexed KB vectors. Local/HF rerankers run in the isolated model worker. Offline mode sets the model worker to local-only loading, so a local/HF reranker must already exist in the model cache. `PI_KNOWLEDGE_RERANKER_RAW_LOGITS=true` switches local/HF rerankers from the Transformers.js text-classification pipeline to direct sequence-classification logits and is intended only for trusted single-logit reranker models; use trusted model repositories or pinned revisions for non-default artifacts. API rerankers send the query and selected candidate chunk content to the configured external service; failures surface by default and do not silently fall back to the local worker.
|
|
57
|
+
|
|
33
58
|
Local embeddings require a real Node 22+ executable for the isolated model worker. `pi-knowledge` first tries Node IPC and automatically falls back to a stdin/stdout JSONL worker transport when the host runtime does not expose `child_process.fork().send()`, which can happen in Windows OMP compatibility layers. On Windows it searches `PI_KNOWLEDGE_NODE_PATH`, persisted `knowledge_configure` settings, common Node, Volta, NVM, Codex `cua_node`, and `PATH` locations before failing. If worker startup still fails, call `knowledge_configure` with the full `node.exe` path such as `C:\Program Files\nodejs\node.exe`; accidental surrounding quotes are ignored. OpenAI-compatible embeddings avoid the local worker entirely.
|
|
34
59
|
|
|
35
|
-
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended.
|
|
60
|
+
API embedding failures intentionally surface by default. Silent fallback can hide bad API keys, wrong base URLs, unsupported model names, or context-window errors and can produce a KB with a different embedding model than intended. Query-time fallback is opt-in; indexing fallback is disabled to prevent mixed vector spaces in one KB.
|
|
36
61
|
|
|
37
62
|
## Runtime Features and Diagnostics
|
|
38
63
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|