pi-knowledge 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +26 -6
- package/dist/index.js +18 -60
- package/dist/src/embedding/instructions.d.ts +33 -0
- package/dist/src/embedding/instructions.js +146 -0
- package/dist/src/embedding/provider.d.ts +25 -8
- package/dist/src/embedding/provider.js +155 -63
- package/dist/src/engine.d.ts +18 -1
- package/dist/src/engine.js +93 -58
- package/dist/src/indexer/embedding-safe-chunks.js +3 -2
- package/dist/src/model-worker-client.d.ts +4 -0
- package/dist/src/model-worker-client.js +14 -2
- package/dist/src/model-worker.js +34 -3
- package/dist/src/status.d.ts +8 -0
- package/dist/src/status.js +93 -0
- package/docs/configuration.md +12 -9
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.14.0] - 2026-11-12
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- Added model-aware embedding input instructions with byte-compatible E5 `on`/`off`, Jina Code task presets, and exact custom query/document instructions (#22).
|
|
9
|
+
- Added instruction-aware embedding signatures and indexed-versus-active compatibility details in `knowledge_status`, `knowledge_doctor`, search warnings, and update progress.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- OpenAI-compatible embedding inputs are decorated once and reused for complete-input budget validation and the HTTP request.
|
|
13
|
+
- Local API fallback is rejected during configuration for Jina/custom instructions that the local E5 model cannot reproduce.
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- Included embedding incompatibility and invalid active embedding configuration in aggregate status health instead of reporting a contradictory healthy state.
|
|
17
|
+
- Isolated PDF/DOCX parsing from Pi/OMP TUI stdio, suppressed all non-fatal PDF.js warning paths, and cleaned up explicit PDF document proxies without changing extracted text (#23).
|
|
18
|
+
|
|
5
19
|
## [0.13.0] - 2026-11-07
|
|
6
20
|
|
|
7
21
|
### Added
|
package/README.md
CHANGED
|
@@ -217,7 +217,7 @@ suggested reindex action. Source-backed knowledge bases can normally use
|
|
|
217
217
|
| `knowledge_show` | List all knowledge bases with stats |
|
|
218
218
|
| `knowledge_status` | Show engine status with health diagnostics (stale, orphans, coverage) |
|
|
219
219
|
| `knowledge_doctor` | Diagnose health score, skipped files, stuck jobs, stale data, and recommended fixes |
|
|
220
|
-
| `knowledge_configure` | Persist runtime prerequisites such as a Windows `node.exe` path for the isolated
|
|
220
|
+
| `knowledge_configure` | Persist runtime prerequisites such as a Windows `node.exe` path for the isolated runtime worker |
|
|
221
221
|
| `knowledge_clear` | Remove all knowledge bases after `confirm: true` |
|
|
222
222
|
| `knowledge_export` | Export a KB to shareable JSONL file |
|
|
223
223
|
| `knowledge_import` | Import a KB from JSONL (re-embeds content) |
|
|
@@ -274,7 +274,27 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
|
274
274
|
# Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
|
|
275
275
|
```
|
|
276
276
|
|
|
277
|
-
|
|
277
|
+
Embedding input instructions must match the selected model. `PI_KNOWLEDGE_EMBEDDING_PREFIXES` accepts:
|
|
278
|
+
|
|
279
|
+
| Value | Query/document input | Intended use |
|
|
280
|
+
|-------|----------------------|--------------|
|
|
281
|
+
| `on` or `e5` | `query: ` / `passage: ` | Default local `multilingual-e5-small`; byte-compatible with existing indexes |
|
|
282
|
+
| `off` or `none` | Raw text | Models explicitly evaluated without instructions |
|
|
283
|
+
| `jina` or `jina-code` | Published Jina Code task prompts | `jinaai/jina-code-embeddings-1.5b` through an OpenAI-compatible endpoint |
|
|
284
|
+
| `custom` | Exact configured query/document instructions | Other instructed embedding models |
|
|
285
|
+
|
|
286
|
+
Jina Code example:
|
|
287
|
+
|
|
288
|
+
```bash
|
|
289
|
+
export PI_KNOWLEDGE_EMBEDDING=openai:jinaai/jina-code-embeddings-1.5b
|
|
290
|
+
export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
291
|
+
export PI_KNOWLEDGE_EMBEDDING_PREFIXES=jina-code
|
|
292
|
+
export PI_KNOWLEDGE_EMBEDDING_TASK=nl2code
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
The Jina preset controls input text only. The embedding server must independently implement the model's required last-token pooling and vector normalization. For custom instructions, set `PI_KNOWLEDGE_EMBEDDING_PREFIXES=custom` plus both `PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION` and `PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION`; values are concatenated verbatim, so include delimiters such as a trailing newline.
|
|
296
|
+
|
|
297
|
+
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Local fallback is rejected during configuration when `jina-code` or `custom` instructions are selected because the local E5 model cannot reproduce those inputs. Inputs are never silently truncated.
|
|
278
298
|
|
|
279
299
|
OpenAI-compatible API vectors are validated and L2-normalized by the client before query use or persistence, so cosine-style dot products do not depend on a provider-specific vector magnitude. This normalization policy is part of the embedding signature. KBs created with an older API signature must run `knowledge_update`; `knowledge_doctor` reports `rebuild_vectors` when the active provider no longer matches stored vector semantics.
|
|
280
300
|
|
|
@@ -282,7 +302,7 @@ OpenAI-compatible embedding inputs use a 20000-character complete-input safety c
|
|
|
282
302
|
|
|
283
303
|
For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
|
|
284
304
|
|
|
285
|
-
Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports
|
|
305
|
+
Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature containing the exact instruction strategy/task/hash. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports the specific mismatch; `knowledge_status` shows indexed versus active model/decoration and includes mismatches in aggregate health. Run `knowledge_update` after changing embedding providers, models, tasks, or instructions.
|
|
286
306
|
|
|
287
307
|
## Configuration
|
|
288
308
|
|
|
@@ -291,8 +311,8 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
|
|
|
291
311
|
| Area | Environment variables |
|
|
292
312
|
|------|-----------------------|
|
|
293
313
|
| Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
|
|
294
|
-
|
|
|
295
|
-
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
314
|
+
| Runtime worker and model cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
|
|
315
|
+
| Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_TASK`, `PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION`, `PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
|
|
296
316
|
| Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
|
|
297
317
|
| Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
|
|
298
318
|
| Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
|
|
@@ -306,7 +326,7 @@ For Hugging Face Text Embeddings Inference rerankers, use `PI_KNOWLEDGE_RERANKER
|
|
|
306
326
|
|
|
307
327
|
Default storage is `~/.pi/knowledge` for Pi and `~/.omp/knowledge` for OMP. Explicit overrides are available with `PI_KNOWLEDGE_DIR` and `OMP_KNOWLEDGE_DIR`. Under the default home OMP root, existing legacy `~/.pi/knowledge` data remains visible when `~/.omp/knowledge` has not been created yet.
|
|
308
328
|
|
|
309
|
-
OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated
|
|
329
|
+
OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated worker startup, Windows-safe worker transport fallback, document parser stdio isolation, and idempotent shutdown. Local embeddings and PDF/DOCX extraction require Node 22+ for the isolated worker; set `PI_KNOWLEDGE_NODE_PATH` to `node.exe` if OMP runs from a non-Node packaged host. Compatibility-sensitive releases should validate both Pi and OMP install/runtime flows.
|
|
310
330
|
|
|
311
331
|
## Large Project Indexing
|
|
312
332
|
|
package/dist/index.js
CHANGED
|
@@ -7,6 +7,7 @@ var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExte
|
|
|
7
7
|
return path;
|
|
8
8
|
};
|
|
9
9
|
import { MAX_SEARCH_RESULTS, MAX_SEARCH_WINDOW } from "./src/search/contracts.js";
|
|
10
|
+
import { formatKnowledgeStatus } from "./src/status.js";
|
|
10
11
|
const Type = {
|
|
11
12
|
Object(properties) {
|
|
12
13
|
const required = Object.entries(properties)
|
|
@@ -253,7 +254,7 @@ export default function (pi) {
|
|
|
253
254
|
pi.registerTool({
|
|
254
255
|
name: "knowledge_configure",
|
|
255
256
|
label: "Knowledge Configure",
|
|
256
|
-
description: "Configure pi-knowledge runtime settings such as the
|
|
257
|
+
description: "Configure pi-knowledge runtime settings such as the isolated worker Node executable",
|
|
257
258
|
approval: "write",
|
|
258
259
|
promptSnippet: "Configure local pi-knowledge runtime prerequisites before indexing",
|
|
259
260
|
promptGuidelines: [
|
|
@@ -264,7 +265,7 @@ export default function (pi) {
|
|
|
264
265
|
],
|
|
265
266
|
parameters: Type.Object({
|
|
266
267
|
node_path: Type.Optional(Type.String({
|
|
267
|
-
description: "Optional full path to a Node 22+ executable for the isolated local
|
|
268
|
+
description: "Optional full path to a Node 22+ executable for the isolated local model and document extraction worker, such as C:\\Program Files\\nodejs\\node.exe",
|
|
268
269
|
})),
|
|
269
270
|
}),
|
|
270
271
|
async execute(_id, params, _signal) {
|
|
@@ -279,7 +280,7 @@ export default function (pi) {
|
|
|
279
280
|
content: [
|
|
280
281
|
{
|
|
281
282
|
type: "text",
|
|
282
|
-
text: `Configured
|
|
283
|
+
text: `Configured isolated worker Node executable: ${configuredPath}`,
|
|
283
284
|
},
|
|
284
285
|
],
|
|
285
286
|
details: { node_path: configuredPath },
|
|
@@ -557,69 +558,26 @@ export default function (pi) {
|
|
|
557
558
|
pi.registerTool({
|
|
558
559
|
name: "knowledge_status",
|
|
559
560
|
label: "Knowledge Status",
|
|
560
|
-
description: "Show knowledge engine status with
|
|
561
|
+
description: "Show knowledge engine status with embedding compatibility, staleness, orphan, and coverage diagnostics",
|
|
561
562
|
approval: "read",
|
|
562
563
|
parameters: Type.Object({}),
|
|
563
564
|
async execute(_id, _params, _signal) {
|
|
564
565
|
if (_signal?.aborted)
|
|
565
566
|
throw new Error("Cancelled");
|
|
566
567
|
const { engine, storage, watcher } = await ensureInitialized();
|
|
567
|
-
const
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
if (kb.source_path)
|
|
581
|
-
lines.push(` source: ${kb.source_path}`);
|
|
582
|
-
if (diag) {
|
|
583
|
-
if (diag.job) {
|
|
584
|
-
const jobAge = Math.round((Date.now() - diag.job.last_progress_at) / 1000);
|
|
585
|
-
const progress = [
|
|
586
|
-
`${diag.job.processed_files} files`,
|
|
587
|
-
`${diag.job.processed_chunks} chunks`,
|
|
588
|
-
diag.job.skipped_total > 0 ? `${diag.job.skipped_total} skipped` : "",
|
|
589
|
-
diag.job.added_chunks > 0 ? `+${diag.job.added_chunks}` : "",
|
|
590
|
-
diag.job.removed_chunks > 0 ? `-${diag.job.removed_chunks}` : "",
|
|
591
|
-
diag.job.unchanged_chunks > 0 ? `=${diag.job.unchanged_chunks}` : "",
|
|
592
|
-
]
|
|
593
|
-
.filter(Boolean)
|
|
594
|
-
.join(", ");
|
|
595
|
-
lines.push(` job: ${diag.job.status}/${diag.job.phase} — ${progress || "no processed items yet"} — last progress ${jobAge}s ago`);
|
|
596
|
-
if (diag.job.message)
|
|
597
|
-
lines.push(` last: ${diag.job.message}`);
|
|
598
|
-
if (diag.job.error_message)
|
|
599
|
-
lines.push(` error: ${diag.job.error_message}`);
|
|
600
|
-
}
|
|
601
|
-
lines.push(` coverage: ${diag.coverage_percent}% (${diag.indexed_files}/${diag.total_source_files} files)`);
|
|
602
|
-
if (diag.stale_files.length > 0)
|
|
603
|
-
lines.push(` ⚠️ stale: ${diag.stale_files.length} files modified since last index`);
|
|
604
|
-
if (diag.orphan_files.length > 0)
|
|
605
|
-
lines.push(` ⚠️ orphans: ${diag.orphan_files.length} chunks reference deleted files`);
|
|
606
|
-
if (diag.stuck_indexing)
|
|
607
|
-
lines.push(` ⚠️ indexing appears stuck for ${Math.round(diag.last_progress_age_ms / 60000)}m; remove and rebuild if no pi process is actively indexing it`);
|
|
608
|
-
if (diag.skipped_files.total > 0)
|
|
609
|
-
lines.push(` skipped: ${diag.skipped_files.total} files (${Object.entries(diag.skipped_files.by_reason)
|
|
610
|
-
.filter(([, count]) => count > 0)
|
|
611
|
-
.map(([reason, count]) => `${reason}: ${count}`)
|
|
612
|
-
.join(", ")})`);
|
|
613
|
-
}
|
|
614
|
-
}
|
|
615
|
-
const totalStale = diagnostics.reduce((n, d) => n + d.stale_files.length, 0);
|
|
616
|
-
const totalOrphans = diagnostics.reduce((n, d) => n + d.orphan_files.length, 0);
|
|
617
|
-
const totalStuck = diagnostics.filter((d) => d.stuck_indexing).length;
|
|
618
|
-
if (totalStale === 0 && totalOrphans === 0 && totalStuck === 0 && kbs.length > 0)
|
|
619
|
-
lines.push("", "Health: ✓ all indexes up to date");
|
|
620
|
-
else if (totalStale > 0 || totalOrphans > 0 || totalStuck > 0)
|
|
621
|
-
lines.push("", `Health: ⚠️ ${totalStale} stale, ${totalOrphans} orphans, ${totalStuck} stuck indexing — run knowledge_update or rebuild affected KBs`);
|
|
622
|
-
return { content: [{ type: "text", text: lines.join("\n") }] };
|
|
568
|
+
const snapshot = engine.statusSnapshot(_signal);
|
|
569
|
+
return {
|
|
570
|
+
content: [
|
|
571
|
+
{
|
|
572
|
+
type: "text",
|
|
573
|
+
text: formatKnowledgeStatus({
|
|
574
|
+
snapshot,
|
|
575
|
+
storagePath: storage.getDefaultKnowledgeDir(),
|
|
576
|
+
watchCount: watcher.getActiveWatcherCount(),
|
|
577
|
+
}),
|
|
578
|
+
},
|
|
579
|
+
],
|
|
580
|
+
};
|
|
623
581
|
},
|
|
624
582
|
});
|
|
625
583
|
pi.registerTool({
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
export type EmbeddingRole = "query" | "passage";
|
|
2
|
+
export type EmbeddingDecorationStrategy = "e5" | "none" | "jina-code" | "custom";
|
|
3
|
+
export type EmbeddingTask = "nl2code" | "code2code" | "code2nl" | "code2completion" | "qa";
|
|
4
|
+
export type EmbeddingPrefixMode = "on" | "off";
|
|
5
|
+
export interface EmbeddingInstructionSet {
|
|
6
|
+
query: string;
|
|
7
|
+
document: string;
|
|
8
|
+
}
|
|
9
|
+
export interface ResolvedEmbeddingDecoration {
|
|
10
|
+
strategy: EmbeddingDecorationStrategy;
|
|
11
|
+
task?: EmbeddingTask;
|
|
12
|
+
instructions: EmbeddingInstructionSet;
|
|
13
|
+
prefixMode: EmbeddingPrefixMode;
|
|
14
|
+
}
|
|
15
|
+
export declare const MAX_INSTRUCTION_CHARS = 512;
|
|
16
|
+
export declare const DEFAULT_EMBEDDING_TASK: EmbeddingTask;
|
|
17
|
+
export declare const ACCEPTED_DECORATION_VALUES: readonly ["on", "off", "e5", "none", "jina", "jina-code", "custom"];
|
|
18
|
+
export declare const ACCEPTED_EMBEDDING_TASKS: readonly ["nl2code", "code2code", "code2nl", "code2completion", "qa"];
|
|
19
|
+
export declare const E5_INSTRUCTIONS: EmbeddingInstructionSet;
|
|
20
|
+
export declare const NO_INSTRUCTIONS: EmbeddingInstructionSet;
|
|
21
|
+
/**
|
|
22
|
+
* Published prompts for jinaai/jina-code-embeddings-1.5b.
|
|
23
|
+
* Source: config_sentence_transformers.json at model revision
|
|
24
|
+
* 39aeb4fb9b60f930934c78ae5d749a46287c248a (verified 2026-11-12).
|
|
25
|
+
* Values already contain their required trailing newline and are concatenated verbatim.
|
|
26
|
+
*/
|
|
27
|
+
export declare const JINA_CODE_INSTRUCTIONS: Readonly<Record<EmbeddingTask, EmbeddingInstructionSet>>;
|
|
28
|
+
export declare function decorateEmbeddingInput(text: string, role: EmbeddingRole, instructions: EmbeddingInstructionSet): string;
|
|
29
|
+
export declare function embeddingInstructionLength(instructions: EmbeddingInstructionSet, role: EmbeddingRole): number;
|
|
30
|
+
export declare function embeddingInstructionSetHash(instructions: EmbeddingInstructionSet): string;
|
|
31
|
+
export declare function embeddingDecorationDescriptor(decoration: ResolvedEmbeddingDecoration): string;
|
|
32
|
+
export declare function resolveEmbeddingDecoration(env: NodeJS.ProcessEnv): ResolvedEmbeddingDecoration;
|
|
33
|
+
export declare function decorationSupportedByLocalModel(strategy: EmbeddingDecorationStrategy): boolean;
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
2
|
+
export const MAX_INSTRUCTION_CHARS = 512;
|
|
3
|
+
export const DEFAULT_EMBEDDING_TASK = "nl2code";
|
|
4
|
+
export const ACCEPTED_DECORATION_VALUES = ["on", "off", "e5", "none", "jina", "jina-code", "custom"];
|
|
5
|
+
export const ACCEPTED_EMBEDDING_TASKS = ["nl2code", "code2code", "code2nl", "code2completion", "qa"];
|
|
6
|
+
const STRATEGY_ALIASES = {
|
|
7
|
+
on: "e5",
|
|
8
|
+
e5: "e5",
|
|
9
|
+
off: "none",
|
|
10
|
+
none: "none",
|
|
11
|
+
jina: "jina-code",
|
|
12
|
+
"jina-code": "jina-code",
|
|
13
|
+
custom: "custom",
|
|
14
|
+
};
|
|
15
|
+
const STRATEGY_ENV = "PI_KNOWLEDGE_EMBEDDING_PREFIXES";
|
|
16
|
+
const TASK_ENV = "PI_KNOWLEDGE_EMBEDDING_TASK";
|
|
17
|
+
const QUERY_INSTRUCTION_ENV = "PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION";
|
|
18
|
+
const DOCUMENT_INSTRUCTION_ENV = "PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION";
|
|
19
|
+
export const E5_INSTRUCTIONS = Object.freeze({ query: "query: ", document: "passage: " });
|
|
20
|
+
export const NO_INSTRUCTIONS = Object.freeze({ query: "", document: "" });
|
|
21
|
+
/**
|
|
22
|
+
* Published prompts for jinaai/jina-code-embeddings-1.5b.
|
|
23
|
+
* Source: config_sentence_transformers.json at model revision
|
|
24
|
+
* 39aeb4fb9b60f930934c78ae5d749a46287c248a (verified 2026-11-12).
|
|
25
|
+
* Values already contain their required trailing newline and are concatenated verbatim.
|
|
26
|
+
*/
|
|
27
|
+
export const JINA_CODE_INSTRUCTIONS = Object.freeze({
|
|
28
|
+
nl2code: Object.freeze({
|
|
29
|
+
query: "Find the most relevant code snippet given the following query:\n",
|
|
30
|
+
document: "Candidate code snippet:\n",
|
|
31
|
+
}),
|
|
32
|
+
code2code: Object.freeze({
|
|
33
|
+
query: "Find an equivalent code snippet given the following code snippet:\n",
|
|
34
|
+
document: "Candidate code snippet:\n",
|
|
35
|
+
}),
|
|
36
|
+
code2nl: Object.freeze({
|
|
37
|
+
query: "Find the most relevant comment given the following code snippet:\n",
|
|
38
|
+
document: "Candidate comment:\n",
|
|
39
|
+
}),
|
|
40
|
+
code2completion: Object.freeze({
|
|
41
|
+
query: "Find the most relevant completion given the following start of code snippet:\n",
|
|
42
|
+
document: "Candidate completion:\n",
|
|
43
|
+
}),
|
|
44
|
+
qa: Object.freeze({
|
|
45
|
+
query: "Find the most relevant answer given the following question:\n",
|
|
46
|
+
document: "Candidate answer:\n",
|
|
47
|
+
}),
|
|
48
|
+
});
|
|
49
|
+
export function decorateEmbeddingInput(text, role, instructions) {
|
|
50
|
+
return `${role === "query" ? instructions.query : instructions.document}${text}`;
|
|
51
|
+
}
|
|
52
|
+
export function embeddingInstructionLength(instructions, role) {
|
|
53
|
+
return (role === "query" ? instructions.query : instructions.document).length;
|
|
54
|
+
}
|
|
55
|
+
export function embeddingInstructionSetHash(instructions) {
|
|
56
|
+
return createHash("sha256")
|
|
57
|
+
.update(JSON.stringify([instructions.query, instructions.document]))
|
|
58
|
+
.digest("hex")
|
|
59
|
+
.slice(0, 12);
|
|
60
|
+
}
|
|
61
|
+
export function embeddingDecorationDescriptor(decoration) {
|
|
62
|
+
if (decoration.strategy === "e5")
|
|
63
|
+
return "q=query:d=passage";
|
|
64
|
+
if (decoration.strategy === "none")
|
|
65
|
+
return "q=query:d=passage:prefixes=off";
|
|
66
|
+
const instructionHash = embeddingInstructionSetHash(decoration.instructions);
|
|
67
|
+
if (decoration.strategy === "jina-code") {
|
|
68
|
+
return `strategy=jina-code:task=${decoration.task ?? DEFAULT_EMBEDDING_TASK}:ih=${instructionHash}`;
|
|
69
|
+
}
|
|
70
|
+
return `strategy=custom:ih=${instructionHash}`;
|
|
71
|
+
}
|
|
72
|
+
function cleanValue(value) {
|
|
73
|
+
const trimmed = value?.trim();
|
|
74
|
+
return trimmed ? trimmed : undefined;
|
|
75
|
+
}
|
|
76
|
+
function assertInstructionValue(name, value) {
|
|
77
|
+
if (value.length > MAX_INSTRUCTION_CHARS) {
|
|
78
|
+
throw new Error(`Embedding instruction ${name} is ${value.length} characters; maximum is ${MAX_INSTRUCTION_CHARS}.`);
|
|
79
|
+
}
|
|
80
|
+
for (const character of value) {
|
|
81
|
+
const code = character.codePointAt(0) ?? 0;
|
|
82
|
+
if (character !== "\n" && (code < 0x20 || (code >= 0x7f && code <= 0x9f))) {
|
|
83
|
+
throw new Error(`Embedding instruction ${name} contains an unsupported control character; only printable text and newline are allowed.`);
|
|
84
|
+
}
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
function resolveCustomInstructions(env) {
|
|
88
|
+
const query = env[QUERY_INSTRUCTION_ENV];
|
|
89
|
+
const document = env[DOCUMENT_INSTRUCTION_ENV];
|
|
90
|
+
const missing = [
|
|
91
|
+
query === undefined ? QUERY_INSTRUCTION_ENV : "",
|
|
92
|
+
document === undefined ? DOCUMENT_INSTRUCTION_ENV : "",
|
|
93
|
+
].filter(Boolean);
|
|
94
|
+
if (missing.length > 0)
|
|
95
|
+
throw new Error(`${STRATEGY_ENV}=custom requires ${missing.join(" and ")}.`);
|
|
96
|
+
if (query === undefined || document === undefined)
|
|
97
|
+
throw new Error("Embedding instruction resolution failed");
|
|
98
|
+
if (!query.trim())
|
|
99
|
+
throw new Error(`${QUERY_INSTRUCTION_ENV} must be a non-empty string.`);
|
|
100
|
+
assertInstructionValue(QUERY_INSTRUCTION_ENV, query);
|
|
101
|
+
assertInstructionValue(DOCUMENT_INSTRUCTION_ENV, document);
|
|
102
|
+
return Object.freeze({ query, document });
|
|
103
|
+
}
|
|
104
|
+
export function resolveEmbeddingDecoration(env) {
|
|
105
|
+
const rawStrategy = cleanValue(env[STRATEGY_ENV])?.toLowerCase();
|
|
106
|
+
const strategy = rawStrategy ? STRATEGY_ALIASES[rawStrategy] : "e5";
|
|
107
|
+
if (!strategy) {
|
|
108
|
+
throw new Error(`Unsupported embedding decoration strategy: ${rawStrategy}. Use ${STRATEGY_ENV}=${ACCEPTED_DECORATION_VALUES.join("|")}.`);
|
|
109
|
+
}
|
|
110
|
+
const customVariablesPresent = env[QUERY_INSTRUCTION_ENV] !== undefined || env[DOCUMENT_INSTRUCTION_ENV] !== undefined;
|
|
111
|
+
if (strategy !== "custom" && customVariablesPresent) {
|
|
112
|
+
throw new Error(`${QUERY_INSTRUCTION_ENV} and ${DOCUMENT_INSTRUCTION_ENV} are valid only when ${STRATEGY_ENV}=custom.`);
|
|
113
|
+
}
|
|
114
|
+
const rawTask = cleanValue(env[TASK_ENV])?.toLowerCase();
|
|
115
|
+
let task;
|
|
116
|
+
if (rawTask) {
|
|
117
|
+
if (strategy !== "jina-code") {
|
|
118
|
+
throw new Error(`${TASK_ENV} is valid only when ${STRATEGY_ENV}=jina-code.`);
|
|
119
|
+
}
|
|
120
|
+
if (!ACCEPTED_EMBEDDING_TASKS.includes(rawTask)) {
|
|
121
|
+
throw new Error(`Unsupported embedding task: ${rawTask}. Use ${TASK_ENV}=${ACCEPTED_EMBEDDING_TASKS.join("|")}.`);
|
|
122
|
+
}
|
|
123
|
+
task = rawTask;
|
|
124
|
+
}
|
|
125
|
+
else if (strategy === "jina-code") {
|
|
126
|
+
task = DEFAULT_EMBEDDING_TASK;
|
|
127
|
+
}
|
|
128
|
+
let instructions;
|
|
129
|
+
if (strategy === "e5")
|
|
130
|
+
instructions = E5_INSTRUCTIONS;
|
|
131
|
+
else if (strategy === "none")
|
|
132
|
+
instructions = NO_INSTRUCTIONS;
|
|
133
|
+
else if (strategy === "jina-code")
|
|
134
|
+
instructions = JINA_CODE_INSTRUCTIONS[task ?? DEFAULT_EMBEDDING_TASK];
|
|
135
|
+
else
|
|
136
|
+
instructions = resolveCustomInstructions(env);
|
|
137
|
+
return Object.freeze({
|
|
138
|
+
strategy,
|
|
139
|
+
task,
|
|
140
|
+
instructions,
|
|
141
|
+
prefixMode: strategy === "e5" ? "on" : "off",
|
|
142
|
+
});
|
|
143
|
+
}
|
|
144
|
+
export function decorationSupportedByLocalModel(strategy) {
|
|
145
|
+
return strategy === "e5" || strategy === "none";
|
|
146
|
+
}
|
|
@@ -1,14 +1,16 @@
|
|
|
1
|
+
import { type EmbeddingDecorationStrategy, type EmbeddingInstructionSet, type EmbeddingPrefixMode, type EmbeddingRole, type EmbeddingTask } from "./instructions.ts";
|
|
2
|
+
export type { EmbeddingDecorationStrategy, EmbeddingInstructionSet, EmbeddingRole, EmbeddingTask, } from "./instructions.ts";
|
|
1
3
|
export type EmbeddingProvider = "local" | "openai";
|
|
2
|
-
export type EmbeddingPrefix = "query" | "passage";
|
|
3
|
-
export type EmbeddingPrefixStrategy = "on" | "off";
|
|
4
4
|
export interface EmbeddingConfig {
|
|
5
5
|
provider: EmbeddingProvider;
|
|
6
6
|
model: string;
|
|
7
7
|
baseUrl?: string;
|
|
8
8
|
maxChars: number;
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
9
|
+
strategy: EmbeddingDecorationStrategy;
|
|
10
|
+
task?: EmbeddingTask;
|
|
11
|
+
instructions: EmbeddingInstructionSet;
|
|
12
|
+
prefixMode: EmbeddingPrefixMode;
|
|
13
|
+
apiFallbackToLocal: boolean;
|
|
12
14
|
pooling: "mean";
|
|
13
15
|
normalize: true;
|
|
14
16
|
}
|
|
@@ -18,6 +20,17 @@ export interface EmbeddingInputMeasurement {
|
|
|
18
20
|
preferredMaximum: number;
|
|
19
21
|
unit: "tokens" | "characters";
|
|
20
22
|
}
|
|
23
|
+
export type EmbeddingCompatibilityCause = "match" | "unrecorded" | "dimension" | "decoration" | "configuration" | "signature_format";
|
|
24
|
+
export interface EmbeddingCompatibility {
|
|
25
|
+
compatible: boolean;
|
|
26
|
+
cause: EmbeddingCompatibilityCause;
|
|
27
|
+
activeSignature: string;
|
|
28
|
+
recordedSignature: string | null;
|
|
29
|
+
activeDecoration: string;
|
|
30
|
+
recordedDecoration?: string;
|
|
31
|
+
activeDimension: number;
|
|
32
|
+
recordedDimension: number | null;
|
|
33
|
+
}
|
|
21
34
|
export declare const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
22
35
|
export declare const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
|
|
23
36
|
export declare const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
|
|
@@ -29,18 +42,22 @@ export declare const API_VECTOR_NORMALIZATION_POLICY = "client-l2-v1";
|
|
|
29
42
|
export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
|
|
30
43
|
export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
|
|
31
44
|
export declare function normalizeEmbeddingVector(vector: Float32Array): Float32Array;
|
|
45
|
+
export declare function describeEmbeddingDecoration(config: EmbeddingConfig): string;
|
|
32
46
|
export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
|
|
47
|
+
export declare function recordedEmbeddingDecoration(signature: string | null): string | undefined;
|
|
48
|
+
export declare function inspectEmbeddingCompatibility(recordedSignature: string | null, recordedDimension: number | null, config: EmbeddingConfig, activeDimension: number): EmbeddingCompatibility;
|
|
49
|
+
export declare function describeEmbeddingCompatibility(compatibility: EmbeddingCompatibility): string;
|
|
33
50
|
export declare function dispose(): Promise<void>;
|
|
34
51
|
export declare function prepareForShutdown(): Promise<void>;
|
|
35
|
-
export declare function measureEmbeddingInputs(texts: string[],
|
|
36
|
-
export declare function embedTextsWithConfig(texts: string[],
|
|
52
|
+
export declare function measureEmbeddingInputs(texts: string[], role: EmbeddingRole, config: EmbeddingConfig, signal?: AbortSignal): Promise<EmbeddingInputMeasurement>;
|
|
53
|
+
export declare function embedTextsWithConfig(texts: string[], role: EmbeddingRole, signal?: AbortSignal, options?: {
|
|
37
54
|
allowApiFallback?: boolean;
|
|
38
55
|
inputValidated?: boolean;
|
|
39
56
|
}): Promise<{
|
|
40
57
|
vectors: Float32Array[];
|
|
41
58
|
config: EmbeddingConfig;
|
|
42
59
|
}>;
|
|
43
|
-
export declare function embedTexts(texts: string[],
|
|
60
|
+
export declare function embedTexts(texts: string[], role: EmbeddingRole, signal?: AbortSignal, options?: {
|
|
44
61
|
allowApiFallback?: boolean;
|
|
45
62
|
inputValidated?: boolean;
|
|
46
63
|
}): Promise<Float32Array[]>;
|