pi-knowledge 0.13.0 → 0.14.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,20 @@
2
2
 
3
3
  ## [Unreleased]
4
4
 
5
+ ## [0.14.0] - 2026-11-12
6
+
7
+ ### Added
8
+ - Added model-aware embedding input instructions with byte-compatible E5 `on`/`off`, Jina Code task presets, and exact custom query/document instructions (#22).
9
+ - Added instruction-aware embedding signatures and indexed-versus-active compatibility details in `knowledge_status`, `knowledge_doctor`, search warnings, and update progress.
10
+
11
+ ### Changed
12
+ - OpenAI-compatible embedding inputs are decorated once and reused for complete-input budget validation and the HTTP request.
13
+ - Local API fallback is rejected during configuration for Jina/custom instructions that the local E5 model cannot reproduce.
14
+
15
+ ### Fixed
16
+ - Included embedding incompatibility and invalid active embedding configuration in aggregate status health instead of reporting a contradictory healthy state.
17
+ - Isolated PDF/DOCX parsing from Pi/OMP TUI stdio, suppressed all non-fatal PDF.js warning paths, and cleaned up explicit PDF document proxies without changing extracted text (#23).
18
+
5
19
  ## [0.13.0] - 2026-11-07
6
20
 
7
21
  ### Added
package/README.md CHANGED
@@ -217,7 +217,7 @@ suggested reindex action. Source-backed knowledge bases can normally use
217
217
  | `knowledge_show` | List all knowledge bases with stats |
218
218
  | `knowledge_status` | Show engine status with health diagnostics (stale, orphans, coverage) |
219
219
  | `knowledge_doctor` | Diagnose health score, skipped files, stuck jobs, stale data, and recommended fixes |
220
- | `knowledge_configure` | Persist runtime prerequisites such as a Windows `node.exe` path for the isolated local model worker |
220
+ | `knowledge_configure` | Persist runtime prerequisites such as a Windows `node.exe` path for the isolated runtime worker |
221
221
  | `knowledge_clear` | Remove all knowledge bases after `confirm: true` |
222
222
  | `knowledge_export` | Export a KB to shareable JSONL file |
223
223
  | `knowledge_import` | Import a KB from JSONL (re-embeds content) |
@@ -274,7 +274,27 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
274
274
  # Set PI_KNOWLEDGE_EMBEDDING_API_KEY or OPENAI_API_KEY only if your endpoint requires bearer auth.
275
275
  ```
276
276
 
277
- API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Inputs are never silently truncated.
277
+ Embedding input instructions must match the selected model. `PI_KNOWLEDGE_EMBEDDING_PREFIXES` accepts:
278
+
279
+ | Value | Query/document input | Intended use |
280
+ |-------|----------------------|--------------|
281
+ | `on` or `e5` | `query: ` / `passage: ` | Default local `multilingual-e5-small`; byte-compatible with existing indexes |
282
+ | `off` or `none` | Raw text | Models explicitly evaluated without instructions |
283
+ | `jina` or `jina-code` | Published Jina Code task prompts | `jinaai/jina-code-embeddings-1.5b` through an OpenAI-compatible endpoint |
284
+ | `custom` | Exact configured query/document instructions | Other instructed embedding models |
285
+
286
+ Jina Code example:
287
+
288
+ ```bash
289
+ export PI_KNOWLEDGE_EMBEDDING=openai:jinaai/jina-code-embeddings-1.5b
290
+ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
291
+ export PI_KNOWLEDGE_EMBEDDING_PREFIXES=jina-code
292
+ export PI_KNOWLEDGE_EMBEDDING_TASK=nl2code
293
+ ```
294
+
295
+ The Jina preset controls input text only. The embedding server must independently implement the model's required last-token pooling and vector normalization. For custom instructions, set `PI_KNOWLEDGE_EMBEDDING_PREFIXES=custom` plus both `PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION` and `PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION`; values are concatenated verbatim, so include delimiters such as a trailing newline.
296
+
297
+ API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Local fallback is rejected during configuration when `jina-code` or `custom` instructions are selected because the local E5 model cannot reproduce those inputs. Inputs are never silently truncated.
278
298
 
279
299
  OpenAI-compatible API vectors are validated and L2-normalized by the client before query use or persistence, so cosine-style dot products do not depend on a provider-specific vector magnitude. This normalization policy is part of the embedding signature. KBs created with an older API signature must run `knowledge_update`; `knowledge_doctor` reports `rebuild_vectors` when the active provider no longer matches stored vector semantics.
280
300
 
@@ -282,7 +302,7 @@ OpenAI-compatible embedding inputs use a 20000-character complete-input safety c
282
302
 
283
303
  For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
284
304
 
285
- Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports a warning; run `knowledge_update` after changing embedding providers or upgrading older KBs.
305
+ Stored KB metadata includes the embedding model, vector dimension, and a non-secret embedding signature containing the exact instruction strategy/task/hash. If the current query embedding is incompatible with a KB's stored vectors, or if an older KB has no signature metadata, `knowledge_search` skips vector retrieval for that KB and reports the specific mismatch; `knowledge_status` shows indexed versus active model/decoration and includes mismatches in aggregate health. Run `knowledge_update` after changing embedding providers, models, tasks, or instructions.
286
306
 
287
307
  ## Configuration
288
308
 
@@ -291,8 +311,8 @@ Full configuration details are in [docs/configuration.md](docs/configuration.md)
291
311
  | Area | Environment variables |
292
312
  |------|-----------------------|
293
313
  | Storage path | `PI_KNOWLEDGE_DIR`, `OMP_KNOWLEDGE_DIR`, `PI_CODING_AGENT_DIR`, `OMP_CODING_AGENT_DIR`, `OMP_PROFILE` |
294
- | Model worker and cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
295
- | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
314
+ | Runtime worker and model cache | `PI_KNOWLEDGE_MODEL_CACHE_DIR`, `PI_KNOWLEDGE_NODE_PATH` |
315
+ | Embedding provider | `PI_KNOWLEDGE_EMBEDDING`, `PI_KNOWLEDGE_EMBEDDING_API_KEY`, `OPENAI_API_KEY`, `PI_KNOWLEDGE_EMBEDDING_BASE_URL`, `OPENAI_BASE_URL`, `PI_KNOWLEDGE_EMBEDDING_PREFIXES`, `PI_KNOWLEDGE_EMBEDDING_TASK`, `PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION`, `PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION`, `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS`, `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE`, `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK` |
296
316
  | Reranker provider | `PI_KNOWLEDGE_RERANKER`, `PI_KNOWLEDGE_RERANKER_REVISION`, `PI_KNOWLEDGE_RERANKER_DTYPE`, `PI_KNOWLEDGE_RERANKER_REMOTE_HOST`, `PI_KNOWLEDGE_RERANKER_REMOTE_PATH_TEMPLATE`, `PI_KNOWLEDGE_RERANKER_RAW_LOGITS`, `PI_KNOWLEDGE_RERANKER_API_ENDPOINT`, `PI_KNOWLEDGE_RERANKER_API_BASE_URL`, `PI_KNOWLEDGE_RERANKER_API_KEY`, `PI_KNOWLEDGE_RERANKER_API_FORMAT`, `PI_KNOWLEDGE_RERANKER_API_TIMEOUT_MS`, `PI_KNOWLEDGE_RERANKER_MAX_DOC_CHARS`, `PI_KNOWLEDGE_RERANKER_API_RESULTS_PATH`, `PI_KNOWLEDGE_RERANKER_API_INDEX_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_FIELD`, `PI_KNOWLEDGE_RERANKER_API_SCORE_DIRECTION` |
297
317
  | Native lifecycle | `PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE`, `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` |
298
318
  | Runtime features | `PI_KNOWLEDGE_WATCH`, `PI_KNOWLEDGE_AUTO_INJECT`, `PI_KNOWLEDGE_STALE_INDEXING_MS`, `PI_KNOWLEDGE_OFFLINE`, `PI_KNOWLEDGE_SEARCH_PROFILE`, `PI_KNOWLEDGE_SEARCH_DEFAULT_LIMIT`, `PI_KNOWLEDGE_SNIPPET_MAX_LENGTH`, `PI_KNOWLEDGE_MIN_HYBRID_SCORE`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MIN`, `PI_KNOWLEDGE_SEARCH_CANDIDATE_MULTIPLIER`, `PI_KNOWLEDGE_ADAPTIVE_CONTEXT_LINES`, `PI_KNOWLEDGE_ADAPTIVE_MAX_CHARS`, `PI_KNOWLEDGE_ADAPTIVE_NEIGHBOR_TARGET`, `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES`, `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` |
@@ -306,7 +326,7 @@ For Hugging Face Text Embeddings Inference rerankers, use `PI_KNOWLEDGE_RERANKER
306
326
 
307
327
  Default storage is `~/.pi/knowledge` for Pi and `~/.omp/knowledge` for OMP. Explicit overrides are available with `PI_KNOWLEDGE_DIR` and `OMP_KNOWLEDGE_DIR`. Under the default home OMP root, existing legacy `~/.pi/knowledge` data remains visible when `~/.omp/knowledge` has not been created yet.
308
328
 
309
- OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated model-worker startup, Windows-safe worker transport fallback, and idempotent shutdown. Local embeddings require Node 22+ for the isolated worker; set `PI_KNOWLEDGE_NODE_PATH` to `node.exe` if OMP runs from a non-Node packaged host. Compatibility-sensitive releases should validate both Pi and OMP install/runtime flows.
329
+ OMP compatibility covers path resolution, packaged entry loading, native SQLite dependency resolution, isolated worker startup, Windows-safe worker transport fallback, document parser stdio isolation, and idempotent shutdown. Local embeddings and PDF/DOCX extraction require Node 22+ for the isolated worker; set `PI_KNOWLEDGE_NODE_PATH` to `node.exe` if OMP runs from a non-Node packaged host. Compatibility-sensitive releases should validate both Pi and OMP install/runtime flows.
310
330
 
311
331
  ## Large Project Indexing
312
332
 
package/dist/index.js CHANGED
@@ -7,6 +7,7 @@ var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExte
7
7
  return path;
8
8
  };
9
9
  import { MAX_SEARCH_RESULTS, MAX_SEARCH_WINDOW } from "./src/search/contracts.js";
10
+ import { formatKnowledgeStatus } from "./src/status.js";
10
11
  const Type = {
11
12
  Object(properties) {
12
13
  const required = Object.entries(properties)
@@ -253,7 +254,7 @@ export default function (pi) {
253
254
  pi.registerTool({
254
255
  name: "knowledge_configure",
255
256
  label: "Knowledge Configure",
256
- description: "Configure pi-knowledge runtime settings such as the local model worker Node executable",
257
+ description: "Configure pi-knowledge runtime settings such as the isolated worker Node executable",
257
258
  approval: "write",
258
259
  promptSnippet: "Configure local pi-knowledge runtime prerequisites before indexing",
259
260
  promptGuidelines: [
@@ -264,7 +265,7 @@ export default function (pi) {
264
265
  ],
265
266
  parameters: Type.Object({
266
267
  node_path: Type.Optional(Type.String({
267
- description: "Optional full path to a Node 22+ executable for the isolated local embedding model worker, such as C:\\Program Files\\nodejs\\node.exe",
268
+ description: "Optional full path to a Node 22+ executable for the isolated local model and document extraction worker, such as C:\\Program Files\\nodejs\\node.exe",
268
269
  })),
269
270
  }),
270
271
  async execute(_id, params, _signal) {
@@ -279,7 +280,7 @@ export default function (pi) {
279
280
  content: [
280
281
  {
281
282
  type: "text",
282
- text: `Configured local model worker Node executable: ${configuredPath}`,
283
+ text: `Configured isolated worker Node executable: ${configuredPath}`,
283
284
  },
284
285
  ],
285
286
  details: { node_path: configuredPath },
@@ -557,69 +558,26 @@ export default function (pi) {
557
558
  pi.registerTool({
558
559
  name: "knowledge_status",
559
560
  label: "Knowledge Status",
560
- description: "Show knowledge engine status with health diagnostics: staleness, orphans, and coverage",
561
+ description: "Show knowledge engine status with embedding compatibility, staleness, orphan, and coverage diagnostics",
561
562
  approval: "read",
562
563
  parameters: Type.Object({}),
563
564
  async execute(_id, _params, _signal) {
564
565
  if (_signal?.aborted)
565
566
  throw new Error("Cancelled");
566
567
  const { engine, storage, watcher } = await ensureInitialized();
567
- const kbs = engine.list();
568
- const watchCount = watcher.getActiveWatcherCount();
569
- const diagnostics = engine.diagnose(_signal);
570
- const lines = [
571
- `Storage: ${storage.getDefaultKnowledgeDir()}`,
572
- `Knowledge bases: ${kbs.length}`,
573
- `Active watchers: ${watchCount}`,
574
- "",
575
- ];
576
- for (const kb of kbs) {
577
- const age = Math.round((Date.now() - kb.updated_at) / 60000);
578
- const diag = diagnostics.find((d) => d.kb_id === kb.id);
579
- lines.push(` "${kb.name}" — ${kb.status} — ${kb.chunk_count} chunks, ${kb.file_count} files — updated ${age}m ago`);
580
- if (kb.source_path)
581
- lines.push(` source: ${kb.source_path}`);
582
- if (diag) {
583
- if (diag.job) {
584
- const jobAge = Math.round((Date.now() - diag.job.last_progress_at) / 1000);
585
- const progress = [
586
- `${diag.job.processed_files} files`,
587
- `${diag.job.processed_chunks} chunks`,
588
- diag.job.skipped_total > 0 ? `${diag.job.skipped_total} skipped` : "",
589
- diag.job.added_chunks > 0 ? `+${diag.job.added_chunks}` : "",
590
- diag.job.removed_chunks > 0 ? `-${diag.job.removed_chunks}` : "",
591
- diag.job.unchanged_chunks > 0 ? `=${diag.job.unchanged_chunks}` : "",
592
- ]
593
- .filter(Boolean)
594
- .join(", ");
595
- lines.push(` job: ${diag.job.status}/${diag.job.phase} — ${progress || "no processed items yet"} — last progress ${jobAge}s ago`);
596
- if (diag.job.message)
597
- lines.push(` last: ${diag.job.message}`);
598
- if (diag.job.error_message)
599
- lines.push(` error: ${diag.job.error_message}`);
600
- }
601
- lines.push(` coverage: ${diag.coverage_percent}% (${diag.indexed_files}/${diag.total_source_files} files)`);
602
- if (diag.stale_files.length > 0)
603
- lines.push(` ⚠️ stale: ${diag.stale_files.length} files modified since last index`);
604
- if (diag.orphan_files.length > 0)
605
- lines.push(` ⚠️ orphans: ${diag.orphan_files.length} chunks reference deleted files`);
606
- if (diag.stuck_indexing)
607
- lines.push(` ⚠️ indexing appears stuck for ${Math.round(diag.last_progress_age_ms / 60000)}m; remove and rebuild if no pi process is actively indexing it`);
608
- if (diag.skipped_files.total > 0)
609
- lines.push(` skipped: ${diag.skipped_files.total} files (${Object.entries(diag.skipped_files.by_reason)
610
- .filter(([, count]) => count > 0)
611
- .map(([reason, count]) => `${reason}: ${count}`)
612
- .join(", ")})`);
613
- }
614
- }
615
- const totalStale = diagnostics.reduce((n, d) => n + d.stale_files.length, 0);
616
- const totalOrphans = diagnostics.reduce((n, d) => n + d.orphan_files.length, 0);
617
- const totalStuck = diagnostics.filter((d) => d.stuck_indexing).length;
618
- if (totalStale === 0 && totalOrphans === 0 && totalStuck === 0 && kbs.length > 0)
619
- lines.push("", "Health: ✓ all indexes up to date");
620
- else if (totalStale > 0 || totalOrphans > 0 || totalStuck > 0)
621
- lines.push("", `Health: ⚠️ ${totalStale} stale, ${totalOrphans} orphans, ${totalStuck} stuck indexing — run knowledge_update or rebuild affected KBs`);
622
- return { content: [{ type: "text", text: lines.join("\n") }] };
568
+ const snapshot = engine.statusSnapshot(_signal);
569
+ return {
570
+ content: [
571
+ {
572
+ type: "text",
573
+ text: formatKnowledgeStatus({
574
+ snapshot,
575
+ storagePath: storage.getDefaultKnowledgeDir(),
576
+ watchCount: watcher.getActiveWatcherCount(),
577
+ }),
578
+ },
579
+ ],
580
+ };
623
581
  },
624
582
  });
625
583
  pi.registerTool({
@@ -0,0 +1,33 @@
1
+ export type EmbeddingRole = "query" | "passage";
2
+ export type EmbeddingDecorationStrategy = "e5" | "none" | "jina-code" | "custom";
3
+ export type EmbeddingTask = "nl2code" | "code2code" | "code2nl" | "code2completion" | "qa";
4
+ export type EmbeddingPrefixMode = "on" | "off";
5
+ export interface EmbeddingInstructionSet {
6
+ query: string;
7
+ document: string;
8
+ }
9
+ export interface ResolvedEmbeddingDecoration {
10
+ strategy: EmbeddingDecorationStrategy;
11
+ task?: EmbeddingTask;
12
+ instructions: EmbeddingInstructionSet;
13
+ prefixMode: EmbeddingPrefixMode;
14
+ }
15
+ export declare const MAX_INSTRUCTION_CHARS = 512;
16
+ export declare const DEFAULT_EMBEDDING_TASK: EmbeddingTask;
17
+ export declare const ACCEPTED_DECORATION_VALUES: readonly ["on", "off", "e5", "none", "jina", "jina-code", "custom"];
18
+ export declare const ACCEPTED_EMBEDDING_TASKS: readonly ["nl2code", "code2code", "code2nl", "code2completion", "qa"];
19
+ export declare const E5_INSTRUCTIONS: EmbeddingInstructionSet;
20
+ export declare const NO_INSTRUCTIONS: EmbeddingInstructionSet;
21
+ /**
22
+ * Published prompts for jinaai/jina-code-embeddings-1.5b.
23
+ * Source: config_sentence_transformers.json at model revision
24
+ * 39aeb4fb9b60f930934c78ae5d749a46287c248a (verified 2026-11-12).
25
+ * Values already contain their required trailing newline and are concatenated verbatim.
26
+ */
27
+ export declare const JINA_CODE_INSTRUCTIONS: Readonly<Record<EmbeddingTask, EmbeddingInstructionSet>>;
28
+ export declare function decorateEmbeddingInput(text: string, role: EmbeddingRole, instructions: EmbeddingInstructionSet): string;
29
+ export declare function embeddingInstructionLength(instructions: EmbeddingInstructionSet, role: EmbeddingRole): number;
30
+ export declare function embeddingInstructionSetHash(instructions: EmbeddingInstructionSet): string;
31
+ export declare function embeddingDecorationDescriptor(decoration: ResolvedEmbeddingDecoration): string;
32
+ export declare function resolveEmbeddingDecoration(env: NodeJS.ProcessEnv): ResolvedEmbeddingDecoration;
33
+ export declare function decorationSupportedByLocalModel(strategy: EmbeddingDecorationStrategy): boolean;
@@ -0,0 +1,146 @@
1
+ import { createHash } from "node:crypto";
2
+ export const MAX_INSTRUCTION_CHARS = 512;
3
+ export const DEFAULT_EMBEDDING_TASK = "nl2code";
4
+ export const ACCEPTED_DECORATION_VALUES = ["on", "off", "e5", "none", "jina", "jina-code", "custom"];
5
+ export const ACCEPTED_EMBEDDING_TASKS = ["nl2code", "code2code", "code2nl", "code2completion", "qa"];
6
+ const STRATEGY_ALIASES = {
7
+ on: "e5",
8
+ e5: "e5",
9
+ off: "none",
10
+ none: "none",
11
+ jina: "jina-code",
12
+ "jina-code": "jina-code",
13
+ custom: "custom",
14
+ };
15
+ const STRATEGY_ENV = "PI_KNOWLEDGE_EMBEDDING_PREFIXES";
16
+ const TASK_ENV = "PI_KNOWLEDGE_EMBEDDING_TASK";
17
+ const QUERY_INSTRUCTION_ENV = "PI_KNOWLEDGE_EMBEDDING_QUERY_INSTRUCTION";
18
+ const DOCUMENT_INSTRUCTION_ENV = "PI_KNOWLEDGE_EMBEDDING_DOCUMENT_INSTRUCTION";
19
+ export const E5_INSTRUCTIONS = Object.freeze({ query: "query: ", document: "passage: " });
20
+ export const NO_INSTRUCTIONS = Object.freeze({ query: "", document: "" });
21
+ /**
22
+ * Published prompts for jinaai/jina-code-embeddings-1.5b.
23
+ * Source: config_sentence_transformers.json at model revision
24
+ * 39aeb4fb9b60f930934c78ae5d749a46287c248a (verified 2026-11-12).
25
+ * Values already contain their required trailing newline and are concatenated verbatim.
26
+ */
27
+ export const JINA_CODE_INSTRUCTIONS = Object.freeze({
28
+ nl2code: Object.freeze({
29
+ query: "Find the most relevant code snippet given the following query:\n",
30
+ document: "Candidate code snippet:\n",
31
+ }),
32
+ code2code: Object.freeze({
33
+ query: "Find an equivalent code snippet given the following code snippet:\n",
34
+ document: "Candidate code snippet:\n",
35
+ }),
36
+ code2nl: Object.freeze({
37
+ query: "Find the most relevant comment given the following code snippet:\n",
38
+ document: "Candidate comment:\n",
39
+ }),
40
+ code2completion: Object.freeze({
41
+ query: "Find the most relevant completion given the following start of code snippet:\n",
42
+ document: "Candidate completion:\n",
43
+ }),
44
+ qa: Object.freeze({
45
+ query: "Find the most relevant answer given the following question:\n",
46
+ document: "Candidate answer:\n",
47
+ }),
48
+ });
49
+ export function decorateEmbeddingInput(text, role, instructions) {
50
+ return `${role === "query" ? instructions.query : instructions.document}${text}`;
51
+ }
52
+ export function embeddingInstructionLength(instructions, role) {
53
+ return (role === "query" ? instructions.query : instructions.document).length;
54
+ }
55
+ export function embeddingInstructionSetHash(instructions) {
56
+ return createHash("sha256")
57
+ .update(JSON.stringify([instructions.query, instructions.document]))
58
+ .digest("hex")
59
+ .slice(0, 12);
60
+ }
61
+ export function embeddingDecorationDescriptor(decoration) {
62
+ if (decoration.strategy === "e5")
63
+ return "q=query:d=passage";
64
+ if (decoration.strategy === "none")
65
+ return "q=query:d=passage:prefixes=off";
66
+ const instructionHash = embeddingInstructionSetHash(decoration.instructions);
67
+ if (decoration.strategy === "jina-code") {
68
+ return `strategy=jina-code:task=${decoration.task ?? DEFAULT_EMBEDDING_TASK}:ih=${instructionHash}`;
69
+ }
70
+ return `strategy=custom:ih=${instructionHash}`;
71
+ }
72
+ function cleanValue(value) {
73
+ const trimmed = value?.trim();
74
+ return trimmed ? trimmed : undefined;
75
+ }
76
+ function assertInstructionValue(name, value) {
77
+ if (value.length > MAX_INSTRUCTION_CHARS) {
78
+ throw new Error(`Embedding instruction ${name} is ${value.length} characters; maximum is ${MAX_INSTRUCTION_CHARS}.`);
79
+ }
80
+ for (const character of value) {
81
+ const code = character.codePointAt(0) ?? 0;
82
+ if (character !== "\n" && (code < 0x20 || (code >= 0x7f && code <= 0x9f))) {
83
+ throw new Error(`Embedding instruction ${name} contains an unsupported control character; only printable text and newline are allowed.`);
84
+ }
85
+ }
86
+ }
87
+ function resolveCustomInstructions(env) {
88
+ const query = env[QUERY_INSTRUCTION_ENV];
89
+ const document = env[DOCUMENT_INSTRUCTION_ENV];
90
+ const missing = [
91
+ query === undefined ? QUERY_INSTRUCTION_ENV : "",
92
+ document === undefined ? DOCUMENT_INSTRUCTION_ENV : "",
93
+ ].filter(Boolean);
94
+ if (missing.length > 0)
95
+ throw new Error(`${STRATEGY_ENV}=custom requires ${missing.join(" and ")}.`);
96
+ if (query === undefined || document === undefined)
97
+ throw new Error("Embedding instruction resolution failed");
98
+ if (!query.trim())
99
+ throw new Error(`${QUERY_INSTRUCTION_ENV} must be a non-empty string.`);
100
+ assertInstructionValue(QUERY_INSTRUCTION_ENV, query);
101
+ assertInstructionValue(DOCUMENT_INSTRUCTION_ENV, document);
102
+ return Object.freeze({ query, document });
103
+ }
104
+ export function resolveEmbeddingDecoration(env) {
105
+ const rawStrategy = cleanValue(env[STRATEGY_ENV])?.toLowerCase();
106
+ const strategy = rawStrategy ? STRATEGY_ALIASES[rawStrategy] : "e5";
107
+ if (!strategy) {
108
+ throw new Error(`Unsupported embedding decoration strategy: ${rawStrategy}. Use ${STRATEGY_ENV}=${ACCEPTED_DECORATION_VALUES.join("|")}.`);
109
+ }
110
+ const customVariablesPresent = env[QUERY_INSTRUCTION_ENV] !== undefined || env[DOCUMENT_INSTRUCTION_ENV] !== undefined;
111
+ if (strategy !== "custom" && customVariablesPresent) {
112
+ throw new Error(`${QUERY_INSTRUCTION_ENV} and ${DOCUMENT_INSTRUCTION_ENV} are valid only when ${STRATEGY_ENV}=custom.`);
113
+ }
114
+ const rawTask = cleanValue(env[TASK_ENV])?.toLowerCase();
115
+ let task;
116
+ if (rawTask) {
117
+ if (strategy !== "jina-code") {
118
+ throw new Error(`${TASK_ENV} is valid only when ${STRATEGY_ENV}=jina-code.`);
119
+ }
120
+ if (!ACCEPTED_EMBEDDING_TASKS.includes(rawTask)) {
121
+ throw new Error(`Unsupported embedding task: ${rawTask}. Use ${TASK_ENV}=${ACCEPTED_EMBEDDING_TASKS.join("|")}.`);
122
+ }
123
+ task = rawTask;
124
+ }
125
+ else if (strategy === "jina-code") {
126
+ task = DEFAULT_EMBEDDING_TASK;
127
+ }
128
+ let instructions;
129
+ if (strategy === "e5")
130
+ instructions = E5_INSTRUCTIONS;
131
+ else if (strategy === "none")
132
+ instructions = NO_INSTRUCTIONS;
133
+ else if (strategy === "jina-code")
134
+ instructions = JINA_CODE_INSTRUCTIONS[task ?? DEFAULT_EMBEDDING_TASK];
135
+ else
136
+ instructions = resolveCustomInstructions(env);
137
+ return Object.freeze({
138
+ strategy,
139
+ task,
140
+ instructions,
141
+ prefixMode: strategy === "e5" ? "on" : "off",
142
+ });
143
+ }
144
+ export function decorationSupportedByLocalModel(strategy) {
145
+ return strategy === "e5" || strategy === "none";
146
+ }
@@ -1,14 +1,16 @@
1
+ import { type EmbeddingDecorationStrategy, type EmbeddingInstructionSet, type EmbeddingPrefixMode, type EmbeddingRole, type EmbeddingTask } from "./instructions.ts";
2
+ export type { EmbeddingDecorationStrategy, EmbeddingInstructionSet, EmbeddingRole, EmbeddingTask, } from "./instructions.ts";
1
3
  export type EmbeddingProvider = "local" | "openai";
2
- export type EmbeddingPrefix = "query" | "passage";
3
- export type EmbeddingPrefixStrategy = "on" | "off";
4
4
  export interface EmbeddingConfig {
5
5
  provider: EmbeddingProvider;
6
6
  model: string;
7
7
  baseUrl?: string;
8
8
  maxChars: number;
9
- queryPrefix: "query";
10
- documentPrefix: "passage";
11
- prefixes: EmbeddingPrefixStrategy;
9
+ strategy: EmbeddingDecorationStrategy;
10
+ task?: EmbeddingTask;
11
+ instructions: EmbeddingInstructionSet;
12
+ prefixMode: EmbeddingPrefixMode;
13
+ apiFallbackToLocal: boolean;
12
14
  pooling: "mean";
13
15
  normalize: true;
14
16
  }
@@ -18,6 +20,17 @@ export interface EmbeddingInputMeasurement {
18
20
  preferredMaximum: number;
19
21
  unit: "tokens" | "characters";
20
22
  }
23
+ export type EmbeddingCompatibilityCause = "match" | "unrecorded" | "dimension" | "decoration" | "configuration" | "signature_format";
24
+ export interface EmbeddingCompatibility {
25
+ compatible: boolean;
26
+ cause: EmbeddingCompatibilityCause;
27
+ activeSignature: string;
28
+ recordedSignature: string | null;
29
+ activeDecoration: string;
30
+ recordedDecoration?: string;
31
+ activeDimension: number;
32
+ recordedDimension: number | null;
33
+ }
21
34
  export declare const CURRENT_EMBEDDING_MODEL = "multilingual-e5-small";
22
35
  export declare const DEFAULT_LOCAL_EMBEDDING_MODEL = "multilingual-e5-small";
23
36
  export declare const DEFAULT_OPENAI_EMBEDDING_MODEL = "text-embedding-3-small";
@@ -29,18 +42,22 @@ export declare const API_VECTOR_NORMALIZATION_POLICY = "client-l2-v1";
29
42
  export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
30
43
  export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
31
44
  export declare function normalizeEmbeddingVector(vector: Float32Array): Float32Array;
45
+ export declare function describeEmbeddingDecoration(config: EmbeddingConfig): string;
32
46
  export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
47
+ export declare function recordedEmbeddingDecoration(signature: string | null): string | undefined;
48
+ export declare function inspectEmbeddingCompatibility(recordedSignature: string | null, recordedDimension: number | null, config: EmbeddingConfig, activeDimension: number): EmbeddingCompatibility;
49
+ export declare function describeEmbeddingCompatibility(compatibility: EmbeddingCompatibility): string;
33
50
  export declare function dispose(): Promise<void>;
34
51
  export declare function prepareForShutdown(): Promise<void>;
35
- export declare function measureEmbeddingInputs(texts: string[], prefix: EmbeddingPrefix, config: EmbeddingConfig, signal?: AbortSignal): Promise<EmbeddingInputMeasurement>;
36
- export declare function embedTextsWithConfig(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
52
+ export declare function measureEmbeddingInputs(texts: string[], role: EmbeddingRole, config: EmbeddingConfig, signal?: AbortSignal): Promise<EmbeddingInputMeasurement>;
53
+ export declare function embedTextsWithConfig(texts: string[], role: EmbeddingRole, signal?: AbortSignal, options?: {
37
54
  allowApiFallback?: boolean;
38
55
  inputValidated?: boolean;
39
56
  }): Promise<{
40
57
  vectors: Float32Array[];
41
58
  config: EmbeddingConfig;
42
59
  }>;
43
- export declare function embedTexts(texts: string[], prefix: EmbeddingPrefix, signal?: AbortSignal, options?: {
60
+ export declare function embedTexts(texts: string[], role: EmbeddingRole, signal?: AbortSignal, options?: {
44
61
  allowApiFallback?: boolean;
45
62
  inputValidated?: boolean;
46
63
  }): Promise<Float32Array[]>;