pi-knowledge 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +139 -26
- package/dist/index.js +15 -4
- package/dist/src/embedding/provider.d.ts +2 -0
- package/dist/src/embedding/provider.js +44 -3
- package/dist/src/engine.js +58 -127
- package/dist/src/search/contracts.d.ts +2 -0
- package/dist/src/search/contracts.js +2 -0
- package/dist/src/search/diversity.d.ts +16 -0
- package/dist/src/search/diversity.js +183 -0
- package/dist/src/search/tuning.d.ts +10 -0
- package/dist/src/search/tuning.js +20 -0
- package/docs/configuration.md +3 -3
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
## [0.13.0] - 2026-11-07
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- Added bounded `knowledge_search` pagination (`limit` 1-50, `offset + limit` up to 800) and diagnostics for per-KB retrieval breadth, merged/filtered/diversity candidate counts, and global-pool truncation.
|
|
9
|
+
- Added client-side validation and L2 normalization for OpenAI-compatible embedding vectors, with a signed normalization policy and `knowledge_doctor` rebuild guidance for older API-backed KBs.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- Reworked the quick start and added verified Pi and Oh My Pi upgrade guidance for npm and local sources, targeted updates, explicit-version pins, safe checkout updates, build activation, source verification, host restarts, and post-upgrade diagnostics.
|
|
13
|
+
- Clarified host-specific storage, opt-in watcher/context injection, model-cache sizing, source-only architecture links, and the distinction between structured result text and custom TUI renderers.
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- Reduced MMR-style diversity ranking from O(n³) repeated redundancy scans to behavior-preserving O(n²) running maxima, with cooperative event-loop yielding and cancellation checks (#21).
|
|
17
|
+
- Bounded merged multi-KB candidates before deep reranking, adaptive expansion, and diversity so query-time work cannot grow unchecked with the number of selected KBs.
|
|
18
|
+
|
|
3
19
|
## [0.12.0] - 2026-10-19
|
|
4
20
|
|
|
5
21
|
### Added
|
package/README.md
CHANGED
|
@@ -21,7 +21,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
21
21
|
|
|
22
22
|
## Highlights
|
|
23
23
|
|
|
24
|
-
- **Local-first project memory**: stores indexes under `~/.pi/knowledge
|
|
24
|
+
- **Local-first project memory**: stores indexes under the active host knowledge directory (`~/.pi/knowledge/` for Pi or `~/.omp/knowledge/` for OMP); no project files are modified.
|
|
25
25
|
- **Hybrid retrieval**: lexical-anchored BM25 + semantic vectors + normalized weighted score fusion, with semantic mode for vector-only conceptual recall.
|
|
26
26
|
- **Code- and document-aware indexing**: recursive AST chunking for supported code, heading-aware Markdown chunks, and an embedding-safe finalizer that preserves oversized multilingual tails without adding a heavy document runtime.
|
|
27
27
|
- **Structured URL indexing**: preserves HTML headings, paragraphs, lists, tables, and entities as lightweight Markdown-like text without a browser, Docling runtime, or new dependency.
|
|
@@ -43,7 +43,7 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
43
43
|
| **Adaptive contextual search** | ✅ | ❌ | ❌ |
|
|
44
44
|
| **Diversity reranking** | ✅ | ❌ | ❌ |
|
|
45
45
|
| **Incremental re-indexing** | ✅ | ❌ | ❌ |
|
|
46
|
-
| **File watcher (auto-update)** | ✅ | ❌ | ❌ |
|
|
46
|
+
| **File watcher (opt-in auto-update)** | ✅ | ❌ | ❌ |
|
|
47
47
|
| **Code-aware chunking** | ✅ (TS/JS/Py/Go/Rust/Java/Bash/C/C++/QML) | ❌ | ❌ |
|
|
48
48
|
| **Symbol/config/heading lookup** | ✅ | ❌ | ❌ |
|
|
49
49
|
| **Local embeddings (zero API)** | ✅ | ❌ | ✅ (qmd) |
|
|
@@ -52,9 +52,9 @@ Unlike `pi-memory` (which manages the agent's own notes), `pi-knowledge` indexes
|
|
|
52
52
|
| **Progress reporting + stuck indexing diagnostics** | ✅ | partial | ❌ |
|
|
53
53
|
| Cross-session persistence | ✅ | ✅ | ✅ |
|
|
54
54
|
| Pi extension native | ✅ | N/A | ✅ |
|
|
55
|
-
| Context injection per turn | ✅ | ❌ | ✅ |
|
|
56
|
-
|
|
|
57
|
-
|
|
|
55
|
+
| Context injection per turn (opt-in) | ✅ | ❌ | ✅ |
|
|
56
|
+
| Structured search result formatting | ✅ | N/A | ❌ |
|
|
57
|
+
| Custom tool TUI renderers | ❌ | N/A | ❌ |
|
|
58
58
|
| RPC mode support | ✅ | N/A | N/A |
|
|
59
59
|
|
|
60
60
|
TUI rendering currently uses Pi's stable default tool rendering plus targeted result formatting. A fuller custom renderer is intentionally not enabled yet because custom renderers must be width-safe across terminal sizes and Pi TUI modes.
|
|
@@ -73,36 +73,137 @@ TUI rendering currently uses Pi's stable default tool rendering plus targeted re
|
|
|
73
73
|
|
|
74
74
|
This is intentionally not a heavy ColBERT-style late-interaction index yet (Khattab and Zaharia 2020). The current product chooses lightweight local embeddings, BM25, query-aware ranking, optional cross-encoder reranking, streamed vector scans, and health diagnostics for commercial usefulness with low setup cost.
|
|
75
75
|
|
|
76
|
-
|
|
76
|
+
Project dogfood drove fixes for score compression, repeated same-file results,
|
|
77
|
+
garbage-query false positives, small-module discoverability, source-versus-test
|
|
78
|
+
ranking, indexing stability, and auto-mode false positives. Existing KBs should
|
|
79
|
+
be rebuilt or updated after upgrades that change indexing text.
|
|
77
80
|
|
|
78
81
|
## Quick Start
|
|
79
82
|
|
|
83
|
+
### Install
|
|
84
|
+
|
|
85
|
+
For Pi:
|
|
86
|
+
|
|
80
87
|
```bash
|
|
81
|
-
# Install for Pi
|
|
82
88
|
pi install npm:pi-knowledge
|
|
89
|
+
```
|
|
83
90
|
|
|
84
|
-
|
|
85
|
-
omp install npm:pi-knowledge
|
|
91
|
+
For OMP:
|
|
86
92
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
omp install npm:pi-knowledge
|
|
93
|
+
```bash
|
|
94
|
+
omp plugin install npm:pi-knowledge
|
|
95
|
+
```
|
|
91
96
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
omp install ./pi-knowledge
|
|
97
|
+
On minimal Ubuntu or Debian systems, install native build tools first if OMP
|
|
98
|
+
must compile `better-sqlite3`:
|
|
95
99
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
100
|
+
```bash
|
|
101
|
+
sudo apt-get update
|
|
102
|
+
sudo apt-get install -y build-essential python3
|
|
103
|
+
omp plugin install npm:pi-knowledge
|
|
104
|
+
```
|
|
99
105
|
|
|
100
|
-
|
|
101
|
-
> Search my knowledge base for "authentication flow"
|
|
106
|
+
For a local checkout, install dependencies and build before registration:
|
|
102
107
|
|
|
103
|
-
|
|
108
|
+
```bash
|
|
109
|
+
cd /path/to/pi-knowledge
|
|
110
|
+
npm ci
|
|
111
|
+
npm run typecheck
|
|
112
|
+
npm test
|
|
113
|
+
npm run build
|
|
114
|
+
|
|
115
|
+
# Register with the host you use:
|
|
116
|
+
pi install ./
|
|
117
|
+
omp plugin link "$(pwd)"
|
|
104
118
|
```
|
|
105
119
|
|
|
120
|
+
Use either the npm package or the local checkout for a host, not both. Check
|
|
121
|
+
Pi with `pi list` and OMP with `omp plugin list --json` when switching sources.
|
|
122
|
+
|
|
123
|
+
### Index and search
|
|
124
|
+
|
|
125
|
+
Ask the agent:
|
|
126
|
+
|
|
127
|
+
```text
|
|
128
|
+
Index my project docs at ./docs as "Project Docs".
|
|
129
|
+
Search my knowledge bases for "authentication flow".
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The agent receives guidance to use `knowledge_symbol_search` and
|
|
133
|
+
`knowledge_search` when knowledge bases are available. Automatic behavior is
|
|
134
|
+
opt-in. Set one or both variables before starting Pi or OMP:
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
export PI_KNOWLEDGE_AUTO_INJECT=true # retrieve context before every model call
|
|
138
|
+
export PI_KNOWLEDGE_WATCH=true # update directory KBs after file changes
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Per-turn injection adds retrieval work and prompt context to every model call.
|
|
142
|
+
File watching applies only to directory-backed knowledge bases.
|
|
143
|
+
|
|
144
|
+
## Upgrade
|
|
145
|
+
|
|
146
|
+
### npm-installed package
|
|
147
|
+
|
|
148
|
+
Update only pi-knowledge in Pi:
|
|
149
|
+
|
|
150
|
+
```bash
|
|
151
|
+
pi update npm:pi-knowledge
|
|
152
|
+
pi list
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Use `pi update --extensions` instead when every installed Pi extension should
|
|
156
|
+
be updated.
|
|
157
|
+
|
|
158
|
+
Update only pi-knowledge in OMP:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
omp plugin upgrade pi-knowledge
|
|
162
|
+
omp plugin list --json
|
|
163
|
+
omp plugin doctor
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Bare `omp plugin upgrade` and `omp update --plugins --check` report marketplace
|
|
167
|
+
plugin status; they do not confirm that this npm plugin was upgraded.
|
|
168
|
+
|
|
169
|
+
Install the unversioned Pi source (`npm:pi-knowledge`) to follow the npm
|
|
170
|
+
`latest` tag. Pi preserves explicitly versioned sources such as
|
|
171
|
+
`npm:pi-knowledge@<version>`. To remove a pin:
|
|
172
|
+
|
|
173
|
+
1. Run `pi list`.
|
|
174
|
+
2. Pass the exact pinned source shown there to `pi remove`.
|
|
175
|
+
3. Run `pi install npm:pi-knowledge`.
|
|
176
|
+
|
|
177
|
+
### Local checkout
|
|
178
|
+
|
|
179
|
+
A linked checkout is updated from Git, not from the npm registry. Finish,
|
|
180
|
+
commit, or stash local edits before pulling. Exit Pi and OMP processes that use
|
|
181
|
+
the checkout before replacing dependencies or build output.
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
cd /path/to/pi-knowledge
|
|
185
|
+
git status --short
|
|
186
|
+
git pull --ff-only # only when the checkout is clean and follows a remote branch
|
|
187
|
+
npm ci
|
|
188
|
+
npm run typecheck
|
|
189
|
+
npm test
|
|
190
|
+
npm run build
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
When actively developing local changes, skip `git pull --ff-only` and build the
|
|
194
|
+
current tree. `extension.js` prefers `dist/index.js`, so source edits are
|
|
195
|
+
inactive until the build succeeds. The local registration only needs to be
|
|
196
|
+
repeated when its path or host settings change.
|
|
197
|
+
|
|
198
|
+
Pi and OMP load extension modules once per process. Start a new host process
|
|
199
|
+
after a successful registry upgrade or local build. Then use `pi list` to
|
|
200
|
+
confirm Pi's selected source, or `omp plugin list --json` to confirm OMP's
|
|
201
|
+
installed version and path.
|
|
202
|
+
|
|
203
|
+
After releases that change indexing text, run `knowledge_doctor` and follow its
|
|
204
|
+
suggested reindex action. Source-backed knowledge bases can normally use
|
|
205
|
+
`knowledge_update`; rebuild only when diagnostics or release notes require it.
|
|
206
|
+
|
|
106
207
|
## Tools
|
|
107
208
|
|
|
108
209
|
| Tool | Description |
|
|
@@ -145,6 +246,8 @@ Mode selection contract:
|
|
|
145
246
|
|
|
146
247
|
Search results use balanced diversity reranking by default so near-duplicate chunks from the same file do not dominate the top results. Diversity scoring considers lexical overlap, same-file line proximity, overlapping adaptive windows, available embedding-vector similarity, file-level interleaving, and a small KB trust multiplier that favors ready file/directory sources over stale, URL, or imported text sources. In `adaptive` mode, code chunks with AST metadata can prefer same-parent/sibling chunks inside the same bounded file window; this is metadata-driven context expansion, not caller/callee analysis. Use `diversity: "off"` only when raw ranking order is needed for diagnostics. Agents can request search diagnostics to inspect mode fallback, selected profile, applied search tuning, ranking coverage, path/source/test boosts, adjusted scores, and provenance such as chunk id, chunk hash, match reason, stale flag, source freshness, and adaptive `source_chunk_ids` where available.
|
|
147
248
|
|
|
249
|
+
`limit` must be an integer from 1 to 50. `offset` must be a non-negative integer, and `offset + limit` cannot exceed the bounded 800-candidate search window. Each selected KB may contribute candidates up to the profile-derived retrieval limit, but merged candidates are globally ranked and capped before deep reranking, adaptive expansion, or diversity work. Diagnostics expose per-KB retrieval breadth, retrieved and filtered counts, the diversity pool size, and whether the global pool was truncated.
|
|
250
|
+
|
|
148
251
|
For best search quality, rebuild or update existing knowledge bases after upgrading. New indexes use contextual retrieval units: embeddings and FTS include file path, file type, Markdown heading breadcrumbs, and AST-derived code structure such as symbol names, scopes, parent symbols, and signatures while returned results keep the original chunk text readable. Before embedding, the complete contextual input is checked against the active provider budget; only oversized chunks are split again along structural boundaries.
|
|
149
252
|
Symbol/config/heading metadata and embedding-safe chunk boundaries are rebuilt during `knowledge_update`. `knowledge_doctor` reports `reindex_for_embedding_budget` for older KBs: file, directory, and URL sources can use `knowledge_update`; imported or inline-text sources should be re-imported or re-added from the original source.
|
|
150
253
|
|
|
@@ -173,6 +276,8 @@ export PI_KNOWLEDGE_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
|
173
276
|
|
|
174
277
|
API embedding failures are surfaced by default so configuration and context-window problems are visible. To explicitly allow query-time local fallback after API failures, set `PI_KNOWLEDGE_EMBEDDING_API_FALLBACK=local`; indexing/import/update document batches still fail instead of creating mixed-provider vector files. Inputs are never silently truncated.
|
|
175
278
|
|
|
279
|
+
OpenAI-compatible API vectors are validated and L2-normalized by the client before query use or persistence, so cosine-style dot products do not depend on a provider-specific vector magnitude. This normalization policy is part of the embedding signature. KBs created with an older API signature must run `knowledge_update`; `knowledge_doctor` reports `rebuild_vectors` when the active provider no longer matches stored vector semantics.
|
|
280
|
+
|
|
176
281
|
OpenAI-compatible embedding inputs use a 20000-character complete-input safety cap by default. Indexing splits oversized chunks before HTTP requests; direct oversized query/document calls fail explicitly. Adjust the cap with `PI_KNOWLEDGE_EMBEDDING_MAX_CHARS` when the remote model has a different context window. Changing the cap changes embedding compatibility and requires `knowledge_update`.
|
|
177
282
|
|
|
178
283
|
For self-hosted servers with smaller request limits, set `PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE` to the maximum number of chunk inputs the server accepts per embedding request.
|
|
@@ -215,17 +320,25 @@ Update and diagnostics paths are also streaming-oriented: changed chunks are emb
|
|
|
215
320
|
|
|
216
321
|
## Architecture
|
|
217
322
|
|
|
218
|
-
[DESIGN.md](
|
|
323
|
+
[DESIGN.md](https://github.com/nczz/pi-knowledge/blob/main/DESIGN.md) is the
|
|
324
|
+
historical technical design. For current behavior contracts, prefer this
|
|
325
|
+
README, the repository's
|
|
326
|
+
[AGENTS.md](https://github.com/nczz/pi-knowledge/blob/main/AGENTS.md),
|
|
327
|
+
[docs/configuration.md](docs/configuration.md), and the source-only
|
|
328
|
+
[ADR and pitfall notes](https://github.com/nczz/pi-knowledge/tree/main/docs).
|
|
329
|
+
The npm package includes the user-facing configuration and offline-mode guides;
|
|
330
|
+
the repository retains the broader design history.
|
|
219
331
|
|
|
220
332
|
## Data Storage
|
|
221
333
|
|
|
222
|
-
All data is stored
|
|
334
|
+
All data is stored under the active host knowledge directory unless overridden;
|
|
335
|
+
indexed projects are never used for internal storage:
|
|
223
336
|
|
|
224
337
|
```
|
|
225
|
-
|
|
338
|
+
<knowledge-dir>/
|
|
226
339
|
├── knowledge.db ← SQLite (metadata + chunks + FTS5 index)
|
|
227
340
|
├── vectors/ ← Embedding vectors per KB (binary)
|
|
228
|
-
└── models/ ←
|
|
341
|
+
└── models/ ← ONNX cache (embedding ~32MB; rerankers add artifacts)
|
|
229
342
|
```
|
|
230
343
|
|
|
231
344
|
- **Backup**: copy the active knowledge directory, usually `~/.pi/knowledge/` or `~/.omp/knowledge/`
|
package/dist/index.js
CHANGED
|
@@ -6,6 +6,7 @@ var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExte
|
|
|
6
6
|
}
|
|
7
7
|
return path;
|
|
8
8
|
};
|
|
9
|
+
import { MAX_SEARCH_RESULTS, MAX_SEARCH_WINDOW } from "./src/search/contracts.js";
|
|
9
10
|
const Type = {
|
|
10
11
|
Object(properties) {
|
|
11
12
|
const required = Object.entries(properties)
|
|
@@ -397,9 +398,19 @@ export default function (pi) {
|
|
|
397
398
|
Type.Literal("code"),
|
|
398
399
|
Type.Literal("docs"),
|
|
399
400
|
])),
|
|
400
|
-
limit: Type.Optional(Type.Number({
|
|
401
|
+
limit: Type.Optional(Type.Number({
|
|
402
|
+
description: "Max results (default comes from search profile/env)",
|
|
403
|
+
minimum: 1,
|
|
404
|
+
maximum: MAX_SEARCH_RESULTS,
|
|
405
|
+
multipleOf: 1,
|
|
406
|
+
})),
|
|
401
407
|
kb_id: Type.Optional(Type.String({ description: "Limit search to a specific KB by ID or exact name" })),
|
|
402
|
-
offset: Type.Optional(Type.Number({
|
|
408
|
+
offset: Type.Optional(Type.Number({
|
|
409
|
+
description: `Pagination offset; offset + limit must not exceed ${MAX_SEARCH_WINDOW}`,
|
|
410
|
+
minimum: 0,
|
|
411
|
+
maximum: MAX_SEARCH_WINDOW - 1,
|
|
412
|
+
multipleOf: 1,
|
|
413
|
+
})),
|
|
403
414
|
file_type: Type.Optional(Type.String({ description: "Filter by file type (e.g. typescript, markdown, python)" })),
|
|
404
415
|
path_pattern: Type.Optional(Type.String({ description: "Filter by file path substring (for example src/engine.ts)" })),
|
|
405
416
|
diversity: Type.Optional(Type.Union([Type.Literal("off"), Type.Literal("balanced"), Type.Literal("strong")])),
|
|
@@ -417,7 +428,7 @@ export default function (pi) {
|
|
|
417
428
|
response.retry_modes?.length ? `Retried: ${response.retry_modes.join(", ")}` : "",
|
|
418
429
|
response.warnings?.length ? `Warnings: ${response.warnings.join(" | ")}` : "",
|
|
419
430
|
diagnostics && response.tuning
|
|
420
|
-
? `Tuning: profile=${response.tuning.requested_profile}->${response.tuning.selected_profile}, limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.candidate_limit}`
|
|
431
|
+
? `Tuning: profile=${response.tuning.requested_profile}->${response.tuning.selected_profile}, limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.diversity_candidates}/${response.tuning.candidate_limit}, retrieval_per_kb=${response.tuning.retrieval_limit_per_kb}, retrieved=${response.tuning.retrieved_candidates}, filtered=${response.tuning.filtered_candidates}, truncated=${response.tuning.candidate_pool_truncated}`
|
|
421
432
|
: "",
|
|
422
433
|
response.suggestions?.length ? `Suggestions: ${response.suggestions.join(" | ")}` : "",
|
|
423
434
|
]
|
|
@@ -446,7 +457,7 @@ export default function (pi) {
|
|
|
446
457
|
? `\nProvenance: chunk=${r.provenance.chunk_id}, hash=${r.provenance.chunk_hash.slice(0, 12)}, reason=${r.provenance.match_reason}, indexed_at=${r.provenance.indexed_at}, source_mtime=${r.provenance.source_mtime ?? "unknown"}, stale=${r.provenance.stale}`
|
|
447
458
|
: "";
|
|
448
459
|
const tuning = diagnostics && response.tuning && i === 0
|
|
449
|
-
? `\nTuning: limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.candidate_limit}`
|
|
460
|
+
? `\nTuning: limit=${response.tuning.limit}, snippet=${response.tuning.snippet_max_length}, min_hybrid=${response.tuning.min_hybrid_score}, candidates=${response.tuning.diversity_candidates}/${response.tuning.candidate_limit}, retrieval_per_kb=${response.tuning.retrieval_limit_per_kb}, retrieved=${response.tuning.retrieved_candidates}, filtered=${response.tuning.filtered_candidates}, truncated=${response.tuning.candidate_pool_truncated}`
|
|
450
461
|
: "";
|
|
451
462
|
return `[${i + 1}] ${r.file_path} (${r.kb_name}, score: ${r.score.toFixed(3)})\n${r.snippet}${diag}${provenance}${tuning}`;
|
|
452
463
|
})
|
|
@@ -25,8 +25,10 @@ export declare const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
|
25
25
|
export declare const LOCAL_EMBEDDING_MAX_TOKENS = 512;
|
|
26
26
|
export declare const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
|
|
27
27
|
export declare const DEFAULT_API_MAX_EMBED_CHARS = 20000;
|
|
28
|
+
export declare const API_VECTOR_NORMALIZATION_POLICY = "client-l2-v1";
|
|
28
29
|
export declare function resolveEmbeddingConfig(env?: NodeJS.ProcessEnv): EmbeddingConfig;
|
|
29
30
|
export declare function embeddingConfigLabel(config: EmbeddingConfig): string;
|
|
31
|
+
export declare function normalizeEmbeddingVector(vector: Float32Array): Float32Array;
|
|
30
32
|
export declare function embeddingSignature(config: EmbeddingConfig, dimension: number): string;
|
|
31
33
|
export declare function dispose(): Promise<void>;
|
|
32
34
|
export declare function prepareForShutdown(): Promise<void>;
|
|
@@ -12,6 +12,7 @@ export const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
|
|
|
12
12
|
export const LOCAL_EMBEDDING_MAX_TOKENS = 512;
|
|
13
13
|
export const LOCAL_EMBEDDING_PREFERRED_TOKENS = 448;
|
|
14
14
|
export const DEFAULT_API_MAX_EMBED_CHARS = 20_000;
|
|
15
|
+
export const API_VECTOR_NORMALIZATION_POLICY = "client-l2-v1";
|
|
15
16
|
const IDLE_TIMEOUT_MS = Number(process.env.PI_KNOWLEDGE_EMBEDDING_IDLE_MS ?? 30_000);
|
|
16
17
|
const ENABLE_NATIVE_IDLE_DISPOSE = process.env.PI_KNOWLEDGE_ENABLE_NATIVE_IDLE_DISPOSE === "true";
|
|
17
18
|
const API_FALLBACK_TO_LOCAL = process.env.PI_KNOWLEDGE_EMBEDDING_API_FALLBACK === "local";
|
|
@@ -109,14 +110,30 @@ export function resolveEmbeddingConfig(env = process.env) {
|
|
|
109
110
|
export function embeddingConfigLabel(config) {
|
|
110
111
|
return config.provider === "local" ? config.model : `${config.provider}:${config.model}`;
|
|
111
112
|
}
|
|
113
|
+
export function normalizeEmbeddingVector(vector) {
|
|
114
|
+
let squaredMagnitude = 0;
|
|
115
|
+
for (const value of vector) {
|
|
116
|
+
if (!Number.isFinite(value))
|
|
117
|
+
throw new Error("Embedding provider returned a non-finite vector value");
|
|
118
|
+
squaredMagnitude += value * value;
|
|
119
|
+
}
|
|
120
|
+
if (!Number.isFinite(squaredMagnitude) || squaredMagnitude <= 0) {
|
|
121
|
+
throw new Error("Embedding provider returned a zero-magnitude vector");
|
|
122
|
+
}
|
|
123
|
+
const magnitude = Math.sqrt(squaredMagnitude);
|
|
124
|
+
for (let index = 0; index < vector.length; index++)
|
|
125
|
+
vector[index] /= magnitude;
|
|
126
|
+
return vector;
|
|
127
|
+
}
|
|
112
128
|
export function embeddingSignature(config, dimension) {
|
|
113
129
|
const base = config.baseUrl ? `:base-sha256=${providerInstanceHash(config.baseUrl)}` : "";
|
|
130
|
+
const normalization = config.provider === "openai" ? API_VECTOR_NORMALIZATION_POLICY : String(config.normalize);
|
|
114
131
|
const parts = [
|
|
115
132
|
`${config.provider}:${config.model}${base}`,
|
|
116
133
|
`dim=${dimension}`,
|
|
117
134
|
`apiMax=${config.provider === "openai" ? config.maxChars : "none"}`,
|
|
118
135
|
`pooling=${config.pooling}`,
|
|
119
|
-
`normalize=${
|
|
136
|
+
`normalize=${normalization}`,
|
|
120
137
|
`q=${config.queryPrefix}`,
|
|
121
138
|
`d=${config.documentPrefix}`,
|
|
122
139
|
];
|
|
@@ -195,8 +212,32 @@ async function embedViaAPI(texts, prefix, config, signal) {
|
|
|
195
212
|
const detail = await res.text().catch(() => "");
|
|
196
213
|
throw new Error(`OpenAI embedding API error: ${res.status}${detail ? ` ${detail.slice(0, 500)}` : ""}`);
|
|
197
214
|
}
|
|
198
|
-
const
|
|
199
|
-
|
|
215
|
+
const payload = (await res.json());
|
|
216
|
+
if (!payload || typeof payload !== "object" || !("data" in payload && Array.isArray(payload.data))) {
|
|
217
|
+
throw new Error("OpenAI embedding API response is missing a data array");
|
|
218
|
+
}
|
|
219
|
+
if (payload.data.length !== texts.length) {
|
|
220
|
+
throw new Error(`OpenAI embedding API returned ${payload.data.length} vectors for ${texts.length} inputs`);
|
|
221
|
+
}
|
|
222
|
+
let expectedDimension;
|
|
223
|
+
return payload.data.map((item, index) => {
|
|
224
|
+
if (!item || typeof item !== "object" || !("embedding" in item && Array.isArray(item.embedding))) {
|
|
225
|
+
throw new Error(`OpenAI embedding API response item ${index + 1} is missing an embedding array`);
|
|
226
|
+
}
|
|
227
|
+
if (expectedDimension !== undefined && item.embedding.length !== expectedDimension) {
|
|
228
|
+
throw new Error(`OpenAI embedding API response item ${index + 1} has dimension ${item.embedding.length}; expected ${expectedDimension}`);
|
|
229
|
+
}
|
|
230
|
+
expectedDimension = item.embedding.length;
|
|
231
|
+
const vector = new Float32Array(item.embedding.length);
|
|
232
|
+
for (let valueIndex = 0; valueIndex < item.embedding.length; valueIndex++) {
|
|
233
|
+
const value = item.embedding[valueIndex];
|
|
234
|
+
if (typeof value !== "number" || !Number.isFinite(value)) {
|
|
235
|
+
throw new Error(`OpenAI embedding API response item ${index + 1} contains a non-finite numeric value at position ${valueIndex}`);
|
|
236
|
+
}
|
|
237
|
+
vector[valueIndex] = value;
|
|
238
|
+
}
|
|
239
|
+
return normalizeEmbeddingVector(vector);
|
|
240
|
+
});
|
|
200
241
|
}
|
|
201
242
|
export async function embedTextsWithConfig(texts, prefix, signal, options = {}) {
|
|
202
243
|
const config = resolveEmbeddingConfig();
|
package/dist/src/engine.js
CHANGED
|
@@ -9,18 +9,19 @@ import { htmlToStructuredMarkdown } from "./indexer/html.js";
|
|
|
9
9
|
import { extractSymbols } from "./indexer/symbols.js";
|
|
10
10
|
import { shutdownModelWorker } from "./model-worker-client.js";
|
|
11
11
|
import { searchBM25 } from "./search/bm25.js";
|
|
12
|
+
import { MAX_SEARCH_WINDOW } from "./search/contracts.js";
|
|
13
|
+
import { diversifyRankedChunks, interleaveByFile } from "./search/diversity.js";
|
|
12
14
|
import { weightedScoreFusion } from "./search/fusion.js";
|
|
13
15
|
import { normalizedQueryText, tokenizeForSearch } from "./search/query.js";
|
|
14
16
|
import { hasAnyLexicalEvidence, hasEnoughLexicalEvidence, normalizeFileTypeFilter, queryCoverage, scoreChunkForQuery, } from "./search/ranking.js";
|
|
15
17
|
import { disposeReranker, prepareRerankerForShutdown, rerank } from "./search/reranker.js";
|
|
16
|
-
import { resolveSearchTuning, summarizeSearchTuning, } from "./search/tuning.js";
|
|
18
|
+
import { resolveSearchTuning, resolveSearchWindow, summarizeSearchTuning, } from "./search/tuning.js";
|
|
17
19
|
import { searchVectorFile } from "./search/vector.js";
|
|
18
20
|
import { countSymbols, createKB, deleteChunksByIds, deleteKB, deleteSymbolsByKB, finishIndexingJob, getChunkById, getChunkCount, getChunksByFile, getFileCount, getKB, getKBByName, getSymbolCount, insertChunks, insertSymbols, iterateChunkIdsByKB, iterateChunksByKB, listKBs, openDatabase, searchSymbols, startIndexingJob, updateIndexingJob, updateKBCounts, updateKBEmbeddingMetadata, updateKBSourceOptions, updateKBStatus, } from "./storage/sqlite.js";
|
|
19
21
|
export { CURRENT_EMBEDDING_MODEL } from "./embedding/provider.js";
|
|
20
22
|
export const CURRENT_URL_STRUCTURE_POLICY = 1;
|
|
21
23
|
export const CURRENT_EMBEDDING_INPUT_POLICY = 2;
|
|
22
24
|
const DEFAULT_INDEX_EMBED_BATCH_SIZE = 64;
|
|
23
|
-
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
24
25
|
function resolveIndexEmbedBatchSize(env = process.env) {
|
|
25
26
|
const raw = env.PI_KNOWLEDGE_EMBEDDING_BATCH_SIZE?.trim();
|
|
26
27
|
if (!raw)
|
|
@@ -160,116 +161,9 @@ function formatBytes(bytes) {
|
|
|
160
161
|
return `${mb.toFixed(1)} MB`;
|
|
161
162
|
return `${(mb / 1024).toFixed(2)} GB`;
|
|
162
163
|
}
|
|
163
|
-
function cosineSimilarity(a, b) {
|
|
164
|
-
if (!a || !b || a.length !== b.length)
|
|
165
|
-
return 0;
|
|
166
|
-
let dot = 0;
|
|
167
|
-
for (let i = 0; i < a.length; i++)
|
|
168
|
-
dot += a[i] * b[i];
|
|
169
|
-
return dot;
|
|
170
|
-
}
|
|
171
164
|
function tokenizeForSimilarity(text) {
|
|
172
165
|
return tokenizeForSearch(text);
|
|
173
166
|
}
|
|
174
|
-
function jaccardSimilarity(a, b) {
|
|
175
|
-
if (a.size === 0 || b.size === 0)
|
|
176
|
-
return 0;
|
|
177
|
-
let intersection = 0;
|
|
178
|
-
for (const token of a) {
|
|
179
|
-
if (b.has(token))
|
|
180
|
-
intersection++;
|
|
181
|
-
}
|
|
182
|
-
return intersection / (a.size + b.size - intersection);
|
|
183
|
-
}
|
|
184
|
-
function lineProximity(a, b) {
|
|
185
|
-
if (a.chunk.file_path !== b.chunk.file_path || a.chunk.kb_id !== b.chunk.kb_id)
|
|
186
|
-
return 0;
|
|
187
|
-
if (a.sourceChunkIds.some((id) => b.sourceChunkIds.includes(id)))
|
|
188
|
-
return 1;
|
|
189
|
-
const overlap = Math.max(0, Math.min(a.endLine, b.endLine) - Math.max(a.startLine, b.startLine) + 1);
|
|
190
|
-
if (overlap > 0)
|
|
191
|
-
return 1;
|
|
192
|
-
const gap = Math.max(a.startLine, b.startLine) - Math.min(a.endLine, b.endLine);
|
|
193
|
-
if (gap <= 20)
|
|
194
|
-
return 0.8;
|
|
195
|
-
if (gap <= 80)
|
|
196
|
-
return 0.45;
|
|
197
|
-
return 0.2;
|
|
198
|
-
}
|
|
199
|
-
function normalizeScores(candidates) {
|
|
200
|
-
const scores = candidates.map((candidate) => candidate.score);
|
|
201
|
-
const min = Math.min(...scores);
|
|
202
|
-
const max = Math.max(...scores);
|
|
203
|
-
const normalized = new Map();
|
|
204
|
-
for (const candidate of candidates) {
|
|
205
|
-
const value = max === min ? 1 : (candidate.score - min) / (max - min);
|
|
206
|
-
normalized.set(candidate.chunk.id, value);
|
|
207
|
-
}
|
|
208
|
-
return normalized;
|
|
209
|
-
}
|
|
210
|
-
function diversifyRankedChunks(candidates, diversity, vectorsByChunkId = new Map()) {
|
|
211
|
-
if (diversity === "off" || candidates.length <= 2)
|
|
212
|
-
return candidates;
|
|
213
|
-
const lambda = diversity === "strong" ? 0.62 : 0.76;
|
|
214
|
-
const normalized = normalizeScores(candidates);
|
|
215
|
-
const tokenSets = new Map(candidates.map((candidate) => [candidate.chunk.id, tokenizeForSimilarity(candidate.content)]));
|
|
216
|
-
const selected = [];
|
|
217
|
-
const remaining = [...candidates];
|
|
218
|
-
while (remaining.length > 0) {
|
|
219
|
-
let bestIndex = 0;
|
|
220
|
-
let bestScore = Number.NEGATIVE_INFINITY;
|
|
221
|
-
for (let i = 0; i < remaining.length; i++) {
|
|
222
|
-
const candidate = remaining[i];
|
|
223
|
-
let redundancy = 0;
|
|
224
|
-
for (const chosen of selected) {
|
|
225
|
-
const lexical = jaccardSimilarity(tokenSets.get(candidate.chunk.id) ?? new Set(), tokenSets.get(chosen.chunk.id) ?? new Set());
|
|
226
|
-
const vector = Math.max(0, cosineSimilarity(vectorsByChunkId.get(candidate.chunk.id), vectorsByChunkId.get(chosen.chunk.id)));
|
|
227
|
-
redundancy = Math.max(redundancy, Math.max(lexical, lineProximity(candidate, chosen), vector * VECTOR_REDUNDANCY_WEIGHT));
|
|
228
|
-
}
|
|
229
|
-
const relevance = normalized.get(candidate.chunk.id) ?? 0;
|
|
230
|
-
const mmrScore = lambda * relevance - (1 - lambda) * redundancy;
|
|
231
|
-
if (mmrScore > bestScore) {
|
|
232
|
-
bestScore = mmrScore;
|
|
233
|
-
bestIndex = i;
|
|
234
|
-
}
|
|
235
|
-
}
|
|
236
|
-
selected.push(remaining.splice(bestIndex, 1)[0]);
|
|
237
|
-
}
|
|
238
|
-
return selected;
|
|
239
|
-
}
|
|
240
|
-
function interleaveByFile(candidates, diversity) {
|
|
241
|
-
if (diversity === "off" || candidates.length <= 2)
|
|
242
|
-
return candidates;
|
|
243
|
-
const buckets = new Map();
|
|
244
|
-
const fileOrder = [];
|
|
245
|
-
for (const candidate of candidates) {
|
|
246
|
-
const key = `${candidate.chunk.kb_id}:${candidate.chunk.file_path}`;
|
|
247
|
-
if (!buckets.has(key)) {
|
|
248
|
-
buckets.set(key, []);
|
|
249
|
-
fileOrder.push(key);
|
|
250
|
-
}
|
|
251
|
-
buckets.get(key)?.push(candidate);
|
|
252
|
-
}
|
|
253
|
-
if (fileOrder.length <= 1)
|
|
254
|
-
return candidates;
|
|
255
|
-
const result = [];
|
|
256
|
-
let round = 0;
|
|
257
|
-
while (result.length < candidates.length) {
|
|
258
|
-
let added = false;
|
|
259
|
-
for (const key of fileOrder) {
|
|
260
|
-
const bucket = buckets.get(key);
|
|
261
|
-
const item = bucket?.[round];
|
|
262
|
-
if (!item)
|
|
263
|
-
continue;
|
|
264
|
-
result.push(item);
|
|
265
|
-
added = true;
|
|
266
|
-
}
|
|
267
|
-
if (!added)
|
|
268
|
-
break;
|
|
269
|
-
round++;
|
|
270
|
-
}
|
|
271
|
-
return result;
|
|
272
|
-
}
|
|
273
167
|
function buildQuerySnippet(content, query, maxLength = 240) {
|
|
274
168
|
const terms = [...tokenizeForSimilarity(query)].sort((a, b) => b.length - a.length);
|
|
275
169
|
const lower = content.toLowerCase();
|
|
@@ -1326,7 +1220,7 @@ export class KnowledgeEngine {
|
|
|
1326
1220
|
};
|
|
1327
1221
|
}
|
|
1328
1222
|
const db = this.db;
|
|
1329
|
-
const { mode = "hybrid", offset
|
|
1223
|
+
const { mode = "hybrid", offset: requestedOffset, kb_id, filters, diversity = "balanced" } = options;
|
|
1330
1224
|
const resolvedMode = retrievalModeFor(mode);
|
|
1331
1225
|
const retrievalMode = resolvedMode === "adaptive" ? "hybrid" : resolvedMode;
|
|
1332
1226
|
const normalizedQuery = normalizedQueryText(query);
|
|
@@ -1348,10 +1242,9 @@ export class KnowledgeEngine {
|
|
|
1348
1242
|
profile: options.profile,
|
|
1349
1243
|
kbSourceTypes: kbs.map((kb) => kb.source_type),
|
|
1350
1244
|
});
|
|
1351
|
-
const
|
|
1352
|
-
|
|
1353
|
-
|
|
1354
|
-
const candidateLimit = Math.max(tuning.candidateMin, offset + limit * tuning.candidateMultiplier);
|
|
1245
|
+
const { limit, offset } = resolveSearchWindow(options.limit, requestedOffset, tuning.defaultLimit);
|
|
1246
|
+
const candidateLimit = Math.min(MAX_SEARCH_WINDOW, Math.max(tuning.candidateMin, offset + limit * tuning.candidateMultiplier));
|
|
1247
|
+
const retrievalLimitPerKb = candidateLimit;
|
|
1355
1248
|
const tuningSummary = summarizeSearchTuning(tuning, limit, candidateLimit);
|
|
1356
1249
|
if (kbs.length === 0) {
|
|
1357
1250
|
return {
|
|
@@ -1365,39 +1258,42 @@ export class KnowledgeEngine {
|
|
|
1365
1258
|
}
|
|
1366
1259
|
const allResults = [];
|
|
1367
1260
|
const vectorsByChunkId = new Map();
|
|
1261
|
+
let queryEmbedding;
|
|
1368
1262
|
for (const kb of kbs) {
|
|
1369
1263
|
throwIfAborted(signal);
|
|
1370
1264
|
if (kb.chunk_count === 0)
|
|
1371
1265
|
continue;
|
|
1372
1266
|
const vectorPath = this.vectorPathFor(kb.id);
|
|
1373
1267
|
if (retrievalMode === "fast") {
|
|
1374
|
-
allResults.push(...searchBM25(db, normalizedQuery || query,
|
|
1268
|
+
allResults.push(...searchBM25(db, normalizedQuery || query, retrievalLimitPerKb, kb.id, { allowOrFallback: false }).map((result) => ({
|
|
1375
1269
|
...result,
|
|
1376
1270
|
score: result.score * kbTrustMultiplier(kb),
|
|
1377
1271
|
})));
|
|
1378
1272
|
}
|
|
1379
1273
|
else if (retrievalMode === "semantic") {
|
|
1380
|
-
|
|
1274
|
+
queryEmbedding ??= await embedQueryWithConfig(query, signal);
|
|
1275
|
+
const { vector: queryVec, config: queryEmbeddingConfig } = queryEmbedding;
|
|
1381
1276
|
throwIfAborted(signal);
|
|
1382
1277
|
if (!canSearchVectors(kb, queryEmbeddingConfig, queryVec.length)) {
|
|
1383
1278
|
warnings.push(embeddingMismatchWarning(kb));
|
|
1384
1279
|
continue;
|
|
1385
1280
|
}
|
|
1386
|
-
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id),
|
|
1281
|
+
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id), retrievalLimitPerKb, signal);
|
|
1387
1282
|
allResults.push(...vectorResults.results.map((result) => ({ ...result, score: result.score * kbTrustMultiplier(kb) })));
|
|
1388
1283
|
for (const [chunkId, vector] of vectorResults.vectorsByChunkId)
|
|
1389
1284
|
vectorsByChunkId.set(chunkId, vector);
|
|
1390
1285
|
}
|
|
1391
1286
|
else {
|
|
1392
1287
|
const semanticHybrid = retrievalMode === "semantic_hybrid";
|
|
1393
|
-
const bm25Results = searchBM25(db, normalizedQuery || query,
|
|
1288
|
+
const bm25Results = searchBM25(db, normalizedQuery || query, retrievalLimitPerKb, kb.id);
|
|
1394
1289
|
if (bm25Results.length === 0 && !semanticHybrid)
|
|
1395
1290
|
continue;
|
|
1396
1291
|
let vecResults = [];
|
|
1397
|
-
|
|
1292
|
+
queryEmbedding ??= await embedQueryWithConfig(query, signal);
|
|
1293
|
+
const { vector: queryVec, config: queryEmbeddingConfig } = queryEmbedding;
|
|
1398
1294
|
throwIfAborted(signal);
|
|
1399
1295
|
if (canSearchVectors(kb, queryEmbeddingConfig, queryVec.length)) {
|
|
1400
|
-
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id),
|
|
1296
|
+
const vectorResults = searchVectorFile(queryVec, vectorPath, iterateChunkIdsByKB(db, kb.id), retrievalLimitPerKb, signal);
|
|
1401
1297
|
vecResults = vectorResults.results;
|
|
1402
1298
|
for (const [chunkId, vector] of vectorResults.vectorsByChunkId)
|
|
1403
1299
|
vectorsByChunkId.set(chunkId, vector);
|
|
@@ -1461,8 +1357,15 @@ export class KnowledgeEngine {
|
|
|
1461
1357
|
return true;
|
|
1462
1358
|
});
|
|
1463
1359
|
}
|
|
1464
|
-
|
|
1465
|
-
|
|
1360
|
+
const candidatePool = filtered.slice(0, candidateLimit);
|
|
1361
|
+
const candidatePoolTruncated = candidatePool.length < filtered.length;
|
|
1362
|
+
const candidateIds = new Set(candidatePool.map((candidate) => candidate.chunkId));
|
|
1363
|
+
for (const chunkId of vectorsByChunkId.keys()) {
|
|
1364
|
+
if (!candidateIds.has(chunkId))
|
|
1365
|
+
vectorsByChunkId.delete(chunkId);
|
|
1366
|
+
}
|
|
1367
|
+
if (mode === "deep" && candidatePool.length > 0) {
|
|
1368
|
+
const candidates = candidatePool
|
|
1466
1369
|
.slice(0, tuning.deepRerankCandidates)
|
|
1467
1370
|
.map((r) => {
|
|
1468
1371
|
const chunk = getChunkById(db, r.chunkId);
|
|
@@ -1489,7 +1392,7 @@ export class KnowledgeEngine {
|
|
|
1489
1392
|
sourceChunkIds: [chunk.id],
|
|
1490
1393
|
});
|
|
1491
1394
|
}
|
|
1492
|
-
const diversified = interleaveByFile(diversifyRankedChunks(ranked, diversity, vectorsByChunkId), diversity);
|
|
1395
|
+
const diversified = interleaveByFile(await diversifyRankedChunks(ranked, diversity, vectorsByChunkId, signal), diversity, signal);
|
|
1493
1396
|
const page = diversified.slice(offset, offset + limit);
|
|
1494
1397
|
const results = page.map((r) => {
|
|
1495
1398
|
const kb = kbById.get(r.chunk.kb_id);
|
|
@@ -1521,11 +1424,17 @@ export class KnowledgeEngine {
|
|
|
1521
1424
|
has_more: offset + limit < diversified.length,
|
|
1522
1425
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1523
1426
|
mode_used: mode,
|
|
1524
|
-
tuning:
|
|
1427
|
+
tuning: {
|
|
1428
|
+
...tuningSummary,
|
|
1429
|
+
retrieved_candidates: unique.length,
|
|
1430
|
+
filtered_candidates: filtered.length,
|
|
1431
|
+
diversity_candidates: ranked.length,
|
|
1432
|
+
candidate_pool_truncated: candidatePoolTruncated,
|
|
1433
|
+
},
|
|
1525
1434
|
};
|
|
1526
1435
|
}
|
|
1527
1436
|
const ranked = [];
|
|
1528
|
-
for (const r of
|
|
1437
|
+
for (const r of candidatePool) {
|
|
1529
1438
|
throwIfAborted(signal);
|
|
1530
1439
|
const chunk = getChunkById(db, r.chunkId);
|
|
1531
1440
|
if (!chunk)
|
|
@@ -1563,7 +1472,7 @@ export class KnowledgeEngine {
|
|
|
1563
1472
|
});
|
|
1564
1473
|
}
|
|
1565
1474
|
}
|
|
1566
|
-
const diversified = interleaveByFile(diversifyRankedChunks(ranked, diversity, vectorsByChunkId), diversity);
|
|
1475
|
+
const diversified = interleaveByFile(await diversifyRankedChunks(ranked, diversity, vectorsByChunkId, signal), diversity, signal);
|
|
1567
1476
|
const total = diversified.length;
|
|
1568
1477
|
const page = diversified.slice(offset, offset + limit);
|
|
1569
1478
|
const results = page.map((r) => ({
|
|
@@ -1596,7 +1505,13 @@ export class KnowledgeEngine {
|
|
|
1596
1505
|
has_more: offset + limit < total,
|
|
1597
1506
|
warnings: warnings.length > 0 ? warnings : undefined,
|
|
1598
1507
|
mode_used: mode,
|
|
1599
|
-
tuning:
|
|
1508
|
+
tuning: {
|
|
1509
|
+
...tuningSummary,
|
|
1510
|
+
retrieved_candidates: unique.length,
|
|
1511
|
+
filtered_candidates: filtered.length,
|
|
1512
|
+
diversity_candidates: ranked.length,
|
|
1513
|
+
candidate_pool_truncated: candidatePoolTruncated,
|
|
1514
|
+
},
|
|
1600
1515
|
suggestions: results.length === 0
|
|
1601
1516
|
? [
|
|
1602
1517
|
"Try mode 'fast' for exact symbols, mode 'semantic' for pure vector search, or mode 'semantic_hybrid' when weak lexical overlap should not block fused vector results.",
|
|
@@ -1768,8 +1683,24 @@ export class KnowledgeEngine {
|
|
|
1768
1683
|
}
|
|
1769
1684
|
}
|
|
1770
1685
|
if (this.db) {
|
|
1686
|
+
const activeEmbeddingConfig = resolveEmbeddingConfig();
|
|
1771
1687
|
for (const kb of kbs) {
|
|
1772
1688
|
throwIfAborted(signal);
|
|
1689
|
+
if ((kb.status === "ready" || kb.status === "stale") &&
|
|
1690
|
+
kb.chunk_count > 0 &&
|
|
1691
|
+
kb.embedding_dimension !== null &&
|
|
1692
|
+
kb.embedding_signature !== embeddingSignature(activeEmbeddingConfig, kb.embedding_dimension)) {
|
|
1693
|
+
const canUpdate = kb.source_path !== null && kb.source_type !== "text";
|
|
1694
|
+
issues.push({
|
|
1695
|
+
severity: "warning",
|
|
1696
|
+
kb_name: kb.name,
|
|
1697
|
+
message: "This KB's vectors are incompatible with the active embedding provider or normalization policy.",
|
|
1698
|
+
action: canUpdate
|
|
1699
|
+
? "Run knowledge_update to rebuild vectors with the active embedding configuration."
|
|
1700
|
+
: "Re-add inline text or re-import the original source with the active embedding configuration.",
|
|
1701
|
+
action_code: "rebuild_vectors",
|
|
1702
|
+
});
|
|
1703
|
+
}
|
|
1773
1704
|
if ((kb.status === "ready" || kb.status === "stale") &&
|
|
1774
1705
|
(storedSourcePolicyVersion(kb.source_options, "embedding_input_policy") ?? 0) < CURRENT_EMBEDDING_INPUT_POLICY) {
|
|
1775
1706
|
const canUpdate = kb.source_path !== null && kb.source_type !== "text";
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
import type { Chunk } from "../storage/sqlite.ts";
|
|
2
|
+
import type { RankingDiagnostics } from "./ranking.ts";
|
|
3
|
+
export type DiversityMode = "off" | "balanced" | "strong" | undefined;
|
|
4
|
+
export interface RankedChunk {
|
|
5
|
+
chunk: Chunk;
|
|
6
|
+
kbName: string;
|
|
7
|
+
ranking?: RankingDiagnostics;
|
|
8
|
+
score: number;
|
|
9
|
+
content: string;
|
|
10
|
+
snippet: string;
|
|
11
|
+
startLine: number;
|
|
12
|
+
endLine: number;
|
|
13
|
+
sourceChunkIds: string[];
|
|
14
|
+
}
|
|
15
|
+
export declare function diversifyRankedChunks(candidates: RankedChunk[], diversity: DiversityMode, vectorsByChunkId?: Map<string, Float32Array>, signal?: AbortSignal): Promise<RankedChunk[]>;
|
|
16
|
+
export declare function interleaveByFile(candidates: RankedChunk[], diversity: DiversityMode, signal?: AbortSignal): RankedChunk[];
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
import { setImmediate as yieldToEventLoop } from "node:timers/promises";
|
|
2
|
+
import { tokenizeForSearch } from "./query.js";
|
|
3
|
+
const VECTOR_REDUNDANCY_WEIGHT = 0.35;
|
|
4
|
+
const PAIR_CHECK_INTERVAL = 1_024;
|
|
5
|
+
const CPU_SLICE_MS = 8;
|
|
6
|
+
function throwIfAborted(signal) {
|
|
7
|
+
if (signal?.aborted)
|
|
8
|
+
throw new Error("Cancelled");
|
|
9
|
+
}
|
|
10
|
+
function cosineSimilarity(a, b) {
|
|
11
|
+
if (!a || !b || a.length !== b.length)
|
|
12
|
+
return 0;
|
|
13
|
+
let dot = 0;
|
|
14
|
+
for (let i = 0; i < a.length; i++)
|
|
15
|
+
dot += a[i] * b[i];
|
|
16
|
+
return dot;
|
|
17
|
+
}
|
|
18
|
+
function jaccardSimilarity(a, b) {
|
|
19
|
+
if (a.size === 0 || b.size === 0)
|
|
20
|
+
return 0;
|
|
21
|
+
const [small, large] = a.size <= b.size ? [a, b] : [b, a];
|
|
22
|
+
let intersection = 0;
|
|
23
|
+
for (const token of small) {
|
|
24
|
+
if (large.has(token))
|
|
25
|
+
intersection++;
|
|
26
|
+
}
|
|
27
|
+
return intersection / (a.size + b.size - intersection);
|
|
28
|
+
}
|
|
29
|
+
function proximityMeta(candidate) {
|
|
30
|
+
return {
|
|
31
|
+
filePath: candidate.chunk.file_path,
|
|
32
|
+
kbId: candidate.chunk.kb_id,
|
|
33
|
+
startLine: candidate.startLine,
|
|
34
|
+
endLine: candidate.endLine,
|
|
35
|
+
sourceIds: new Set(candidate.sourceChunkIds),
|
|
36
|
+
};
|
|
37
|
+
}
|
|
38
|
+
function lineProximity(a, b) {
|
|
39
|
+
if (a.filePath !== b.filePath || a.kbId !== b.kbId)
|
|
40
|
+
return 0;
|
|
41
|
+
const [small, large] = a.sourceIds.size <= b.sourceIds.size ? [a.sourceIds, b.sourceIds] : [b.sourceIds, a.sourceIds];
|
|
42
|
+
for (const id of small) {
|
|
43
|
+
if (large.has(id))
|
|
44
|
+
return 1;
|
|
45
|
+
}
|
|
46
|
+
const overlap = Math.max(0, Math.min(a.endLine, b.endLine) - Math.max(a.startLine, b.startLine) + 1);
|
|
47
|
+
if (overlap > 0)
|
|
48
|
+
return 1;
|
|
49
|
+
const gap = Math.max(a.startLine, b.startLine) - Math.min(a.endLine, b.endLine);
|
|
50
|
+
if (gap <= 20)
|
|
51
|
+
return 0.8;
|
|
52
|
+
if (gap <= 80)
|
|
53
|
+
return 0.45;
|
|
54
|
+
return 0.2;
|
|
55
|
+
}
|
|
56
|
+
function normalizeScores(candidates) {
|
|
57
|
+
let min = Number.POSITIVE_INFINITY;
|
|
58
|
+
let max = Number.NEGATIVE_INFINITY;
|
|
59
|
+
for (const candidate of candidates) {
|
|
60
|
+
if (candidate.score < min)
|
|
61
|
+
min = candidate.score;
|
|
62
|
+
if (candidate.score > max)
|
|
63
|
+
max = candidate.score;
|
|
64
|
+
}
|
|
65
|
+
const normalized = new Float64Array(candidates.length);
|
|
66
|
+
for (let i = 0; i < candidates.length; i++) {
|
|
67
|
+
normalized[i] = max === min ? 1 : (candidates[i].score - min) / (max - min);
|
|
68
|
+
}
|
|
69
|
+
return normalized;
|
|
70
|
+
}
|
|
71
|
+
function firstUnusedIndex(used) {
|
|
72
|
+
for (let i = 0; i < used.length; i++) {
|
|
73
|
+
if (used[i] === 0)
|
|
74
|
+
return i;
|
|
75
|
+
}
|
|
76
|
+
throw new Error("Diversity selection exhausted its candidate pool unexpectedly");
|
|
77
|
+
}
|
|
78
|
+
export async function diversifyRankedChunks(candidates, diversity, vectorsByChunkId = new Map(), signal) {
|
|
79
|
+
throwIfAborted(signal);
|
|
80
|
+
if (diversity === "off" || candidates.length <= 2)
|
|
81
|
+
return candidates;
|
|
82
|
+
const lambda = diversity === "strong" ? 0.62 : 0.76;
|
|
83
|
+
const n = candidates.length;
|
|
84
|
+
const tokenSets = new Array(n);
|
|
85
|
+
const vectors = new Array(n);
|
|
86
|
+
const metas = new Array(n);
|
|
87
|
+
const relevance = normalizeScores(candidates);
|
|
88
|
+
for (let i = 0; i < n; i++) {
|
|
89
|
+
const candidate = candidates[i];
|
|
90
|
+
tokenSets[i] = tokenizeForSearch(candidate.content);
|
|
91
|
+
vectors[i] = vectorsByChunkId.get(candidate.chunk.id);
|
|
92
|
+
metas[i] = proximityMeta(candidate);
|
|
93
|
+
}
|
|
94
|
+
const pairRedundancy = (a, b) => {
|
|
95
|
+
let best = lineProximity(metas[a], metas[b]);
|
|
96
|
+
const lexical = jaccardSimilarity(tokenSets[a], tokenSets[b]);
|
|
97
|
+
if (lexical > best)
|
|
98
|
+
best = lexical;
|
|
99
|
+
const va = vectors[a];
|
|
100
|
+
const vb = vectors[b];
|
|
101
|
+
if (va !== undefined && vb !== undefined) {
|
|
102
|
+
const vector = Math.max(0, cosineSimilarity(va, vb)) * VECTOR_REDUNDANCY_WEIGHT;
|
|
103
|
+
if (vector > best)
|
|
104
|
+
best = vector;
|
|
105
|
+
}
|
|
106
|
+
return best;
|
|
107
|
+
};
|
|
108
|
+
const redundancy = new Float64Array(n);
|
|
109
|
+
const used = new Uint8Array(n);
|
|
110
|
+
const selected = new Array(n);
|
|
111
|
+
let pairsSinceCheck = 0;
|
|
112
|
+
let sliceStartedAt = performance.now();
|
|
113
|
+
for (let round = 0; round < n; round++) {
|
|
114
|
+
throwIfAborted(signal);
|
|
115
|
+
let bestIndex = firstUnusedIndex(used);
|
|
116
|
+
let bestScore = Number.NEGATIVE_INFINITY;
|
|
117
|
+
for (let i = 0; i < n; i++) {
|
|
118
|
+
if (used[i] === 1)
|
|
119
|
+
continue;
|
|
120
|
+
const mmrScore = lambda * relevance[i] - (1 - lambda) * redundancy[i];
|
|
121
|
+
if (mmrScore > bestScore) {
|
|
122
|
+
bestScore = mmrScore;
|
|
123
|
+
bestIndex = i;
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
used[bestIndex] = 1;
|
|
127
|
+
selected[round] = candidates[bestIndex];
|
|
128
|
+
for (let i = 0; i < n; i++) {
|
|
129
|
+
if (used[i] === 1)
|
|
130
|
+
continue;
|
|
131
|
+
const pair = pairRedundancy(bestIndex, i);
|
|
132
|
+
if (pair > redundancy[i])
|
|
133
|
+
redundancy[i] = pair;
|
|
134
|
+
pairsSinceCheck++;
|
|
135
|
+
if (pairsSinceCheck < PAIR_CHECK_INTERVAL)
|
|
136
|
+
continue;
|
|
137
|
+
pairsSinceCheck = 0;
|
|
138
|
+
throwIfAborted(signal);
|
|
139
|
+
if (performance.now() - sliceStartedAt < CPU_SLICE_MS)
|
|
140
|
+
continue;
|
|
141
|
+
await yieldToEventLoop();
|
|
142
|
+
throwIfAborted(signal);
|
|
143
|
+
sliceStartedAt = performance.now();
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
return selected;
|
|
147
|
+
}
|
|
148
|
+
export function interleaveByFile(candidates, diversity, signal) {
|
|
149
|
+
throwIfAborted(signal);
|
|
150
|
+
if (diversity === "off" || candidates.length <= 2)
|
|
151
|
+
return candidates;
|
|
152
|
+
const buckets = new Map();
|
|
153
|
+
const fileOrder = [];
|
|
154
|
+
for (const candidate of candidates) {
|
|
155
|
+
throwIfAborted(signal);
|
|
156
|
+
const key = `${candidate.chunk.kb_id}:${candidate.chunk.file_path}`;
|
|
157
|
+
let bucket = buckets.get(key);
|
|
158
|
+
if (!bucket) {
|
|
159
|
+
bucket = [];
|
|
160
|
+
buckets.set(key, bucket);
|
|
161
|
+
fileOrder.push(key);
|
|
162
|
+
}
|
|
163
|
+
bucket.push(candidate);
|
|
164
|
+
}
|
|
165
|
+
if (fileOrder.length <= 1)
|
|
166
|
+
return candidates;
|
|
167
|
+
const result = [];
|
|
168
|
+
let round = 0;
|
|
169
|
+
while (result.length < candidates.length) {
|
|
170
|
+
let added = false;
|
|
171
|
+
for (const key of fileOrder) {
|
|
172
|
+
const item = buckets.get(key)?.[round];
|
|
173
|
+
if (!item)
|
|
174
|
+
continue;
|
|
175
|
+
result.push(item);
|
|
176
|
+
added = true;
|
|
177
|
+
}
|
|
178
|
+
if (!added)
|
|
179
|
+
break;
|
|
180
|
+
round++;
|
|
181
|
+
}
|
|
182
|
+
return result;
|
|
183
|
+
}
|
|
@@ -22,6 +22,11 @@ export interface SearchTuningSummary {
|
|
|
22
22
|
snippet_max_length: number;
|
|
23
23
|
min_hybrid_score: number;
|
|
24
24
|
candidate_limit: number;
|
|
25
|
+
retrieval_limit_per_kb: number;
|
|
26
|
+
retrieved_candidates: number;
|
|
27
|
+
filtered_candidates: number;
|
|
28
|
+
diversity_candidates: number;
|
|
29
|
+
candidate_pool_truncated: boolean;
|
|
25
30
|
candidate_min: number;
|
|
26
31
|
candidate_multiplier: number;
|
|
27
32
|
adaptive_context_lines: number;
|
|
@@ -36,6 +41,11 @@ interface SearchTuningInput {
|
|
|
36
41
|
profile?: SearchProfile;
|
|
37
42
|
kbSourceTypes: string[];
|
|
38
43
|
}
|
|
44
|
+
export interface SearchWindow {
|
|
45
|
+
limit: number;
|
|
46
|
+
offset: number;
|
|
47
|
+
}
|
|
48
|
+
export declare function resolveSearchWindow(limit: number | undefined, offset: number | undefined, defaultLimit: number): SearchWindow;
|
|
39
49
|
export declare function resolveSearchTuning(input: SearchTuningInput): SearchTuning;
|
|
40
50
|
export declare function summarizeSearchTuning(tuning: SearchTuning, limit: number, candidateLimit: number): SearchTuningSummary;
|
|
41
51
|
export {};
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { MAX_SEARCH_RESULTS, MAX_SEARCH_WINDOW } from "./contracts.js";
|
|
1
2
|
import { MIN_HYBRID_SCORE } from "./ranking.js";
|
|
2
3
|
const PROFILE_DEFAULTS = {
|
|
3
4
|
balanced: {
|
|
@@ -152,6 +153,20 @@ function chooseRuntimeProfile(input) {
|
|
|
152
153
|
return "long_context";
|
|
153
154
|
return "balanced";
|
|
154
155
|
}
|
|
156
|
+
export function resolveSearchWindow(limit, offset, defaultLimit) {
|
|
157
|
+
const resolvedLimit = limit ?? defaultLimit;
|
|
158
|
+
if (!Number.isInteger(resolvedLimit) || resolvedLimit < 1 || resolvedLimit > MAX_SEARCH_RESULTS) {
|
|
159
|
+
throw new Error(`Search limit must be an integer from 1 to ${MAX_SEARCH_RESULTS}`);
|
|
160
|
+
}
|
|
161
|
+
const resolvedOffset = offset ?? 0;
|
|
162
|
+
if (!Number.isInteger(resolvedOffset) || resolvedOffset < 0) {
|
|
163
|
+
throw new Error("Search offset must be a non-negative integer");
|
|
164
|
+
}
|
|
165
|
+
if (resolvedOffset + resolvedLimit > MAX_SEARCH_WINDOW) {
|
|
166
|
+
throw new Error(`Search offset + limit must not exceed ${MAX_SEARCH_WINDOW}; narrow the query or use a more specific knowledge base`);
|
|
167
|
+
}
|
|
168
|
+
return { limit: resolvedLimit, offset: resolvedOffset };
|
|
169
|
+
}
|
|
155
170
|
export function resolveSearchTuning(input) {
|
|
156
171
|
const requestedProfile = input.profile ?? parseProfile(process.env.PI_KNOWLEDGE_SEARCH_PROFILE) ?? "auto";
|
|
157
172
|
const selectedProfile = requestedProfile === "auto" ? chooseRuntimeProfile(input) : requestedProfile;
|
|
@@ -179,6 +194,11 @@ export function summarizeSearchTuning(tuning, limit, candidateLimit) {
|
|
|
179
194
|
snippet_max_length: tuning.snippetMaxLength,
|
|
180
195
|
min_hybrid_score: tuning.minHybridScore,
|
|
181
196
|
candidate_limit: candidateLimit,
|
|
197
|
+
retrieval_limit_per_kb: candidateLimit,
|
|
198
|
+
retrieved_candidates: 0,
|
|
199
|
+
filtered_candidates: 0,
|
|
200
|
+
diversity_candidates: 0,
|
|
201
|
+
candidate_pool_truncated: false,
|
|
182
202
|
candidate_min: tuning.candidateMin,
|
|
183
203
|
candidate_multiplier: tuning.candidateMultiplier,
|
|
184
204
|
adaptive_context_lines: tuning.adaptiveContextLines,
|
package/docs/configuration.md
CHANGED
|
@@ -33,7 +33,7 @@ Default data storage is `~/.pi/knowledge` under Pi and `~/.omp/knowledge` under
|
|
|
33
33
|
| `PI_KNOWLEDGE_EMBEDDING_IDLE_MS` | `30000` | Idle timer used only when native idle disposal coordination is enabled. Mainly for lifecycle stress tests. |
|
|
34
34
|
| `PI_KNOWLEDGE_OFFLINE` | unset | Use with a pre-populated model cache for offline local model operation. See `docs/offline-mode.md`. |
|
|
35
35
|
|
|
36
|
-
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, non-default prefix strategy, pooling, normalization, API input cap, and a hash of the API base URL when applicable. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character/prefix settings or after upgrading
|
|
36
|
+
Each KB stores the embedding model label, vector dimension, and a non-secret embedding signature that includes provider/model semantics, query/document prefixes, non-default prefix strategy, pooling, normalization policy, API input cap, and a hash of the API base URL when applicable. OpenAI-compatible response vectors are validated and L2-normalized by the client before use or persistence; the `client-l2-v1` policy is part of API signatures. `knowledge_search` skips vector retrieval for KBs whose stored signature or dimension is missing or incompatible with the current query embedding and uses BM25-only fallback for hybrid searches. Run `knowledge_update` to rebuild vectors after changing embedding provider/model/max-character/prefix settings or after upgrading an API-backed KB created before client-side normalization was signed.
|
|
37
37
|
|
|
38
38
|
## Reranker
|
|
39
39
|
|
|
@@ -83,9 +83,9 @@ API embedding failures intentionally surface by default. Silent fallback can hid
|
|
|
83
83
|
| `PI_KNOWLEDGE_DEEP_RERANK_CANDIDATES` | profile default | Override how many filtered candidates `deep` mode sends to the reranker. Clamped to 5-100. |
|
|
84
84
|
| `PI_KNOWLEDGE_DEEP_RERANK_TOPK_MULTIPLIER` | profile default | Override deep rerank top-k multiplier relative to the effective result limit. Clamped to 1-10. |
|
|
85
85
|
|
|
86
|
-
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, candidate pool
|
|
86
|
+
Search diagnostics include result provenance and applied search tuning in both text output and structured `details`: selected profile, effective limit, snippet length, hybrid threshold, global candidate limit, per-KB retrieval limit, retrieved/filtered/diversity candidate counts, global-pool truncation, chunk id, chunk hash, match reason, indexed timestamp, stale flag, and source mtime when the original source file is available. This provenance is stored in SQLite and derived from indexed source metadata; it does not create or modify files in the indexed project.
|
|
87
87
|
|
|
88
|
-
`knowledge_search` supports `file_type`, `path_pattern`, `profile`, and
|
|
88
|
+
`knowledge_search` supports `file_type`, substring `path_pattern`, `profile`, integer `limit` from 1 to 50, and non-negative integer `offset`. `offset + limit` cannot exceed the 800-candidate global search window. Each KB retrieves up to the profile-derived candidate limit, then merged results are ranked, filtered, and globally capped before deep reranking, adaptive expansion, or diversity. Search profiles tune result count, snippets, hybrid strictness, candidate breadth, adaptive context, and deep rerank breadth at runtime. `auto` chooses from query shape, mode, and KB source type; explicit tool parameters win over env overrides, env overrides win over profile defaults, and profile defaults win over built-in compatibility values. `hybrid` mode remains lexical-anchored to prevent low-evidence semantic false positives; use `semantic` for pure vector search, or `semantic_hybrid` when weak lexical overlap should not block vector-backed weighted fusion. `adaptive` uses bounded same-file context expansion; when AST metadata is present it can prefer same-parent/sibling chunks without claiming caller/callee graph semantics.
|
|
89
89
|
|
|
90
90
|
`knowledge_doctor` emits both human-readable issues and machine-readable actions in structured `details`. Action codes include `run_update`, `rebuild_kb`, `wait_for_indexing`, `review_skipped_scope`, `rebuild_vectors`, `check_source`, and `none`. Agents should prefer these action codes over parsing the English action text.
|
|
91
91
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-knowledge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Local-first RAG knowledge base for Pi and OMP coding agents with hybrid code/doc search, reranking, diagnostics, and large-project indexing",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|