@maxgfr/codeindex 2.11.1 → 2.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/MIGRATION.md +2 -1
- package/docs/SEMANTIC.md +242 -0
- package/package.json +3 -2
- package/scripts/engine.d.mts +16 -4
- package/scripts/engine.mjs +106 -35
- package/src/callers.ts +0 -0
- package/src/embed/model.ts +50 -11
- package/src/engine-cli.ts +13 -17
- package/src/engine.ts +5 -1
- package/src/lang/common.ts +18 -7
- package/src/mcp.ts +81 -12
- package/src/types.ts +10 -4
package/docs/MIGRATION.md
CHANGED
|
@@ -119,7 +119,8 @@ note only) — so wiring it on is safe before an asset exists.
|
|
|
119
119
|
| **ultrasec** | walker + gitignore + scope + symlink guard | `scanRepo` (`scope`, `gitignore` — its own semantics, ported here) |
|
|
120
120
|
| | per-language defs/imports/calls extraction | `extractCode` (`symbols`/`refs`/`calls`) |
|
|
121
121
|
| | `resolveImport` | `resolveImport` |
|
|
122
|
-
| | `buildGraph` (import+call edges, symbolDefs
|
|
122
|
+
| | `buildGraph` (import+call edges, symbolDefs) | `buildGraph` + `resolveCallEdges` |
|
|
123
|
+
| | `callersBySymbol`, `enclosingSymbol` (raw-recall taint-BFS input) | `buildRawCallerIndex` (issue #8) — every name-matched call site keyed by the raw callee name, no def resolution or gating, `enclosingSymbol` computed per site. `buildCallerIndex` is **NOT** a substitute here: it is def-resolved and gated (language-family filter, JS/TS import gate, same-file self-declaration skip) and will silently drop sites a recall consumer needs. Both are bounded by `FileRecord.calls`'s per-file 512-call cap (dedup by name+line) — a file with more raw call sites than that loses sites upstream of either function. |
|
|
123
124
|
| | keeps | taint source→sink enumeration, external scanners, EPSS/KEV/CVSS, SARIF |
|
|
124
125
|
|
|
125
126
|
## Golden-diff adjudication (every migration)
|
package/docs/SEMANTIC.md
ADDED
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
# Deterministic static-embedding tier (semantic search)
|
|
2
|
+
|
|
3
|
+
codeindex's default search is keyless BM25 lexical ranking (see `search`). The
|
|
4
|
+
**semantic tier** (v2.10.0) adds embedding-based retrieval that is still
|
|
5
|
+
**keyless, zero-dependency, and byte-deterministic** — the same doctrine as the
|
|
6
|
+
rest of the engine, and the reason it does *not* use onnxruntime / transformers.js
|
|
7
|
+
(those are threaded and dequantize int8 to fp32 for matmul, so they are never
|
|
8
|
+
byte-stable across architectures).
|
|
9
|
+
|
|
10
|
+
It is **strictly opt-in by asset presence**, exactly like the tree-sitter
|
|
11
|
+
grammar tier: with no model on disk the engine silently stays lexical.
|
|
12
|
+
|
|
13
|
+
## Why a *static* embedding model
|
|
14
|
+
|
|
15
|
+
A static embedding (model2vec / potion style) is a plain lookup table:
|
|
16
|
+
`token → row vector`. There is **no neural forward pass** — "inference" is:
|
|
17
|
+
|
|
18
|
+
```
|
|
19
|
+
tokenize → gather rows → mean-pool → L2-normalize → int8-quantize
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Every step is pure JS and chosen to be byte-identical everywhere:
|
|
23
|
+
|
|
24
|
+
| step | how determinism is guaranteed |
|
|
25
|
+
|---|---|
|
|
26
|
+
| tokenize | `foldText` (NFKD + strip combining marks) → camelCase/ACRONYM split → lowercase → split on non-alphanumeric |
|
|
27
|
+
| wordpiece | greedy longest-match against the vocab, BERT-style `##` continuations; an unsplittable word → the `[UNK]` row |
|
|
28
|
+
| gather | model rows widen float→double exactly |
|
|
29
|
+
| mean-pool | IEEE-754 double add/div in a fixed token order |
|
|
30
|
+
| L2-norm | division by `sqrt(Σv²)` — `sqrt` is correctly-rounded per IEEE-754 |
|
|
31
|
+
| quantize | `unit × 127`, **round-half-to-even** (not `Math.round`), clamp to `[-127,127]` → int8 |
|
|
32
|
+
|
|
33
|
+
Ranking is a **pure integer dot product** of int8 vectors. Because the fixed
|
|
34
|
+
`1/127` scale is shared by every vector, for a given query the integer dot
|
|
35
|
+
`Σ q·c` is monotonic with the cosine estimate — no per-vector float scale is
|
|
36
|
+
stored or multiplied, so nothing about ranking can drift between platforms.
|
|
37
|
+
|
|
38
|
+
**Key insight:** the corpus *and* the query are encoded by the *same* JS encoder,
|
|
39
|
+
so self-consistency is all that is required — fidelity to the original HF
|
|
40
|
+
tokenizer is not. The JS tokenizer deliberately differs from the Python
|
|
41
|
+
reference; that gap is assumed and harmless.
|
|
42
|
+
|
|
43
|
+
## Activation (opt-in by asset)
|
|
44
|
+
|
|
45
|
+
Resolution order (`resolveEmbedModelDir`):
|
|
46
|
+
|
|
47
|
+
1. `CODEINDEX_EMBED_DIR` (explicit override — wins outright)
|
|
48
|
+
2. `<repo>/.codeindex/models/`
|
|
49
|
+
3. `<cwd>/.codeindex/models/`
|
|
50
|
+
|
|
51
|
+
A directory "has a model" when it contains `model.json`. Nothing found →
|
|
52
|
+
`undefined` → silent lexical degradation. **Models are never in the npm
|
|
53
|
+
tarball** (`package.json` `files` is unchanged; the pack-smoke test asserts no
|
|
54
|
+
`model.json` / `*.safetensors` / `embeddings.bin` ships).
|
|
55
|
+
|
|
56
|
+
### `model.json` shape
|
|
57
|
+
|
|
58
|
+
```json
|
|
59
|
+
{
|
|
60
|
+
"modelId": "…",
|
|
61
|
+
"dim": 256,
|
|
62
|
+
"unk": "[UNK]",
|
|
63
|
+
"vocab": ["[UNK]", "auth", "##token", "..."],
|
|
64
|
+
"weights": [[0.0, ...], [1.0, ...], ...]
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
`vocab[i]` is the token whose row is `weights[i]` (length `dim`). Human-authorable
|
|
69
|
+
JSON so a tiny fixture is committable; a real `pull`ed model uses the same shape
|
|
70
|
+
at scale.
|
|
71
|
+
|
|
72
|
+
### The official asset
|
|
73
|
+
|
|
74
|
+
`codeindex embed pull` (no env) fetches the published **`embed-model-v1`** GitHub
|
|
75
|
+
release: a conversion of [`minishlab/potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M)
|
|
76
|
+
(revision `bf8b0566…`, **MIT**) — 256-dim, a 27 559-token WordPiece vocab, int8
|
|
77
|
+
rows at one global scale. The download is **sha256-verified** against
|
|
78
|
+
`EMBED_ASSET_SHA256` (`src/embed/model.ts`) before it is written; a mismatch
|
|
79
|
+
fails the pull and writes nothing. The asset is **never** committed to git or
|
|
80
|
+
shipped in the npm tarball — it lives only in the release. Reproduce it
|
|
81
|
+
byte-for-byte with the pinned toolchain in [`scripts/embed-asset/`](../scripts/embed-asset/).
|
|
82
|
+
|
|
83
|
+
## CLI
|
|
84
|
+
|
|
85
|
+
```sh
|
|
86
|
+
# Effective mode (none/static/endpoint), model, EMBED_VERSION, endpoint + its
|
|
87
|
+
# reachability (JSON). Precedence: endpoint > static model.
|
|
88
|
+
codeindex embed status --repo <dir>
|
|
89
|
+
|
|
90
|
+
# Fetch the official static-embedding asset (published as the `embed-model-v1`
|
|
91
|
+
# GitHub release) into CODEINDEX_EMBED_DIR (or <repo>/.codeindex/models/). The
|
|
92
|
+
# built-in default download is sha256-verified before it is written; set
|
|
93
|
+
# CODEINDEX_EMBED_URL to fetch a custom/mirrored model.json instead (un-verified).
|
|
94
|
+
codeindex embed pull --repo <dir>
|
|
95
|
+
|
|
96
|
+
# Build embeddings.bin from the repo into --out <dir> (static tier only).
|
|
97
|
+
codeindex embed build --repo <dir> --out <dir>
|
|
98
|
+
|
|
99
|
+
# Print (or --run) the docker command that starts the rich-tier embedding server.
|
|
100
|
+
codeindex embed serve # prints the one-liner
|
|
101
|
+
codeindex embed serve --run # runs it (needs docker)
|
|
102
|
+
|
|
103
|
+
# Search: RRF-fuse an embedding tier + lexical. Degrades to lexical (exit 0)
|
|
104
|
+
# when no tier is available/reachable.
|
|
105
|
+
codeindex search "http client retry" --repo <dir> --semantic
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`codeindex index --out <dir>` additionally writes `embeddings.bin` next to
|
|
109
|
+
`graph.json` **iff** a model is present.
|
|
110
|
+
|
|
111
|
+
## Artifact: `embeddings.bin`
|
|
112
|
+
|
|
113
|
+
```
|
|
114
|
+
"CIE1" 4-byte ASCII magic
|
|
115
|
+
uint32 LE header length
|
|
116
|
+
UTF-8 JSON header { embedVersion, modelId, dim, count, records:[{file,symbol,line}] }
|
|
117
|
+
int8 body count × dim signed bytes (row-major)
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The header carries **no absolute path and no timestamp**; records follow scan
|
|
121
|
+
order (files sorted by `rel`, symbols in declaration order). Two builds of an
|
|
122
|
+
unchanged repo produce byte-identical `embeddings.bin`. A dedicated
|
|
123
|
+
`EMBED_VERSION` (independent of `SCHEMA_VERSION`) plus `modelId` + `dim` in the
|
|
124
|
+
header invalidate a stale or foreign artifact.
|
|
125
|
+
|
|
126
|
+
Granularity is **per-symbol** (name + signature + file summary + path segments),
|
|
127
|
+
with a **per-file** fallback record for symbol-less files (docs, config) so
|
|
128
|
+
every file with content is represented.
|
|
129
|
+
|
|
130
|
+
## Fusion
|
|
131
|
+
|
|
132
|
+
Semantic and lexical rankings live on incomparable scales (BM25 score vs integer
|
|
133
|
+
dot product), so `searchSemantic` fuses them by **rank** via the shared `rrf`
|
|
134
|
+
helper (k=60) — never a linear score blend. A `SemanticSearchResult` extends the
|
|
135
|
+
lexical `SearchResult` additively with `semanticSymbol` (the corpus symbol whose
|
|
136
|
+
embedding was closest for that file).
|
|
137
|
+
|
|
138
|
+
## Degradation matrix
|
|
139
|
+
|
|
140
|
+
| present | behavior |
|
|
141
|
+
|---|---|
|
|
142
|
+
| nothing | BM25 lexical |
|
|
143
|
+
| + fuzzy | BM25 + trigram fallback for `df==0` terms (v2.9.0) |
|
|
144
|
+
| + model asset | RRF-fused deterministic static semantic search |
|
|
145
|
+
| + HTTP endpoint (`CODEINDEX_EMBED_ENDPOINT`) | rich tier — **wins over a static model** (v2.11.0) |
|
|
146
|
+
| `--semantic`, nothing available | lexical + stderr note, **exit 0** |
|
|
147
|
+
| endpoint set but unreachable/timeout | lexical + stderr note, **exit 0** (never falls back to the static model) |
|
|
148
|
+
|
|
149
|
+
## Library
|
|
150
|
+
|
|
151
|
+
```ts
|
|
152
|
+
import {
|
|
153
|
+
resolveEmbedModelDir, loadEmbedModel,
|
|
154
|
+
buildEmbeddingIndex, serializeEmbeddings, deserializeEmbeddings,
|
|
155
|
+
searchSemantic, encode, EMBED_VERSION,
|
|
156
|
+
} from "@maxgfr/codeindex";
|
|
157
|
+
|
|
158
|
+
const model = loadEmbedModel(resolveEmbedModelDir("/repo"));
|
|
159
|
+
if (model) {
|
|
160
|
+
const scan = scanRepo("/repo");
|
|
161
|
+
const index = buildEmbeddingIndex(scan, model); // deterministic
|
|
162
|
+
const results = searchSemantic(scan, "http retry", index, { model });
|
|
163
|
+
}
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`searchSemantic(scan, query, index, opts)` with no `opts.model` (or no index)
|
|
167
|
+
returns the pure lexical ranking — the same silent degradation the CLI uses.
|
|
168
|
+
|
|
169
|
+
## The rich tier: containerized HTTP embedding endpoint (v2.11.0)
|
|
170
|
+
|
|
171
|
+
The **rich tier** lets the engine consume a real neural embedding model (e.g.
|
|
172
|
+
all-MiniLM-L6-v2) served by a local container, instead of the static lookup
|
|
173
|
+
table. It is opt-in via one env var:
|
|
174
|
+
|
|
175
|
+
```sh
|
|
176
|
+
CODEINDEX_EMBED_ENDPOINT=http://localhost:8756 \
|
|
177
|
+
codeindex search "auth token" --repo . --semantic
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
### Precedence
|
|
181
|
+
|
|
182
|
+
`endpoint > static model > none`. Setting `CODEINDEX_EMBED_ENDPOINT` is an
|
|
183
|
+
**explicit** user intent, so it wins over any local `model.json`. `embed status`
|
|
184
|
+
reports the effective `mode`. If the endpoint is defined but
|
|
185
|
+
unreachable/times-out/malformed, the search degrades straight to **lexical**
|
|
186
|
+
(stderr note, exit 0) — it does **not** silently fall back to the static model.
|
|
187
|
+
|
|
188
|
+
### How the float vectors join the deterministic pipeline
|
|
189
|
+
|
|
190
|
+
The engine POSTs corpus and query texts to the endpoint, receives **float**
|
|
191
|
+
vectors, then runs them through the **exact same** tail as the static tier —
|
|
192
|
+
`quantize()` in `encode.ts`: L2-normalize → int8 at the fixed `1/127` scale
|
|
193
|
+
(round-half-to-even, clamp). Ranking stays a pure integer dot product. So the
|
|
194
|
+
only tier-specific step is the encoder; fusion and ranking are shared code.
|
|
195
|
+
|
|
196
|
+
Corpus embeddings for the endpoint tier are built **at search time**
|
|
197
|
+
(`buildEndpointIndex`) and are **never serialized** to `embeddings.bin`: unlike
|
|
198
|
+
the static tier, endpoint vectors are float and provider-dependent, so this tier
|
|
199
|
+
is **deterministic per image digest**, not byte-golden. Pin the digest for
|
|
200
|
+
reproducibility:
|
|
201
|
+
|
|
202
|
+
```sh
|
|
203
|
+
docker run -d -p 8756:8756 ghcr.io/maxgfr/codeindex-embed@sha256:<digest>
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
### HTTP protocol (implement your own server)
|
|
207
|
+
|
|
208
|
+
`CODEINDEX_EMBED_ENDPOINT` is the server **base URL**. The client derives:
|
|
209
|
+
|
|
210
|
+
| method + path | request | response |
|
|
211
|
+
|---|---|---|
|
|
212
|
+
| `POST {base}/embed` | `{ "texts": ["…", …] }` | `{ "vectors": [[…float…], …] }` (same order, one row per text) |
|
|
213
|
+
| `GET {base}/healthz` | — | `200` (any body) when ready |
|
|
214
|
+
|
|
215
|
+
Notes: any embedding dimension is accepted (the engine quantizes whatever it
|
|
216
|
+
receives); vectors need not be pre-normalized (the engine L2-normalizes anyway);
|
|
217
|
+
the request times out after `CODEINDEX_EMBED_TIMEOUT_MS` (default 30 000).
|
|
218
|
+
`codeindex embed serve` prints/`--run`s the docker command for the official
|
|
219
|
+
image; the **library never orchestrates docker** (that is CLI-only).
|
|
220
|
+
|
|
221
|
+
The reference server is `docker/embed/` (transformers.js + all-MiniLM-L6-v2,
|
|
222
|
+
model baked in at build, offline at run, non-root, `:8756`), published to
|
|
223
|
+
`ghcr.io/maxgfr/codeindex-embed` by `.github/workflows/embed-image.yml`.
|
|
224
|
+
|
|
225
|
+
### Library
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
import {
|
|
229
|
+
buildEndpointIndex, encodeQueryViaEndpoint, probeEndpoint,
|
|
230
|
+
searchSemantic, quantize,
|
|
231
|
+
} from "@maxgfr/codeindex";
|
|
232
|
+
|
|
233
|
+
if (await probeEndpoint("http://localhost:8756")) {
|
|
234
|
+
const scan = scanRepo("/repo");
|
|
235
|
+
const index = await buildEndpointIndex(scan); // float → int8 corpus
|
|
236
|
+
const queryVec = await encodeQueryViaEndpoint("auth token");
|
|
237
|
+
const results = searchSemantic(scan, "auth token", index, { queryVec });
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
`embedViaEndpoint(texts, opts)` is the low-level client (`{ texts }` →
|
|
242
|
+
`{ vectors }`).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@maxgfr/codeindex",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.12.0",
|
|
4
4
|
"description": "Self-contained, deterministic repo-indexing engine: walk + language detection + symbol/import extraction (tree-sitter AST with regex fallback) + import resolution + typed cross-file link-graph + analytics. Ships as a single zero-dependency engine.mjs that consumer tools vendor.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"packageManager": "pnpm@10.33.0",
|
|
@@ -23,7 +23,8 @@
|
|
|
23
23
|
"scripts/cli.mjs",
|
|
24
24
|
"scripts/grammars",
|
|
25
25
|
"src",
|
|
26
|
-
"docs/MIGRATION.md"
|
|
26
|
+
"docs/MIGRATION.md",
|
|
27
|
+
"docs/SEMANTIC.md"
|
|
27
28
|
],
|
|
28
29
|
"engines": {
|
|
29
30
|
"node": ">=18"
|
package/scripts/engine.d.mts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
declare const ENGINE_VERSION = "2.
|
|
1
|
+
declare const ENGINE_VERSION = "2.12.0";
|
|
2
2
|
declare const SCHEMA_VERSION = 4;
|
|
3
|
-
declare const EXTRACTOR_VERSION =
|
|
3
|
+
declare const EXTRACTOR_VERSION = 9;
|
|
4
4
|
type FileKind = "code" | "doc" | "config" | "asset" | "other";
|
|
5
5
|
type EdgeKind = "contains" | "doc-link" | "import" | "call" | "use" | "mention";
|
|
6
6
|
type Tier = 0 | 1 | 2;
|
|
@@ -337,6 +337,14 @@ type CallerIndex = Map<string, CallerEntry>;
|
|
|
337
337
|
declare function computeImportPairs(scan: RepoScan): Set<string>;
|
|
338
338
|
declare function buildCallerIndex(scan: RepoScan, importPairs?: Set<string>, opts?: CallerIndexOptions): CallerIndex;
|
|
339
339
|
declare function enclosingSymbol(scan: RepoScan, file: string, line: number): CodeSymbol | undefined;
|
|
340
|
+
interface RawCallerSite {
|
|
341
|
+
file: string;
|
|
342
|
+
line: number;
|
|
343
|
+
receiver?: string;
|
|
344
|
+
enclosingSymbol?: CodeSymbol;
|
|
345
|
+
}
|
|
346
|
+
type RawCallerIndex = Map<string, RawCallerSite[]>;
|
|
347
|
+
declare function buildRawCallerIndex(scan: RepoScan): RawCallerIndex;
|
|
340
348
|
|
|
341
349
|
declare function symbolsOverview(scan: RepoScan, rel: string): CodeSymbol[];
|
|
342
350
|
interface SymbolMatch extends CodeSymbol {
|
|
@@ -512,7 +520,11 @@ interface StaticEmbedModel {
|
|
|
512
520
|
declare function resolveEmbedModelDir(repo?: string): string | undefined;
|
|
513
521
|
declare function hasEmbedModel(repo?: string): boolean;
|
|
514
522
|
declare function loadEmbedModel(dir?: string): StaticEmbedModel | undefined;
|
|
515
|
-
|
|
523
|
+
interface EmbedPullTarget {
|
|
524
|
+
url: string;
|
|
525
|
+
sha256?: string;
|
|
526
|
+
}
|
|
527
|
+
declare function resolveEmbedPullUrl(): EmbedPullTarget;
|
|
516
528
|
|
|
517
529
|
declare function basicTokenize(text: string): string[];
|
|
518
530
|
declare function wordpiece(word: string, model: StaticEmbedModel): number[];
|
|
@@ -693,4 +705,4 @@ declare function rrf<T>(lists: T[][], keyOf: (item: T) => string, k?: number): M
|
|
|
693
705
|
|
|
694
706
|
declare function runCli(argv: string[]): Promise<void>;
|
|
695
707
|
|
|
696
|
-
export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
|
|
708
|
+
export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawCallerIndex, type RawCallerSite, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildRawCallerIndex, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
|
package/scripts/engine.mjs
CHANGED
|
@@ -14,9 +14,9 @@ var ENGINE_VERSION, SCHEMA_VERSION, EXTRACTOR_VERSION;
|
|
|
14
14
|
var init_types = __esm({
|
|
15
15
|
"src/types.ts"() {
|
|
16
16
|
"use strict";
|
|
17
|
-
ENGINE_VERSION = "2.
|
|
17
|
+
ENGINE_VERSION = "2.12.0";
|
|
18
18
|
SCHEMA_VERSION = 4;
|
|
19
|
-
EXTRACTOR_VERSION =
|
|
19
|
+
EXTRACTOR_VERSION = 9;
|
|
20
20
|
}
|
|
21
21
|
});
|
|
22
22
|
|
|
@@ -737,8 +737,8 @@ function extractReexports(rel, content, localSymbols) {
|
|
|
737
737
|
const out2 = [];
|
|
738
738
|
const seen = /* @__PURE__ */ new Set();
|
|
739
739
|
const lineAt = (idx) => content.slice(0, idx).split(/\r?\n/).length;
|
|
740
|
-
const
|
|
741
|
-
for (const s of localSymbols) if (!
|
|
740
|
+
const localDeclOf = /* @__PURE__ */ new Map();
|
|
741
|
+
for (const s of localSymbols) if (!localDeclOf.has(s.name)) localDeclOf.set(s.name, s);
|
|
742
742
|
const named = /export\s*\{([\s\S]*?)\}\s*(?:from\s*['"]([^'"]+)['"])?\s*;?/g;
|
|
743
743
|
let m;
|
|
744
744
|
while ((m = named.exec(content)) && out2.length < 60) {
|
|
@@ -750,12 +750,13 @@ function extractReexports(rel, content, localSymbols) {
|
|
|
750
750
|
const name2 = as ? as[2] : p;
|
|
751
751
|
if (!/^[A-Za-z_$][\w$]*$/.test(name2) || name2 === "default" || seen.has(name2)) continue;
|
|
752
752
|
seen.add(name2);
|
|
753
|
-
const
|
|
753
|
+
const decl = !from ? localDeclOf.get(orig) : void 0;
|
|
754
754
|
out2.push({
|
|
755
755
|
name: name2,
|
|
756
|
-
kind:
|
|
756
|
+
kind: decl?.kind ?? "reexport",
|
|
757
757
|
file: rel,
|
|
758
|
-
line: lineAt(m.index),
|
|
758
|
+
line: decl ? decl.line : lineAt(m.index),
|
|
759
|
+
...decl?.endLine !== void 0 ? { endLine: decl.endLine } : {},
|
|
759
760
|
signature: from ? `export { ${name2} } from "${from}"` : `export { ${name2} }`,
|
|
760
761
|
exported: true,
|
|
761
762
|
lang
|
|
@@ -7731,8 +7732,11 @@ function buildCallerIndex(scan2, importPairs, opts = {}) {
|
|
|
7731
7732
|
function enclosingSymbol(scan2, file, line) {
|
|
7732
7733
|
const f = scan2.files.find((x) => x.rel === file);
|
|
7733
7734
|
if (!f?.symbols.length) return void 0;
|
|
7735
|
+
return enclosingAmong(f.symbols, line);
|
|
7736
|
+
}
|
|
7737
|
+
function enclosingAmong(symbols, line) {
|
|
7734
7738
|
let best;
|
|
7735
|
-
for (const s of
|
|
7739
|
+
for (const s of symbols) {
|
|
7736
7740
|
if (REFERENCE_KINDS3.has(s.kind)) continue;
|
|
7737
7741
|
if (s.line > line) continue;
|
|
7738
7742
|
if (s.endLine !== void 0 && line > s.endLine) continue;
|
|
@@ -7742,6 +7746,29 @@ function enclosingSymbol(scan2, file, line) {
|
|
|
7742
7746
|
}
|
|
7743
7747
|
return best;
|
|
7744
7748
|
}
|
|
7749
|
+
function buildRawCallerIndex(scan2) {
|
|
7750
|
+
const byName = /* @__PURE__ */ new Map();
|
|
7751
|
+
for (const f of scan2.files) {
|
|
7752
|
+
if (!f.calls?.length) continue;
|
|
7753
|
+
const symbols = f.symbols.filter((s) => !REFERENCE_KINDS3.has(s.kind));
|
|
7754
|
+
for (const c2 of f.calls) {
|
|
7755
|
+
const site = { file: f.rel, line: c2.line };
|
|
7756
|
+
if (c2.receiver !== void 0) site.receiver = c2.receiver;
|
|
7757
|
+
const enc = enclosingAmong(symbols, c2.line);
|
|
7758
|
+
if (enc) site.enclosingSymbol = enc;
|
|
7759
|
+
let arr = byName.get(c2.name);
|
|
7760
|
+
if (!arr) byName.set(c2.name, arr = []);
|
|
7761
|
+
arr.push(site);
|
|
7762
|
+
}
|
|
7763
|
+
}
|
|
7764
|
+
const index = /* @__PURE__ */ new Map();
|
|
7765
|
+
for (const name2 of [...byName.keys()].sort(byStr)) {
|
|
7766
|
+
const sites = byName.get(name2);
|
|
7767
|
+
sites.sort((a, b) => byStr(a.file, b.file) || a.line - b.line);
|
|
7768
|
+
index.set(name2, sites);
|
|
7769
|
+
}
|
|
7770
|
+
return index;
|
|
7771
|
+
}
|
|
7745
7772
|
var REFERENCE_KINDS3;
|
|
7746
7773
|
var init_callers = __esm({
|
|
7747
7774
|
"src/callers.ts"() {
|
|
@@ -9374,6 +9401,7 @@ var init_bm25 = __esm({
|
|
|
9374
9401
|
});
|
|
9375
9402
|
|
|
9376
9403
|
// src/embed/model.ts
|
|
9404
|
+
import { createHash as createHash2 } from "crypto";
|
|
9377
9405
|
import { existsSync as existsSync3, readFileSync as readFileSync5 } from "fs";
|
|
9378
9406
|
import { join as join10 } from "path";
|
|
9379
9407
|
function resolveEmbedModelDir(repo) {
|
|
@@ -9419,15 +9447,30 @@ function loadEmbedModel(dir) {
|
|
|
9419
9447
|
return { modelId, dim, unk, unkId, vocabSize, vocab: vmap, weights: flat };
|
|
9420
9448
|
}
|
|
9421
9449
|
function resolveEmbedPullUrl() {
|
|
9422
|
-
const
|
|
9423
|
-
|
|
9450
|
+
const env = process.env.CODEINDEX_EMBED_URL;
|
|
9451
|
+
if (env && env.trim()) return { url: env.trim() };
|
|
9452
|
+
return { url: DEFAULT_EMBED_URL, sha256: EMBED_ASSET_SHA256 };
|
|
9453
|
+
}
|
|
9454
|
+
async function fetchEmbedModel(url, expectedSha256) {
|
|
9455
|
+
const res = await fetch(url);
|
|
9456
|
+
if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`);
|
|
9457
|
+
const body2 = await res.text();
|
|
9458
|
+
if (expectedSha256) {
|
|
9459
|
+
const got = createHash2("sha256").update(body2).digest("hex");
|
|
9460
|
+
if (got !== expectedSha256) {
|
|
9461
|
+
throw new Error(`sha256 mismatch: expected ${expectedSha256}, got ${got}`);
|
|
9462
|
+
}
|
|
9463
|
+
}
|
|
9464
|
+
return body2;
|
|
9424
9465
|
}
|
|
9425
|
-
var EMBED_VERSION, DEFAULT_EMBED_DIRNAME;
|
|
9466
|
+
var EMBED_VERSION, DEFAULT_EMBED_DIRNAME, DEFAULT_EMBED_URL, EMBED_ASSET_SHA256;
|
|
9426
9467
|
var init_model = __esm({
|
|
9427
9468
|
"src/embed/model.ts"() {
|
|
9428
9469
|
"use strict";
|
|
9429
9470
|
EMBED_VERSION = 1;
|
|
9430
9471
|
DEFAULT_EMBED_DIRNAME = "models";
|
|
9472
|
+
DEFAULT_EMBED_URL = "https://github.com/maxgfr/codeindex/releases/download/embed-model-v1/model.json";
|
|
9473
|
+
EMBED_ASSET_SHA256 = "163ad053eab4e9a80d421ed4164f32292c83290f02fbbe6fe4b9b1cd6ea18d34";
|
|
9431
9474
|
}
|
|
9432
9475
|
});
|
|
9433
9476
|
|
|
@@ -10129,7 +10172,9 @@ var init_viz = __esm({
|
|
|
10129
10172
|
// src/mcp.ts
|
|
10130
10173
|
var mcp_exports = {};
|
|
10131
10174
|
__export(mcp_exports, {
|
|
10132
|
-
|
|
10175
|
+
memoizedEmbeddingIndex: () => memoizedEmbeddingIndex,
|
|
10176
|
+
runMcpServer: () => runMcpServer,
|
|
10177
|
+
scanFingerprint: () => scanFingerprint
|
|
10133
10178
|
});
|
|
10134
10179
|
import { createInterface } from "readline";
|
|
10135
10180
|
function str(v) {
|
|
@@ -10138,6 +10183,19 @@ function str(v) {
|
|
|
10138
10183
|
function strArray(v) {
|
|
10139
10184
|
return Array.isArray(v) && v.every((x) => typeof x === "string") && v.length ? v : void 0;
|
|
10140
10185
|
}
|
|
10186
|
+
function errMessage(e) {
|
|
10187
|
+
return e instanceof Error ? e.message : String(e);
|
|
10188
|
+
}
|
|
10189
|
+
function scanFingerprint(scan2) {
|
|
10190
|
+
return sha1(scan2.files.map((f) => `${f.rel}:${f.hash}`).join("\n"));
|
|
10191
|
+
}
|
|
10192
|
+
async function memoizedEmbeddingIndex(key, build) {
|
|
10193
|
+
const cacheKey = `${key.mode}:${key.identity}:${scanFingerprint(key.scan)}`;
|
|
10194
|
+
if (embeddingIndexCache && embeddingIndexCache.key === cacheKey) return embeddingIndexCache.index;
|
|
10195
|
+
const index = await build();
|
|
10196
|
+
embeddingIndexCache = { key: cacheKey, index };
|
|
10197
|
+
return index;
|
|
10198
|
+
}
|
|
10141
10199
|
async function callTool(name2, args2) {
|
|
10142
10200
|
const repo = str(args2.repo);
|
|
10143
10201
|
if (!repo) throw new Error("`repo` is required (absolute path to the repository root)");
|
|
@@ -10278,19 +10336,35 @@ async function callTool(name2, args2) {
|
|
|
10278
10336
|
const endpoint = resolveEmbedEndpoint();
|
|
10279
10337
|
if (endpoint) {
|
|
10280
10338
|
try {
|
|
10281
|
-
const index = await buildEndpointIndex(scan2);
|
|
10339
|
+
const index = await memoizedEmbeddingIndex({ mode: "endpoint", identity: endpoint, scan: scan2 }, () => buildEndpointIndex(scan2));
|
|
10282
10340
|
const queryVec = await encodeQueryViaEndpoint(query);
|
|
10283
|
-
|
|
10284
|
-
|
|
10285
|
-
|
|
10341
|
+
const results2 = searchSemantic(scan2, query, index, { queryVec, limit, fuzzy });
|
|
10342
|
+
return JSON.stringify({ results: results2, tier: "endpoint" }, null, 2);
|
|
10343
|
+
} catch (e) {
|
|
10344
|
+
const results2 = searchIndex(scan2, query, { limit, fuzzy });
|
|
10345
|
+
return JSON.stringify(
|
|
10346
|
+
{ results: results2, tier: "lexical", degradedReason: `embedding endpoint failed: ${errMessage(e)}` },
|
|
10347
|
+
null,
|
|
10348
|
+
2
|
|
10349
|
+
);
|
|
10286
10350
|
}
|
|
10287
10351
|
}
|
|
10288
10352
|
const modelDir = resolveEmbedModelDir(repo);
|
|
10289
10353
|
const model = modelDir ? loadEmbedModel(modelDir) : void 0;
|
|
10290
10354
|
if (model) {
|
|
10291
|
-
const index =
|
|
10292
|
-
|
|
10355
|
+
const index = await memoizedEmbeddingIndex(
|
|
10356
|
+
{ mode: "static", identity: `${modelDir}#${model.modelId}`, scan: scan2 },
|
|
10357
|
+
() => buildEmbeddingIndex(scan2, model)
|
|
10358
|
+
);
|
|
10359
|
+
const results2 = searchSemantic(scan2, query, index, { model, limit, fuzzy });
|
|
10360
|
+
return JSON.stringify({ results: results2, tier: "static" }, null, 2);
|
|
10293
10361
|
}
|
|
10362
|
+
const results = searchIndex(scan2, query, { limit, fuzzy });
|
|
10363
|
+
return JSON.stringify(
|
|
10364
|
+
{ results, tier: "lexical", degradedReason: "no embedding endpoint or static model configured \u2014 see embed_status" },
|
|
10365
|
+
null,
|
|
10366
|
+
2
|
|
10367
|
+
);
|
|
10294
10368
|
}
|
|
10295
10369
|
return JSON.stringify(searchIndex(scan2, query, { limit, fuzzy }), null, 2);
|
|
10296
10370
|
}
|
|
@@ -10371,7 +10445,7 @@ async function runMcpServer() {
|
|
|
10371
10445
|
}
|
|
10372
10446
|
}
|
|
10373
10447
|
}
|
|
10374
|
-
var repoProp, scopeProps, TOOLS;
|
|
10448
|
+
var repoProp, scopeProps, TOOLS, embeddingIndexCache;
|
|
10375
10449
|
var init_mcp = __esm({
|
|
10376
10450
|
"src/mcp.ts"() {
|
|
10377
10451
|
"use strict";
|
|
@@ -10398,6 +10472,7 @@ var init_mcp = __esm({
|
|
|
10398
10472
|
init_embed();
|
|
10399
10473
|
init_search();
|
|
10400
10474
|
init_endpoint();
|
|
10475
|
+
init_hash();
|
|
10401
10476
|
repoProp = { repo: { type: "string", description: "Absolute path to the repository root" } };
|
|
10402
10477
|
scopeProps = {
|
|
10403
10478
|
scope: { type: "string", description: "Restrict to one directory (repo-relative)" },
|
|
@@ -10610,7 +10685,7 @@ var init_mcp = __esm({
|
|
|
10610
10685
|
},
|
|
10611
10686
|
{
|
|
10612
10687
|
name: "search",
|
|
10613
|
-
description: 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default \u2014 deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse
|
|
10688
|
+
description: 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default \u2014 deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse an embedding tier (HTTP endpoint, else a local static model) with lexical \u2014 the response then wraps the ranked list as `{ results, tier, degradedReason? }`, `tier` being "endpoint"/"static" when fusion happened or "lexical" (with `degradedReason`) when it did not (see embed_status). Without `semantic`, the response is the bare ranked array, unchanged.',
|
|
10614
10689
|
inputSchema: {
|
|
10615
10690
|
type: "object",
|
|
10616
10691
|
properties: {
|
|
@@ -10624,7 +10699,7 @@ var init_mcp = __esm({
|
|
|
10624
10699
|
},
|
|
10625
10700
|
semantic: {
|
|
10626
10701
|
type: "boolean",
|
|
10627
|
-
description:
|
|
10702
|
+
description: 'RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. The response reports the effective tier as a top-level `tier` field ("endpoint"/"static" on success, "lexical" plus `degradedReason` when neither is available/reachable) instead of degrading silently \u2014 see embed_status.'
|
|
10628
10703
|
}
|
|
10629
10704
|
},
|
|
10630
10705
|
required: ["repo", "query"]
|
|
@@ -11116,8 +11191,9 @@ Commands:
|
|
|
11116
11191
|
embed status Effective mode (none/static/endpoint), model +
|
|
11117
11192
|
EMBED_VERSION, and endpoint reachability (JSON)
|
|
11118
11193
|
embed build Write embeddings.bin into --out <dir> (static tier)
|
|
11119
|
-
embed pull Fetch the model asset into CODEINDEX_EMBED_DIR
|
|
11120
|
-
<repo>/.codeindex/models/)
|
|
11194
|
+
embed pull Fetch the official model asset into CODEINDEX_EMBED_DIR
|
|
11195
|
+
(or <repo>/.codeindex/models/); sha256-verified. Override
|
|
11196
|
+
the source with CODEINDEX_EMBED_URL
|
|
11121
11197
|
embed serve Print (or --run) the docker command that starts the
|
|
11122
11198
|
containerized embedding server (rich tier)
|
|
11123
11199
|
rules Architecture rules (forbidden edges, cycles, orphans) validated
|
|
@@ -11397,26 +11473,20 @@ async function runCli(argv) {
|
|
|
11397
11473
|
process.stderr.write(`codeindex: ${index.records.length} embedding records \u2192 ${flags2.out}/embeddings.bin (model ${model.modelId})
|
|
11398
11474
|
`);
|
|
11399
11475
|
} else if (sub === "pull") {
|
|
11400
|
-
const url = resolveEmbedPullUrl();
|
|
11401
|
-
if (!url) {
|
|
11402
|
-
process.stderr.write(
|
|
11403
|
-
"codeindex: no model URL configured. The official static-embedding asset is not published yet.\nSet CODEINDEX_EMBED_URL to a model.json URL (optionally CODEINDEX_EMBED_DIR as the destination), then re-run `codeindex embed pull`.\n"
|
|
11404
|
-
);
|
|
11405
|
-
process.exitCode = 1;
|
|
11406
|
-
return;
|
|
11407
|
-
}
|
|
11476
|
+
const { url, sha256 } = resolveEmbedPullUrl();
|
|
11408
11477
|
const destDir = process.env.CODEINDEX_EMBED_DIR ?? join12(flags2.repo, ".codeindex", "models");
|
|
11409
11478
|
mkdirSync2(destDir, { recursive: true });
|
|
11410
11479
|
process.stderr.write(`codeindex: fetching model from ${url} \u2192 ${join12(destDir, "model.json")}
|
|
11411
11480
|
`);
|
|
11412
|
-
|
|
11413
|
-
|
|
11414
|
-
|
|
11481
|
+
let body2;
|
|
11482
|
+
try {
|
|
11483
|
+
body2 = await fetchEmbedModel(url, sha256);
|
|
11484
|
+
} catch (e) {
|
|
11485
|
+
process.stderr.write(`codeindex: pull failed \u2014 ${e instanceof Error ? e.message : String(e)} (nothing written)
|
|
11415
11486
|
`);
|
|
11416
11487
|
process.exitCode = 1;
|
|
11417
11488
|
return;
|
|
11418
11489
|
}
|
|
11419
|
-
const body2 = await res.text();
|
|
11420
11490
|
try {
|
|
11421
11491
|
JSON.parse(body2);
|
|
11422
11492
|
} catch {
|
|
@@ -11508,6 +11578,7 @@ export {
|
|
|
11508
11578
|
buildGraph,
|
|
11509
11579
|
buildIndexArtifacts,
|
|
11510
11580
|
buildModules,
|
|
11581
|
+
buildRawCallerIndex,
|
|
11511
11582
|
buildResolveContext,
|
|
11512
11583
|
buildSymbolIndex,
|
|
11513
11584
|
byKey,
|
package/src/callers.ts
CHANGED
|
Binary file
|
package/src/embed/model.ts
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
1
2
|
import { existsSync, readFileSync } from "node:fs";
|
|
2
3
|
import { join } from "node:path";
|
|
3
4
|
|
|
@@ -20,12 +21,24 @@ import { join } from "node:path";
|
|
|
20
21
|
// graph.json / symbols.json consumers.
|
|
21
22
|
export const EMBED_VERSION = 1;
|
|
22
23
|
|
|
23
|
-
// The default
|
|
24
|
-
// URL is intentionally NOT hard-coded yet (see resolveEmbedPullUrl): the asset
|
|
25
|
-
// is unpublished, so `pull` fails cleanly asking for CODEINDEX_EMBED_URL rather
|
|
26
|
-
// than fetching a phantom.
|
|
24
|
+
// The default sub-directory a `pull` writes into and a `status` reports.
|
|
27
25
|
export const DEFAULT_EMBED_DIRNAME = "models";
|
|
28
26
|
|
|
27
|
+
// The official static-embedding asset, published as the dedicated GitHub release
|
|
28
|
+
// `embed-model-v1` (tag deliberately outside the `v<semver>` semantic-release
|
|
29
|
+
// namespace, so publishing it triggers no CI). `pull` uses this when
|
|
30
|
+
// CODEINDEX_EMBED_URL is unset. Reproduce the asset byte-for-byte with the pinned
|
|
31
|
+
// toolchain in scripts/embed-asset/.
|
|
32
|
+
export const DEFAULT_EMBED_URL =
|
|
33
|
+
"https://github.com/maxgfr/codeindex/releases/download/embed-model-v1/model.json";
|
|
34
|
+
|
|
35
|
+
// sha256 of the published model.json. `pull` verifies the downloaded bytes
|
|
36
|
+
// against this ONLY when it fetched the built-in default (a custom
|
|
37
|
+
// CODEINDEX_EMBED_URL is the user's own mirror and keeps the un-verified
|
|
38
|
+
// behavior). Regenerate alongside DEFAULT_EMBED_URL when re-cutting the asset:
|
|
39
|
+
// `shasum -a 256 model.json`.
|
|
40
|
+
export const EMBED_ASSET_SHA256 = "163ad053eab4e9a80d421ed4164f32292c83290f02fbbe6fe4b9b1cd6ea18d34";
|
|
41
|
+
|
|
29
42
|
// A loaded static-embedding model. `vocab` maps a wordpiece token to its row
|
|
30
43
|
// index; `weights` is the flat row-major (vocabSize × dim) matrix as IEEE-754
|
|
31
44
|
// doubles (float rows widen to double exactly, so mean-pool stays deterministic).
|
|
@@ -107,11 +120,37 @@ export function loadEmbedModel(dir?: string): StaticEmbedModel | undefined {
|
|
|
107
120
|
return { modelId, dim, unk, unkId, vocabSize, vocab: vmap, weights: flat };
|
|
108
121
|
}
|
|
109
122
|
|
|
110
|
-
//
|
|
111
|
-
//
|
|
112
|
-
//
|
|
113
|
-
// the
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
123
|
+
// What `embed pull` fetches, and whether to verify it. CODEINDEX_EMBED_URL wins
|
|
124
|
+
// outright (the user's explicit override / private mirror) and carries NO
|
|
125
|
+
// sha256, so a custom asset keeps the current un-verified behavior. With no env,
|
|
126
|
+
// we fall back to the built-in official asset AND its pinned sha256, so `pull`
|
|
127
|
+
// can prove integrity of the default download.
|
|
128
|
+
export interface EmbedPullTarget {
|
|
129
|
+
url: string;
|
|
130
|
+
sha256?: string; // present only for the built-in default → verify
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
export function resolveEmbedPullUrl(): EmbedPullTarget {
|
|
134
|
+
const env = process.env.CODEINDEX_EMBED_URL;
|
|
135
|
+
if (env && env.trim()) return { url: env.trim() };
|
|
136
|
+
return { url: DEFAULT_EMBED_URL, sha256: EMBED_ASSET_SHA256 };
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
// Fetch a model.json body over HTTP, following redirects (GitHub release assets
|
|
140
|
+
// 302 to a CDN — native fetch follows by default). When `expectedSha256` is
|
|
141
|
+
// given, the downloaded bytes are hashed and MUST match or this throws with a
|
|
142
|
+
// clear message and returns nothing — so a corrupt/tampered default asset never
|
|
143
|
+
// gets written. A non-2xx response also throws. Verification and I/O are split
|
|
144
|
+
// from the CLI so both the success and the sha-mismatch paths are unit-testable.
|
|
145
|
+
export async function fetchEmbedModel(url: string, expectedSha256?: string): Promise<string> {
|
|
146
|
+
const res = await fetch(url);
|
|
147
|
+
if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`);
|
|
148
|
+
const body = await res.text();
|
|
149
|
+
if (expectedSha256) {
|
|
150
|
+
const got = createHash("sha256").update(body).digest("hex");
|
|
151
|
+
if (got !== expectedSha256) {
|
|
152
|
+
throw new Error(`sha256 mismatch: expected ${expectedSha256}, got ${got}`);
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
return body;
|
|
117
156
|
}
|
package/src/engine-cli.ts
CHANGED
|
@@ -19,7 +19,7 @@ import { symbolComplexity, riskHotspots } from "./complexity.js";
|
|
|
19
19
|
import { renderMermaid } from "./viz.js";
|
|
20
20
|
import { searchIndex } from "./bm25.js";
|
|
21
21
|
import { checkRules, parseRules } from "./rules.js";
|
|
22
|
-
import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel, resolveEmbedPullUrl } from "./embed/model.js";
|
|
22
|
+
import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel, resolveEmbedPullUrl, fetchEmbedModel } from "./embed/model.js";
|
|
23
23
|
import { buildEmbeddingIndex, serializeEmbeddings } from "./embed/index.js";
|
|
24
24
|
import { searchSemantic } from "./embed/search.js";
|
|
25
25
|
import {
|
|
@@ -55,8 +55,9 @@ Commands:
|
|
|
55
55
|
embed status Effective mode (none/static/endpoint), model +
|
|
56
56
|
EMBED_VERSION, and endpoint reachability (JSON)
|
|
57
57
|
embed build Write embeddings.bin into --out <dir> (static tier)
|
|
58
|
-
embed pull Fetch the model asset into CODEINDEX_EMBED_DIR
|
|
59
|
-
<repo>/.codeindex/models/)
|
|
58
|
+
embed pull Fetch the official model asset into CODEINDEX_EMBED_DIR
|
|
59
|
+
(or <repo>/.codeindex/models/); sha256-verified. Override
|
|
60
|
+
the source with CODEINDEX_EMBED_URL
|
|
60
61
|
embed serve Print (or --run) the docker command that starts the
|
|
61
62
|
containerized embedding server (rich tier)
|
|
62
63
|
rules Architecture rules (forbidden edges, cycles, orphans) validated
|
|
@@ -387,26 +388,21 @@ export async function runCli(argv: string[]): Promise<void> {
|
|
|
387
388
|
writeFileSync(join(flags.out, "embeddings.bin"), serializeEmbeddings(index));
|
|
388
389
|
process.stderr.write(`codeindex: ${index.records.length} embedding records → ${flags.out}/embeddings.bin (model ${model.modelId})\n`);
|
|
389
390
|
} else if (sub === "pull") {
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
process.stderr.write(
|
|
394
|
-
"codeindex: no model URL configured. The official static-embedding asset is not published yet.\n" +
|
|
395
|
-
"Set CODEINDEX_EMBED_URL to a model.json URL (optionally CODEINDEX_EMBED_DIR as the destination), then re-run `codeindex embed pull`.\n",
|
|
396
|
-
);
|
|
397
|
-
process.exitCode = 1;
|
|
398
|
-
return;
|
|
399
|
-
}
|
|
391
|
+
// Default: the official published asset + its pinned sha256. A user-set
|
|
392
|
+
// CODEINDEX_EMBED_URL overrides both (mirror/custom model, no verification).
|
|
393
|
+
const { url, sha256 } = resolveEmbedPullUrl();
|
|
400
394
|
const destDir = process.env.CODEINDEX_EMBED_DIR ?? join(flags.repo, ".codeindex", "models");
|
|
401
395
|
mkdirSync(destDir, { recursive: true });
|
|
402
396
|
process.stderr.write(`codeindex: fetching model from ${url} → ${join(destDir, "model.json")}\n`);
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
397
|
+
let body: string;
|
|
398
|
+
try {
|
|
399
|
+
// Follows redirects (GitHub → CDN) and verifies sha256 for the default asset.
|
|
400
|
+
body = await fetchEmbedModel(url, sha256);
|
|
401
|
+
} catch (e) {
|
|
402
|
+
process.stderr.write(`codeindex: pull failed — ${e instanceof Error ? e.message : String(e)} (nothing written)\n`);
|
|
406
403
|
process.exitCode = 1;
|
|
407
404
|
return;
|
|
408
405
|
}
|
|
409
|
-
const body = await res.text();
|
|
410
406
|
try {
|
|
411
407
|
JSON.parse(body);
|
|
412
408
|
} catch {
|
package/src/engine.ts
CHANGED
|
@@ -55,7 +55,7 @@ export { buildModules, isTestFile, tierForPath } from "./modules.js";
|
|
|
55
55
|
export type { ModuleInfo } from "./modules.js";
|
|
56
56
|
export { buildGraph, uniqueSymbolDefs } from "./graph.js";
|
|
57
57
|
export { resolveCallEdges } from "./calls.js";
|
|
58
|
-
export { buildCallerIndex, enclosingSymbol, computeImportPairs } from "./callers.js";
|
|
58
|
+
export { buildCallerIndex, buildRawCallerIndex, enclosingSymbol, computeImportPairs } from "./callers.js";
|
|
59
59
|
export { symbolsOverview, findSymbol, findReferences } from "./query.js";
|
|
60
60
|
export { resolveUniqueSymbol, replaceSymbolBody, insertAfterSymbol, insertBeforeSymbol } from "./edit.js";
|
|
61
61
|
export type { EditResult } from "./edit.js";
|
|
@@ -143,6 +143,10 @@ export type { ArchRule, ForbiddenEdgeRule, BuiltinRule, RuleSeverity, RuleViolat
|
|
|
143
143
|
// Caller-index recall mode (issue #7) — options for buildCallerIndex above.
|
|
144
144
|
export type { CallerIndexOptions } from "./callers.js";
|
|
145
145
|
|
|
146
|
+
// Raw-recall caller index (issue #8) — ungated, def-resolution-free companion
|
|
147
|
+
// to buildCallerIndex above; see its JSDoc in callers.ts for the contract.
|
|
148
|
+
export type { RawCallerIndex, RawCallerSite } from "./callers.js";
|
|
149
|
+
|
|
146
150
|
// Behavioral analytics (git-history mining) + the token-budgeted repo map.
|
|
147
151
|
export { changeCoupling, rankHotspots } from "./coupling.js";
|
|
148
152
|
export type { ChangeCoupling, CouplingOptions, Hotspot } from "./coupling.js";
|
package/src/lang/common.ts
CHANGED
|
@@ -75,11 +75,14 @@ const REEXPORT_EXTS = new Set([".ts", ".tsx", ".mts", ".cts", ".js", ".jsx", ".m
|
|
|
75
75
|
//
|
|
76
76
|
// An ALIAS with no `from` clause (`export { b as c }`) renames an in-file
|
|
77
77
|
// declaration — `localSymbols` (already extracted by the AST or regex tier)
|
|
78
|
-
// lets us resolve `b` and mirror ITS kind
|
|
79
|
-
//
|
|
78
|
+
// lets us resolve `b` and mirror ITS kind, declaration line, and endLine (AST
|
|
79
|
+
// tier only) onto `c` (e.g. "function" at b's own line), so the alias reads
|
|
80
|
+
// as the real symbol it is — citeable at its actual declaration — rather than
|
|
81
|
+
// the generic "reexport" pinned to the export statement's line.
|
|
80
82
|
// A true cross-module re-export (`export { b as c } from "./mod"`) has no
|
|
81
83
|
// local `b` to resolve — and an alias the local pass genuinely can't see
|
|
82
|
-
// (destructured/ambient/etc.) falls back the same way — both keep "reexport"
|
|
84
|
+
// (destructured/ambient/etc.) falls back the same way — both keep "reexport"
|
|
85
|
+
// and cite the export statement's own line, the only line they have.
|
|
83
86
|
//
|
|
84
87
|
// Shared by extractCode (extract/code.ts) AND the standalone extractSymbols
|
|
85
88
|
// (lang/registry.ts) — ultradoc and other direct extractSymbols consumers hit
|
|
@@ -92,8 +95,11 @@ export function extractReexports(rel: string, content: string, localSymbols: Cod
|
|
|
92
95
|
const out: CodeSymbol[] = [];
|
|
93
96
|
const seen = new Set<string>();
|
|
94
97
|
const lineAt = (idx: number): number => content.slice(0, idx).split(/\r?\n/).length;
|
|
95
|
-
|
|
96
|
-
|
|
98
|
+
// Keyed on the whole CodeSymbol (not just kind) so the alias branch below
|
|
99
|
+
// can also cite the resolved declaration's own line/endLine, not just mirror
|
|
100
|
+
// its kind.
|
|
101
|
+
const localDeclOf = new Map<string, CodeSymbol>();
|
|
102
|
+
for (const s of localSymbols) if (!localDeclOf.has(s.name)) localDeclOf.set(s.name, s);
|
|
97
103
|
|
|
98
104
|
const named = /export\s*\{([\s\S]*?)\}\s*(?:from\s*['"]([^'"]+)['"])?\s*;?/g;
|
|
99
105
|
let m: RegExpExecArray | null;
|
|
@@ -106,9 +112,14 @@ export function extractReexports(rel: string, content: string, localSymbols: Cod
|
|
|
106
112
|
const name = as ? as[2]! : p;
|
|
107
113
|
if (!/^[A-Za-z_$][\w$]*$/.test(name) || name === "default" || seen.has(name)) continue;
|
|
108
114
|
seen.add(name);
|
|
109
|
-
|
|
115
|
+
// A resolved decl means this is a same-file alias: cite ITS line (and
|
|
116
|
+
// endLine, when the AST tier populated one) rather than the export
|
|
117
|
+
// statement's — an unresolved alias or a `from`-clause re-export has no
|
|
118
|
+
// local declaration to point at, so it keeps lineAt(m.index) below.
|
|
119
|
+
const decl = !from ? localDeclOf.get(orig) : undefined;
|
|
110
120
|
out.push({
|
|
111
|
-
name, kind:
|
|
121
|
+
name, kind: decl?.kind ?? "reexport", file: rel, line: decl ? decl.line : lineAt(m.index),
|
|
122
|
+
...(decl?.endLine !== undefined ? { endLine: decl.endLine } : {}),
|
|
112
123
|
signature: from ? `export { ${name} } from "${from}"` : `export { ${name} }`,
|
|
113
124
|
exported: true, lang,
|
|
114
125
|
});
|
package/src/mcp.ts
CHANGED
|
@@ -10,7 +10,7 @@ import { ENGINE_VERSION } from "./types.js";
|
|
|
10
10
|
import { ensureGrammars, allGrammarKeys } from "./ast/loader.js";
|
|
11
11
|
import { buildIndexArtifacts } from "./pipeline.js";
|
|
12
12
|
import { renderGraphJson } from "./render/graph-json.js";
|
|
13
|
-
import { scanRepo } from "./scan.js";
|
|
13
|
+
import { scanRepo, type RepoScan } from "./scan.js";
|
|
14
14
|
import { buildCallerIndex } from "./callers.js";
|
|
15
15
|
import { detectWorkspaces } from "./workspaces.js";
|
|
16
16
|
import { gitChurn } from "./git.js";
|
|
@@ -26,9 +26,10 @@ import { writeMemory, readMemory, deleteMemory, listMemories } from "./memory.js
|
|
|
26
26
|
import { searchIndex } from "./bm25.js";
|
|
27
27
|
import { checkRules, parseRules } from "./rules.js";
|
|
28
28
|
import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel } from "./embed/model.js";
|
|
29
|
-
import { buildEmbeddingIndex } from "./embed/index.js";
|
|
29
|
+
import { buildEmbeddingIndex, type EmbeddingIndex } from "./embed/index.js";
|
|
30
30
|
import { searchSemantic } from "./embed/search.js";
|
|
31
31
|
import { resolveEmbedEndpoint, buildEndpointIndex, encodeQueryViaEndpoint, probeEndpoint } from "./embed/endpoint.js";
|
|
32
|
+
import { sha1 } from "./hash.js";
|
|
32
33
|
|
|
33
34
|
interface RpcRequest {
|
|
34
35
|
jsonrpc: "2.0";
|
|
@@ -270,7 +271,7 @@ const TOOLS = [
|
|
|
270
271
|
{
|
|
271
272
|
name: "search",
|
|
272
273
|
description:
|
|
273
|
-
'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default — deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse
|
|
274
|
+
'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default — deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse an embedding tier (HTTP endpoint, else a local static model) with lexical — the response then wraps the ranked list as `{ results, tier, degradedReason? }`, `tier` being "endpoint"/"static" when fusion happened or "lexical" (with `degradedReason`) when it did not (see embed_status). Without `semantic`, the response is the bare ranked array, unchanged.',
|
|
274
275
|
inputSchema: {
|
|
275
276
|
type: "object",
|
|
276
277
|
properties: {
|
|
@@ -286,7 +287,7 @@ const TOOLS = [
|
|
|
286
287
|
semantic: {
|
|
287
288
|
type: "boolean",
|
|
288
289
|
description:
|
|
289
|
-
|
|
290
|
+
'RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. The response reports the effective tier as a top-level `tier` field ("endpoint"/"static" on success, "lexical" plus `degradedReason` when neither is available/reachable) instead of degrading silently — see embed_status.',
|
|
290
291
|
},
|
|
291
292
|
},
|
|
292
293
|
required: ["repo", "query"],
|
|
@@ -320,6 +321,50 @@ function str(v: unknown): string | undefined {
|
|
|
320
321
|
function strArray(v: unknown): string[] | undefined {
|
|
321
322
|
return Array.isArray(v) && v.every((x) => typeof x === "string") && v.length ? (v as string[]) : undefined;
|
|
322
323
|
}
|
|
324
|
+
function errMessage(e: unknown): string {
|
|
325
|
+
return e instanceof Error ? e.message : String(e);
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
// --- embedding index memoization --------------------------------------------
|
|
329
|
+
// The MCP server process is long-lived, but every `search` call used to redo
|
|
330
|
+
// the FULL corpus embedding build — N `buildEndpointIndex` HTTP round-trips,
|
|
331
|
+
// or a full `buildEmbeddingIndex` re-encode pass — even when nothing in the
|
|
332
|
+
// repo changed between requests. Memoize the last build behind a fingerprint
|
|
333
|
+
// of the scan contents plus the tier's identity, so an unchanged repo reuses
|
|
334
|
+
// the cached index and any file add/edit/remove (or a switch of endpoint/model)
|
|
335
|
+
// still rebuilds. RepoScan carries no fingerprint of its own (checked
|
|
336
|
+
// scan.ts/types.ts) — every FileRecord already carries a content hash, so
|
|
337
|
+
// hashing the (rel, hash) pairs is the staleness oracle scan.ts itself uses.
|
|
338
|
+
export function scanFingerprint(scan: RepoScan): string {
|
|
339
|
+
return sha1(scan.files.map((f) => `${f.rel}:${f.hash}`).join("\n"));
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
export interface EmbeddingIndexCacheKey {
|
|
343
|
+
mode: "endpoint" | "static";
|
|
344
|
+
// Distinguishes cache entries across configs sharing the same scan: the
|
|
345
|
+
// endpoint URL, or the model dir + modelId for the static tier.
|
|
346
|
+
identity: string;
|
|
347
|
+
scan: RepoScan;
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
// A SINGLE entry — never an unbounded map — holding the most recent build.
|
|
351
|
+
let embeddingIndexCache: { key: string; index: EmbeddingIndex } | undefined;
|
|
352
|
+
|
|
353
|
+
// Reuse the cached index when (mode, identity, scanFingerprint) matches the
|
|
354
|
+
// last build; otherwise call `build` and cache its result. A failed build is
|
|
355
|
+
// NEVER cached (matches today's per-call error behavior: the next request
|
|
356
|
+
// retries from scratch, and a still-valid previous entry — under a different
|
|
357
|
+
// key — is left untouched).
|
|
358
|
+
export async function memoizedEmbeddingIndex(
|
|
359
|
+
key: EmbeddingIndexCacheKey,
|
|
360
|
+
build: () => Promise<EmbeddingIndex> | EmbeddingIndex,
|
|
361
|
+
): Promise<EmbeddingIndex> {
|
|
362
|
+
const cacheKey = `${key.mode}:${key.identity}:${scanFingerprint(key.scan)}`;
|
|
363
|
+
if (embeddingIndexCache && embeddingIndexCache.key === cacheKey) return embeddingIndexCache.index;
|
|
364
|
+
const index = await build();
|
|
365
|
+
embeddingIndexCache = { key: cacheKey, index };
|
|
366
|
+
return index;
|
|
367
|
+
}
|
|
323
368
|
|
|
324
369
|
async function callTool(name: string, args: Record<string, unknown>): Promise<string> {
|
|
325
370
|
const repo = str(args.repo);
|
|
@@ -459,25 +504,49 @@ async function callTool(name: string, args: Record<string, unknown>): Promise<st
|
|
|
459
504
|
const limit = typeof args.limit === "number" ? args.limit : undefined;
|
|
460
505
|
const fuzzy = typeof args.fuzzy === "boolean" ? args.fuzzy : undefined;
|
|
461
506
|
if (args.semantic === true) {
|
|
507
|
+
// semantic:true changes the response SHAPE (wraps the ranked list with a
|
|
508
|
+
// `tier`/`degradedReason?`) so a caller can tell "fusion happened" apart
|
|
509
|
+
// from "degraded to lexical" — see the `search` tool description. This
|
|
510
|
+
// branch is the ONLY place that shape appears; plain lexical search below
|
|
511
|
+
// stays the bare array, byte-compat for existing consumers.
|
|
462
512
|
const endpoint = resolveEmbedEndpoint();
|
|
463
513
|
if (endpoint) {
|
|
464
514
|
// Rich tier — endpoint takes PRECEDENCE over a local static model. An
|
|
465
|
-
// unreachable/malformed endpoint degrades to lexical
|
|
515
|
+
// unreachable/malformed endpoint degrades to lexical, now with a reason.
|
|
516
|
+
// The corpus index is memoized per (endpoint, scan state) — the query
|
|
517
|
+
// itself is always re-encoded fresh (it differs per call).
|
|
466
518
|
try {
|
|
467
|
-
const index = await buildEndpointIndex(scan);
|
|
519
|
+
const index = await memoizedEmbeddingIndex({ mode: "endpoint", identity: endpoint, scan }, () => buildEndpointIndex(scan));
|
|
468
520
|
const queryVec = await encodeQueryViaEndpoint(query);
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
521
|
+
const results = searchSemantic(scan, query, index, { queryVec, limit, fuzzy });
|
|
522
|
+
return JSON.stringify({ results, tier: "endpoint" }, null, 2);
|
|
523
|
+
} catch (e) {
|
|
524
|
+
const results = searchIndex(scan, query, { limit, fuzzy });
|
|
525
|
+
return JSON.stringify(
|
|
526
|
+
{ results, tier: "lexical", degradedReason: `embedding endpoint failed: ${errMessage(e)}` },
|
|
527
|
+
null,
|
|
528
|
+
2,
|
|
529
|
+
);
|
|
472
530
|
}
|
|
473
531
|
}
|
|
474
532
|
const modelDir = resolveEmbedModelDir(repo);
|
|
475
533
|
const model = modelDir ? loadEmbedModel(modelDir) : undefined;
|
|
476
534
|
if (model) {
|
|
477
|
-
const index =
|
|
478
|
-
|
|
535
|
+
const index = await memoizedEmbeddingIndex(
|
|
536
|
+
{ mode: "static", identity: `${modelDir}#${model.modelId}`, scan },
|
|
537
|
+
() => buildEmbeddingIndex(scan, model),
|
|
538
|
+
);
|
|
539
|
+
const results = searchSemantic(scan, query, index, { model, limit, fuzzy });
|
|
540
|
+
return JSON.stringify({ results, tier: "static" }, null, 2);
|
|
479
541
|
}
|
|
480
|
-
//
|
|
542
|
+
// Opt-in tier not activated (no endpoint, no model asset) — degrade to
|
|
543
|
+
// lexical with a reason instead of failing silently.
|
|
544
|
+
const results = searchIndex(scan, query, { limit, fuzzy });
|
|
545
|
+
return JSON.stringify(
|
|
546
|
+
{ results, tier: "lexical", degradedReason: "no embedding endpoint or static model configured — see embed_status" },
|
|
547
|
+
null,
|
|
548
|
+
2,
|
|
549
|
+
);
|
|
481
550
|
}
|
|
482
551
|
return JSON.stringify(searchIndex(scan, query, { limit, fuzzy }), null, 2);
|
|
483
552
|
}
|
package/src/types.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Single source of truth for the engine version the bundle reports. Kept in
|
|
2
2
|
// lockstep with package.json by the release pipeline. Do not edit by hand
|
|
3
3
|
// outside a release.
|
|
4
|
-
export const ENGINE_VERSION = "2.
|
|
4
|
+
export const ENGINE_VERSION = "2.12.0";
|
|
5
5
|
|
|
6
6
|
// Bumped whenever the on-disk artifact shape changes, so a consumer can reject
|
|
7
7
|
// an index written by an incompatible engine instead of misreading it. The
|
|
@@ -30,9 +30,15 @@ export const SCHEMA_VERSION = 4;
|
|
|
30
30
|
// "reexport" when it resolves in-file. v8 fixes the C/C++ regex tier
|
|
31
31
|
// reporting a function DEFINITION as a call to itself (`void load(void) {`
|
|
32
32
|
// yielding a spurious call `load@<defline>`, found during the ultrasec
|
|
33
|
-
// consumer migration) by excluding call candidates
|
|
34
|
-
//
|
|
35
|
-
|
|
33
|
+
// consumer migration) by excluding, from call candidates, only the
|
|
34
|
+
// definition's own token on its definition line — not every occurrence of
|
|
35
|
+
// that name+line pair, which would also drop genuine same-line calls (dense
|
|
36
|
+
// one-liners packing several definitions, one-line recursion); v9 makes a
|
|
37
|
+
// resolved export-alias symbol also cite the original declaration's own line
|
|
38
|
+
// (and endLine, when the AST tier populated one) instead of the export
|
|
39
|
+
// statement's line — a citation-precision fix (issue #9) for consumers
|
|
40
|
+
// (ultradoc) that use file:line as evidence.
|
|
41
|
+
export const EXTRACTOR_VERSION = 9;
|
|
36
42
|
|
|
37
43
|
// How a file is classified. `code` gets symbol/import extraction; `doc` gets
|
|
38
44
|
// link/heading extraction; the rest are catalogued but not deeply parsed.
|