@maxgfr/codeindex 2.11.1 → 2.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/MIGRATION.md CHANGED
@@ -119,7 +119,8 @@ note only) — so wiring it on is safe before an asset exists.
119
119
  | **ultrasec** | walker + gitignore + scope + symlink guard | `scanRepo` (`scope`, `gitignore` — its own semantics, ported here) |
120
120
  | | per-language defs/imports/calls extraction | `extractCode` (`symbols`/`refs`/`calls`) |
121
121
  | | `resolveImport` | `resolveImport` |
122
- | | `buildGraph` (import+call edges, symbolDefs, callersBySymbol, enclosingSymbol) | `buildGraph` + `resolveCallEdges` + `buildCallerIndex` + `enclosingSymbol` |
122
+ | | `buildGraph` (import+call edges, symbolDefs) | `buildGraph` + `resolveCallEdges` |
123
+ | | `callersBySymbol`, `enclosingSymbol` (raw-recall taint-BFS input) | `buildRawCallerIndex` (issue #8) — every name-matched call site keyed by the raw callee name, no def resolution or gating, `enclosingSymbol` computed per site. `buildCallerIndex` is **NOT** a substitute here: it is def-resolved and gated (language-family filter, JS/TS import gate, same-file self-declaration skip) and will silently drop sites a recall consumer needs. Both are bounded by `FileRecord.calls`'s per-file 512-call cap (dedup by name+line) — a file with more raw call sites than that loses sites upstream of either function. |
123
124
  | | keeps | taint source→sink enumeration, external scanners, EPSS/KEV/CVSS, SARIF |
124
125
 
125
126
  ## Golden-diff adjudication (every migration)
@@ -0,0 +1,242 @@
1
+ # Deterministic static-embedding tier (semantic search)
2
+
3
+ codeindex's default search is keyless BM25 lexical ranking (see `search`). The
4
+ **semantic tier** (v2.10.0) adds embedding-based retrieval that is still
5
+ **keyless, zero-dependency, and byte-deterministic** — the same doctrine as the
6
+ rest of the engine, and the reason it does *not* use onnxruntime / transformers.js
7
+ (those are threaded and dequantize int8 to fp32 for matmul, so they are never
8
+ byte-stable across architectures).
9
+
10
+ It is **strictly opt-in by asset presence**, exactly like the tree-sitter
11
+ grammar tier: with no model on disk the engine silently stays lexical.
12
+
13
+ ## Why a *static* embedding model
14
+
15
+ A static embedding (model2vec / potion style) is a plain lookup table:
16
+ `token → row vector`. There is **no neural forward pass** — "inference" is:
17
+
18
+ ```
19
+ tokenize → gather rows → mean-pool → L2-normalize → int8-quantize
20
+ ```
21
+
22
+ Every step is pure JS and chosen to be byte-identical everywhere:
23
+
24
+ | step | how determinism is guaranteed |
25
+ |---|---|
26
+ | tokenize | `foldText` (NFKD + strip combining marks) → camelCase/ACRONYM split → lowercase → split on non-alphanumeric |
27
+ | wordpiece | greedy longest-match against the vocab, BERT-style `##` continuations; an unsplittable word → the `[UNK]` row |
28
+ | gather | model rows widen float→double exactly |
29
+ | mean-pool | IEEE-754 double add/div in a fixed token order |
30
+ | L2-norm | division by `sqrt(Σv²)` — `sqrt` is correctly-rounded per IEEE-754 |
31
+ | quantize | `unit × 127`, **round-half-to-even** (not `Math.round`), clamp to `[-127,127]` → int8 |
32
+
33
+ Ranking is a **pure integer dot product** of int8 vectors. Because the fixed
34
+ `1/127` scale is shared by every vector, for a given query the integer dot
35
+ `Σ q·c` is monotonic with the cosine estimate — no per-vector float scale is
36
+ stored or multiplied, so nothing about ranking can drift between platforms.
37
+
38
+ **Key insight:** the corpus *and* the query are encoded by the *same* JS encoder,
39
+ so self-consistency is all that is required — fidelity to the original HF
40
+ tokenizer is not. The JS tokenizer deliberately differs from the Python
41
+ reference; that gap is assumed and harmless.
42
+
43
+ ## Activation (opt-in by asset)
44
+
45
+ Resolution order (`resolveEmbedModelDir`):
46
+
47
+ 1. `CODEINDEX_EMBED_DIR` (explicit override — wins outright)
48
+ 2. `<repo>/.codeindex/models/`
49
+ 3. `<cwd>/.codeindex/models/`
50
+
51
+ A directory "has a model" when it contains `model.json`. Nothing found →
52
+ `undefined` → silent lexical degradation. **Models are never in the npm
53
+ tarball** (`package.json` `files` is unchanged; the pack-smoke test asserts no
54
+ `model.json` / `*.safetensors` / `embeddings.bin` ships).
55
+
56
+ ### `model.json` shape
57
+
58
+ ```json
59
+ {
60
+ "modelId": "…",
61
+ "dim": 256,
62
+ "unk": "[UNK]",
63
+ "vocab": ["[UNK]", "auth", "##token", "..."],
64
+ "weights": [[0.0, ...], [1.0, ...], ...]
65
+ }
66
+ ```
67
+
68
+ `vocab[i]` is the token whose row is `weights[i]` (length `dim`). Human-authorable
69
+ JSON so a tiny fixture is committable; a real `pull`ed model uses the same shape
70
+ at scale.
71
+
72
+ ### The official asset
73
+
74
+ `codeindex embed pull` (no env) fetches the published **`embed-model-v1`** GitHub
75
+ release: a conversion of [`minishlab/potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M)
76
+ (revision `bf8b0566…`, **MIT**) — 256-dim, a 27 559-token WordPiece vocab, int8
77
+ rows at one global scale. The download is **sha256-verified** against
78
+ `EMBED_ASSET_SHA256` (`src/embed/model.ts`) before it is written; a mismatch
79
+ fails the pull and writes nothing. The asset is **never** committed to git or
80
+ shipped in the npm tarball — it lives only in the release. Reproduce it
81
+ byte-for-byte with the pinned toolchain in [`scripts/embed-asset/`](../scripts/embed-asset/).
82
+
83
+ ## CLI
84
+
85
+ ```sh
86
+ # Effective mode (none/static/endpoint), model, EMBED_VERSION, endpoint + its
87
+ # reachability (JSON). Precedence: endpoint > static model.
88
+ codeindex embed status --repo <dir>
89
+
90
+ # Fetch the official static-embedding asset (published as the `embed-model-v1`
91
+ # GitHub release) into CODEINDEX_EMBED_DIR (or <repo>/.codeindex/models/). The
92
+ # built-in default download is sha256-verified before it is written; set
93
+ # CODEINDEX_EMBED_URL to fetch a custom/mirrored model.json instead (un-verified).
94
+ codeindex embed pull --repo <dir>
95
+
96
+ # Build embeddings.bin from the repo into --out <dir> (static tier only).
97
+ codeindex embed build --repo <dir> --out <dir>
98
+
99
+ # Print (or --run) the docker command that starts the rich-tier embedding server.
100
+ codeindex embed serve # prints the one-liner
101
+ codeindex embed serve --run # runs it (needs docker)
102
+
103
+ # Search: RRF-fuse an embedding tier + lexical. Degrades to lexical (exit 0)
104
+ # when no tier is available/reachable.
105
+ codeindex search "http client retry" --repo <dir> --semantic
106
+ ```
107
+
108
+ `codeindex index --out <dir>` additionally writes `embeddings.bin` next to
109
+ `graph.json` **iff** a model is present.
110
+
111
+ ## Artifact: `embeddings.bin`
112
+
113
+ ```
114
+ "CIE1" 4-byte ASCII magic
115
+ uint32 LE header length
116
+ UTF-8 JSON header { embedVersion, modelId, dim, count, records:[{file,symbol,line}] }
117
+ int8 body count × dim signed bytes (row-major)
118
+ ```
119
+
120
+ The header carries **no absolute path and no timestamp**; records follow scan
121
+ order (files sorted by `rel`, symbols in declaration order). Two builds of an
122
+ unchanged repo produce byte-identical `embeddings.bin`. A dedicated
123
+ `EMBED_VERSION` (independent of `SCHEMA_VERSION`) plus `modelId` + `dim` in the
124
+ header invalidate a stale or foreign artifact.
125
+
126
+ Granularity is **per-symbol** (name + signature + file summary + path segments),
127
+ with a **per-file** fallback record for symbol-less files (docs, config) so
128
+ every file with content is represented.
129
+
130
+ ## Fusion
131
+
132
+ Semantic and lexical rankings live on incomparable scales (BM25 score vs integer
133
+ dot product), so `searchSemantic` fuses them by **rank** via the shared `rrf`
134
+ helper (k=60) — never a linear score blend. A `SemanticSearchResult` extends the
135
+ lexical `SearchResult` additively with `semanticSymbol` (the corpus symbol whose
136
+ embedding was closest for that file).
137
+
138
+ ## Degradation matrix
139
+
140
+ | present | behavior |
141
+ |---|---|
142
+ | nothing | BM25 lexical |
143
+ | + fuzzy | BM25 + trigram fallback for `df==0` terms (v2.9.0) |
144
+ | + model asset | RRF-fused deterministic static semantic search |
145
+ | + HTTP endpoint (`CODEINDEX_EMBED_ENDPOINT`) | rich tier — **wins over a static model** (v2.11.0) |
146
+ | `--semantic`, nothing available | lexical + stderr note, **exit 0** |
147
+ | endpoint set but unreachable/timeout | lexical + stderr note, **exit 0** (never falls back to the static model) |
148
+
149
+ ## Library
150
+
151
+ ```ts
152
+ import {
153
+ resolveEmbedModelDir, loadEmbedModel,
154
+ buildEmbeddingIndex, serializeEmbeddings, deserializeEmbeddings,
155
+ searchSemantic, encode, EMBED_VERSION,
156
+ } from "@maxgfr/codeindex";
157
+
158
+ const model = loadEmbedModel(resolveEmbedModelDir("/repo"));
159
+ if (model) {
160
+ const scan = scanRepo("/repo");
161
+ const index = buildEmbeddingIndex(scan, model); // deterministic
162
+ const results = searchSemantic(scan, "http retry", index, { model });
163
+ }
164
+ ```
165
+
166
+ `searchSemantic(scan, query, index, opts)` with no `opts.model` (or no index)
167
+ returns the pure lexical ranking — the same silent degradation the CLI uses.
168
+
169
+ ## The rich tier: containerized HTTP embedding endpoint (v2.11.0)
170
+
171
+ The **rich tier** lets the engine consume a real neural embedding model (e.g.
172
+ all-MiniLM-L6-v2) served by a local container, instead of the static lookup
173
+ table. It is opt-in via one env var:
174
+
175
+ ```sh
176
+ CODEINDEX_EMBED_ENDPOINT=http://localhost:8756 \
177
+ codeindex search "auth token" --repo . --semantic
178
+ ```
179
+
180
+ ### Precedence
181
+
182
+ `endpoint > static model > none`. Setting `CODEINDEX_EMBED_ENDPOINT` is an
183
+ **explicit** user intent, so it wins over any local `model.json`. `embed status`
184
+ reports the effective `mode`. If the endpoint is defined but
185
+ unreachable/times-out/malformed, the search degrades straight to **lexical**
186
+ (stderr note, exit 0) — it does **not** silently fall back to the static model.
187
+
188
+ ### How the float vectors join the deterministic pipeline
189
+
190
+ The engine POSTs corpus and query texts to the endpoint, receives **float**
191
+ vectors, then runs them through the **exact same** tail as the static tier —
192
+ `quantize()` in `encode.ts`: L2-normalize → int8 at the fixed `1/127` scale
193
+ (round-half-to-even, clamp). Ranking stays a pure integer dot product. So the
194
+ only tier-specific step is the encoder; fusion and ranking are shared code.
195
+
196
+ Corpus embeddings for the endpoint tier are built **at search time**
197
+ (`buildEndpointIndex`) and are **never serialized** to `embeddings.bin`: unlike
198
+ the static tier, endpoint vectors are float and provider-dependent, so this tier
199
+ is **deterministic per image digest**, not byte-golden. Pin the digest for
200
+ reproducibility:
201
+
202
+ ```sh
203
+ docker run -d -p 8756:8756 ghcr.io/maxgfr/codeindex-embed@sha256:<digest>
204
+ ```
205
+
206
+ ### HTTP protocol (implement your own server)
207
+
208
+ `CODEINDEX_EMBED_ENDPOINT` is the server **base URL**. The client derives:
209
+
210
+ | method + path | request | response |
211
+ |---|---|---|
212
+ | `POST {base}/embed` | `{ "texts": ["…", …] }` | `{ "vectors": [[…float…], …] }` (same order, one row per text) |
213
+ | `GET {base}/healthz` | — | `200` (any body) when ready |
214
+
215
+ Notes: any embedding dimension is accepted (the engine quantizes whatever it
216
+ receives); vectors need not be pre-normalized (the engine L2-normalizes anyway);
217
+ the request times out after `CODEINDEX_EMBED_TIMEOUT_MS` (default 30 000).
218
+ `codeindex embed serve` prints/`--run`s the docker command for the official
219
+ image; the **library never orchestrates docker** (that is CLI-only).
220
+
221
+ The reference server is `docker/embed/` (transformers.js + all-MiniLM-L6-v2,
222
+ model baked in at build, offline at run, non-root, `:8756`), published to
223
+ `ghcr.io/maxgfr/codeindex-embed` by `.github/workflows/embed-image.yml`.
224
+
225
+ ### Library
226
+
227
+ ```ts
228
+ import {
229
+ buildEndpointIndex, encodeQueryViaEndpoint, probeEndpoint,
230
+ searchSemantic, quantize,
231
+ } from "@maxgfr/codeindex";
232
+
233
+ if (await probeEndpoint("http://localhost:8756")) {
234
+ const scan = scanRepo("/repo");
235
+ const index = await buildEndpointIndex(scan); // float → int8 corpus
236
+ const queryVec = await encodeQueryViaEndpoint("auth token");
237
+ const results = searchSemantic(scan, "auth token", index, { queryVec });
238
+ }
239
+ ```
240
+
241
+ `embedViaEndpoint(texts, opts)` is the low-level client (`{ texts }` →
242
+ `{ vectors }`).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@maxgfr/codeindex",
3
- "version": "2.11.1",
3
+ "version": "2.12.0",
4
4
  "description": "Self-contained, deterministic repo-indexing engine: walk + language detection + symbol/import extraction (tree-sitter AST with regex fallback) + import resolution + typed cross-file link-graph + analytics. Ships as a single zero-dependency engine.mjs that consumer tools vendor.",
5
5
  "type": "module",
6
6
  "packageManager": "pnpm@10.33.0",
@@ -23,7 +23,8 @@
23
23
  "scripts/cli.mjs",
24
24
  "scripts/grammars",
25
25
  "src",
26
- "docs/MIGRATION.md"
26
+ "docs/MIGRATION.md",
27
+ "docs/SEMANTIC.md"
27
28
  ],
28
29
  "engines": {
29
30
  "node": ">=18"
@@ -1,6 +1,6 @@
1
- declare const ENGINE_VERSION = "2.11.1";
1
+ declare const ENGINE_VERSION = "2.12.0";
2
2
  declare const SCHEMA_VERSION = 4;
3
- declare const EXTRACTOR_VERSION = 8;
3
+ declare const EXTRACTOR_VERSION = 9;
4
4
  type FileKind = "code" | "doc" | "config" | "asset" | "other";
5
5
  type EdgeKind = "contains" | "doc-link" | "import" | "call" | "use" | "mention";
6
6
  type Tier = 0 | 1 | 2;
@@ -337,6 +337,14 @@ type CallerIndex = Map<string, CallerEntry>;
337
337
  declare function computeImportPairs(scan: RepoScan): Set<string>;
338
338
  declare function buildCallerIndex(scan: RepoScan, importPairs?: Set<string>, opts?: CallerIndexOptions): CallerIndex;
339
339
  declare function enclosingSymbol(scan: RepoScan, file: string, line: number): CodeSymbol | undefined;
340
+ interface RawCallerSite {
341
+ file: string;
342
+ line: number;
343
+ receiver?: string;
344
+ enclosingSymbol?: CodeSymbol;
345
+ }
346
+ type RawCallerIndex = Map<string, RawCallerSite[]>;
347
+ declare function buildRawCallerIndex(scan: RepoScan): RawCallerIndex;
340
348
 
341
349
  declare function symbolsOverview(scan: RepoScan, rel: string): CodeSymbol[];
342
350
  interface SymbolMatch extends CodeSymbol {
@@ -512,7 +520,11 @@ interface StaticEmbedModel {
512
520
  declare function resolveEmbedModelDir(repo?: string): string | undefined;
513
521
  declare function hasEmbedModel(repo?: string): boolean;
514
522
  declare function loadEmbedModel(dir?: string): StaticEmbedModel | undefined;
515
- declare function resolveEmbedPullUrl(): string | undefined;
523
+ interface EmbedPullTarget {
524
+ url: string;
525
+ sha256?: string;
526
+ }
527
+ declare function resolveEmbedPullUrl(): EmbedPullTarget;
516
528
 
517
529
  declare function basicTokenize(text: string): string[];
518
530
  declare function wordpiece(word: string, model: StaticEmbedModel): number[];
@@ -693,4 +705,4 @@ declare function rrf<T>(lists: T[][], keyOf: (item: T) => string, k?: number): M
693
705
 
694
706
  declare function runCli(argv: string[]): Promise<void>;
695
707
 
696
- export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
708
+ export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawCallerIndex, type RawCallerSite, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildRawCallerIndex, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
@@ -14,9 +14,9 @@ var ENGINE_VERSION, SCHEMA_VERSION, EXTRACTOR_VERSION;
14
14
  var init_types = __esm({
15
15
  "src/types.ts"() {
16
16
  "use strict";
17
- ENGINE_VERSION = "2.11.1";
17
+ ENGINE_VERSION = "2.12.0";
18
18
  SCHEMA_VERSION = 4;
19
- EXTRACTOR_VERSION = 8;
19
+ EXTRACTOR_VERSION = 9;
20
20
  }
21
21
  });
22
22
 
@@ -737,8 +737,8 @@ function extractReexports(rel, content, localSymbols) {
737
737
  const out2 = [];
738
738
  const seen = /* @__PURE__ */ new Set();
739
739
  const lineAt = (idx) => content.slice(0, idx).split(/\r?\n/).length;
740
- const localKindOf = /* @__PURE__ */ new Map();
741
- for (const s of localSymbols) if (!localKindOf.has(s.name)) localKindOf.set(s.name, s.kind);
740
+ const localDeclOf = /* @__PURE__ */ new Map();
741
+ for (const s of localSymbols) if (!localDeclOf.has(s.name)) localDeclOf.set(s.name, s);
742
742
  const named = /export\s*\{([\s\S]*?)\}\s*(?:from\s*['"]([^'"]+)['"])?\s*;?/g;
743
743
  let m;
744
744
  while ((m = named.exec(content)) && out2.length < 60) {
@@ -750,12 +750,13 @@ function extractReexports(rel, content, localSymbols) {
750
750
  const name2 = as ? as[2] : p;
751
751
  if (!/^[A-Za-z_$][\w$]*$/.test(name2) || name2 === "default" || seen.has(name2)) continue;
752
752
  seen.add(name2);
753
- const mirroredKind = !from ? localKindOf.get(orig) : void 0;
753
+ const decl = !from ? localDeclOf.get(orig) : void 0;
754
754
  out2.push({
755
755
  name: name2,
756
- kind: mirroredKind ?? "reexport",
756
+ kind: decl?.kind ?? "reexport",
757
757
  file: rel,
758
- line: lineAt(m.index),
758
+ line: decl ? decl.line : lineAt(m.index),
759
+ ...decl?.endLine !== void 0 ? { endLine: decl.endLine } : {},
759
760
  signature: from ? `export { ${name2} } from "${from}"` : `export { ${name2} }`,
760
761
  exported: true,
761
762
  lang
@@ -7731,8 +7732,11 @@ function buildCallerIndex(scan2, importPairs, opts = {}) {
7731
7732
  function enclosingSymbol(scan2, file, line) {
7732
7733
  const f = scan2.files.find((x) => x.rel === file);
7733
7734
  if (!f?.symbols.length) return void 0;
7735
+ return enclosingAmong(f.symbols, line);
7736
+ }
7737
+ function enclosingAmong(symbols, line) {
7734
7738
  let best;
7735
- for (const s of f.symbols) {
7739
+ for (const s of symbols) {
7736
7740
  if (REFERENCE_KINDS3.has(s.kind)) continue;
7737
7741
  if (s.line > line) continue;
7738
7742
  if (s.endLine !== void 0 && line > s.endLine) continue;
@@ -7742,6 +7746,29 @@ function enclosingSymbol(scan2, file, line) {
7742
7746
  }
7743
7747
  return best;
7744
7748
  }
7749
+ function buildRawCallerIndex(scan2) {
7750
+ const byName = /* @__PURE__ */ new Map();
7751
+ for (const f of scan2.files) {
7752
+ if (!f.calls?.length) continue;
7753
+ const symbols = f.symbols.filter((s) => !REFERENCE_KINDS3.has(s.kind));
7754
+ for (const c2 of f.calls) {
7755
+ const site = { file: f.rel, line: c2.line };
7756
+ if (c2.receiver !== void 0) site.receiver = c2.receiver;
7757
+ const enc = enclosingAmong(symbols, c2.line);
7758
+ if (enc) site.enclosingSymbol = enc;
7759
+ let arr = byName.get(c2.name);
7760
+ if (!arr) byName.set(c2.name, arr = []);
7761
+ arr.push(site);
7762
+ }
7763
+ }
7764
+ const index = /* @__PURE__ */ new Map();
7765
+ for (const name2 of [...byName.keys()].sort(byStr)) {
7766
+ const sites = byName.get(name2);
7767
+ sites.sort((a, b) => byStr(a.file, b.file) || a.line - b.line);
7768
+ index.set(name2, sites);
7769
+ }
7770
+ return index;
7771
+ }
7745
7772
  var REFERENCE_KINDS3;
7746
7773
  var init_callers = __esm({
7747
7774
  "src/callers.ts"() {
@@ -9374,6 +9401,7 @@ var init_bm25 = __esm({
9374
9401
  });
9375
9402
 
9376
9403
  // src/embed/model.ts
9404
+ import { createHash as createHash2 } from "crypto";
9377
9405
  import { existsSync as existsSync3, readFileSync as readFileSync5 } from "fs";
9378
9406
  import { join as join10 } from "path";
9379
9407
  function resolveEmbedModelDir(repo) {
@@ -9419,15 +9447,30 @@ function loadEmbedModel(dir) {
9419
9447
  return { modelId, dim, unk, unkId, vocabSize, vocab: vmap, weights: flat };
9420
9448
  }
9421
9449
  function resolveEmbedPullUrl() {
9422
- const url = process.env.CODEINDEX_EMBED_URL;
9423
- return url && url.trim() ? url.trim() : void 0;
9450
+ const env = process.env.CODEINDEX_EMBED_URL;
9451
+ if (env && env.trim()) return { url: env.trim() };
9452
+ return { url: DEFAULT_EMBED_URL, sha256: EMBED_ASSET_SHA256 };
9453
+ }
9454
+ async function fetchEmbedModel(url, expectedSha256) {
9455
+ const res = await fetch(url);
9456
+ if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`);
9457
+ const body2 = await res.text();
9458
+ if (expectedSha256) {
9459
+ const got = createHash2("sha256").update(body2).digest("hex");
9460
+ if (got !== expectedSha256) {
9461
+ throw new Error(`sha256 mismatch: expected ${expectedSha256}, got ${got}`);
9462
+ }
9463
+ }
9464
+ return body2;
9424
9465
  }
9425
- var EMBED_VERSION, DEFAULT_EMBED_DIRNAME;
9466
+ var EMBED_VERSION, DEFAULT_EMBED_DIRNAME, DEFAULT_EMBED_URL, EMBED_ASSET_SHA256;
9426
9467
  var init_model = __esm({
9427
9468
  "src/embed/model.ts"() {
9428
9469
  "use strict";
9429
9470
  EMBED_VERSION = 1;
9430
9471
  DEFAULT_EMBED_DIRNAME = "models";
9472
+ DEFAULT_EMBED_URL = "https://github.com/maxgfr/codeindex/releases/download/embed-model-v1/model.json";
9473
+ EMBED_ASSET_SHA256 = "163ad053eab4e9a80d421ed4164f32292c83290f02fbbe6fe4b9b1cd6ea18d34";
9431
9474
  }
9432
9475
  });
9433
9476
 
@@ -10129,7 +10172,9 @@ var init_viz = __esm({
10129
10172
  // src/mcp.ts
10130
10173
  var mcp_exports = {};
10131
10174
  __export(mcp_exports, {
10132
- runMcpServer: () => runMcpServer
10175
+ memoizedEmbeddingIndex: () => memoizedEmbeddingIndex,
10176
+ runMcpServer: () => runMcpServer,
10177
+ scanFingerprint: () => scanFingerprint
10133
10178
  });
10134
10179
  import { createInterface } from "readline";
10135
10180
  function str(v) {
@@ -10138,6 +10183,19 @@ function str(v) {
10138
10183
  function strArray(v) {
10139
10184
  return Array.isArray(v) && v.every((x) => typeof x === "string") && v.length ? v : void 0;
10140
10185
  }
10186
+ function errMessage(e) {
10187
+ return e instanceof Error ? e.message : String(e);
10188
+ }
10189
+ function scanFingerprint(scan2) {
10190
+ return sha1(scan2.files.map((f) => `${f.rel}:${f.hash}`).join("\n"));
10191
+ }
10192
+ async function memoizedEmbeddingIndex(key, build) {
10193
+ const cacheKey = `${key.mode}:${key.identity}:${scanFingerprint(key.scan)}`;
10194
+ if (embeddingIndexCache && embeddingIndexCache.key === cacheKey) return embeddingIndexCache.index;
10195
+ const index = await build();
10196
+ embeddingIndexCache = { key: cacheKey, index };
10197
+ return index;
10198
+ }
10141
10199
  async function callTool(name2, args2) {
10142
10200
  const repo = str(args2.repo);
10143
10201
  if (!repo) throw new Error("`repo` is required (absolute path to the repository root)");
@@ -10278,19 +10336,35 @@ async function callTool(name2, args2) {
10278
10336
  const endpoint = resolveEmbedEndpoint();
10279
10337
  if (endpoint) {
10280
10338
  try {
10281
- const index = await buildEndpointIndex(scan2);
10339
+ const index = await memoizedEmbeddingIndex({ mode: "endpoint", identity: endpoint, scan: scan2 }, () => buildEndpointIndex(scan2));
10282
10340
  const queryVec = await encodeQueryViaEndpoint(query);
10283
- return JSON.stringify(searchSemantic(scan2, query, index, { queryVec, limit, fuzzy }), null, 2);
10284
- } catch {
10285
- return JSON.stringify(searchIndex(scan2, query, { limit, fuzzy }), null, 2);
10341
+ const results2 = searchSemantic(scan2, query, index, { queryVec, limit, fuzzy });
10342
+ return JSON.stringify({ results: results2, tier: "endpoint" }, null, 2);
10343
+ } catch (e) {
10344
+ const results2 = searchIndex(scan2, query, { limit, fuzzy });
10345
+ return JSON.stringify(
10346
+ { results: results2, tier: "lexical", degradedReason: `embedding endpoint failed: ${errMessage(e)}` },
10347
+ null,
10348
+ 2
10349
+ );
10286
10350
  }
10287
10351
  }
10288
10352
  const modelDir = resolveEmbedModelDir(repo);
10289
10353
  const model = modelDir ? loadEmbedModel(modelDir) : void 0;
10290
10354
  if (model) {
10291
- const index = buildEmbeddingIndex(scan2, model);
10292
- return JSON.stringify(searchSemantic(scan2, query, index, { model, limit, fuzzy }), null, 2);
10355
+ const index = await memoizedEmbeddingIndex(
10356
+ { mode: "static", identity: `${modelDir}#${model.modelId}`, scan: scan2 },
10357
+ () => buildEmbeddingIndex(scan2, model)
10358
+ );
10359
+ const results2 = searchSemantic(scan2, query, index, { model, limit, fuzzy });
10360
+ return JSON.stringify({ results: results2, tier: "static" }, null, 2);
10293
10361
  }
10362
+ const results = searchIndex(scan2, query, { limit, fuzzy });
10363
+ return JSON.stringify(
10364
+ { results, tier: "lexical", degradedReason: "no embedding endpoint or static model configured \u2014 see embed_status" },
10365
+ null,
10366
+ 2
10367
+ );
10294
10368
  }
10295
10369
  return JSON.stringify(searchIndex(scan2, query, { limit, fuzzy }), null, 2);
10296
10370
  }
@@ -10371,7 +10445,7 @@ async function runMcpServer() {
10371
10445
  }
10372
10446
  }
10373
10447
  }
10374
- var repoProp, scopeProps, TOOLS;
10448
+ var repoProp, scopeProps, TOOLS, embeddingIndexCache;
10375
10449
  var init_mcp = __esm({
10376
10450
  "src/mcp.ts"() {
10377
10451
  "use strict";
@@ -10398,6 +10472,7 @@ var init_mcp = __esm({
10398
10472
  init_embed();
10399
10473
  init_search();
10400
10474
  init_endpoint();
10475
+ init_hash();
10401
10476
  repoProp = { repo: { type: "string", description: "Absolute path to the repository root" } };
10402
10477
  scopeProps = {
10403
10478
  scope: { type: "string", description: "Restrict to one directory (repo-relative)" },
@@ -10610,7 +10685,7 @@ var init_mcp = __esm({
10610
10685
  },
10611
10686
  {
10612
10687
  name: "search",
10613
- description: 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default \u2014 deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse the deterministic static-embedding tier when a model asset is present (degrades to lexical otherwise \u2014 see embed_status).',
10688
+ description: 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default \u2014 deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse an embedding tier (HTTP endpoint, else a local static model) with lexical \u2014 the response then wraps the ranked list as `{ results, tier, degradedReason? }`, `tier` being "endpoint"/"static" when fusion happened or "lexical" (with `degradedReason`) when it did not (see embed_status). Without `semantic`, the response is the bare ranked array, unchanged.',
10614
10689
  inputSchema: {
10615
10690
  type: "object",
10616
10691
  properties: {
@@ -10624,7 +10699,7 @@ var init_mcp = __esm({
10624
10699
  },
10625
10700
  semantic: {
10626
10701
  type: "boolean",
10627
- description: "RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. Degrades silently to lexical-only when neither is available/reachable \u2014 see embed_status."
10702
+ description: 'RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. The response reports the effective tier as a top-level `tier` field ("endpoint"/"static" on success, "lexical" plus `degradedReason` when neither is available/reachable) instead of degrading silently \u2014 see embed_status.'
10628
10703
  }
10629
10704
  },
10630
10705
  required: ["repo", "query"]
@@ -11116,8 +11191,9 @@ Commands:
11116
11191
  embed status Effective mode (none/static/endpoint), model +
11117
11192
  EMBED_VERSION, and endpoint reachability (JSON)
11118
11193
  embed build Write embeddings.bin into --out <dir> (static tier)
11119
- embed pull Fetch the model asset into CODEINDEX_EMBED_DIR (or
11120
- <repo>/.codeindex/models/) \u2014 needs CODEINDEX_EMBED_URL
11194
+ embed pull Fetch the official model asset into CODEINDEX_EMBED_DIR
11195
+ (or <repo>/.codeindex/models/); sha256-verified. Override
11196
+ the source with CODEINDEX_EMBED_URL
11121
11197
  embed serve Print (or --run) the docker command that starts the
11122
11198
  containerized embedding server (rich tier)
11123
11199
  rules Architecture rules (forbidden edges, cycles, orphans) validated
@@ -11397,26 +11473,20 @@ async function runCli(argv) {
11397
11473
  process.stderr.write(`codeindex: ${index.records.length} embedding records \u2192 ${flags2.out}/embeddings.bin (model ${model.modelId})
11398
11474
  `);
11399
11475
  } else if (sub === "pull") {
11400
- const url = resolveEmbedPullUrl();
11401
- if (!url) {
11402
- process.stderr.write(
11403
- "codeindex: no model URL configured. The official static-embedding asset is not published yet.\nSet CODEINDEX_EMBED_URL to a model.json URL (optionally CODEINDEX_EMBED_DIR as the destination), then re-run `codeindex embed pull`.\n"
11404
- );
11405
- process.exitCode = 1;
11406
- return;
11407
- }
11476
+ const { url, sha256 } = resolveEmbedPullUrl();
11408
11477
  const destDir = process.env.CODEINDEX_EMBED_DIR ?? join12(flags2.repo, ".codeindex", "models");
11409
11478
  mkdirSync2(destDir, { recursive: true });
11410
11479
  process.stderr.write(`codeindex: fetching model from ${url} \u2192 ${join12(destDir, "model.json")}
11411
11480
  `);
11412
- const res = await fetch(url);
11413
- if (!res.ok) {
11414
- process.stderr.write(`codeindex: pull failed \u2014 HTTP ${res.status} from ${url}
11481
+ let body2;
11482
+ try {
11483
+ body2 = await fetchEmbedModel(url, sha256);
11484
+ } catch (e) {
11485
+ process.stderr.write(`codeindex: pull failed \u2014 ${e instanceof Error ? e.message : String(e)} (nothing written)
11415
11486
  `);
11416
11487
  process.exitCode = 1;
11417
11488
  return;
11418
11489
  }
11419
- const body2 = await res.text();
11420
11490
  try {
11421
11491
  JSON.parse(body2);
11422
11492
  } catch {
@@ -11508,6 +11578,7 @@ export {
11508
11578
  buildGraph,
11509
11579
  buildIndexArtifacts,
11510
11580
  buildModules,
11581
+ buildRawCallerIndex,
11511
11582
  buildResolveContext,
11512
11583
  buildSymbolIndex,
11513
11584
  byKey,
package/src/callers.ts CHANGED
Binary file
@@ -1,3 +1,4 @@
1
+ import { createHash } from "node:crypto";
1
2
  import { existsSync, readFileSync } from "node:fs";
2
3
  import { join } from "node:path";
3
4
 
@@ -20,12 +21,24 @@ import { join } from "node:path";
20
21
  // graph.json / symbols.json consumers.
21
22
  export const EMBED_VERSION = 1;
22
23
 
23
- // The default model-name a `pull` writes and a `status` reports. The real asset
24
- // URL is intentionally NOT hard-coded yet (see resolveEmbedPullUrl): the asset
25
- // is unpublished, so `pull` fails cleanly asking for CODEINDEX_EMBED_URL rather
26
- // than fetching a phantom.
24
+ // The default sub-directory a `pull` writes into and a `status` reports.
27
25
  export const DEFAULT_EMBED_DIRNAME = "models";
28
26
 
27
+ // The official static-embedding asset, published as the dedicated GitHub release
28
+ // `embed-model-v1` (tag deliberately outside the `v<semver>` semantic-release
29
+ // namespace, so publishing it triggers no CI). `pull` uses this when
30
+ // CODEINDEX_EMBED_URL is unset. Reproduce the asset byte-for-byte with the pinned
31
+ // toolchain in scripts/embed-asset/.
32
+ export const DEFAULT_EMBED_URL =
33
+ "https://github.com/maxgfr/codeindex/releases/download/embed-model-v1/model.json";
34
+
35
+ // sha256 of the published model.json. `pull` verifies the downloaded bytes
36
+ // against this ONLY when it fetched the built-in default (a custom
37
+ // CODEINDEX_EMBED_URL is the user's own mirror and keeps the un-verified
38
+ // behavior). Regenerate alongside DEFAULT_EMBED_URL when re-cutting the asset:
39
+ // `shasum -a 256 model.json`.
40
+ export const EMBED_ASSET_SHA256 = "163ad053eab4e9a80d421ed4164f32292c83290f02fbbe6fe4b9b1cd6ea18d34";
41
+
29
42
  // A loaded static-embedding model. `vocab` maps a wordpiece token to its row
30
43
  // index; `weights` is the flat row-major (vocabSize × dim) matrix as IEEE-754
31
44
  // doubles (float rows widen to double exactly, so mean-pool stays deterministic).
@@ -107,11 +120,37 @@ export function loadEmbedModel(dir?: string): StaticEmbedModel | undefined {
107
120
  return { modelId, dim, unk, unkId, vocabSize, vocab: vmap, weights: flat };
108
121
  }
109
122
 
110
- // Resolve the URL `embed pull` fetches from. The official asset is not published
111
- // yet, so there is no built-in default: the user must point CODEINDEX_EMBED_URL
112
- // at a model.json (or a directory serving one). Returns undefined when unset —
113
- // the caller turns that into a clean, actionable failure, never a crash.
114
- export function resolveEmbedPullUrl(): string | undefined {
115
- const url = process.env.CODEINDEX_EMBED_URL;
116
- return url && url.trim() ? url.trim() : undefined;
123
+ // What `embed pull` fetches, and whether to verify it. CODEINDEX_EMBED_URL wins
124
+ // outright (the user's explicit override / private mirror) and carries NO
125
+ // sha256, so a custom asset keeps the current un-verified behavior. With no env,
126
+ // we fall back to the built-in official asset AND its pinned sha256, so `pull`
127
+ // can prove integrity of the default download.
128
+ export interface EmbedPullTarget {
129
+ url: string;
130
+ sha256?: string; // present only for the built-in default → verify
131
+ }
132
+
133
+ export function resolveEmbedPullUrl(): EmbedPullTarget {
134
+ const env = process.env.CODEINDEX_EMBED_URL;
135
+ if (env && env.trim()) return { url: env.trim() };
136
+ return { url: DEFAULT_EMBED_URL, sha256: EMBED_ASSET_SHA256 };
137
+ }
138
+
139
+ // Fetch a model.json body over HTTP, following redirects (GitHub release assets
140
+ // 302 to a CDN — native fetch follows by default). When `expectedSha256` is
141
+ // given, the downloaded bytes are hashed and MUST match or this throws with a
142
+ // clear message and returns nothing — so a corrupt/tampered default asset never
143
+ // gets written. A non-2xx response also throws. Verification and I/O are split
144
+ // from the CLI so both the success and the sha-mismatch paths are unit-testable.
145
+ export async function fetchEmbedModel(url: string, expectedSha256?: string): Promise<string> {
146
+ const res = await fetch(url);
147
+ if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`);
148
+ const body = await res.text();
149
+ if (expectedSha256) {
150
+ const got = createHash("sha256").update(body).digest("hex");
151
+ if (got !== expectedSha256) {
152
+ throw new Error(`sha256 mismatch: expected ${expectedSha256}, got ${got}`);
153
+ }
154
+ }
155
+ return body;
117
156
  }
package/src/engine-cli.ts CHANGED
@@ -19,7 +19,7 @@ import { symbolComplexity, riskHotspots } from "./complexity.js";
19
19
  import { renderMermaid } from "./viz.js";
20
20
  import { searchIndex } from "./bm25.js";
21
21
  import { checkRules, parseRules } from "./rules.js";
22
- import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel, resolveEmbedPullUrl } from "./embed/model.js";
22
+ import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel, resolveEmbedPullUrl, fetchEmbedModel } from "./embed/model.js";
23
23
  import { buildEmbeddingIndex, serializeEmbeddings } from "./embed/index.js";
24
24
  import { searchSemantic } from "./embed/search.js";
25
25
  import {
@@ -55,8 +55,9 @@ Commands:
55
55
  embed status Effective mode (none/static/endpoint), model +
56
56
  EMBED_VERSION, and endpoint reachability (JSON)
57
57
  embed build Write embeddings.bin into --out <dir> (static tier)
58
- embed pull Fetch the model asset into CODEINDEX_EMBED_DIR (or
59
- <repo>/.codeindex/models/) needs CODEINDEX_EMBED_URL
58
+ embed pull Fetch the official model asset into CODEINDEX_EMBED_DIR
59
+ (or <repo>/.codeindex/models/); sha256-verified. Override
60
+ the source with CODEINDEX_EMBED_URL
60
61
  embed serve Print (or --run) the docker command that starts the
61
62
  containerized embedding server (rich tier)
62
63
  rules Architecture rules (forbidden edges, cycles, orphans) validated
@@ -387,26 +388,21 @@ export async function runCli(argv: string[]): Promise<void> {
387
388
  writeFileSync(join(flags.out, "embeddings.bin"), serializeEmbeddings(index));
388
389
  process.stderr.write(`codeindex: ${index.records.length} embedding records → ${flags.out}/embeddings.bin (model ${model.modelId})\n`);
389
390
  } else if (sub === "pull") {
390
- const url = resolveEmbedPullUrl();
391
- if (!url) {
392
- // Clean, actionable failure the official asset is not published yet.
393
- process.stderr.write(
394
- "codeindex: no model URL configured. The official static-embedding asset is not published yet.\n" +
395
- "Set CODEINDEX_EMBED_URL to a model.json URL (optionally CODEINDEX_EMBED_DIR as the destination), then re-run `codeindex embed pull`.\n",
396
- );
397
- process.exitCode = 1;
398
- return;
399
- }
391
+ // Default: the official published asset + its pinned sha256. A user-set
392
+ // CODEINDEX_EMBED_URL overrides both (mirror/custom model, no verification).
393
+ const { url, sha256 } = resolveEmbedPullUrl();
400
394
  const destDir = process.env.CODEINDEX_EMBED_DIR ?? join(flags.repo, ".codeindex", "models");
401
395
  mkdirSync(destDir, { recursive: true });
402
396
  process.stderr.write(`codeindex: fetching model from ${url} → ${join(destDir, "model.json")}\n`);
403
- const res = await fetch(url);
404
- if (!res.ok) {
405
- process.stderr.write(`codeindex: pull failed HTTP ${res.status} from ${url}\n`);
397
+ let body: string;
398
+ try {
399
+ // Follows redirects (GitHub CDN) and verifies sha256 for the default asset.
400
+ body = await fetchEmbedModel(url, sha256);
401
+ } catch (e) {
402
+ process.stderr.write(`codeindex: pull failed — ${e instanceof Error ? e.message : String(e)} (nothing written)\n`);
406
403
  process.exitCode = 1;
407
404
  return;
408
405
  }
409
- const body = await res.text();
410
406
  try {
411
407
  JSON.parse(body);
412
408
  } catch {
package/src/engine.ts CHANGED
@@ -55,7 +55,7 @@ export { buildModules, isTestFile, tierForPath } from "./modules.js";
55
55
  export type { ModuleInfo } from "./modules.js";
56
56
  export { buildGraph, uniqueSymbolDefs } from "./graph.js";
57
57
  export { resolveCallEdges } from "./calls.js";
58
- export { buildCallerIndex, enclosingSymbol, computeImportPairs } from "./callers.js";
58
+ export { buildCallerIndex, buildRawCallerIndex, enclosingSymbol, computeImportPairs } from "./callers.js";
59
59
  export { symbolsOverview, findSymbol, findReferences } from "./query.js";
60
60
  export { resolveUniqueSymbol, replaceSymbolBody, insertAfterSymbol, insertBeforeSymbol } from "./edit.js";
61
61
  export type { EditResult } from "./edit.js";
@@ -143,6 +143,10 @@ export type { ArchRule, ForbiddenEdgeRule, BuiltinRule, RuleSeverity, RuleViolat
143
143
  // Caller-index recall mode (issue #7) — options for buildCallerIndex above.
144
144
  export type { CallerIndexOptions } from "./callers.js";
145
145
 
146
+ // Raw-recall caller index (issue #8) — ungated, def-resolution-free companion
147
+ // to buildCallerIndex above; see its JSDoc in callers.ts for the contract.
148
+ export type { RawCallerIndex, RawCallerSite } from "./callers.js";
149
+
146
150
  // Behavioral analytics (git-history mining) + the token-budgeted repo map.
147
151
  export { changeCoupling, rankHotspots } from "./coupling.js";
148
152
  export type { ChangeCoupling, CouplingOptions, Hotspot } from "./coupling.js";
@@ -75,11 +75,14 @@ const REEXPORT_EXTS = new Set([".ts", ".tsx", ".mts", ".cts", ".js", ".jsx", ".m
75
75
  //
76
76
  // An ALIAS with no `from` clause (`export { b as c }`) renames an in-file
77
77
  // declaration — `localSymbols` (already extracted by the AST or regex tier)
78
- // lets us resolve `b` and mirror ITS kind onto `c` (e.g. "function"), so the
79
- // alias reads as the real symbol it is rather than the generic "reexport".
78
+ // lets us resolve `b` and mirror ITS kind, declaration line, and endLine (AST
79
+ // tier only) onto `c` (e.g. "function" at b's own line), so the alias reads
80
+ // as the real symbol it is — citeable at its actual declaration — rather than
81
+ // the generic "reexport" pinned to the export statement's line.
80
82
  // A true cross-module re-export (`export { b as c } from "./mod"`) has no
81
83
  // local `b` to resolve — and an alias the local pass genuinely can't see
82
- // (destructured/ambient/etc.) falls back the same way — both keep "reexport".
84
+ // (destructured/ambient/etc.) falls back the same way — both keep "reexport"
85
+ // and cite the export statement's own line, the only line they have.
83
86
  //
84
87
  // Shared by extractCode (extract/code.ts) AND the standalone extractSymbols
85
88
  // (lang/registry.ts) — ultradoc and other direct extractSymbols consumers hit
@@ -92,8 +95,11 @@ export function extractReexports(rel: string, content: string, localSymbols: Cod
92
95
  const out: CodeSymbol[] = [];
93
96
  const seen = new Set<string>();
94
97
  const lineAt = (idx: number): number => content.slice(0, idx).split(/\r?\n/).length;
95
- const localKindOf = new Map<string, string>();
96
- for (const s of localSymbols) if (!localKindOf.has(s.name)) localKindOf.set(s.name, s.kind);
98
+ // Keyed on the whole CodeSymbol (not just kind) so the alias branch below
99
+ // can also cite the resolved declaration's own line/endLine, not just mirror
100
+ // its kind.
101
+ const localDeclOf = new Map<string, CodeSymbol>();
102
+ for (const s of localSymbols) if (!localDeclOf.has(s.name)) localDeclOf.set(s.name, s);
97
103
 
98
104
  const named = /export\s*\{([\s\S]*?)\}\s*(?:from\s*['"]([^'"]+)['"])?\s*;?/g;
99
105
  let m: RegExpExecArray | null;
@@ -106,9 +112,14 @@ export function extractReexports(rel: string, content: string, localSymbols: Cod
106
112
  const name = as ? as[2]! : p;
107
113
  if (!/^[A-Za-z_$][\w$]*$/.test(name) || name === "default" || seen.has(name)) continue;
108
114
  seen.add(name);
109
- const mirroredKind = !from ? localKindOf.get(orig) : undefined;
115
+ // A resolved decl means this is a same-file alias: cite ITS line (and
116
+ // endLine, when the AST tier populated one) rather than the export
117
+ // statement's — an unresolved alias or a `from`-clause re-export has no
118
+ // local declaration to point at, so it keeps lineAt(m.index) below.
119
+ const decl = !from ? localDeclOf.get(orig) : undefined;
110
120
  out.push({
111
- name, kind: mirroredKind ?? "reexport", file: rel, line: lineAt(m.index),
121
+ name, kind: decl?.kind ?? "reexport", file: rel, line: decl ? decl.line : lineAt(m.index),
122
+ ...(decl?.endLine !== undefined ? { endLine: decl.endLine } : {}),
112
123
  signature: from ? `export { ${name} } from "${from}"` : `export { ${name} }`,
113
124
  exported: true, lang,
114
125
  });
package/src/mcp.ts CHANGED
@@ -10,7 +10,7 @@ import { ENGINE_VERSION } from "./types.js";
10
10
  import { ensureGrammars, allGrammarKeys } from "./ast/loader.js";
11
11
  import { buildIndexArtifacts } from "./pipeline.js";
12
12
  import { renderGraphJson } from "./render/graph-json.js";
13
- import { scanRepo } from "./scan.js";
13
+ import { scanRepo, type RepoScan } from "./scan.js";
14
14
  import { buildCallerIndex } from "./callers.js";
15
15
  import { detectWorkspaces } from "./workspaces.js";
16
16
  import { gitChurn } from "./git.js";
@@ -26,9 +26,10 @@ import { writeMemory, readMemory, deleteMemory, listMemories } from "./memory.js
26
26
  import { searchIndex } from "./bm25.js";
27
27
  import { checkRules, parseRules } from "./rules.js";
28
28
  import { EMBED_VERSION, resolveEmbedModelDir, loadEmbedModel } from "./embed/model.js";
29
- import { buildEmbeddingIndex } from "./embed/index.js";
29
+ import { buildEmbeddingIndex, type EmbeddingIndex } from "./embed/index.js";
30
30
  import { searchSemantic } from "./embed/search.js";
31
31
  import { resolveEmbedEndpoint, buildEndpointIndex, encodeQueryViaEndpoint, probeEndpoint } from "./embed/endpoint.js";
32
+ import { sha1 } from "./hash.js";
32
33
 
33
34
  interface RpcRequest {
34
35
  jsonrpc: "2.0";
@@ -270,7 +271,7 @@ const TOOLS = [
270
271
  {
271
272
  name: "search",
272
273
  description:
273
- 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default — deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse the deterministic static-embedding tier when a model asset is present (degrades to lexical otherwise see embed_status).',
274
+ 'Natural-language-ish lexical search: BM25 ranking (k1=1.2, b=0.75) over symbol names (camelCase/snake_case subtokens), file path segments, markdown headings and summary lines. NOT embeddings by default — deterministic, diacritic-folded, zero API keys. Answers "where is auth handled?"-style queries with ranked files, matched terms and top symbols. Query terms with zero document frequency get a deterministic trigram-fuzzy fallback (typo-tolerant) unless `fuzzy: false`. Set `semantic: true` to RRF-fuse an embedding tier (HTTP endpoint, else a local static model) with lexical the response then wraps the ranked list as `{ results, tier, degradedReason? }`, `tier` being "endpoint"/"static" when fusion happened or "lexical" (with `degradedReason`) when it did not (see embed_status). Without `semantic`, the response is the bare ranked array, unchanged.',
274
275
  inputSchema: {
275
276
  type: "object",
276
277
  properties: {
@@ -286,7 +287,7 @@ const TOOLS = [
286
287
  semantic: {
287
288
  type: "boolean",
288
289
  description:
289
- "RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. Degrades silently to lexical-only when neither is available/reachable — see embed_status.",
290
+ 'RRF-fuse an embedding tier with lexical (default false). Precedence: the HTTP endpoint (CODEINDEX_EMBED_ENDPOINT) if set, else a local static model. The response reports the effective tier as a top-level `tier` field ("endpoint"/"static" on success, "lexical" plus `degradedReason` when neither is available/reachable) instead of degrading silently — see embed_status.',
290
291
  },
291
292
  },
292
293
  required: ["repo", "query"],
@@ -320,6 +321,50 @@ function str(v: unknown): string | undefined {
320
321
  function strArray(v: unknown): string[] | undefined {
321
322
  return Array.isArray(v) && v.every((x) => typeof x === "string") && v.length ? (v as string[]) : undefined;
322
323
  }
324
+ function errMessage(e: unknown): string {
325
+ return e instanceof Error ? e.message : String(e);
326
+ }
327
+
328
+ // --- embedding index memoization --------------------------------------------
329
+ // The MCP server process is long-lived, but every `search` call used to redo
330
+ // the FULL corpus embedding build — N `buildEndpointIndex` HTTP round-trips,
331
+ // or a full `buildEmbeddingIndex` re-encode pass — even when nothing in the
332
+ // repo changed between requests. Memoize the last build behind a fingerprint
333
+ // of the scan contents plus the tier's identity, so an unchanged repo reuses
334
+ // the cached index and any file add/edit/remove (or a switch of endpoint/model)
335
+ // still rebuilds. RepoScan carries no fingerprint of its own (checked
336
+ // scan.ts/types.ts) — every FileRecord already carries a content hash, so
337
+ // hashing the (rel, hash) pairs is the staleness oracle scan.ts itself uses.
338
+ export function scanFingerprint(scan: RepoScan): string {
339
+ return sha1(scan.files.map((f) => `${f.rel}:${f.hash}`).join("\n"));
340
+ }
341
+
342
+ export interface EmbeddingIndexCacheKey {
343
+ mode: "endpoint" | "static";
344
+ // Distinguishes cache entries across configs sharing the same scan: the
345
+ // endpoint URL, or the model dir + modelId for the static tier.
346
+ identity: string;
347
+ scan: RepoScan;
348
+ }
349
+
350
+ // A SINGLE entry — never an unbounded map — holding the most recent build.
351
+ let embeddingIndexCache: { key: string; index: EmbeddingIndex } | undefined;
352
+
353
+ // Reuse the cached index when (mode, identity, scanFingerprint) matches the
354
+ // last build; otherwise call `build` and cache its result. A failed build is
355
+ // NEVER cached (matches today's per-call error behavior: the next request
356
+ // retries from scratch, and a still-valid previous entry — under a different
357
+ // key — is left untouched).
358
+ export async function memoizedEmbeddingIndex(
359
+ key: EmbeddingIndexCacheKey,
360
+ build: () => Promise<EmbeddingIndex> | EmbeddingIndex,
361
+ ): Promise<EmbeddingIndex> {
362
+ const cacheKey = `${key.mode}:${key.identity}:${scanFingerprint(key.scan)}`;
363
+ if (embeddingIndexCache && embeddingIndexCache.key === cacheKey) return embeddingIndexCache.index;
364
+ const index = await build();
365
+ embeddingIndexCache = { key: cacheKey, index };
366
+ return index;
367
+ }
323
368
 
324
369
  async function callTool(name: string, args: Record<string, unknown>): Promise<string> {
325
370
  const repo = str(args.repo);
@@ -459,25 +504,49 @@ async function callTool(name: string, args: Record<string, unknown>): Promise<st
459
504
  const limit = typeof args.limit === "number" ? args.limit : undefined;
460
505
  const fuzzy = typeof args.fuzzy === "boolean" ? args.fuzzy : undefined;
461
506
  if (args.semantic === true) {
507
+ // semantic:true changes the response SHAPE (wraps the ranked list with a
508
+ // `tier`/`degradedReason?`) so a caller can tell "fusion happened" apart
509
+ // from "degraded to lexical" — see the `search` tool description. This
510
+ // branch is the ONLY place that shape appears; plain lexical search below
511
+ // stays the bare array, byte-compat for existing consumers.
462
512
  const endpoint = resolveEmbedEndpoint();
463
513
  if (endpoint) {
464
514
  // Rich tier — endpoint takes PRECEDENCE over a local static model. An
465
- // unreachable/malformed endpoint degrades to lexical (no throw).
515
+ // unreachable/malformed endpoint degrades to lexical, now with a reason.
516
+ // The corpus index is memoized per (endpoint, scan state) — the query
517
+ // itself is always re-encoded fresh (it differs per call).
466
518
  try {
467
- const index = await buildEndpointIndex(scan);
519
+ const index = await memoizedEmbeddingIndex({ mode: "endpoint", identity: endpoint, scan }, () => buildEndpointIndex(scan));
468
520
  const queryVec = await encodeQueryViaEndpoint(query);
469
- return JSON.stringify(searchSemantic(scan, query, index, { queryVec, limit, fuzzy }), null, 2);
470
- } catch {
471
- return JSON.stringify(searchIndex(scan, query, { limit, fuzzy }), null, 2);
521
+ const results = searchSemantic(scan, query, index, { queryVec, limit, fuzzy });
522
+ return JSON.stringify({ results, tier: "endpoint" }, null, 2);
523
+ } catch (e) {
524
+ const results = searchIndex(scan, query, { limit, fuzzy });
525
+ return JSON.stringify(
526
+ { results, tier: "lexical", degradedReason: `embedding endpoint failed: ${errMessage(e)}` },
527
+ null,
528
+ 2,
529
+ );
472
530
  }
473
531
  }
474
532
  const modelDir = resolveEmbedModelDir(repo);
475
533
  const model = modelDir ? loadEmbedModel(modelDir) : undefined;
476
534
  if (model) {
477
- const index = buildEmbeddingIndex(scan, model);
478
- return JSON.stringify(searchSemantic(scan, query, index, { model, limit, fuzzy }), null, 2);
535
+ const index = await memoizedEmbeddingIndex(
536
+ { mode: "static", identity: `${modelDir}#${model.modelId}`, scan },
537
+ () => buildEmbeddingIndex(scan, model),
538
+ );
539
+ const results = searchSemantic(scan, query, index, { model, limit, fuzzy });
540
+ return JSON.stringify({ results, tier: "static" }, null, 2);
479
541
  }
480
- // Degrade to lexical (opt-in tier not activated) — no throw.
542
+ // Opt-in tier not activated (no endpoint, no model asset) — degrade to
543
+ // lexical with a reason instead of failing silently.
544
+ const results = searchIndex(scan, query, { limit, fuzzy });
545
+ return JSON.stringify(
546
+ { results, tier: "lexical", degradedReason: "no embedding endpoint or static model configured — see embed_status" },
547
+ null,
548
+ 2,
549
+ );
481
550
  }
482
551
  return JSON.stringify(searchIndex(scan, query, { limit, fuzzy }), null, 2);
483
552
  }
package/src/types.ts CHANGED
@@ -1,7 +1,7 @@
1
1
  // Single source of truth for the engine version the bundle reports. Kept in
2
2
  // lockstep with package.json by the release pipeline. Do not edit by hand
3
3
  // outside a release.
4
- export const ENGINE_VERSION = "2.11.1";
4
+ export const ENGINE_VERSION = "2.12.0";
5
5
 
6
6
  // Bumped whenever the on-disk artifact shape changes, so a consumer can reject
7
7
  // an index written by an incompatible engine instead of misreading it. The
@@ -30,9 +30,15 @@ export const SCHEMA_VERSION = 4;
30
30
  // "reexport" when it resolves in-file. v8 fixes the C/C++ regex tier
31
31
  // reporting a function DEFINITION as a call to itself (`void load(void) {`
32
32
  // yielding a spurious call `load@<defline>`, found during the ultrasec
33
- // consumer migration) by excluding call candidates whose name+line match one
34
- // of the file's own extracted symbols.
35
- export const EXTRACTOR_VERSION = 8;
33
+ // consumer migration) by excluding, from call candidates, only the
34
+ // definition's own token on its definition line — not every occurrence of
35
+ // that name+line pair, which would also drop genuine same-line calls (dense
36
+ // one-liners packing several definitions, one-line recursion); v9 makes a
37
+ // resolved export-alias symbol also cite the original declaration's own line
38
+ // (and endLine, when the AST tier populated one) instead of the export
39
+ // statement's line — a citation-precision fix (issue #9) for consumers
40
+ // (ultradoc) that use file:line as evidence.
41
+ export const EXTRACTOR_VERSION = 9;
36
42
 
37
43
  // How a file is classified. `code` gets symbol/import extraction; `doc` gets
38
44
  // link/heading extraction; the rest are catalogued but not deeply parsed.