@maxgfr/codeindex 2.11.1 → 2.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -1
- package/docs/MIGRATION.md +39 -1
- package/docs/SEMANTIC.md +249 -0
- package/package.json +3 -2
- package/scripts/engine.d.mts +25 -6
- package/scripts/engine.mjs +179 -76
- package/src/ast/extract.ts +7 -5
- package/src/callers.ts +0 -0
- package/src/embed/model.ts +73 -25
- package/src/engine-cli.ts +31 -21
- package/src/engine.ts +6 -2
- package/src/extract/code.ts +9 -5
- package/src/lang/common.ts +18 -7
- package/src/lang/js-ts.ts +1 -1
- package/src/mcp.ts +111 -15
- package/src/scan.ts +9 -1
- package/src/types.ts +13 -4
- package/src/walk.ts +15 -2
package/README.md
CHANGED
|
@@ -129,9 +129,13 @@ and the `embeddings.bin` artifact are byte-identical across builds and platforms
|
|
|
129
129
|
It is **opt-in by asset**: with no model on disk the engine silently stays
|
|
130
130
|
lexical, and `--semantic` without a model returns lexical results on **exit 0**
|
|
131
131
|
(a stderr note only). Models are **never** shipped in the package; a model is
|
|
132
|
-
resolved from `CODEINDEX_EMBED_DIR` or `<repo>/.codeindex/models/`.
|
|
132
|
+
resolved from `CODEINDEX_EMBED_DIR` or `<repo>/.codeindex/models/`. Getting one
|
|
133
|
+
is zero-config: `codeindex embed pull` fetches the official `embed-model-v1`
|
|
134
|
+
release asset, sha256-verified before anything is written.
|
|
133
135
|
|
|
134
136
|
```sh
|
|
137
|
+
codeindex embed pull --repo . # fetch the official model asset into
|
|
138
|
+
# CODEINDEX_EMBED_DIR (or <repo>/.codeindex/models/); sha256-verified
|
|
135
139
|
codeindex embed status --repo . # effective mode + reachability (JSON)
|
|
136
140
|
codeindex embed build --repo . --out .codeindex # write embeddings.bin
|
|
137
141
|
codeindex search "http client retry" --repo . --semantic
|
package/docs/MIGRATION.md
CHANGED
|
@@ -93,6 +93,43 @@ embeddings live in a separate `embeddings.bin` sidecar keyed by a dedicated
|
|
|
93
93
|
`--semantic` without a model degrades to lexical results on **exit 0** (a stderr
|
|
94
94
|
note only) — so wiring it on is safe before an asset exists.
|
|
95
95
|
|
|
96
|
+
## v2.12.0 — two return-shape changes (check your call sites)
|
|
97
|
+
|
|
98
|
+
Not additive: two public return shapes changed in this release without a
|
|
99
|
+
compat flag, so a consumer re-pinning across v2.12.0 must check both call
|
|
100
|
+
sites. Artifact schemas are untouched (`SCHEMA_VERSION` / `EMBED_VERSION`
|
|
101
|
+
unchanged) — this is API shape only.
|
|
102
|
+
|
|
103
|
+
- `resolveEmbedPullUrl()` now returns an `EmbedPullTarget`
|
|
104
|
+
(`{ url: string; sha256?: string }`) — it previously returned
|
|
105
|
+
`string | undefined`. It always resolves: `CODEINDEX_EMBED_URL` wins
|
|
106
|
+
outright and carries **no** `sha256` (a custom mirror keeps the
|
|
107
|
+
un-verified behavior); with no env it falls back to the built-in official
|
|
108
|
+
asset **with** its pinned `sha256` so `embed pull` verifies the default
|
|
109
|
+
download. Replace `const url = resolveEmbedPullUrl()` with
|
|
110
|
+
`const { url, sha256 } = resolveEmbedPullUrl()`. The `EmbedPullTarget`
|
|
111
|
+
type is exported from the barrel.
|
|
112
|
+
- MCP `search` with `semantic: true` now returns
|
|
113
|
+
`{ results, tier, degradedReason? }` — it previously returned the bare
|
|
114
|
+
ranked array. `tier` is `"endpoint" | "static" | "lexical"` and
|
|
115
|
+
`degradedReason` is present only when the semantic tier degraded to
|
|
116
|
+
lexical, so a caller can tell "fusion happened" apart from "degraded".
|
|
117
|
+
Plain lexical `search` (no `semantic: true`) still returns the bare
|
|
118
|
+
array, byte-compatible with existing consumers.
|
|
119
|
+
|
|
120
|
+
## v2.13.0 — `.codeindex` excluded from the walk
|
|
121
|
+
|
|
122
|
+
`.codeindex/` — the engine's own output directory (index artifacts, pulled
|
|
123
|
+
models, MCP memories) — joined `IGNORE_DIRS`, so `walk`/`scanRepo` no longer
|
|
124
|
+
descend into it and memories stop entering BM25/embedding results (previously
|
|
125
|
+
every `write_memory` also churned the scan fingerprint, busting the MCP
|
|
126
|
+
server's memoized embedding index). Access memories through the MCP memory
|
|
127
|
+
tools (`read_memory` / `list_memories` / `delete_memory`), never `search`.
|
|
128
|
+
This is a file-set change only — extraction shape and `SCHEMA_VERSION` are
|
|
129
|
+
untouched, so **no re-pin is required**; a consumer that deliberately wants
|
|
130
|
+
`.codeindex` walked can pass `ignoreDirs` (replace semantics, also new in
|
|
131
|
+
this release) with its own set.
|
|
132
|
+
|
|
96
133
|
## Per-skill mapping (what to replace with what)
|
|
97
134
|
|
|
98
135
|
| Skill | Replace | With (engine export) |
|
|
@@ -119,7 +156,8 @@ note only) — so wiring it on is safe before an asset exists.
|
|
|
119
156
|
| **ultrasec** | walker + gitignore + scope + symlink guard | `scanRepo` (`scope`, `gitignore` — its own semantics, ported here) |
|
|
120
157
|
| | per-language defs/imports/calls extraction | `extractCode` (`symbols`/`refs`/`calls`) |
|
|
121
158
|
| | `resolveImport` | `resolveImport` |
|
|
122
|
-
| | `buildGraph` (import+call edges, symbolDefs
|
|
159
|
+
| | `buildGraph` (import+call edges, symbolDefs) | `buildGraph` + `resolveCallEdges` |
|
|
160
|
+
| | `callersBySymbol`, `enclosingSymbol` (raw-recall taint-BFS input) | `buildRawCallerIndex` (issue #8) — every name-matched call site keyed by the raw callee name, no def resolution or gating, `enclosingSymbol` computed per site. `buildCallerIndex` is **NOT** a substitute here: it is def-resolved and gated (language-family filter, JS/TS import gate, same-file self-declaration skip) and will silently drop sites a recall consumer needs. Both are bounded by `FileRecord.calls`'s per-file 512-call cap (dedup by name+line) — a file with more raw call sites than that loses sites upstream of either function. |
|
|
123
161
|
| | keeps | taint source→sink enumeration, external scanners, EPSS/KEV/CVSS, SARIF |
|
|
124
162
|
|
|
125
163
|
## Golden-diff adjudication (every migration)
|
package/docs/SEMANTIC.md
ADDED
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
# Deterministic static-embedding tier (semantic search)
|
|
2
|
+
|
|
3
|
+
codeindex's default search is keyless BM25 lexical ranking (see `search`). The
|
|
4
|
+
**semantic tier** (v2.10.0) adds embedding-based retrieval that is still
|
|
5
|
+
**keyless, zero-dependency, and byte-deterministic** — the same doctrine as the
|
|
6
|
+
rest of the engine, and the reason it does *not* use onnxruntime / transformers.js
|
|
7
|
+
(those are threaded and dequantize int8 to fp32 for matmul, so they are never
|
|
8
|
+
byte-stable across architectures).
|
|
9
|
+
|
|
10
|
+
It is **strictly opt-in by asset presence**, exactly like the tree-sitter
|
|
11
|
+
grammar tier: with no model on disk the engine silently stays lexical.
|
|
12
|
+
|
|
13
|
+
## Why a *static* embedding model
|
|
14
|
+
|
|
15
|
+
A static embedding (model2vec / potion style) is a plain lookup table:
|
|
16
|
+
`token → row vector`. There is **no neural forward pass** — "inference" is:
|
|
17
|
+
|
|
18
|
+
```
|
|
19
|
+
tokenize → gather rows → mean-pool → L2-normalize → int8-quantize
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Every step is pure JS and chosen to be byte-identical everywhere:
|
|
23
|
+
|
|
24
|
+
| step | how determinism is guaranteed |
|
|
25
|
+
|---|---|
|
|
26
|
+
| tokenize | `foldText` (NFKD + strip combining marks) → camelCase/ACRONYM split → lowercase → split on non-alphanumeric |
|
|
27
|
+
| wordpiece | greedy longest-match against the vocab, BERT-style `##` continuations; an unsplittable word → the `[UNK]` row |
|
|
28
|
+
| gather | model rows widen float→double exactly |
|
|
29
|
+
| mean-pool | IEEE-754 double add/div in a fixed token order |
|
|
30
|
+
| L2-norm | division by `sqrt(Σv²)` — `sqrt` is correctly-rounded per IEEE-754 |
|
|
31
|
+
| quantize | `unit × 127`, **round-half-to-even** (not `Math.round`), clamp to `[-127,127]` → int8 |
|
|
32
|
+
|
|
33
|
+
Ranking is a **pure integer dot product** of int8 vectors. Because the fixed
|
|
34
|
+
`1/127` scale is shared by every vector, for a given query the integer dot
|
|
35
|
+
`Σ q·c` is monotonic with the cosine estimate — no per-vector float scale is
|
|
36
|
+
stored or multiplied, so nothing about ranking can drift between platforms.
|
|
37
|
+
|
|
38
|
+
**Key insight:** the corpus *and* the query are encoded by the *same* JS encoder,
|
|
39
|
+
so self-consistency is all that is required — fidelity to the original HF
|
|
40
|
+
tokenizer is not. The JS tokenizer deliberately differs from the Python
|
|
41
|
+
reference; that gap is assumed and harmless.
|
|
42
|
+
|
|
43
|
+
## Activation (opt-in by asset)
|
|
44
|
+
|
|
45
|
+
Resolution order (`resolveEmbedModelDir`):
|
|
46
|
+
|
|
47
|
+
1. `CODEINDEX_EMBED_DIR` (explicit override — wins outright)
|
|
48
|
+
2. `<repo>/.codeindex/models/`
|
|
49
|
+
3. `<cwd>/.codeindex/models/`
|
|
50
|
+
|
|
51
|
+
A directory "has a model" when it contains `model.json`. Nothing found →
|
|
52
|
+
`undefined` → silent lexical degradation. **Models are never in the npm
|
|
53
|
+
tarball** (`package.json` `files` is unchanged; the pack-smoke test asserts no
|
|
54
|
+
`model.json` / `*.safetensors` / `embeddings.bin` ships).
|
|
55
|
+
|
|
56
|
+
### `model.json` shape
|
|
57
|
+
|
|
58
|
+
```json
|
|
59
|
+
{
|
|
60
|
+
"modelId": "…",
|
|
61
|
+
"dim": 256,
|
|
62
|
+
"unk": "[UNK]",
|
|
63
|
+
"vocab": ["[UNK]", "auth", "##token", "..."],
|
|
64
|
+
"weights": [[0.0, ...], [1.0, ...], ...]
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
`vocab[i]` is the token whose row is `weights[i]` (length `dim`). Human-authorable
|
|
69
|
+
JSON so a tiny fixture is committable; a real `pull`ed model uses the same shape
|
|
70
|
+
at scale.
|
|
71
|
+
|
|
72
|
+
### The official asset
|
|
73
|
+
|
|
74
|
+
`codeindex embed pull` (no env) fetches the published **`embed-model-v1`** GitHub
|
|
75
|
+
release: a conversion of [`minishlab/potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M)
|
|
76
|
+
(revision `bf8b0566…`, **MIT**) — 256-dim, a 27 559-token WordPiece vocab, int8
|
|
77
|
+
rows at one global scale. The download is **sha256-verified** against
|
|
78
|
+
`EMBED_ASSET_SHA256` (`src/embed/model.ts`) before it is written; a mismatch
|
|
79
|
+
fails the pull and writes nothing. The asset is **never** committed to git or
|
|
80
|
+
shipped in the npm tarball — it lives only in the release. Reproduce it
|
|
81
|
+
byte-for-byte with the pinned toolchain in [`scripts/embed-asset/`](https://github.com/maxgfr/codeindex/tree/main/scripts/embed-asset).
|
|
82
|
+
|
|
83
|
+
## CLI
|
|
84
|
+
|
|
85
|
+
```sh
|
|
86
|
+
# Effective mode (none/static/endpoint), model, EMBED_VERSION, endpoint + its
|
|
87
|
+
# reachability (JSON). Precedence: endpoint > static model.
|
|
88
|
+
codeindex embed status --repo <dir>
|
|
89
|
+
|
|
90
|
+
# Fetch the official static-embedding asset (published as the `embed-model-v1`
|
|
91
|
+
# GitHub release) into CODEINDEX_EMBED_DIR (or <repo>/.codeindex/models/). The
|
|
92
|
+
# built-in default download is sha256-verified before it is written; set
|
|
93
|
+
# CODEINDEX_EMBED_URL to fetch a custom/mirrored model.json instead (un-verified).
|
|
94
|
+
codeindex embed pull --repo <dir>
|
|
95
|
+
|
|
96
|
+
# Build embeddings.bin from the repo into --out <dir> (static tier only).
|
|
97
|
+
codeindex embed build --repo <dir> --out <dir>
|
|
98
|
+
|
|
99
|
+
# Print (or --run) the docker command that starts the rich-tier embedding server.
|
|
100
|
+
codeindex embed serve # prints the one-liner
|
|
101
|
+
codeindex embed serve --run # runs it (needs docker)
|
|
102
|
+
|
|
103
|
+
# Search: RRF-fuse an embedding tier + lexical. Degrades to lexical (exit 0)
|
|
104
|
+
# when no tier is available/reachable.
|
|
105
|
+
codeindex search "http client retry" --repo <dir> --semantic
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`codeindex index --out <dir>` additionally writes `embeddings.bin` next to
|
|
109
|
+
`graph.json` **iff** a model is present.
|
|
110
|
+
|
|
111
|
+
## Artifact: `embeddings.bin`
|
|
112
|
+
|
|
113
|
+
### Layout (the `deserializeEmbeddings` contract)
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
offset 0 "CIE1" 4-byte ASCII magic
|
|
117
|
+
offset 4 uint32 LE header length (headerLen)
|
|
118
|
+
offset 8 UTF-8 JSON header { embedVersion, modelId, dim, count, records:[{file,symbol,line}] }
|
|
119
|
+
offset 8+headerLen int8 body count × dim signed bytes (row-major)
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
This is exactly what `deserializeEmbeddings` (`src/embed/index.ts`) reads back:
|
|
123
|
+
it throws on a bad magic (a corrupt or foreign file fails loudly instead of
|
|
124
|
+
being misread), parses the JSON header, and slices the packed body at
|
|
125
|
+
`8 + headerLen`.
|
|
126
|
+
|
|
127
|
+
The header carries **no absolute path and no timestamp**; records follow scan
|
|
128
|
+
order (files sorted by `rel`, symbols in declaration order). Two builds of an
|
|
129
|
+
unchanged repo produce byte-identical `embeddings.bin`. A dedicated
|
|
130
|
+
`EMBED_VERSION` (independent of `SCHEMA_VERSION`) plus `modelId` + `dim` in the
|
|
131
|
+
header invalidate a stale or foreign artifact.
|
|
132
|
+
|
|
133
|
+
Granularity is **per-symbol** (name + signature + file summary + path segments),
|
|
134
|
+
with a **per-file** fallback record for symbol-less files (docs, config) so
|
|
135
|
+
every file with content is represented.
|
|
136
|
+
|
|
137
|
+
## Fusion
|
|
138
|
+
|
|
139
|
+
Semantic and lexical rankings live on incomparable scales (BM25 score vs integer
|
|
140
|
+
dot product), so `searchSemantic` fuses them by **rank** via the shared `rrf`
|
|
141
|
+
helper (k=60) — never a linear score blend. A `SemanticSearchResult` extends the
|
|
142
|
+
lexical `SearchResult` additively with `semanticSymbol` (the corpus symbol whose
|
|
143
|
+
embedding was closest for that file).
|
|
144
|
+
|
|
145
|
+
## Degradation matrix
|
|
146
|
+
|
|
147
|
+
| present | behavior |
|
|
148
|
+
|---|---|
|
|
149
|
+
| nothing | BM25 lexical |
|
|
150
|
+
| + fuzzy | BM25 + trigram fallback for `df==0` terms (v2.9.0) |
|
|
151
|
+
| + model asset | RRF-fused deterministic static semantic search |
|
|
152
|
+
| + HTTP endpoint (`CODEINDEX_EMBED_ENDPOINT`) | rich tier — **wins over a static model** (v2.11.0) |
|
|
153
|
+
| `--semantic`, nothing available | lexical + stderr note, **exit 0** |
|
|
154
|
+
| endpoint set but unreachable/timeout | lexical + stderr note, **exit 0** (never falls back to the static model) |
|
|
155
|
+
|
|
156
|
+
## Library
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
import {
|
|
160
|
+
resolveEmbedModelDir, loadEmbedModel,
|
|
161
|
+
buildEmbeddingIndex, serializeEmbeddings, deserializeEmbeddings,
|
|
162
|
+
searchSemantic, encode, EMBED_VERSION,
|
|
163
|
+
} from "@maxgfr/codeindex";
|
|
164
|
+
|
|
165
|
+
const model = loadEmbedModel(resolveEmbedModelDir("/repo"));
|
|
166
|
+
if (model) {
|
|
167
|
+
const scan = scanRepo("/repo");
|
|
168
|
+
const index = buildEmbeddingIndex(scan, model); // deterministic
|
|
169
|
+
const results = searchSemantic(scan, "http retry", index, { model });
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
`searchSemantic(scan, query, index, opts)` with no `opts.model` (or no index)
|
|
174
|
+
returns the pure lexical ranking — the same silent degradation the CLI uses.
|
|
175
|
+
|
|
176
|
+
## The rich tier: containerized HTTP embedding endpoint (v2.11.0)
|
|
177
|
+
|
|
178
|
+
The **rich tier** lets the engine consume a real neural embedding model (e.g.
|
|
179
|
+
all-MiniLM-L6-v2) served by a local container, instead of the static lookup
|
|
180
|
+
table. It is opt-in via one env var:
|
|
181
|
+
|
|
182
|
+
```sh
|
|
183
|
+
CODEINDEX_EMBED_ENDPOINT=http://localhost:8756 \
|
|
184
|
+
codeindex search "auth token" --repo . --semantic
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
### Precedence
|
|
188
|
+
|
|
189
|
+
`endpoint > static model > none`. Setting `CODEINDEX_EMBED_ENDPOINT` is an
|
|
190
|
+
**explicit** user intent, so it wins over any local `model.json`. `embed status`
|
|
191
|
+
reports the effective `mode`. If the endpoint is defined but
|
|
192
|
+
unreachable/times-out/malformed, the search degrades straight to **lexical**
|
|
193
|
+
(stderr note, exit 0) — it does **not** silently fall back to the static model.
|
|
194
|
+
|
|
195
|
+
### How the float vectors join the deterministic pipeline
|
|
196
|
+
|
|
197
|
+
The engine POSTs corpus and query texts to the endpoint, receives **float**
|
|
198
|
+
vectors, then runs them through the **exact same** tail as the static tier —
|
|
199
|
+
`quantize()` in `encode.ts`: L2-normalize → int8 at the fixed `1/127` scale
|
|
200
|
+
(round-half-to-even, clamp). Ranking stays a pure integer dot product. So the
|
|
201
|
+
only tier-specific step is the encoder; fusion and ranking are shared code.
|
|
202
|
+
|
|
203
|
+
Corpus embeddings for the endpoint tier are built **at search time**
|
|
204
|
+
(`buildEndpointIndex`) and are **never serialized** to `embeddings.bin`: unlike
|
|
205
|
+
the static tier, endpoint vectors are float and provider-dependent, so this tier
|
|
206
|
+
is **deterministic per image digest**, not byte-golden. Pin the digest for
|
|
207
|
+
reproducibility:
|
|
208
|
+
|
|
209
|
+
```sh
|
|
210
|
+
docker run -d -p 8756:8756 ghcr.io/maxgfr/codeindex-embed@sha256:<digest>
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
### HTTP protocol (implement your own server)
|
|
214
|
+
|
|
215
|
+
`CODEINDEX_EMBED_ENDPOINT` is the server **base URL**. The client derives:
|
|
216
|
+
|
|
217
|
+
| method + path | request | response |
|
|
218
|
+
|---|---|---|
|
|
219
|
+
| `POST {base}/embed` | `{ "texts": ["…", …] }` | `{ "vectors": [[…float…], …] }` (same order, one row per text) |
|
|
220
|
+
| `GET {base}/healthz` | — | `200` (any body) when ready |
|
|
221
|
+
|
|
222
|
+
Notes: any embedding dimension is accepted (the engine quantizes whatever it
|
|
223
|
+
receives); vectors need not be pre-normalized (the engine L2-normalizes anyway);
|
|
224
|
+
the request times out after `CODEINDEX_EMBED_TIMEOUT_MS` (default 30 000).
|
|
225
|
+
`codeindex embed serve` prints/`--run`s the docker command for the official
|
|
226
|
+
image; the **library never orchestrates docker** (that is CLI-only).
|
|
227
|
+
|
|
228
|
+
The reference server is `docker/embed/` (transformers.js + all-MiniLM-L6-v2,
|
|
229
|
+
model baked in at build, offline at run, non-root, `:8756`), published to
|
|
230
|
+
`ghcr.io/maxgfr/codeindex-embed` by `.github/workflows/embed-image.yml`.
|
|
231
|
+
|
|
232
|
+
### Library
|
|
233
|
+
|
|
234
|
+
```ts
|
|
235
|
+
import {
|
|
236
|
+
buildEndpointIndex, encodeQueryViaEndpoint, probeEndpoint,
|
|
237
|
+
searchSemantic, quantize,
|
|
238
|
+
} from "@maxgfr/codeindex";
|
|
239
|
+
|
|
240
|
+
if (await probeEndpoint("http://localhost:8756")) {
|
|
241
|
+
const scan = scanRepo("/repo");
|
|
242
|
+
const index = await buildEndpointIndex(scan); // float → int8 corpus
|
|
243
|
+
const queryVec = await encodeQueryViaEndpoint("auth token");
|
|
244
|
+
const results = searchSemantic(scan, "auth token", index, { queryVec });
|
|
245
|
+
}
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
`embedViaEndpoint(texts, opts)` is the low-level client (`{ texts }` →
|
|
249
|
+
`{ vectors }`).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@maxgfr/codeindex",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.13.0",
|
|
4
4
|
"description": "Self-contained, deterministic repo-indexing engine: walk + language detection + symbol/import extraction (tree-sitter AST with regex fallback) + import resolution + typed cross-file link-graph + analytics. Ships as a single zero-dependency engine.mjs that consumer tools vendor.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"packageManager": "pnpm@10.33.0",
|
|
@@ -23,7 +23,8 @@
|
|
|
23
23
|
"scripts/cli.mjs",
|
|
24
24
|
"scripts/grammars",
|
|
25
25
|
"src",
|
|
26
|
-
"docs/MIGRATION.md"
|
|
26
|
+
"docs/MIGRATION.md",
|
|
27
|
+
"docs/SEMANTIC.md"
|
|
27
28
|
],
|
|
28
29
|
"engines": {
|
|
29
30
|
"node": ">=18"
|
package/scripts/engine.d.mts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
declare const ENGINE_VERSION = "2.
|
|
1
|
+
declare const ENGINE_VERSION = "2.13.0";
|
|
2
2
|
declare const SCHEMA_VERSION = 4;
|
|
3
|
-
declare const EXTRACTOR_VERSION =
|
|
3
|
+
declare const EXTRACTOR_VERSION = 10;
|
|
4
4
|
type FileKind = "code" | "doc" | "config" | "asset" | "other";
|
|
5
5
|
type EdgeKind = "contains" | "doc-link" | "import" | "call" | "use" | "mention";
|
|
6
6
|
type Tier = 0 | 1 | 2;
|
|
@@ -121,6 +121,7 @@ interface WalkOptions {
|
|
|
121
121
|
maxFileBytes?: number;
|
|
122
122
|
maxFiles?: number;
|
|
123
123
|
gitignore?: boolean;
|
|
124
|
+
ignoreDirs?: string[];
|
|
124
125
|
}
|
|
125
126
|
interface WalkedFile {
|
|
126
127
|
rel: string;
|
|
@@ -153,8 +154,10 @@ interface ScanOptions {
|
|
|
153
154
|
exclude?: string[];
|
|
154
155
|
scope?: string;
|
|
155
156
|
gitignore?: boolean;
|
|
157
|
+
ignoreDirs?: string[];
|
|
156
158
|
maxBytes?: number;
|
|
157
159
|
maxFiles?: number;
|
|
160
|
+
maxCallsPerFile?: number;
|
|
158
161
|
out?: string;
|
|
159
162
|
cache?: Map<string, {
|
|
160
163
|
hash: string;
|
|
@@ -202,7 +205,9 @@ interface CodeInfo {
|
|
|
202
205
|
}[];
|
|
203
206
|
importedNames?: string[];
|
|
204
207
|
}
|
|
205
|
-
declare function extractCode(rel: string, ext: string, content: string
|
|
208
|
+
declare function extractCode(rel: string, ext: string, content: string, opts?: {
|
|
209
|
+
maxCallsPerFile?: number;
|
|
210
|
+
}): CodeInfo;
|
|
206
211
|
|
|
207
212
|
interface MarkdownInfo {
|
|
208
213
|
title?: string;
|
|
@@ -229,7 +234,9 @@ interface AstResult {
|
|
|
229
234
|
}[];
|
|
230
235
|
importedNames: string[];
|
|
231
236
|
}
|
|
232
|
-
declare function extractAst(rel: string, ext: string, content: string
|
|
237
|
+
declare function extractAst(rel: string, ext: string, content: string, opts?: {
|
|
238
|
+
maxCalls?: number;
|
|
239
|
+
}): AstResult | undefined;
|
|
233
240
|
|
|
234
241
|
type Resolution = {
|
|
235
242
|
kind: "resolved";
|
|
@@ -337,6 +344,14 @@ type CallerIndex = Map<string, CallerEntry>;
|
|
|
337
344
|
declare function computeImportPairs(scan: RepoScan): Set<string>;
|
|
338
345
|
declare function buildCallerIndex(scan: RepoScan, importPairs?: Set<string>, opts?: CallerIndexOptions): CallerIndex;
|
|
339
346
|
declare function enclosingSymbol(scan: RepoScan, file: string, line: number): CodeSymbol | undefined;
|
|
347
|
+
interface RawCallerSite {
|
|
348
|
+
file: string;
|
|
349
|
+
line: number;
|
|
350
|
+
receiver?: string;
|
|
351
|
+
enclosingSymbol?: CodeSymbol;
|
|
352
|
+
}
|
|
353
|
+
type RawCallerIndex = Map<string, RawCallerSite[]>;
|
|
354
|
+
declare function buildRawCallerIndex(scan: RepoScan): RawCallerIndex;
|
|
340
355
|
|
|
341
356
|
declare function symbolsOverview(scan: RepoScan, rel: string): CodeSymbol[];
|
|
342
357
|
interface SymbolMatch extends CodeSymbol {
|
|
@@ -512,7 +527,11 @@ interface StaticEmbedModel {
|
|
|
512
527
|
declare function resolveEmbedModelDir(repo?: string): string | undefined;
|
|
513
528
|
declare function hasEmbedModel(repo?: string): boolean;
|
|
514
529
|
declare function loadEmbedModel(dir?: string): StaticEmbedModel | undefined;
|
|
515
|
-
|
|
530
|
+
interface EmbedPullTarget {
|
|
531
|
+
url: string;
|
|
532
|
+
sha256?: string;
|
|
533
|
+
}
|
|
534
|
+
declare function resolveEmbedPullUrl(): EmbedPullTarget;
|
|
516
535
|
|
|
517
536
|
declare function basicTokenize(text: string): string[];
|
|
518
537
|
declare function wordpiece(word: string, model: StaticEmbedModel): number[];
|
|
@@ -693,4 +712,4 @@ declare function rrf<T>(lists: T[][], keyOf: (item: T) => string, k?: number): M
|
|
|
693
712
|
|
|
694
713
|
declare function runCli(argv: string[]): Promise<void>;
|
|
695
714
|
|
|
696
|
-
export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
|
|
715
|
+
export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbedPullTarget, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawCallerIndex, type RawCallerSite, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildRawCallerIndex, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
|