@maxgfr/codeindex 2.12.0 → 2.14.0

package/README.md

@@ -8,9 +8,8 @@ import resolution, a typed cross-file link-graph, and graph analytics — shippe
 as a single zero-dependency `engine.mjs` that consumer tools **vendor** (copy
 into their repo) instead of installing.
 
-Extracted from [ultraindex](https://github.com/maxgfr/ultraindex) 5.1.0's core,
-and consumed by the ultra* skill family (ultraindex, ultradoc, ultrasec,
-ultraeval, reconstruct, construct, ultra11y).
+Designed for downstream tools — agent skills, CLIs, CI gates — that vendor the
+engine as a single file instead of taking an npm dependency.
 
 ## What it does
 
@@ -44,7 +43,28 @@ const { scan, graph, symbols } = buildIndexArtifacts("/path/to/repo");
 
 The AST tier is optional: without a `grammars/` directory next to the bundle
 the engine silently uses its regex tier. Only tools that want AST precision
-(e.g. ultraindex) also vendor `scripts/grammars/` (~17 MiB of wasm).
+also vendor `scripts/grammars/` (~17 MiB of wasm).
+
+### Slim grammars (pull instead of vendor)
+
+Consumers that want AST precision but not the ~17 MiB of vendored wasm can
+`codeindex grammars pull` the grammars once into a shared, per-machine cache
+(`<XDG_CACHE_HOME|~/.cache>/codeindex/grammars/<ENGINE_VERSION>`) instead:
+
+```sh
+codeindex grammars status   # active tier (adjacent/env/cache/none) + whether a pull is needed
+codeindex grammars pull     # fetch the per-release grammars asset, sha256-verified, into the cache
+```
+
+Resolution is **adjacent > env > cache > regex**: a bundle-adjacent `grammars/`
+still wins if present (offline setups are untouched), then
+`CODEINDEX_GRAMMARS_DIR`, then the pulled cache. `pull` fetches the official
+`grammars-<version>.tar.gz` release asset (its `.sha256` sidecar is verified
+before anything is written) and extracts it atomically; the same wasm bytes
+produce **byte-identical** AST extraction from the cache as from a vendored dir.
+It is fully **offline-safe**: with no grammars resolvable anywhere — and after a
+failed or absent pull — the engine silently falls back to the regex tier exactly
+as it does today; a pull never throws into indexing.
 
 ## Use from npm
 
@@ -62,8 +82,8 @@ const scan = scanRepo("/path/to/repo");
 ```
 
 The CLI ships in the same package — see **Use as a CLI** below for the global
-install command. Skills should still prefer vendoring: it keeps their own
-bundle single-file and pinned to an exact commit without an npm dependency.
+install command. Consumer tools should still prefer vendoring: it keeps their
+own bundle single-file and pinned to an exact commit without an npm dependency.
 
 ## Use as a CLI
 
@@ -129,9 +149,13 @@ and the `embeddings.bin` artifact are byte-identical across builds and platforms
 It is **opt-in by asset**: with no model on disk the engine silently stays
 lexical, and `--semantic` without a model returns lexical results on **exit 0**
 (a stderr note only). Models are **never** shipped in the package; a model is
-resolved from `CODEINDEX_EMBED_DIR` or `<repo>/.codeindex/models/`.
+resolved from `CODEINDEX_EMBED_DIR` or `<repo>/.codeindex/models/`. Getting one
+is zero-config: `codeindex embed pull` fetches the official `embed-model-v1`
+release asset, sha256-verified before anything is written.
 
 ```sh
+codeindex embed pull   --repo .              # fetch the official model asset into
+                                             # CODEINDEX_EMBED_DIR (or <repo>/.codeindex/models/); sha256-verified
 codeindex embed status --repo .              # effective mode + reachability (JSON)
 codeindex embed build  --repo . --out .codeindex   # write embeddings.bin
 codeindex search "http client retry" --repo . --semantic
@@ -182,8 +206,8 @@ into their own CLIs); `cli.mjs` is the thin standalone CLI/MCP wrapper.
 ## Versioning
 
 - `ENGINE_VERSION` — the release tag, embedded greppably in the bundle.
-- `SCHEMA_VERSION` — the `graph.json`/`symbols.json` shape (continues
-  ultraindex's lineage; currently 4). Consumers reject mismatched artifacts.
+- `SCHEMA_VERSION` — the `graph.json`/`symbols.json` shape (currently 4).
+  Consumers reject mismatched artifacts.
 - `EXTRACTOR_VERSION` — the extraction output shape; incremental caches keyed
   on it are discarded wholesale when it bumps.
 
@@ -214,8 +238,8 @@ pnpm check:build   # proves the committed bundle is byte-reproducible
 pnpm test:e2e      # opt-in: pinned real-repo builds with ratchets
 ```
 
-The compat suite pins the exact bytes ultraindex 5.1.0 produced for the
-`mini-repo` fixture — the proof that extraction was lossless.
+The compat suite pins golden bytes for the `mini-repo` fixture — the proof
+that extraction stays lossless across releases.
 
 ## License
 

package/docs/MIGRATION.md

@@ -1,4 +1,4 @@
-# Migrating a skill onto the codeindex engine
+# Migrating a consumer tool onto the codeindex engine
 
 The engine ships as two files, released together at every tag:
 
@@ -6,12 +6,12 @@ The engine ships as two files, released together at every tag:
 - `scripts/engine.d.mts` — TypeScript declarations for the bundle
 
 Consumers **vendor** them (commit a copy) — never an npm dependency — so every
-skill stays standalone-installable.
+consumer stays standalone-installable.
 
 ## Vendoring steps
 
-1. Copy `scripts/sync-engine.mjs` (reference implementation lives in the
-   ultraeval repo) into your repo and run:
+1. Add a small `scripts/sync-engine.mjs` to your repo (a ~50-line fetch script
+   with the behavior described below) and run:
 
    ```sh
    node scripts/sync-engine.mjs --ref v1.0.0
@@ -40,8 +40,7 @@ skill stays standalone-installable.
 
 The AST tier is optional: without a `grammars/` directory next to the vendored
 bundle the engine uses its regex tier (15 languages). Only vendor
-`scripts/grammars/` (~17 MiB wasm) if you need AST-exact symbols (ultraindex
-does; nobody else should).
+`scripts/grammars/` (~17 MiB wasm) if you need AST-exact symbols.
 
 ## Version constants
 
@@ -53,7 +52,7 @@ does; nobody else should).
 
 `buildGraph(...)` / `buildIndexArtifacts(...)` accept
 `meta: { version, schemaVersion }` so a consumer stamps its own identity into
-artifacts it persists (ultraindex does this to keep its graph.json lineage).
+artifacts it persists and keeps its own `graph.json` lineage.
 
 ## v2.9.0 — `search` trigram fuzzy fallback
 
@@ -93,39 +92,159 @@ embeddings live in a separate `embeddings.bin` sidecar keyed by a dedicated
 `--semantic` without a model degrades to lexical results on **exit 0** (a stderr
 note only) — so wiring it on is safe before an asset exists.
 
-## Per-skill mapping (what to replace with what)
+## v2.12.0 — two return-shape changes (check your call sites)
 
-| Skill | Replace | With (engine export) |
-|---|---|---|
-| **ultraeval** | `walkFiles`, `SKIP_DIRS` | `scanRepo` (gitignore on by default) |
-| | `importsOf` + `resolveImport` (JS/TS+py regex) | `FileRecord.refs` (kind `import`) + `buildResolveContext`/`resolveImport` |
-| | `gitChurn`, `changedFiles` | `gitChurn`, `changedSince` |
-| | keeps | cycle DFS, hotspot scoring, todos/maxIndent, ANALYSIS.md rendering |
-| **construct** | walker clone + language histogram + test count | `scanRepo` (`.languages`) + `isTestPath` |
-| **ultraindex** | the whole core (walk/scan/extract/resolve/modules/graph/calls/analytics/renderers) | vendored engine + shim `src/engine.ts` re-exporting it; pass `meta: { version: VERSION, schemaVersion: SCHEMA_VERSION }` to keep graph.json byte-lineage |
-| | keeps | store/manifest, encyclopedia, merge/entries, find/ask, check/verify, embeddings, orchestrate |
-| **ultradoc** | walker, `EXT_LANG` map, symbol `RULES` + `applyExportLists` | `scanRepo`, `extToLang`/`languageOf`, `extractSymbols`/`buildSymbolIndex` |
-| | `PKG_MANIFESTS` workspace probing | `detectWorkspaces` |
-| | `rgSearch` + JS fallback | `grepRepo` |
-| | keeps | issues/PRs/SO/web retrieval, citations, DOC.md |
-| **ultra11y** | walker + glob scoping | `walk`/`scanRepo` with `include`/`exclude` |
-| | tsconfig path-alias resolution | `buildResolveContext` + `resolveImport` |
-| | keeps | its JSX component graph and every WCAG check |
-| **reconstruct** | walker + gitignore parser + `categorize` | `scanRepo`, `categorize` |
-| | `EXT_LANGUAGE` | `extToLang` |
-| | workspace detection + dep graph + cycle + topo | `detectWorkspaces` (packages/dependsOn/cycle/topoOrder) |
-| | `resolveModule` | `resolveImport` |
-| | keeps | framework adapters (routes), data-model inference, PRD rendering |
-| **ultrasec** | walker + gitignore + scope + symlink guard | `scanRepo` (`scope`, `gitignore` — its own semantics, ported here) |
-| | per-language defs/imports/calls extraction | `extractCode` (`symbols`/`refs`/`calls`) |
-| | `resolveImport` | `resolveImport` |
-| | `buildGraph` (import+call edges, symbolDefs) | `buildGraph` + `resolveCallEdges` |
-| | `callersBySymbol`, `enclosingSymbol` (raw-recall taint-BFS input) | `buildRawCallerIndex` (issue #8) — every name-matched call site keyed by the raw callee name, no def resolution or gating, `enclosingSymbol` computed per site. `buildCallerIndex` is **NOT** a substitute here: it is def-resolved and gated (language-family filter, JS/TS import gate, same-file self-declaration skip) and will silently drop sites a recall consumer needs. Both are bounded by `FileRecord.calls`'s per-file 512-call cap (dedup by name+line) — a file with more raw call sites than that loses sites upstream of either function. |
-| | keeps | taint source→sink enumeration, external scanners, EPSS/KEV/CVSS, SARIF |
+Not additive: two public return shapes changed in this release without a
+compat flag, so a consumer re-pinning across v2.12.0 must check both call
+sites. Artifact schemas are untouched (`SCHEMA_VERSION` / `EMBED_VERSION`
+unchanged) — this is API shape only.
+
+- `resolveEmbedPullUrl()` now returns an `EmbedPullTarget`
+  (`{ url: string; sha256?: string }`) — it previously returned
+  `string | undefined`. It always resolves: `CODEINDEX_EMBED_URL` wins
+  outright and carries **no** `sha256` (a custom mirror keeps the
+  un-verified behavior); with no env it falls back to the built-in official
+  asset **with** its pinned `sha256` so `embed pull` verifies the default
+  download. Replace `const url = resolveEmbedPullUrl()` with
+  `const { url, sha256 } = resolveEmbedPullUrl()`. The `EmbedPullTarget`
+  type is exported from the barrel.
+- MCP `search` with `semantic: true` now returns
+  `{ results, tier, degradedReason? }` — it previously returned the bare
+  ranked array. `tier` is `"endpoint" | "static" | "lexical"` and
+  `degradedReason` is present only when the semantic tier degraded to
+  lexical, so a caller can tell "fusion happened" apart from "degraded".
+  Plain lexical `search` (no `semantic: true`) still returns the bare
+  array, byte-compatible with existing consumers.
+
+## v2.13.0 — `.codeindex` excluded from the walk
+
+`.codeindex/` — the engine's own output directory (index artifacts, pulled
+models, MCP memories) — joined `IGNORE_DIRS`, so `walk`/`scanRepo` no longer
+descend into it and memories stop entering BM25/embedding results (previously
+every `write_memory` also churned the scan fingerprint, busting the MCP
+server's memoized embedding index). Access memories through the MCP memory
+tools (`read_memory` / `list_memories` / `delete_memory`), never `search`.
+This is a file-set change only — extraction shape and `SCHEMA_VERSION` are
+untouched, so **no re-pin is required**; a consumer that deliberately wants
+`.codeindex` walked can pass `ignoreDirs` (replace semantics, also new in
+this release) with its own set.
+
+## v2.14.0 — incremental fastpaths (all additive, no re-pin required)
+
+Pure fastpaths: every entry point still produces byte-identical artifacts for
+unchanged inputs, so **no re-pin is required** and no consumer needs to act.
+`SCHEMA_VERSION` / `EMBED_VERSION` / `EXTRACTOR_VERSION` are untouched; the new
+surface is additive (semver-minor).
+
+- **`RepoScan.contentUnchanged` / `RepoScan.cacheDirty` + `ScanOptions.precomputedWalk`.**
+  Two derived read-only flags: `contentUnchanged` is true when a `cache` was
+  supplied and every kept file reused its cached record (stat fastpath or exact
+  content-hash) with an unchanged file set; `cacheDirty` is true when persisting
+  the cache would change any byte (a hash/size/mtime drift, a file-set
+  difference, or no cache at all). `precomputedWalk` lets a caller that already
+  walked hand its `WalkResult` to `scanRepo` instead of re-walking — it must
+  come from `walk(root, <the same options>)`.
+- **`buildArtifactsFromScan(scan, opts)` export.** The downstream half of
+  `buildIndexArtifacts` (resolve → graph → communities → centrality → symbol
+  index) split out as its own export; `buildIndexArtifacts` is now `scanRepo` +
+  this call. A consumer already holding a `RepoScan` builds artifacts without
+  re-walking; the extracted body is verbatim, so output is byte-identical.
+- **`cache.json` additive meta keys.** Writes gain a fixed-order `meta` block —
+  `engineVersion`, `commit`, `graphSha1`, `symbolsSha1`, and `embed`
+  (`{ embedVersion, modelId, sha1 }`) only when `embeddings.bin` was written.
+  Old engines ignore these keys (they only check schema/extractor); an old cache
+  lacking them never fastpaths but still reuses records. `cache.json` embeds
+  mtimes so it never was cross-machine byte-reproducible — no determinism
+  surface changes.
+- **CLI `index` fastpath.** `index` skips `buildArtifactsFromScan`, both renders
+  and every artifact write when a guard proves the run would reproduce the
+  on-disk bytes: `scan.contentUnchanged`, `meta.engineVersion` matches,
+  `meta.commit` matches the scan's commit (graph.json embeds the commit — an
+  identical tree under a new HEAD rebuilds), the sha1 of the on-disk
+  graph/symbols equals the recorded shas, and the embed leg holds (no model, or
+  the model's embedVersion+modelId and `embeddings.bin` sha match — a model swap
+  rebuilds the sidecar). Any failure — deleted, truncated or tampered artifacts
+  included — falls through to the full rebuild that rewrites everything, so the
+  fastpath self-heals on corruption; `cache.json` is rewritten on the fastpath
+  only when `scan.cacheDirty`.
+- **MCP session scan + artifacts cache.** The long-lived server memoizes a
+  single scan and its artifacts across tool calls: `getScan` re-runs `scanRepo`
+  with the prior scan re-expressed as its `cache`, so scan.ts's stat/hash oracle
+  decides freshness and an unchanged repo returns the SAME `RepoScan` object,
+  while `getArtifacts` lazily runs `buildArtifactsFromScan` memoized on scan
+  object identity (rendered strings are never cached). Any successful edit tool
+  (`replace_symbol_body`, `insert_after_/insert_before_symbol`) drops the entry
+  — a controlled write landing in the same mtime tick at the same byte count
+  would fool the (size, mtime) fastpath; `write_memory` needs no invalidation
+  (`.codeindex/` is off the walk since v2.13.0).
+- **`runMcpServer` serverInfo override.** `runMcpServer(opts?)` accepts
+  `{ serverInfo?: { name?, version? } }` so a consumer embedding the server
+  announces its own identity in the `initialize` response; omitted fields keep
+  the `{ name: "codeindex", version: ENGINE_VERSION }` defaults, the zero-arg
+  call is unchanged, and `McpServerOptions` is exported from the barrel.
+- **Lazy grammar warm — covering-set guarantee.** The CLI and MCP server warm
+  only the grammars for languages actually present (`grammarKeysForExts` over
+  the walked extensions) rather than every grammar at startup. The walk's
+  extension set is a superset of what `scanRepo` keeps (scope/include/exclude
+  only filter further), so every extracted file has its grammar loaded before
+  extraction and AST output stays byte-identical — including a language whose
+  first file appears mid-session (the MCP warm re-derives per call). The
+  **pre-existing cache-tier caveat is unchanged**: a record reused by hash may
+  have been extracted under a different grammar tier.
+- **Slim grammars-pull tier (`grammars pull` / `grammars status`).** The AST
+  wasm sidecar (`scripts/grammars/`, ~17 MiB) stays optional and opt-in by
+  presence, but a consumer that vendors only `engine.mjs` no longer has to
+  vendor the wasm to get AST-exact symbols. `resolveGrammarsTier` /
+  `resolveGrammarsDir` now resolve in order **adjacent > env > cache > regex**:
+  the bundle-adjacent `grammars/` dir wins if present (the offline, no-network
+  story is untouched), then `CODEINDEX_GRAMMARS_DIR`, then the shared
+  version-scoped cache `sharedGrammarsCacheDir()`
+  (`<XDG_CACHE_HOME|~/.cache>/codeindex/grammars/<ENGINE_VERSION>`), else nothing
+  resolvable → the regex tier exactly as today. `codeindex grammars pull`
+  fetches the per-release `grammars-<ENGINE_VERSION>.tar.gz` asset (built and
+  uploaded to the `v<ENGINE_VERSION>` tag by the release workflow) plus its
+  `.sha256` sidecar, verifies the digest, and extracts atomically into that
+  cache with a zero-dep inline ustar reader (path-traversal-guarded, no spawned
+  `tar`); it is idempotent (a matching marker skips the ~22 MB download) and
+  `CODEINDEX_GRAMMARS_URL` overrides the source (private mirror, unverified,
+  like the embed-pull precedent). `codeindex grammars status` reports the active
+  tier, resolved dir, pinned `ENGINE_VERSION`, and whether a pull is needed
+  (JSON). The **guarantee**: the same committed wasm bytes loaded from the cache
+  produce byte-identical AST extraction as from a bundle-adjacent dir (same wasm
+  → same AST → same symbols), so `SCHEMA_VERSION` / `EXTRACTOR_VERSION` are
+  untouched and **no re-pin is required**. **Offline-safe**: `grammars pull`
+  never runs during indexing, and a failed/absent pull only ever leaves the
+  cache empty — it never throws into the scan, which silently uses the regex
+  tier. New exports: `resolveGrammarsTier`, `resolveGrammarsDir`,
+  `sharedGrammarsCacheDir`, `GrammarsTier` / `GrammarsTierName`,
+  `resolveGrammarsPullTarget`, `fetchGrammarsTarball`, `fetchExpectedSha256`,
+  `extractGrammarsTarball`, `GrammarsPullTarget`.
+
+## Typical mapping (what to replace with what)
+
+What a consumer usually deletes from its own codebase, and the engine export
+that replaces it:
+
+| Hand-rolled piece | Engine replacement |
+|---|---|
+| file walker + skip lists + gitignore parser | `walk` / `scanRepo` (gitignore on by default; `include`/`exclude`/`scope`) |
+| extension→language map | `extToLang` / `languageOf` |
+| per-language symbol/import/call regexes | `extractCode` / `extractSymbols` (`symbols`/`refs`/`calls`), `buildSymbolIndex` |
+| import resolution (tsconfig paths, package `exports`, go.mod, Cargo…) | `buildResolveContext` + `resolveImport` |
+| workspace/monorepo probing | `detectWorkspaces` (packages/dependsOn/cycle/topoOrder) |
+| dependency/link graph construction | `buildGraph` + `resolveCallEdges` |
+| grep with ripgrep + JS fallback | `grepRepo` |
+| git churn / changed-files helpers | `gitChurn`, `changedSince` |
+| language histogram, test detection | `scanRepo` (`.languages`) + `isTestPath` |
+| caller lookup, precision-gated | `buildCallerIndex` (def-resolved and gated: language-family filter, JS/TS import gate, same-file self-declaration skip) |
+| caller lookup, raw recall (e.g. taint-BFS input) | `buildRawCallerIndex` (issue #8) — every name-matched call site keyed by the raw callee name, no def resolution or gating, `enclosingSymbol` computed per site. `buildCallerIndex` is **NOT** a substitute here: its gates silently drop sites a recall consumer needs. Both are bounded by `FileRecord.calls`'s per-file 512-call cap (dedup by name+line) — a file with more raw call sites than that loses sites upstream of either function. |
+
+What a consumer keeps is everything above the index: its own scoring,
+rendering, retrieval and domain logic.
 
 ## Golden-diff adjudication (every migration)
 
-Capture the skill's load-bearing artifact **before** touching code (a committed
+Capture the consumer's load-bearing artifact **before** touching code (a committed
 snapshot test), migrate, then adjudicate every diff:
 
 - **Accept + document**: file-set changes from better ignore rules (gitignore

package/docs/SEMANTIC.md

@@ -78,7 +78,7 @@ rows at one global scale. The download is **sha256-verified** against
 `EMBED_ASSET_SHA256` (`src/embed/model.ts`) before it is written; a mismatch
 fails the pull and writes nothing. The asset is **never** committed to git or
 shipped in the npm tarball — it lives only in the release. Reproduce it
-byte-for-byte with the pinned toolchain in [`scripts/embed-asset/`](../scripts/embed-asset/).
+byte-for-byte with the pinned toolchain in [`scripts/embed-asset/`](https://github.com/maxgfr/codeindex/tree/main/scripts/embed-asset).
 
 ## CLI
 
@@ -110,13 +110,20 @@ codeindex search "http client retry" --repo <dir> --semantic
 
 ## Artifact: `embeddings.bin`
 
+### Layout (the `deserializeEmbeddings` contract)
+
 ```
-"CIE1"                     4-byte ASCII magic
-uint32 LE header length
-UTF-8 JSON header          { embedVersion, modelId, dim, count, records:[{file,symbol,line}] }
-int8 body                  count × dim signed bytes (row-major)
+offset 0            "CIE1"            4-byte ASCII magic
+offset 4            uint32 LE         header length (headerLen)
+offset 8            UTF-8 JSON header { embedVersion, modelId, dim, count, records:[{file,symbol,line}] }
+offset 8+headerLen  int8 body         count × dim signed bytes (row-major)
 ```
 
+This is exactly what `deserializeEmbeddings` (`src/embed/index.ts`) reads back:
+it throws on a bad magic (a corrupt or foreign file fails loudly instead of
+being misread), parses the JSON header, and slices the packed body at
+`8 + headerLen`.
+
 The header carries **no absolute path and no timestamp**; records follow scan
 order (files sorted by `rel`, symbols in declaration order). Two builds of an
 unchanged repo produce byte-identical `embeddings.bin`. A dedicated

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@maxgfr/codeindex",
-  "version": "2.12.0",
+  "version": "2.14.0",
   "description": "Self-contained, deterministic repo-indexing engine: walk + language detection + symbol/import extraction (tree-sitter AST with regex fallback) + import resolution + typed cross-file link-graph + analytics. Ships as a single zero-dependency engine.mjs that consumer tools vendor.",
   "type": "module",
   "packageManager": "pnpm@10.33.0",

package/scripts/engine.d.mts

@@ -1,6 +1,6 @@
-declare const ENGINE_VERSION = "2.12.0";
+declare const ENGINE_VERSION = "2.14.0";
 declare const SCHEMA_VERSION = 4;
-declare const EXTRACTOR_VERSION = 9;
+declare const EXTRACTOR_VERSION = 10;
 type FileKind = "code" | "doc" | "config" | "asset" | "other";
 type EdgeKind = "contains" | "doc-link" | "import" | "call" | "use" | "mention";
 type Tier = 0 | 1 | 2;
@@ -121,6 +121,7 @@ interface WalkOptions {
     maxFileBytes?: number;
     maxFiles?: number;
     gitignore?: boolean;
+    ignoreDirs?: string[];
 }
 interface WalkedFile {
     rel: string;
@@ -147,14 +148,18 @@ interface RepoScan {
     mtimes: Map<string, number>;
     capped: boolean;
     excluded: number;
+    contentUnchanged: boolean;
+    cacheDirty: boolean;
 }
 interface ScanOptions {
     include?: string[];
     exclude?: string[];
     scope?: string;
     gitignore?: boolean;
+    ignoreDirs?: string[];
     maxBytes?: number;
     maxFiles?: number;
+    maxCallsPerFile?: number;
     out?: string;
     cache?: Map<string, {
         hash: string;
@@ -163,6 +168,7 @@ interface ScanOptions {
         mtimeMs?: number;
     }>;
     fullHash?: boolean;
+    precomputedWalk?: WalkResult;
 }
 declare function scanRepo(root: string, opts?: ScanOptions): RepoScan;
 
@@ -202,7 +208,9 @@ interface CodeInfo {
     }[];
     importedNames?: string[];
 }
-declare function extractCode(rel: string, ext: string, content: string): CodeInfo;
+declare function extractCode(rel: string, ext: string, content: string, opts?: {
+    maxCallsPerFile?: number;
+}): CodeInfo;
 
 interface MarkdownInfo {
     title?: string;
@@ -213,8 +221,22 @@ interface MarkdownInfo {
 declare function extractMarkdown(content: string): MarkdownInfo;
 
 declare function grammarKeyForExt(ext: string): string | undefined;
+type GrammarsTierName = "adjacent" | "env" | "cache" | "none";
+interface GrammarsTier {
+    tier: GrammarsTierName;
+    dir?: string;
+    cacheDir: string;
+}
+declare function sharedGrammarsCacheDir(): string;
+declare function resolveGrammarsTier(opts?: {
+    moduleDir?: string;
+}): GrammarsTier;
+declare function resolveGrammarsDir(opts?: {
+    moduleDir?: string;
+}): string | undefined;
 declare function ensureGrammars(keys: Iterable<string>): Promise<void>;
 declare function allGrammarKeys(): string[];
+declare function grammarKeysForExts(exts: Iterable<string>): string[];
 declare function grammarReady(key: string): boolean;
 
 interface AstResult {
@@ -229,7 +251,20 @@ interface AstResult {
     }[];
     importedNames: string[];
 }
-declare function extractAst(rel: string, ext: string, content: string): AstResult | undefined;
+declare function extractAst(rel: string, ext: string, content: string, opts?: {
+    maxCalls?: number;
+}): AstResult | undefined;
+
+declare const DEFAULT_GRAMMARS_URL = "https://github.com/maxgfr/codeindex/releases/download/v2.14.0/grammars-2.14.0.tar.gz";
+interface GrammarsPullTarget {
+    url: string;
+    sha256Url?: string;
+}
+declare function resolveGrammarsPullTarget(): GrammarsPullTarget;
+declare function fetchGrammarsTarball(url: string, expectedSha256?: string): Promise<Uint8Array>;
+declare function fetchExpectedSha256(url: string): Promise<string>;
+declare function extractTarInto(rawTar: Uint8Array, destDir: string): string[];
+declare function extractGrammarsTarball(bytes: Uint8Array, destDir: string): string[];
 
 type Resolution = {
     kind: "resolved";
@@ -442,6 +477,7 @@ interface IndexArtifacts {
     symbols: SymbolIndex;
 }
 declare function buildIndexArtifacts(repo: string, opts?: BuildIndexOptions): IndexArtifacts;
+declare function buildArtifactsFromScan(scan: RepoScan, opts?: BuildIndexOptions): IndexArtifacts;
 
 declare function headCommit(dir: string): string | undefined;
 interface DiffFile {
@@ -672,9 +708,15 @@ interface MermaidOptions {
 }
 declare function renderMermaid(graph: Graph, opts?: MermaidOptions): string;
 
-declare function runMcpServer(): Promise<void>;
+interface McpServerOptions {
+    serverInfo?: {
+        name?: string;
+        version?: string;
+    };
+}
+declare function runMcpServer(opts?: McpServerOptions): Promise<void>;
 
-declare function sha1(s: string): string;
+declare function sha1(s: string | Uint8Array): string;
 declare function shortHash(s: string, n?: number): string;
 
 declare function byStr(a: string, b: string): number;
@@ -705,4 +747,4 @@ declare function rrf<T>(lists: T[][], keyOf: (item: T) => string, k?: number): M
 
 declare function runCli(argv: string[]): Promise<void>;
 
-export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawCallerIndex, type RawCallerSite, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildRawCallerIndex, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractMarkdown, extractSymbols, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };
+export { type ArchRule, type BuildIndexOptions, type BuiltinRule, type CallerEntry, type CallerIndex, type CallerIndexOptions, type CallerSite, type ChangeCoupling, type CodeInfo, type CodeSymbol, type CouplingOptions, DEFAULT_GRAMMARS_URL, DEFAULT_MAX_FILES, type DeadSymbol, type DiffFile, type DiffSpec, EMBED_VERSION, ENGINE_VERSION, EXTRACTOR_VERSION, type Edge, type EdgeKind, type EditResult, type EmbedEndpointOptions, type EmbedPullTarget, type EmbeddingIndex, type EmbeddingRecord, type EmbeddingUnit, type FileCategory, type FileKind, type FileNode, type FileRecord, type FindSymbolOptions, type ForbiddenEdgeRule, type GrammarsPullTarget, type GrammarsTier, type GrammarsTierName, type Graph, type GrepOptions, type Hotspot, type Hunk, type IgnoreRule, type IndexArtifacts, MARKDOWN_EXT, type MarkdownInfo, type McpServerOptions, type MermaidOptions, type ModuleInfo, type ModuleNode, type RawCallerIndex, type RawCallerSite, type RawRef, type RenderScipOptions, type RepoMapOptions, type RepoScan, type Resolution, type ResolveContext, type RiskHotspot, type RuleSeverity, type RuleViolation, SCHEMA_VERSION, type ScanOptions, type SearchHit, type SearchOptions, type SearchResult, type SemanticSearchOptions, type SemanticSearchResult, type ShResult, type StaticEmbedModel, type SurpriseEdge, type SymbolComplexity, type SymbolIndex, type SymbolMatch, type SymbolReferences, type TestMap, type Tier, type WalkOptions, type WalkResult, type WalkedFile, type WorkspaceInfo, type WorkspaceKind, type WorkspacePackage, allGrammarKeys, applyCentrality, basicTokenize, betweennessOf, buildArtifactsFromScan, buildCallerIndex, buildEmbeddingIndex, buildEndpointIndex, buildGraph, buildIndexArtifacts, buildModules, buildRawCallerIndex, buildResolveContext, buildSymbolIndex, byKey, byStr, categorize, changeCoupling, changedSince, checkRules, classify, clip, clipInline, communityOf, compileGlobs, complexityOfSource, computeImportPairs, computeSurprises, computeSymbolRefs, computeTestMap, deleteMemory, deserializeEmbeddings, detectCommunities, detectWorkspaces, diffFiles, diffHunks, embedEndpointUrl, embedViaEndpoint, embeddingUnits, enclosingSymbol, encode, encodeQueryViaEndpoint, ensureGrammars, escapeRegExp, extToLang, extractAst, extractCode, extractGrammarsTarball, extractMarkdown, extractSymbols, extractTarInto, fetchExpectedSha256, fetchGrammarsTarball, findDeadCode, findReferences, findSymbol, foldText, gitChurn, grammarKeyForExt, grammarKeysForExts, grammarReady, grepRepo, hasEmbedModel, have, headCommit, healthzUrl, insertAfterSymbol, insertBeforeSymbol, intDot, isCode, isDoc, isGitWorktree, isIgnored, isSurprising, isTestFile, isTestPath, keywords, languageOf, listMemories, loadEmbedModel, pagerankOf, parseGitignore, parseRules, probeEndpoint, quantize, rankHotspots, rankedKeywords, readMemory, readText, renderGraphJson, renderMermaid, renderRepoMap, renderScip, renderSymbolsJson, replaceSymbolBody, resolveBaseRef, resolveCallEdges, resolveDocLink, resolveEmbedEndpoint, resolveEmbedModelDir, resolveEmbedPullUrl, resolveGrammarsDir, resolveGrammarsPullTarget, resolveGrammarsTier, resolveImport, resolveUniqueSymbol, riskHotspots, roundHalfToEven, rrf, runCli, runMcpServer, scanRepo, searchIndex, searchSemantic, serializeEmbeddings, sh, sha1, sharedGrammarsCacheDir, shortHash, slugify, subtokens, symbolComplexity, symbolsOverview, testsForModule, tierForPath, tokenize, uniqueSymbolDefs, untestedModules, untrackedFiles, walk, wordpiece, writeMemory };