@nithin554/code-crawler 0.1.53 → 0.1.55

package/README.md

@@ -80,9 +80,19 @@ and answers them with bounded recursive SQL queries.
   bare/`this`/`super` calls, static/qualified calls, and superclass chains.
   Unresolvable calls become `unknown_symbol` phantom nodes with `resolved:false` edges,
   so the graph stays traversable.
-- **Idempotent, file-scoped indexing** — a full re-crawl refreshes changed files, rebuilds
-  derived dynamic edges, and removes files that disappeared, without duplicating the graph.
-  Scoped crawls (`fileFilter` or a forced `language`) never remove files outside their scope.
+- **Incremental, file-scoped indexing** — each crawl records a per-file manifest
+  (`size`, mtime, SHA-256) and replays the cached parse result for files whose content is
+  unchanged, so a re-crawl re-parses only what actually changed. Changes are reported as
+  `filesAdded` / `filesUpdated` / `filesUnchanged` / `filesRemoved`. Derived dynamic edges
+  and orphan pruning are rebuilt only when something changed. A full re-crawl still removes
+  files that disappeared, without duplicating the graph, and scoped crawls (`fileFilter` or a
+  forced `language`) never remove files outside their scope.
+- **FTS5-backed symbol search** — symbol lookup runs against an external-content FTS5 index
+  with BM25 ranking and exact/prefix boosting, falling back to a `LIKE` scan only when FTS
+  finds nothing. Search does not degrade as the graph grows.
+- **Versioned schema with automatic migration** — the database carries a schema version and
+  upgrades itself in place on open. A database written by a newer build is rejected with a
+  clear message instead of being read as if the shape were known.
 - **One project per database** — indexing a different project into the same database fails
   with a `clear_index` hint instead of silently wiping the first index.
 - **Analytics** — hotspots (fan-in/fan-out/advice counts) and dead-code candidates.
@@ -97,7 +107,8 @@ Requires Node.js ≥ 20.
 ```bash
 npm install
 npm run build        # tsc → dist/
-npm test             # vitest (135 tests)
+npm test             # vitest (174 tests; run with CODECRAWLER_BENCH=1 for the scale test)
+npm run bench        # crawl/query benchmark on a generated ~137k-line project
 ```
 
 ## CLI
@@ -116,9 +127,101 @@ Environment variables:
 | `CODECRAWLER_DB` | `~/.code-crawler/index.db` | Database path |
 | `CODECRAWLER_EXCLUDE_DIRS` | `.git,node_modules,dist,build,target,...` | Dirs skipped during crawl |
 | `CODECRAWLER_DEFAULT_LIMIT` | `50` | Default result cap |
-| `CODECRAWLER_MAX_DEPTH` | `10` | Traversal depth cap |
+| `CODECRAWLER_MAX_DEPTH` | `10` | Traversal depth cap (1–100) |
+| `CODECRAWLER_MAX_FILE_BYTES` | `1000000` | Files larger than this are skipped, not parsed |
+| `CODECRAWLER_RESPECT_GITIGNORE` | `true` | Honor the project's `.gitignore` during discovery |
 | `CODECRAWLER_HELPFUL_ERRORS` | `true` | Fuzzy suggestions on symbol misses |
 
+Invalid values never crash the server: booleans fall back to their default and
+integers are clamped to their documented range, each with a one-line warning on
+stderr.
+
+## What gets indexed
+
+`crawl` walks the project and skips `.git`, `node_modules`, build output, and
+other `CODECRAWLER_EXCLUDE_DIRS` entries. On top of that:
+
+- the project's **`.gitignore`** is honored (disable with
+  `CODECRAWLER_RESPECT_GITIGNORE=false`),
+- an optional **`.crawlerrc`** in the project root can narrow or widen discovery
+  with gitignore-style globs:
+
+  ```json
+  { "include": ["src/", "packages/core/"], "exclude": ["**/generated/"] }
+  ```
+
+  When `include` is non-empty, only matching files are indexed. A malformed
+  `.crawlerrc` is ignored with a warning rather than failing the crawl.
+- files above `CODECRAWLER_MAX_FILE_BYTES` and files that look binary are
+  skipped; `crawl_project` reports them in `skippedFiles` with a reason.
+- per-file parse failures never abort the crawl; they are reported in
+  `parseErrors`.
+
+Only the root `.gitignore` is read — nested `.gitignore` files are not yet
+supported.
+
+## Incremental crawls
+
+A crawl records a manifest row per file (`size`, mtime, SHA-256, language). On the
+next crawl a file whose size and truncated mtime both match is served from a cached
+parse result instead of being re-read and re-parsed; a file whose mtime changed but
+whose hash did not is also replayed from cache. Resolution, phantom-node creation and
+derived edges still run over the whole in-memory file set, so cross-file edges stay
+correct — only I/O, parsing and database writes are skipped.
+
+`crawl_project` reports the split:
+
+| Field | Meaning |
+| --- | --- |
+| `filesAdded` | First seen by this crawl |
+| `filesUpdated` | Content changed, re-parsed |
+| `filesUnchanged` | Replayed from the parse cache |
+| `filesRemoved` | Disappeared from disk and were pruned |
+
+**Known limitation:** because an unchanged file's edges are not rewritten, a call that
+becomes *newly resolvable* only because some other file changed (for example a method
+gained an overload, or a class was introduced that a previously-ambiguous call could
+now bind to) will not be upgraded in the untouched file. Touch that file, or run
+`clear_index`, to force full resolution.
+
+## Benchmarks
+
+`npm run bench` generates a synthetic TypeScript project (575 modules, ~137k lines,
+~9.8k methods), crawls it, and reports crawl phases and query latency percentiles.
+`tests/perf.test.ts` asserts the same budgets and is opt-in, because a realistic
+corpus takes time to crawl:
+
+```bash
+npm run bench                                        # human-readable report
+CODECRAWLER_BENCH=1 npm test -- tests/perf.test.ts   # enforce the budgets
+CODECRAWLER_BENCH_LINES=20000 npm run bench          # smaller corpus
+```
+
+Measured on a Windows dev laptop (Node 20, warm disk cache); absolute numbers vary,
+the *ratios* and the index usage are what matter:
+
+| Phase | Duration | Added | Updated | Unchanged |
+| --- | --- | --- | --- | --- |
+| Cold crawl | 22.5 s | 575 | 0 | 0 |
+| Re-crawl, nothing changed | 0.91 s | 0 | 0 | 575 |
+| Re-crawl after a one-file edit | 0.75 s | 0 | 1 | 574 |
+
+| Query | p50 | p95 | Budget (p95) |
+| --- | --- | --- | --- |
+| `search_symbols` (exact) | 0.69 ms | 1.96 ms | 50 ms |
+| `search_symbols` (prefix) | 1.60 ms | 4.09 ms | 50 ms |
+| `get_callers` (depth 2) | 0.16 ms | 0.21 ms | 50 ms |
+| `get_callees` (depth 2) | 0.11 ms | 0.15 ms | 50 ms |
+| `get_subgraph` (radius 2) | 0.16 ms | 0.27 ms | 50 ms |
+| `find_entry_points` | 0.12 ms | 0.29 ms | 50 ms |
+| `find_hotspots` | 145.5 ms | 181.4 ms | 1000 ms |
+| `find_dead_code` | 99.8 ms | 290.3 ms | 1000 ms |
+
+The index is 66 MB for those 14.4k nodes / 38.8k edges. `find_hotspots` and
+`find_dead_code` are whole-project aggregations over every edge and node, which is why
+their band is an order of magnitude looser than the symbol lookups; both were profiled
+with `EXPLAIN QUERY PLAN` and are served by `idx_edges_target_kind`.
+
 ## CI & publishing
 
 GitHub Actions runs on every pull request (`pull-request.yml`): `npm ci` →

package/dist/config.d.ts

@@ -7,6 +7,10 @@ export interface Config {
     defaultLimit: number;
     /** Cap on transitive traversal depth to keep queries bounded. */
     maxDepth: number;
+    /** Files larger than this are skipped instead of parsed. */
+    maxFileBytes: number;
+    /** Honor the project's .gitignore when discovering source files. */
+    respectGitignore: boolean;
     /** When true, errors include symbol-search suggestions. */
     helpfulErrors: boolean;
 }

package/dist/config.js

@@ -7,25 +7,56 @@ import os from "node:os";
  * point the server at different databases without code changes.
  */
 const DEFAULT_DB_PATH = path.join(os.homedir(), ".code-crawler", "index.db");
+function warn(message) {
+    process.stderr.write(`code-crawler: ${message}\n`);
+}
 function envStr(name, fallback) {
     const v = process.env[name];
     return v && v.length > 0 ? v : fallback;
 }
 function envBool(name, fallback) {
-    const v = process.env[name];
-    if (v === undefined)
+    const raw = process.env[name];
+    if (raw === undefined || raw.trim() === "")
+        return fallback;
+    const v = raw.trim().toLowerCase();
+    if (v === "1" || v === "true" || v === "yes" || v === "on")
+        return true;
+    if (v === "0" || v === "false" || v === "no" || v === "off")
+        return false;
+    warn(`${name}="${raw}" is not a boolean (use true/false); using ${fallback}`);
+    return fallback;
+}
+/** Parse an integer, clamping out-of-range values and warning about both. */
+function envInt(name, fallback, min, max) {
+    const raw = process.env[name];
+    if (raw === undefined || raw.trim() === "")
+        return fallback;
+    const parsed = Number(raw);
+    if (!Number.isFinite(parsed)) {
+        warn(`${name}="${raw}" is not a number; using ${fallback}`);
         return fallback;
-    return v === "1" || v.toLowerCase() === "true";
+    }
+    const truncated = Math.trunc(parsed);
+    if (truncated < min || truncated > max) {
+        const clamped = Math.min(max, Math.max(min, truncated));
+        warn(`${name}="${raw}" must be between ${min} and ${max}; using ${clamped}`);
+        return clamped;
+    }
+    return truncated;
 }
 export function loadConfig() {
     return {
         dbPath: envStr("CODECRAWLER_DB", DEFAULT_DB_PATH),
-        excludeDirs: envStr("CODECRAWLER_EXCLUDE_DIRS", ".git,node_modules,dist,build,target,.idea,.venv,.gradle,out,.next,coverage")
-            .split(",")
-            .map((s) => s.trim())
-            .filter((s) => s.length > 0),
-        defaultLimit: Number(envStr("CODECRAWLER_DEFAULT_LIMIT", "50")),
-        maxDepth: Number(envStr("CODECRAWLER_MAX_DEPTH", "10")),
+        excludeDirs: [
+            ...new Set(envStr("CODECRAWLER_EXCLUDE_DIRS", ".git,node_modules,dist,build,target,.idea,.venv,.gradle,out,.next,coverage")
+                .split(",")
+                .map((s) => s.trim())
+                .filter((s) => s.length > 0)),
+        ],
+        defaultLimit: envInt("CODECRAWLER_DEFAULT_LIMIT", 50, 1, 5000),
+        maxDepth: envInt("CODECRAWLER_MAX_DEPTH", 10, 1, 100),
+        maxFileBytes: envInt("CODECRAWLER_MAX_FILE_BYTES", 1_000_000, 1024, 64 * 1024 * 1024),
+        respectGitignore: envBool("CODECRAWLER_RESPECT_GITIGNORE", true),
         helpfulErrors: envBool("CODECRAWLER_HELPFUL_ERRORS", true),
     };
 }

package/dist/config.js.map

@@ -1 +1 @@
-{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,MAAM,SAAS,CAAC;AAEzB;;;;;GAKG;AAEH,MAAM,eAAe,GAAG,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,EAAE,eAAe,EAAE,UAAU,CAAC,CAAC;AAE7E,SAAS,MAAM,CAAC,IAAY,EAAE,QAAgB;IAC5C,MAAM,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC5B,OAAO,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC;AAC1C,CAAC;AAED,SAAS,OAAO,CAAC,IAAY,EAAE,QAAiB;IAC9C,MAAM,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC5B,IAAI,CAAC,KAAK,SAAS;QAAE,OAAO,QAAQ,CAAC;IACrC,OAAO,CAAC,KAAK,GAAG,IAAI,CAAC,CAAC,WAAW,EAAE,KAAK,MAAM,CAAC;AACjD,CAAC;AAeD,MAAM,UAAU,UAAU;IACxB,OAAO;QACL,MAAM,EAAE,MAAM,CAAC,gBAAgB,EAAE,eAAe,CAAC;QACjD,WAAW,EAAE,MAAM,CACjB,0BAA0B,EAC1B,4EAA4E,CAC7E;aACE,KAAK,CAAC,GAAG,CAAC;aACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;aACpB,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC;QAC9B,YAAY,EAAE,MAAM,CAAC,MAAM,CAAC,2BAA2B,EAAE,IAAI,CAAC,CAAC;QAC/D,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,uBAAuB,EAAE,IAAI,CAAC,CAAC;QACvD,aAAa,EAAE,OAAO,CAAC,4BAA4B,EAAE,IAAI,CAAC;KAC3D,CAAC;AACJ,CAAC"}
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,MAAM,SAAS,CAAC;AAEzB;;;;;GAKG;AAEH,MAAM,eAAe,GAAG,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,EAAE,eAAe,EAAE,UAAU,CAAC,CAAC;AAE7E,SAAS,IAAI,CAAC,OAAe;IAC3B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,iBAAiB,OAAO,IAAI,CAAC,CAAC;AACrD,CAAC;AAED,SAAS,MAAM,CAAC,IAAY,EAAE,QAAgB;IAC5C,MAAM,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC5B,OAAO,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC;AAC1C,CAAC;AAED,SAAS,OAAO,CAAC,IAAY,EAAE,QAAiB;IAC9C,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC9B,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE;QAAE,OAAO,QAAQ,CAAC;IAC5D,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;IACnC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,MAAM,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,KAAK,IAAI;QAAE,OAAO,IAAI,CAAC;IACxE,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,OAAO,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,KAAK;QAAE,OAAO,KAAK,CAAC;IAC1E,IAAI,CAAC,GAAG,IAAI,KAAK,GAAG,8CAA8C,QAAQ,EAAE,CAAC,CAAC;IAC9E,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,6EAA6E;AAC7E,SAAS,MAAM,CAAC,IAAY,EAAE,QAAgB,EAAE,GAAW,EAAE,GAAW;IACtE,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC9B,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE;QAAE,OAAO,QAAQ,CAAC;IAC5D,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC3B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;QAC7B,IAAI,CAAC,GAAG,IAAI,KAAK,GAAG,4BAA4B,QAAQ,EAAE,CAAC,CAAC;QAC5D,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;IACrC,IAAI,SAAS,GAAG,GAAG,IAAI,SAAS,GAAG,GAAG,EAAE,CAAC;QACvC,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC,CAAC;QACxD,IAAI,CAAC,GAAG,IAAI,KAAK,GAAG,qBAAqB,GAAG,QAAQ,GAAG,WAAW,OAAO,EAAE,CAAC,CAAC;QAC7E,OAAO,OAAO,CAAC;IACjB,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAmBD,MAAM,UAAU,UAAU;IACxB,OAAO;QACL,MAAM,EAAE,MAAM,CAAC,gBAAgB,EAAE,eAAe,CAAC;QACjD,WAAW,EAAE;YACX,GAAG,IAAI,GAAG,CACR,MAAM,CACJ,0BAA0B,EAC1B,4EAA4E,CAC7E;iBACE,KAAK,CAAC,GAAG,CAAC;iBACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;iBACpB,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,CAC/B;SACF;QACD,YAAY,EAAE,MAAM,CAAC,2BAA2B,EAAE,EAAE,EAAE,CAAC,EAAE,IAAI,CAAC;QAC9D,QAAQ,EAAE,MAAM,CAAC,uBAAuB,EAAE,EAAE,EAAE,CAAC,EAAE,GAAG,CAAC;QACrD,YAAY,EAAE,MAAM,CAAC,4BAA4B,EAAE,SAAS,EAAE,IAAI,EAAE,EAAE,GAAG,IAAI,GAAG,IAAI,CAAC;QACrF,gBAAgB,EAAE,OAAO,CAAC,+BAA+B,EAAE,IAAI,CAAC;QAChE,aAAa,EAAE,OAAO,CAAC,4BAA4B,EAAE,IAAI,CAAC;KAC3D,CAAC;AACJ,CAAC"}

package/dist/crawler.d.ts

@@ -11,12 +11,40 @@ export interface CrawlOptions {
     language?: string;
     /** Optional include filter: path must contain this substring. */
     fileFilter?: string;
+    /** Honor the project's .gitignore (default true). */
+    respectGitignore?: boolean;
+    /** Skip files larger than this many bytes. */
+    maxFileBytes?: number;
+    /** Called as parsing/persisting advances, for progress reporting. */
+    onProgress?: (progress: CrawlProgress) => void;
+}
+export interface CrawlProgress {
+    phase: "parse" | "persist";
+    processed: number;
+    total: number;
+}
+/** A discovered file that was deliberately not parsed. */
+export interface SkippedFile {
+    path: string;
+    reason: "too-large" | "binary";
 }
 export interface CrawlResult {
     projectPath: string;
     filesScanned: number;
+    /** Files whose parse result is in the index after this crawl. */
     filesParsed: number;
+    /** Files parsed for the first time by this crawl. */
+    filesAdded: number;
+    /** Files re-parsed because their content changed. */
+    filesUpdated: number;
+    /** Files replayed from the parse cache because their content was identical. */
+    filesUnchanged: number;
     filesRemoved: number;
+    /** Files skipped because they are too large or look binary. */
+    filesSkipped: number;
+    skippedFiles: SkippedFile[];
+    /** Per-file parse failures (paths and messages, capped for output size). */
+    parseErrors: string[];
     /** False when the crawl was scoped (fileFilter/language) and stale pruning was skipped. */
     pruned: boolean;
     /** File-less shared nodes deleted because nothing references them anymore. */
@@ -46,8 +74,20 @@ export interface CrawlResult {
     languages: string[];
     durationMs: number;
 }
-/** Discover source files under a project, respecting exclusion dirs. */
-export declare function discoverSourceFiles(projectPath: string, excludeDirs: string[], language?: string): string[];
+export interface DiscoverOptions {
+    language?: string;
+    /** Honor the project's .gitignore (default true). */
+    respectGitignore?: boolean;
+}
+/**
+ * Discover source files under a project.
+ *
+ * Honors the hardcoded/configured exclusion directory names, the project's
+ * `.gitignore`, and an optional `.crawlerrc` with gitignore-style `include`
+ * and `exclude` globs. When `include` is present only matching files are
+ * indexed.
+ */
+export declare function discoverSourceFiles(projectPath: string, excludeDirs: string[], options?: string | DiscoverOptions): string[];
 /** Convert a ParsedFile into graph nodes + edges (declarations only). */
 export declare function buildDeclarationGraph(parsed: ParsedFile): {
     nodes: GraphNode[];

package/dist/crawler.js

@@ -1,15 +1,79 @@
 import fs from "node:fs";
 import path from "node:path";
+import { createHash } from "node:crypto";
+import ignore from "ignore";
 import { annotationNodeId, externalNodeId, fieldNodeId, importNodeId } from "./graph/schema.js";
 import { languageForFile } from "./parser/languages/index.js";
+import { decodeParsedFile, encodeParsedFile } from "./parser/cache.js";
 import { buildTypeIndex, phantomNodeId } from "./resolver/types.js";
 import { resolveJavaFile } from "./resolver/javaResolver.js";
 import { resolveGenericFile, resolveInjectionTarget, resolveReflectiveTarget, resolveRouteHandler } from "./resolver/genericResolver.js";
 import { normalizeTypeText, resolveTypeCandidates } from "./resolver/common.js";
 import { buildDynamicEdges, isDecoratorAnnotation } from "./resolver/dynamic.js";
-/** Discover source files under a project, respecting exclusion dirs. */
-export function discoverSourceFiles(projectPath, excludeDirs, language) {
+/** Optional project-local crawl configuration. */
+const CRAWLER_RC = ".crawlerrc";
+function loadCrawlerRc(projectPath) {
+    const file = path.join(projectPath, CRAWLER_RC);
+    if (!fs.existsSync(file))
+        return {};
+    try {
+        const parsed = JSON.parse(fs.readFileSync(file, "utf8"));
+        if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {
+            throw new Error("expected a JSON object");
+        }
+        const record = parsed;
+        const strings = (value, field) => {
+            if (value === undefined)
+                return undefined;
+            if (!Array.isArray(value) || value.some((v) => typeof v !== "string")) {
+                throw new Error(`"${field}" must be an array of strings`);
+            }
+            return value;
+        };
+        return {
+            include: strings(record.include, "include"),
+            exclude: strings(record.exclude, "exclude"),
+        };
+    }
+    catch (err) {
+        process.stderr.write(`code-crawler: ignoring ${CRAWLER_RC}: ${err.message}\n`);
+        return {};
+    }
+}
+/**
+ * Discover source files under a project.
+ *
+ * Honors the hardcoded/configured exclusion directory names, the project's
+ * `.gitignore`, and an optional `.crawlerrc` with gitignore-style `include`
+ * and `exclude` globs. When `include` is present only matching files are
+ * indexed.
+ */
+export function discoverSourceFiles(projectPath, excludeDirs, options = {}) {
+    const opts = typeof options === "string" ? { language: options } : options;
     const excluded = new Set(excludeDirs.map((d) => d.replace(/\/+$/, "")));
+    const rc = loadCrawlerRc(projectPath);
+    const excludeMatcher = ignore().add(rc.exclude ?? []);
+    const includeMatcher = ignore().add(rc.include ?? []);
+    const hasIncludes = (rc.include?.length ?? 0) > 0;
+    if (opts.respectGitignore !== false) {
+        try {
+            const gitignorePath = path.join(projectPath, ".gitignore");
+            if (fs.existsSync(gitignorePath)) {
+                excludeMatcher.add(fs.readFileSync(gitignorePath, "utf8"));
+            }
+        }
+        catch {
+            // Unreadable .gitignore: fall back to indexing everything.
+        }
+    }
+    const ignored = (rel, isDir) => {
+        const candidate = isDir ? `${rel}/` : rel;
+        if (excludeMatcher.ignores(candidate))
+            return true;
+        if (hasIncludes && !includeMatcher.ignores(candidate))
+            return true;
+        return false;
+    };
     const out = [];
     function walk(dir) {
         let entries;
@@ -21,11 +85,15 @@ export function discoverSourceFiles(projectPath, excludeDirs, language) {
         }
         for (const entry of entries) {
             const full = path.join(dir, entry.name);
+            const rel = path.relative(projectPath, full).split(path.sep).join("/");
             if (entry.isDirectory()) {
-                if (!excluded.has(entry.name))
-                    walk(full);
+                if (excluded.has(entry.name) || ignored(rel, true))
+                    continue;
+                walk(full);
             }
-            else if (entry.isFile() && languageForFile(entry.name, language)) {
+            else if (entry.isFile() &&
+                languageForFile(entry.name, opts.language) &&
+                !ignored(rel, false)) {
                 out.push(full);
             }
         }
@@ -38,7 +106,7 @@ export function buildDeclarationGraph(parsed) {
     const nodes = [];
     const edges = [];
     const fileId = parsed.filePath;
-    const fileLines = sourceLineCount(parsed.filePath);
+    const fileLines = parsed.lineCount ?? sourceLineCount(parsed.filePath);
     nodes.push({
         id: fileId,
         kind: "file",
@@ -599,30 +667,125 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
     // Stale pruning is only meaningful for a full scan. fileFilter and a forced
     // language narrow the scan, so files outside that scope must be left alone.
     const scopedScan = Boolean(options.fileFilter || options.language);
-    const files = discoverSourceFiles(absPath, excludeDirs, options.language);
+    const files = discoverSourceFiles(absPath, excludeDirs, {
+        language: options.language,
+        respectGitignore: options.respectGitignore,
+    });
     if (options.fileFilter) {
         const filter = options.fileFilter;
         files.splice(0, files.length, ...files.filter((f) => f.includes(filter)));
     }
-    // Parse every file (grammar is cached, so subsequent parses are fast).
+    const maxFileBytes = options.maxFileBytes ?? 1_000_000;
+    const skipSampleBytes = 8192;
+    // Incremental crawl: the manifest records size + mtime + content hash for
+    // every indexed file, together with its parse result. A file whose metadata
+    // is unchanged is replayed from the cache; a file whose metadata changed but
+    // whose hash matches was only touched and is replayed too.
+    const manifest = store.listFileRecords();
     const parsedFiles = [];
     const parseErrors = [];
+    const skippedFiles = [];
+    const changedPaths = new Set();
+    const recordsToWrite = new Map();
+    let filesAdded = 0;
+    let filesUpdated = 0;
+    let filesUnchanged = 0;
+    const pendingParses = [];
+    // Pass 1: stat only. Cheap enough to run over every discovered file and it
+    // decides whether any real parsing (or indexing) work is needed at all.
     for (const file of files) {
         try {
-            const source = fs.readFileSync(file, "utf8");
+            const stat = fs.statSync(file);
+            if (stat.size > maxFileBytes) {
+                skippedFiles.push({ path: file, reason: "too-large" });
+                continue;
+            }
+            const record = manifest.get(file);
+            const mtimeMs = Math.trunc(stat.mtimeMs);
+            if (record && record.size === stat.size && record.mtimeMs === mtimeMs) {
+                const cached = decodeParsedFile(record.summary);
+                if (cached) {
+                    filesUnchanged++;
+                    parsedFiles.push(cached);
+                    continue;
+                }
+            }
+            pendingParses.push({ path: file, size: stat.size, mtimeMs });
+        }
+        catch (err) {
+            if (parseErrors.length < 50) {
+                parseErrors.push(`${file}: ${err.message}`);
+            }
+        }
+    }
+    // Pass 2: read, hash and (only when the content really changed) parse.
+    // Each file is read once: the same buffer provides the binary sniff, the
+    // content hash, the source text, and the line count.
+    let processedFiles = 0;
+    for (const pending of pendingParses) {
+        const file = pending.path;
+        try {
+            const buffer = fs.readFileSync(file);
+            if (buffer.subarray(0, skipSampleBytes).includes(0)) {
+                skippedFiles.push({ path: file, reason: "binary" });
+                continue;
+            }
             const config = languageForFile(file, options.language);
             if (!config)
                 continue;
-            parsedFiles.push(await config.parseFile(source, file, { projectRoot: absPath }));
+            const sha256 = createHash("sha256").update(buffer).digest("hex");
+            const record = manifest.get(file);
+            if (record && record.sha256 === sha256) {
+                const cached = decodeParsedFile(record.summary);
+                if (cached) {
+                    // Same content, new timestamp: refresh metadata without re-parsing.
+                    filesUnchanged++;
+                    parsedFiles.push(cached);
+                    recordsToWrite.set(file, { ...record, size: pending.size, mtimeMs: pending.mtimeMs });
+                    continue;
+                }
+            }
+            const source = buffer.toString("utf8");
+            const parsed = await config.parseFile(source, file, { projectRoot: absPath });
+            const full = { ...parsed, lineCount: source.split("\n").length };
+            parsedFiles.push(full);
+            changedPaths.add(file);
+            if (record)
+                filesUpdated++;
+            else
+                filesAdded++;
+            recordsToWrite.set(file, {
+                path: file,
+                size: pending.size,
+                mtimeMs: pending.mtimeMs,
+                sha256,
+                language: full.language,
+                summary: encodeParsedFile(full),
+            });
         }
         catch (err) {
-            parseErrors.push(`${file}: ${err.message}`);
+            if (parseErrors.length < 50) {
+                parseErrors.push(`${file}: ${err.message}`);
+            }
+        }
+        finally {
+            processedFiles++;
+            options.onProgress?.({
+                phase: "parse",
+                processed: processedFiles,
+                total: pendingParses.length,
+            });
+            // Parsing is synchronous once a grammar is loaded; yield so the stdio
+            // server can answer requests during long crawls.
+            await new Promise((resolve) => setImmediate(resolve));
         }
     }
     // Global index for cross-file resolution.
     const index = buildTypeIndex(parsedFiles);
     // Build nodes + edges per file and persist file-scoped.
-    const previouslyIndexed = new Set(store.raw("SELECT DISTINCT file_path FROM nodes WHERE file_path IS NOT NULL").map((r) => r.file_path));
+    const previouslyIndexed = new Set(store
+        .raw("SELECT DISTINCT file_path FROM nodes WHERE file_path IS NOT NULL")
+        .map((r) => r.file_path));
     const currentFiles = new Set(files);
     let filesRemoved = 0;
     let orphansRemoved = 0;
@@ -685,6 +848,16 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
                 entryPointIds.add(node.id);
         }
     }
+    // Nodes owned by files this crawl rewrote; their rows already carry the
+    // markers computed above, so only the rest need a targeted update.
+    const changedNodeIds = new Set();
+    for (const parsed of parsedFiles) {
+        if (!changedPaths.has(parsed.filePath))
+            continue;
+        for (const node of declarations.get(parsed.filePath)?.nodes ?? []) {
+            changedNodeIds.add(node.id);
+        }
+    }
     // Hierarchy metadata is emitted with whatever name the source used
     // (`implements Payment`), which is not always the fully-qualified node id.
     // Resolve imports first, then fall back to unique simple names.
@@ -715,6 +888,7 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
         const resolved = candidates?.length === 1 ? candidates[0] : undefined;
         return resolved ? { ...edge, target: resolved } : edge;
     };
+    let persistedFiles = 0;
     store.withTransaction(() => {
         for (const parsed of parsedFiles) {
             const decl = declarations.get(parsed.filePath);
@@ -724,17 +898,21 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
                 externalDeps.add(pkg);
             for (const p of calls.phantomNodes)
                 allNodeIds.add(p.id);
-            // Phantom nodes must exist as edge endpoints (FK constraint).
-            for (const p of calls.phantomNodes)
-                store.ensureNode(p);
-            // Drop edges whose endpoints aren't part of the indexed project
-            // (external EXTENDS/IMPLEMENTS/IMPORTS targets).
-            const edges = decl.edges
-                .map((edge) => normalizeHierarchyTarget(edge, parsed.filePath))
-                .concat(calls.edges, deps.edges)
-                .filter((e) => allNodeIds.has(e.source) && allNodeIds.has(e.target));
-            const nodes = decl.nodes.concat(calls.phantomNodes, deps.nodes);
-            store.replaceFile(parsed.filePath, nodes, edges);
+            // Unchanged files keep their existing rows: re-resolving them would
+            // rewrite identical data and defeat the point of an incremental crawl.
+            if (changedPaths.has(parsed.filePath)) {
+                // Phantom nodes must exist as edge endpoints (FK constraint).
+                for (const p of calls.phantomNodes)
+                    store.ensureNode(p);
+                // Drop edges whose endpoints aren't part of the indexed project
+                // (external EXTENDS/IMPLEMENTS/IMPORTS targets).
+                const edges = decl.edges
+                    .map((edge) => normalizeHierarchyTarget(edge, parsed.filePath))
+                    .concat(calls.edges, deps.edges)
+                    .filter((e) => allNodeIds.has(e.source) && allNodeIds.has(e.target));
+                const nodes = decl.nodes.concat(calls.phantomNodes, deps.nodes);
+                store.replaceFile(parsed.filePath, nodes, edges);
+            }
             for (const edge of calls.edges) {
                 if (allNodeIds.has(edge.source) && allNodeIds.has(edge.target)) {
                     allCallEdges.push(edge);
@@ -744,6 +922,12 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
             totalResolved += calls.resolved;
             totalUnresolved += calls.unresolved;
             previouslyIndexed.delete(parsed.filePath);
+            persistedFiles++;
+            options.onProgress?.({
+                phase: "persist",
+                processed: persistedFiles,
+                total: parsedFiles.length,
+            });
         }
         // Remove files that disappeared from the project. Only a full scan knows
         // the complete file set, and only files under this project belong to it.
@@ -755,16 +939,30 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
                 filesRemoved++;
             }
         }
+        // Refresh the manifest for parsed files and for untouched-but-retimed ones.
+        for (const record of recordsToWrite.values())
+            store.putFileRecord(record);
+        // Entry points can be registered in one file and declared in another. Only
+        // changed files were rewritten, so handlers living in unchanged files need
+        // the marker applied directly.
+        for (const [id, mark] of entryPointMarks) {
+            if (changedNodeIds.has(id))
+                continue;
+            store.markEntryPoint(id, mark.kind, mark.detail ?? mark.kind);
+        }
         // Rebuild project-wide derived edges: virtual dispatch and AOP advice.
-        store.deleteEdgesByKind(["DYNAMIC_DISPATCH", "ADVISES"]);
-        const dynamic = buildDynamicEdges(parsedFiles, index, allNodeIds, allCallEdges);
-        store.upsertEdges(dynamic.edges);
-        dynamicDispatchCount = dynamic.dispatchCount;
-        adviceCount = dynamic.adviceCount;
-        aspectCount = dynamic.aspects;
-        // Shared file-less nodes (packages, annotations, external symbols) outlive
-        // the files that created them; collect the ones nothing references now.
-        orphansRemoved = store.pruneOrphanNodes();
+        // Nothing changed, so the existing derived edges are still correct.
+        if (changedPaths.size > 0 || filesRemoved > 0) {
+            store.deleteEdgesByKind(["DYNAMIC_DISPATCH", "ADVISES"]);
+            const dynamic = buildDynamicEdges(parsedFiles, index, allNodeIds, allCallEdges);
+            store.upsertEdges(dynamic.edges);
+            dynamicDispatchCount = dynamic.dispatchCount;
+            adviceCount = dynamic.adviceCount;
+            aspectCount = dynamic.aspects;
+            // Shared file-less nodes (packages, annotations, external symbols)
+            // outlive the files that created them; collect the unreferenced ones.
+            orphansRemoved = store.pruneOrphanNodes();
+        }
     });
     store.setMeta("last_crawl_at", new Date().toISOString());
     store.setMeta("last_project_path", absPath);
@@ -773,7 +971,13 @@ export async function crawlProject(store, projectPath, excludeDirs, options = {}
         projectPath: absPath,
         filesScanned: files.length,
         filesParsed: parsedFiles.length,
+        filesAdded,
+        filesUpdated,
+        filesUnchanged,
         filesRemoved,
+        filesSkipped: skippedFiles.length,
+        skippedFiles,
+        parseErrors,
         pruned: !scopedScan,
         orphansRemoved,
         nodes: stats.nodes,