codebase-contextualizer-cli 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Roopak Yadav
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,242 @@
+# Codebase Contextualizer CLI
+
+Query your codebase in plain English, completely offline. Codebase Contextualizer CLI indexes local JavaScript-family projects with Tree-sitter, creates local embeddings with `@xenova/transformers`, stores vectors in SQLite, and searches code semantics from your terminal without sending source code to external APIs.
+
+```bash
+npx codebase-contextualizer-cli index .
+npx codebase-contextualizer-cli search "where are files hashed with sha256" .
+```
+
+## Highlights
+
+- Zero external API dependencies: embeddings and search run locally.
+- Sub-0.08ms p95 vector retrieval latency over 1,000 local benchmark queries.
+- Worker-thread indexing keeps parsing and embedding work off the main V8 event loop.
+- Tree-sitter semantic chunks for JavaScript-family files.
+- SQLite vector storage in `.contextualizer/vector.db`.
+- JSON output for scripts, demos, and CI-friendly inspection.
+
+## Installation
+
+Run without installing:
+
+```bash
+npx codebase-contextualizer-cli <command> [arguments]
+```
+
+Install globally:
+
+```bash
+npm install -g codebase-contextualizer-cli
+codebase-contextualizer <command> [arguments]
+```
+
+Requirements:
+
+- Node.js 20 or newer
+- npm 10 or newer recommended
+
+## Usage
+
+Index a codebase:
+
+```bash
+codebase-contextualizer index .
+```
+
+Index with a fixed worker count:
+
+```bash
+codebase-contextualizer index . --workers 4
+```
+
+Check cache drift without writing updates:
+
+```bash
+codebase-contextualizer status .
+```
+
+Search indexed code:
+
+```bash
+codebase-contextualizer search "hash files with sha256" .
+```
+
+Return machine-readable output:
+
+```bash
+codebase-contextualizer search "worker pool queue" . --json
+```
+
+## CLI Examples
+
+```bash
+$ codebase-contextualizer index .
+Index complete: C:\projects\my-app
+Cache: C:\projects\my-app\.contextualizer\cache.json
+Scanned files: 42
+New: 42
+Modified: 0
+Unchanged: 0
+Removed: 0
+Embedded chunks: 118
+Vector database: C:\projects\my-app\.contextualizer\vector.db
+Persisted chunks: 118
+```
+
+```bash
+$ codebase-contextualizer search "where is the worker queue drained" .
+Search: where is the worker queue drained
+Target: C:\projects\my-app
+Database: C:\projects\my-app\.contextualizer\vector.db
+
+1. drainQueue (function)
+Score: 0.8124
+File: src/worker-pool.js
+Line: 74-118
+Code:
+function drainQueue() {
+  ...
+}
+```
+
+```bash
+$ codebase-contextualizer status . --json
+{
+  "root": "C:\\projects\\my-app",
+  "cacheExists": true,
+  "saved": false,
+  "counts": {
+    "scanned": 42,
+    "new": 0,
+    "modified": 1,
+    "unchanged": 41,
+    "removed": 0
+  }
+}
+```
+
+## Features
+
+- Offline semantic code search with no SaaS API calls.
+- Incremental indexing with SHA-256 file hashes.
+- `.gitignore`-aware traversal.
+- Worker pool architecture for CPU-heavy parsing and embedding.
+- Zero-copy transfer of `Float32Array` embeddings from workers to the main thread.
+- SQLite persistence with WAL mode and transactional writes.
+- Graceful Ctrl+C cleanup for workers and database handles.
+
+## System Architecture
+
+The CLI is split into two execution planes.
+
+**Main thread**
+
+- Parses commands with `commander`.
+- Walks directories asynchronously while respecting `.gitignore`.
+- Hashes source files with streamed SHA-256.
+- Uses the cache at `.contextualizer/cache.json` to process only new or modified files.
+- Owns all SQLite writes through `better-sqlite3`.
+
+**Worker pool**
+
+- Uses native Node.js `worker_threads`.
+- Defaults to `Math.min(4, Math.max(1, os.cpus().length - 1))` workers.
+- Each worker initializes its own Tree-sitter parser and singleton local embedding pipeline.
+- Workers read files, extract JavaScript functions/classes/methods, build semantic payloads, and return chunk metadata plus vector tensors.
+
+This design avoids the primary Node.js performance trap: running CPU-heavy AST parsing and ONNX inference on the main V8 event loop. The main thread remains responsible for orchestration and storage, while worker threads absorb the expensive compute path.
+
+## Memory Management
+
+Embeddings are represented as `Float32Array` instances end to end. Workers convert model output into compact typed arrays before returning results to the main process.
+
+The worker response uses zero-copy transfer:
+
+```js
+parentPort.postMessage(payload, transferList);
+```
+
+Each vector's underlying `ArrayBuffer` is included in `transferList`, so ownership moves from the worker to the main thread instead of cloning megabytes of embedding data. This prevents avoidable garbage collection pressure and keeps large-codebase indexing predictable.
+
+## Storage Engine
+
+Vectors are persisted in `.contextualizer/vector.db` with `better-sqlite3`.
+
+The database uses:
+
+- `PRAGMA journal_mode = WAL` for concurrent-friendly write behavior.
+- A `files` table keyed by path and hash.
+- A `chunks` table containing symbol metadata, line ranges, source snippets, and vector BLOBs.
+- Bulk transactional writes with `db.transaction()` so chunk inserts are committed as a batch.
+- Binary vector storage via `Buffer.from(embedding.buffer)`.
+
+At search time, BLOBs are reconstructed as typed arrays:
+
+```js
+new Float32Array(
+  row.embedding.buffer,
+  row.embedding.byteOffset,
+  row.embedding.byteLength / Float32Array.BYTES_PER_ELEMENT
+);
+```
+
+Cosine similarity is computed over typed arrays and the top matches are ranked in memory.
+
+## Benchmark
+
+Run the search latency benchmark against the current directory:
+
+```bash
+node scripts/benchmark.js
+```
+
+Run against another indexed target:
+
+```bash
+node scripts/benchmark.js ./src "authentication logic"
+```
+
+The benchmark opens `.contextualizer/vector.db`, embeds a dummy query once, runs 1,000 cosine-similarity scans, and reports average, p50, p90, and p95 latency in milliseconds.
+
+Performance: achieves sub-0.0763ms p95 vector retrieval latency over 1,000 local queries.
+
+## Current Language Support
+
+The parser dispatch layer currently routes JavaScript-family files:
+
+- `.js`
+- `.jsx`
+- `.mjs`
+- `.cjs`
+
+Unsupported source extensions are skipped gracefully. The architecture is ready for additional Tree-sitter grammars through the parser router.
+
+## Local Package Testing
+
+From the repository root:
+
+```bash
+npm install
+npm link
+codebase-contextualizer --help
+codebase-contextualizer status . --json
+npm pack --dry-run
+npx --yes . --help
+```
+
+Remove the global link when finished:
+
+```bash
+npm unlink -g codebase-contextualizer-cli
+```
+
+## Engineering Notes
+
+- No external embedding APIs are used.
+- Unchanged files are not re-parsed or re-embedded.
+- Indexing skips source files over 1 MB, common binary extensions, and obvious minified bundles before they reach the parser.
+- Ctrl+C aborts active indexing, terminates worker threads, and closes open SQLite handles.
+- SQLite writes happen on the main thread after worker results return.
+- Search loads the local embedding model as a singleton for query vectors.
+- The system favors explicit typed-array and BLOB boundaries over generic JavaScript arrays.

package/bin/contextualizer.js

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+
+require("../index.js");

package/index.js

@@ -0,0 +1,248 @@
+#!/usr/bin/env node
+
+const { Command, InvalidArgumentError } = require("commander");
+const { isAbortError } = require("./src/abort");
+const { getDefaultWorkerCount, indexTarget, statusTarget } = require("./src/indexer");
+const { searchTarget } = require("./src/search");
+const { runCleanup } = require("./src/shutdown");
+
+const DEFAULT_HASH_CONCURRENCY = 16;
+const shutdownController = new AbortController();
+let shutdownStarted = false;
+
+async function shutdown(signalName) {
+  if (shutdownStarted) {
+    process.exit(signalName === "SIGTERM" ? 143 : 130);
+  }
+
+  shutdownStarted = true;
+  const error = new Error(`Interrupted by ${signalName}`);
+  error.name = "AbortError";
+  shutdownController.abort(error);
+
+  console.error("");
+  console.error("Interrupted. Cleaning up workers and database handles...");
+  await runCleanup();
+  process.exit(signalName === "SIGTERM" ? 143 : 130);
+}
+
+process.on("SIGINT", () => {
+  void shutdown("SIGINT");
+});
+
+process.on("SIGTERM", () => {
+  void shutdown("SIGTERM");
+});
+
+function parsePositiveInteger(value) {
+  const parsed = Number.parseInt(value, 10);
+
+  if (!Number.isInteger(parsed) || parsed < 1) {
+    throw new InvalidArgumentError("must be a positive integer");
+  }
+
+  return parsed;
+}
+
+function summarize(result, saved) {
+  return {
+    root: result.root,
+    cachePath: result.cachePath,
+    cacheExists: result.cacheExists,
+    saved,
+    counts: result.counts,
+    changes: result.changes,
+    database: result.database,
+    embeddings: result.embeddings
+      ? {
+          counts: result.embeddings.counts,
+          files: result.embeddings.files.map((file) => ({
+            relativePath: file.relativePath,
+            ok: file.ok,
+            skipped: Boolean(file.skipped),
+            reason: file.reason,
+            error: file.error,
+            chunkCount: file.chunks.length,
+          })),
+        }
+      : undefined,
+    errors: result.errors,
+  };
+}
+
+function printChangedFiles(label, files) {
+  if (files.length === 0) {
+    return;
+  }
+
+  console.log(`${label}:`);
+  for (const file of files.slice(0, 20)) {
+    console.log(`  ${file}`);
+  }
+
+  if (files.length > 20) {
+    console.log(`  ...and ${files.length - 20} more`);
+  }
+}
+
+function printResult(result, mode, json) {
+  const saved = mode === "index";
+
+  if (json) {
+    console.log(JSON.stringify(summarize(result, saved), null, 2));
+    return;
+  }
+
+  console.log(`${mode === "index" ? "Index complete" : "Status"}: ${result.root}`);
+  console.log(`Cache: ${result.cachePath}${result.cacheExists ? "" : " (new)"}`);
+  console.log(`Scanned files: ${result.counts.scanned}`);
+  console.log(`New: ${result.counts.new}`);
+  console.log(`Modified: ${result.counts.modified}`);
+  console.log(`Unchanged: ${result.counts.unchanged}`);
+  console.log(`Removed: ${result.counts.removed}`);
+
+  printChangedFiles("New files", result.changes.new);
+  printChangedFiles("Modified files", result.changes.modified);
+  printChangedFiles("Removed files", result.changes.removed);
+
+  if (result.embeddings) {
+    console.log(`Embedded chunks: ${result.embeddings.counts.chunks}`);
+    console.log(`Vector database: ${result.database.databasePath}`);
+    console.log(`Persisted chunks: ${result.database.chunksWritten}`);
+
+    for (const file of result.embeddings.files) {
+      if (!file.ok) {
+        console.log(`Failed to embed ${file.relativePath}: ${file.error}`);
+      } else if (file.skipped) {
+        console.log(`Skipped ${file.relativePath}: ${file.reason}`);
+      } else {
+        console.log(`Successfully embedded ${file.chunks.length} chunks from ${file.relativePath}`);
+      }
+    }
+  }
+
+  if (result.errors.length > 0) {
+    console.log("Traversal warnings:");
+    for (const error of result.errors.slice(0, 10)) {
+      console.log(`  ${error.path}: ${error.message}`);
+    }
+
+    if (result.errors.length > 10) {
+      console.log(`  ...and ${result.errors.length - 10} more`);
+    }
+  }
+}
+
+function trimSnippet(code, maxLines = 16) {
+  const lines = code.trim().split(/\r?\n/);
+  const clipped = lines.slice(0, maxLines);
+
+  if (lines.length > maxLines) {
+    clipped.push("...");
+  }
+
+  return clipped.join("\n");
+}
+
+function printSearchResults(result, json) {
+  if (json) {
+    console.log(JSON.stringify(result, null, 2));
+    return;
+  }
+
+  console.log(`Search: ${result.query}`);
+  console.log(`Target: ${result.root}`);
+  console.log(`Database: ${result.databasePath}`);
+
+  if (result.results.length === 0) {
+    console.log("No indexed chunks found.");
+    return;
+  }
+
+  for (const [index, match] of result.results.entries()) {
+    const lineRange = match.startLine === match.endLine
+      ? `${match.startLine}`
+      : `${match.startLine}-${match.endLine}`;
+
+    console.log("");
+    console.log(`${index + 1}. ${match.name} (${match.type})`);
+    console.log(`Score: ${match.score.toFixed(4)}`);
+    console.log(`File: ${match.file}`);
+    console.log(`Line: ${lineRange}`);
+    console.log("Code:");
+    console.log(trimSnippet(match.code));
+  }
+}
+
+async function runIndex(target, options) {
+  const result = await indexTarget(target, {
+    hashConcurrency: options.concurrency,
+    signal: shutdownController.signal,
+    workerCount: options.workers,
+  });
+
+  printResult(result, "index", options.json);
+}
+
+async function runStatus(target, options) {
+  const result = await statusTarget(target, {
+    hashConcurrency: options.concurrency,
+    signal: shutdownController.signal,
+  });
+
+  printResult(result, "status", options.json);
+}
+
+async function runSearch(query, target, options) {
+  const result = await searchTarget(target || ".", query, {
+    limit: options.limit,
+    signal: shutdownController.signal,
+  });
+
+  printSearchResults(result, options.json);
+}
+
+const program = new Command();
+
+program
+  .name("codebase-contextualizer")
+  .description("Index local codebases for offline semantic search.")
+  .version("0.1.0");
+
+program
+  .command("index")
+  .description("Walk a target directory, hash source files, and update the local cache.")
+  .argument("<target>", "directory to index")
+  .option("-c, --concurrency <number>", "concurrent file hash operations", parsePositiveInteger, DEFAULT_HASH_CONCURRENCY)
+  .option("-w, --workers <number>", "worker thread count for parsing and embedding, capped at 4", parsePositiveInteger, getDefaultWorkerCount())
+  .option("--json", "print machine-readable output")
+  .action(runIndex);
+
+program
+  .command("status")
+  .description("Show cache drift without writing cache updates.")
+  .argument("<target>", "directory to inspect")
+  .option("-c, --concurrency <number>", "concurrent file hash operations", parsePositiveInteger, DEFAULT_HASH_CONCURRENCY)
+  .option("--json", "print machine-readable output")
+  .action(runStatus);
+
+program
+  .command("search")
+  .description("Search indexed chunks with a local semantic query.")
+  .argument("<query>", "natural language query")
+  .argument("[target]", "indexed target directory", ".")
+  .option("-n, --limit <number>", "maximum ranked results", parsePositiveInteger, 5)
+  .option("--json", "print machine-readable output")
+  .action(runSearch);
+
+program.parseAsync(process.argv).catch(async (error) => {
+  await runCleanup();
+
+  if (isAbortError(error)) {
+    process.exitCode = 130;
+    return;
+  }
+
+  console.error(`Error: ${error.message}`);
+  process.exitCode = 1;
+});

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "codebase-contextualizer-cli",
+  "version": "1.0.0",
+  "description": "Local semantic code search CLI using Tree-sitter, worker threads, local embeddings, and SQLite vector storage.",
+  "main": "index.js",
+  "bin": {
+    "codebase-contextualizer": "bin/contextualizer.js"
+  },
+  "files": [
+    "bin",
+    "src",
+    "scripts",
+    "index.js",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "start": "node index.js"
+  },
+  "keywords": [
+    "cli",
+    "search",
+    "semantic",
+    "embeddings",
+    "vector",
+    "sqlite",
+    "tree-sitter",
+    "transformers"
+  ],
+  "author": "Roopak Yadav",
+  "license": "MIT",
+  "engines": {
+    "node": ">=20"
+  },
+  "dependencies": {
+    "@xenova/transformers": "^2.17.2",
+    "better-sqlite3": "^12.9.0",
+    "commander": "^14.0.3",
+    "ignore": "^7.0.5",
+    "tree-sitter": "^0.25.0",
+    "tree-sitter-javascript": "^0.25.0"
+  }
+}