devlensio 0.6.2 → 1.0.1

package/README.md

@@ -3,7 +3,11 @@
 [![npm: devlensio](https://img.shields.io/badge/npm-devlensio-cb3837?logo=npm)](https://www.npmjs.com/package/devlensio)
 [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
 
-The core engine behind [DevLens](https://github.com/devlensio/devlensOSS) — the codebase visualizer. It turns a TypeScript / JavaScript / React / Next.js / Node.js repository into a **typed code graph** with functional summaries, technical summaries, and security analysis on every node.
+The core engine behind [DevLens](https://github.com/devlensio/devlensOSS) — the codebase visualizer.
+
+Point it at any repository and it builds a **typed code graph**: every file, function, class, route, and data model becomes a *node*, connected by *edges* that show what imports what, what calls what, and how data flows. Each node gets an importance score and — optionally — an AI-generated functional, technical, and security summary.
+
+It understands **6 languages out of the box** — TypeScript, JavaScript, Python, Java, Go, and Rust — with framework-aware parsing (React, Next.js, Express, Django, FastAPI, Spring Boot, Gin, Axum, and more). No tree-sitter, no generic AST guesswork: every language uses its own mature, native parser.
 
 The user-facing tools — CLI, MCP server, Agent Skill, and Web UI — all live in [DevLens OSS](https://github.com/devlensio/devlensOSS) and consume this package.
 
@@ -11,34 +15,52 @@ The user-facing tools — CLI, MCP server, Agent Skill, and Web UI — all live
 
 ## What it does
 
+Analyzing a repo runs through a simple 7-step pipeline:
+
 ```
 Repo path
    │
-[1] Fingerprint     → detect language, framework, router, state manager, data layer
-[2] Route detection → extract routes (Next.js, React Router, Express, Fastify, Koa)
-[3] AST parsing      → walk every .ts/.tsx/.js/.jsx → extract nodes with types
-[4] Edge detection   → map CALLS, IMPORTS, PROP_PASS, WRITES_TO, and 12 more edge types
-[5] Scoring          → multi-pass importance scoring (no AI, deterministic)
-[6] Clustering       → assign cohesive clusters
-[7] Summarize (opt)  → topological LLM summaries — functional + technical + security
+[1] Fingerprint     → which language & framework? (React, Express, Django, Spring, Gin, Axum …)
+[2] Route detection → every URL the app serves, with the handler behind each route
+[3] AST parsing      → per-language parser walks the source → typed nodes (files, functions, classes, structs, traits …)
+[4] Edge detection   → links nodes: CALLS, IMPORTS, HANDLES, IMPLEMENTS, EXTENDS, READS_FROM, WRITES_TO + more
+[5] Scoring          → deterministic importance scoring (no AI)
+[6] Clustering       → groups related nodes into cohesive clusters
+[7] Summarize (opt)  → LLM summaries — functional + technical + security
    │
    ▼
-Graph saved to ~/.devlens  →  queried via the traversal API / CLI / MCP / UI
+Graph saved to ~/.devlens → queried via the traversal API / CLI / MCP / UI
 ```
 
 Structural analysis is fast and deterministic. Summarization is the only step that calls an LLM — and unchanged nodes are reused across commits (90%+ free on re-runs).
 
 ---
 
+## Supported languages & frameworks
+
+Every language uses a **native parser** — no tree-sitter. TypeScript/JavaScript is parsed inline by the engine; Python, Java, Go, and Rust each run a small language-native extractor that the engine orchestrates over JSON.
+
+| Language | Extractor | Runtime needed on the analyzing machine | Frameworks detected (routes + data layer) |
+| :-- | :-- | :-- | :-- |
+| **TypeScript / JavaScript** | Inline (`ts-morph`) | none | React (incl. React Router), Next.js, Express, Fastify, Koa, Hono, Elysia, Bun |
+| **Python** | Native (stdlib `ast`) | Python 3.11+ (private venv, auto-created) | Django, Flask, FastAPI, DRF · SQLAlchemy / Django ORM |
+| **Java** | Native (JavaParser) | JVM 17+ | Spring Boot, Quarkus · JPA |
+| **Go** | Native (prebuilt static binary) | none | net/http, Gin, Echo, Fiber · GORM, database/sql |
+| **Rust** | Native (prebuilt static binary) | none | Axum, Actix-web, Rocket, utoipa-axum · Diesel |
+
+**Zero-toolchain languages.** Go and Rust ship as prebuilt static binaries for Linux / macOS / Windows (amd64 + arm64), and the Java extractor ships as a prebuilt fat jar — no compiler, Maven, or Gradle needed where the analysis runs. Python creates a private `.venv` on install (idempotent, skipped if `python3` isn't found).
+
+---
+
 ## Install
 
 ```bash
-npm install devlensio
-# or
 bun add devlensio
+# or
+npm install devlensio
 ```
 
-Requires Node 18+ (or Bun). An LLM provider key is only needed for AI summarization — structural analysis works offline.
+The engine runs on **Bun** — its published entry imports from `bun` (used by the job queue), so plain-Node loading isn't supported. No LLM key is needed for structural analysis — it's fully offline and deterministic. Keys are only needed for AI summaries. See the language table above for per-language runtime requirements.
 
 ---
 
@@ -105,23 +127,52 @@ Also exported: all core types (`CodeNode`, `CodeEdge`, `NodeType`, `EdgeType`, `
 
 ## Node & edge types
 
-**Node types**
-
-| Type | What it represents |
-| :-- | :-- |
-| `COMPONENT` | React / UI component |
-| `HOOK` | React custom hook |
-| `FUNCTION` | Plain function |
-| `STATE_STORE` | State management (Zustand, Redux, etc.) |
-| `UTILITY` | Utility / helper module |
-| `FILE` | File-level node |
-| `ROUTE` | Application route |
-| `TEST` | Test file |
-| `THIRD_PARTY` | External dependency |
+**Node types** — everything the graph knows about
 
-**Edge types**
-
-`CALLS`, `IMPORTS`, `READS_FROM`, `WRITES_TO`, `PROP_PASS`, `EMITS`, `LISTENS`, `WRAPPED_BY`, `GUARDS`, `HANDLES`, `TESTS`, `USES`, `NEXTJS_API_CALL`, `NAVIGATES_TO`
+| Type | What it means | Where you'll see it |
+| :-- | :-- | :-- |
+| `COMPONENT` | A React / UI component — something that renders UI | TS/JS |
+| `HOOK` | A React custom hook (`useX`), including the state/functions it returns | TS/JS |
+| `FUNCTION` | Any plain function — helper, callback, utility, serverless handler | TS/JS, Python, Go, Rust |
+| `STATE_STORE` | A central state container (Zustand, Redux, …) | TS/JS |
+| `UTILITY` | A helper module — code that isn't UI and isn't a component | TS/JS |
+| `CLASS` | A class — a blueprint for objects (`class User {}`). JS ships props/state types & decorators for React class components | TS/JS, Python, Java |
+| `METHOD` | A function attached to a class — shown as `ClassName.method` | TS/JS, Python, Java, Go, Rust |
+| `INTERFACE` | A type contract — the shape implementing types must satisfy | Java, Go |
+| `ENUM` | A fixed set of named values (e.g. `Status.Active`) | Java, Rust |
+| `STRUCT` | A plain data structure with fields (record-like) | Go, Rust |
+| `TRAIT` | Rust's version of an interface — a set of behaviors a type can implement | Rust |
+| `IMPL_BLOCK` | A Rust `impl` block that adds methods & behavior to a type | Rust |
+| `FILE` | One source file — the root node the rest of that file attaches to | every language |
+| `ROUTE` | An application route — a URL the app serves, with the handler behind it | every language |
+| `TEST` | A test file (`.test.tsx`, `_test.go`, `test_*.py`, …) | every language |
+| `STORY` | A Storybook story file | TS/JS |
+| `GHOST` | An invisible "event" node that ties event emitters to listeners via `EMITS` / `LISTENS` | TS/JS event graph |
+| `THIRD_PARTY` | An external package/dependency — shown but not parsed | every language |
+| `MODULE` / `PACKAGE` | Namespace / package node | reserved — not emitted yet |
+
+**Edge types** — the arrows between nodes
+
+| Edge | What it means | Where you'll see it |
+| :-- | :-- | :-- |
+| `CALLS` | A calls B — one function/method invokes another | all languages |
+| `IMPORTS` | A file imports another file or package | all languages |
+| `READS_FROM` | A reads data from B — a state store, model, or DB query (`User.objects.get`, `session.query(User)`, `db.Find`) | state + data layers (TS/JS, Python, Java, Go ORM) |
+| `WRITES_TO` | A writes/updates data in B (`store.set(...)`, `user.save()`, `db.Create(...)`) | state + data layers (same as above) |
+| `PROP_PASS` | A React prop flows from a parent component to a child | TS/JS |
+| `EMITS` | A emits an event | TS/JS event graph |
+| `LISTENS` | A subscribes to an event | TS/JS event graph |
+| `WRAPPED_BY` | A is wrapped by B (e.g. a context provider wraps its consumers) | TS/JS |
+| `GUARDS` | A guards B — a route guard / middleware protecting a route or handler | route layers |
+| `HANDLES` | A route is handled by a handler — the controller/viewset/function behind a URL | all languages' routes |
+| `TESTS` | A test file verifies the code it points to | all languages |
+| `USES` | A JSX component uses an external function/hook internally | TS/JS |
+| `NEXTJS_API_CALL` | A component fetches a Next.js API route | TS/JS Next.js |
+| `NAVIGATES_TO` | Client-side navigation points to a route | TS/JS |
+| `IMPLEMENTS` | A type implements a contract — class implements an interface, Go struct implements an interface, Rust type implements a trait | Java, Go, Rust, Python (ABC/Protocol) |
+| `EXTENDS` | A inherits / embeds B — class extends a base class, Go struct embeds another, Rust supertrait | class-based languages |
+| `EXPORTS` | reserved — declared, not yet emitted | — |
+| `THROWS` | reserved — declared, not yet emitted | — |
 
 Each node carries: **importance score** + **functional summary** + **technical summary** + **security assessment** (severity + notes).
 
@@ -236,9 +287,10 @@ A custom-model entry is always available — essential for OpenRouter's huge cat
 src/
 ├── fingerprint/    # Detect framework, language, router, state, data layer
 ├── filesystem/     # Route detection (Next.js, React Router, Express, etc.)
-├── parser/         # AST extraction → nodes (ts-morph)
+├── parser/         # AST extraction → nodes (ts-morph) — the inline JS/TS extractor
+├── extractors/     # Extractor registry, language detection, subprocess runner
 ├── graph/          # Edge detectors, traversal API, lookup maps
-├── scoring/        # Multi-pass importance scoring + noise filtering
+├── scoring/        # Multi-pass importance scoring + noise filtering + pruning
 ├── clustering/     # Cohesive cluster computation
 ├── summarizer/     # LLM summarization pipeline, prompts, checkpoints
 │   └── providers/  # Generic OpenAI & Anthropic clients, model discovery
@@ -247,7 +299,13 @@ src/
 ├── storage/        # File-based graph persistence (~/.devlens)
 ├── config/         # Provider config resolution (types, catalog, writer, env)
 ├── server/         # HTTP API server (consumed by Web UI)
-└── debug/          # Export and dev utilities
+└── debug/          # Export and validation utilities
+
+extractors/             # Native subprocess extractors (not part of the TS build)
+├── python/             # Python extractor (stdlib `ast`, pip package, auto-venv)
+├── java/               # Java extractor (JavaParser, prebuilt fat jar)
+├── go/                 # Go extractor (go/ast + go/types, prebuilt static binary)
+└── rust/               # Rust extractor (syn, prebuilt static binary)
 ```
 
 ---

package/dist/extractors/detectLanguage.d.ts

@@ -0,0 +1,2 @@
+import { Language } from "../index.js";
+export declare function detectLanguage(repoPath: string): Language;

package/dist/extractors/detectLanguage.js

@@ -0,0 +1,33 @@
+// Detects the primary langauge of a repo by checking for the manifest files
+import fs from "fs";
+import path from "path";
+export function detectLanguage(repoPath) {
+    // first for JS/TS, check for package.json first then use tsconfig.json to distinguish between JS and TS
+    if (fs.existsSync(`${repoPath}/package.json`)) {
+        if (fs.existsSync(`${repoPath}/tsconfig.json`)) {
+            return "typescript";
+        }
+        return "javascript";
+    }
+    // secondly lets check for python.
+    if (fs.existsSync(path.join(repoPath, "requirements.txt")) ||
+        fs.existsSync(path.join(repoPath, "pyproject.toml")) ||
+        fs.existsSync(path.join(repoPath, "setup.py"))) {
+        return "python";
+    }
+    // Go
+    if (fs.existsSync(path.join(repoPath, "go.mod"))) {
+        return "go";
+    }
+    // Rust
+    if (fs.existsSync(path.join(repoPath, "Cargo.toml"))) {
+        return "rust";
+    }
+    // Java
+    if (fs.existsSync(path.join(repoPath, "pom.xml")) ||
+        fs.existsSync(path.join(repoPath, "build.gradle")) ||
+        fs.existsSync(path.join(repoPath, "build.gradle.kts"))) {
+        return "java";
+    }
+    return "unknown";
+}

package/dist/extractors/index.d.ts

@@ -0,0 +1,6 @@
+import { Language } from "../types.js";
+import { ExtractorResult, LanguageExtractor } from "./types.js";
+export declare function defaultParseResult(stdout: string): ExtractorResult;
+export declare function getExtractor(language: Language): LanguageExtractor | undefined;
+export declare const INLINE_LANGUAGES: Set<Language>;
+export declare function commandExists(command: string): boolean;

package/dist/extractors/index.js

@@ -0,0 +1,155 @@
+// src/extractors/index.ts
+import { fileURLToPath } from "url";
+import fs from "fs";
+import path from "path";
+// runner.ts is the main file entry point for the extractors. It contains the runExtractor() function which is called by the pipeline.
+// Extractor registry - maps langauge to their extractor config.
+// JS/TS is NOT here (it will be handled inline by the runner.ts)
+// All subprocesses extractors must return the JSON (ExtractorResults) format
+export function defaultParseResult(stdout) {
+    return JSON.parse(stdout);
+}
+// Subprocess Extractor Registry
+// These extractors are spawned as child processes.
+// ── extractor artifact resolution ────────────────────────────────────────────
+// The subprocess extractors (python venv, java jar, go/rust static binaries)
+// are DATA files inside the devlensio package. In a normal install they sit at
+// `<pkg>/extractors/…` and resolve relative to this module. But when devlensio
+// is BUNDLED into a standalone binary (`bun build --compile`, which is how the
+// `@devlensio/cli` and MCP server ship), `import.meta.url` points at the binary
+// itself, so the URL-relative path resolves to `/extractors/…` (nonexistent).
+// The fallbacks below find the real extractor root for that case.
+function resolveExtractorsRoot() {
+    // 1. Env override (CI / unusual layouts): DEVLENS_EXTRACTORS_DIR = the
+    //    `extractors/` directory itself.
+    const env = process.env.DEVLENS_EXTRACTORS_DIR;
+    if (env) {
+        const candidate = path.resolve(env);
+        if (fs.existsSync(path.join(candidate, "python")) && fs.existsSync(path.join(candidate, "java"))) {
+            return candidate;
+        }
+    }
+    // 2. Normal install: `<pkg>/dist/extractors/../../extractors` = `<pkg>/extractors`.
+    const viaModule = fileURLToPath(new URL("../../extractors", import.meta.url));
+    if (fs.existsSync(viaModule))
+        return viaModule;
+    // 3. Bundled binary: walk up from this bundle's own directory AND from the
+    //    cwd to find `<…>/node_modules/devlensio/extractors`. Covers running the
+    //    compiled CLI inside a project that depends on devlensio, and global
+    //    installs where devlensio sits hoisted beside the CLI package.
+    const scanRoots = [
+        fileURLToPath(new URL(".", import.meta.url)),
+        process.cwd(),
+    ];
+    for (const start of scanRoots) {
+        let dir = path.resolve(start);
+        for (;;) {
+            const candidate = path.join(dir, "node_modules", "devlensio", "extractors");
+            if (fs.existsSync(candidate))
+                return candidate;
+            const parent = path.dirname(dir);
+            if (parent === dir)
+                break;
+            dir = parent;
+        }
+    }
+    return null;
+}
+const extractorsRoot = resolveExtractorsRoot();
+/** Absolute path to an extractor artifact under the resolved root (or null). */
+function extractorArtifact(rel) {
+    return extractorsRoot ? path.join(extractorsRoot, rel) : null;
+}
+function resolvePythonCommand() {
+    const venvBin = process.platform === "win32" ? "Scripts/python.exe" : "bin/python";
+    const venvPython = extractorArtifact(`python/.venv/${venvBin}`) ??
+        // Last resort: URL-relative (pre-fix behavior for direct-from-dist runs).
+        fileURLToPath(new URL(`../../extractors/python/.venv/${venvBin}`, import.meta.url));
+    return fs.existsSync(venvPython) ? venvPython : "python3";
+}
+function platformDir() {
+    return process.platform === "win32"
+        ? "windows-amd64"
+        : process.platform === "darwin"
+            ? `darwin-${process.arch === "arm64" ? "arm64" : "amd64"}`
+            : `linux-${process.arch === "arm64" ? "arm64" : "amd64"}`;
+}
+function resolveJavaJarPath() {
+    const rel = "java/devlens_java_extractor.jar";
+    return extractorArtifact(rel) ?? fileURLToPath(new URL(`../../extractors/${rel}`, import.meta.url));
+}
+function resolveGoBinaryPath() {
+    const exe = process.platform === "win32" ? ".exe" : "";
+    const rel = `go/bin/${platformDir()}/devlens_go_extractor${exe}`;
+    return extractorArtifact(rel) ?? fileURLToPath(new URL(`../../extractors/${rel}`, import.meta.url));
+}
+function resolveRustBinaryPath() {
+    const exe = process.platform === "win32" ? ".exe" : "";
+    const rel = `rust/bin/${platformDir()}/devlens_rust_extractor${exe}`;
+    return extractorArtifact(rel) ?? fileURLToPath(new URL(`../../extractors/${rel}`, import.meta.url));
+}
+const SUBPROCESS_EXTRACTORS = {
+    python: {
+        language: "python",
+        // Absolute venv python when the postinstall created one (works from
+        // node_modules); fall back to PATH python3 in dev/other setups.
+        command: resolvePythonCommand(),
+        args: ["-m", "devlens_extractors_python"],
+        parseResult: defaultParseResult,
+    },
+    java: {
+        language: "java",
+        command: "java",
+        // Absolute path — the runner spawns with cwd=repoPath, so a bare jar
+        // name would be looked up inside the analyzed repo. fileURLToPath
+        // also decodes percent-escapes (spaces in the path).
+        args: ["-jar", resolveJavaJarPath()],
+        parseResult: defaultParseResult,
+    },
+    go: {
+        language: "go",
+        // Absolute per-platform static binary path — the runner spawns with
+        // cwd=repoPath, so a bare name would be looked up inside the analyzed
+        // repo. Cross-compiled at publish time by prepack → build.mjs.
+        command: resolveGoBinaryPath(),
+        args: [],
+        parseResult: defaultParseResult,
+    },
+    rust: {
+        language: "rust",
+        // Absolute per-platform static binary path — the runner spawns with
+        // cwd=repoPath, so a bare name would be looked up inside the analyzed
+        // repo. Cross-compiled at publish time by prepack → build.mjs.
+        command: resolveRustBinaryPath(),
+        args: [],
+        parseResult: defaultParseResult,
+    }
+};
+// Public API to get the extractor config
+export function getExtractor(language) {
+    return SUBPROCESS_EXTRACTORS[language];
+}
+// langauges handled inline meaning JS/TS
+export const INLINE_LANGUAGES = new Set(["javascript", "typescript"]);
+// Does `command` resolve? Absolute path → existsSync; bare name → PATH scan
+// (cross-platform: PATHEXT on win32).
+export function commandExists(command) {
+    if (command.includes("/") || command.includes("\\")) {
+        return fs.existsSync(command);
+    }
+    const pathEnv = process.env.PATH || "";
+    const exts = process.platform === "win32"
+        ? (process.env.PATHEXT || ".EXE;.CMD;.BAT;.COM").toLowerCase().split(";")
+        : [""];
+    for (const dir of pathEnv.split(process.platform === "win32" ? ";" : ":")) {
+        if (!dir)
+            continue;
+        for (const ext of exts) {
+            if (fs.existsSync(path.join(dir, command + ext))
+                || fs.existsSync(path.join(dir, command.toUpperCase() + ext))) {
+                return true;
+            }
+        }
+    }
+    return false;
+}

package/dist/extractors/runner.d.ts

@@ -0,0 +1,7 @@
+import { ExtractorInput, ExtractorResult, LanguageExtractor } from "./types.js";
+export declare function runSubprocessExtractor(extractor: LanguageExtractor, input: ExtractorInput, timeoutMs?: number): Promise<ExtractorResult>;
+export declare function runInlineExtractor(input: ExtractorInput, onStep?: (step: "fingerprint" | "filesystem" | "parse" | "edges" | "scoring") => void): Promise<ExtractorResult>;
+export declare function runExtractor(repoPath: string, options?: {
+    includeThirdPartyLibs?: string[];
+    onStep?: (step: "fingerprint" | "filesystem" | "parse" | "edges" | "scoring") => void;
+}): Promise<ExtractorResult>;

package/dist/extractors/runner.js

@@ -0,0 +1,194 @@
+// src/extractors/runner.ts
+//
+// The Runner — two execution modes:
+//   1. Subprocess: spawns Python/Java/Go/Rust extractor as child process
+//   2. Inline:     calls existing ts-morph code directly (no subprocess)
+//
+// The pipeline calls runExtractor() which auto-detects language and routes.
+// This is basically the entry point for the extractor execution.
+/*
+What Actually Happens (Timeline)
+spawn() creates the child process and returns immediately. The child is now running but waiting for input on stdin. Meanwhile, your Node.js code continues executing line by line.
+
+Here's the real execution order:
+
+
+TIME 0ms:   spawn("python3", ["-m", "devlens_extractors_python"])
+            → child process starts, waits for stdin input
+
+TIME 1ms:   child.stdout.on("data", ...)   ← registers callback (doesn't run yet)
+TIME 2ms:   child.stderr.on("data", ...)   ← registers callback (doesn't run yet)
+TIME 3ms:   setTimeout(...)                ← registers timer (doesn't fire yet)
+TIME 4ms:   child.on("error", ...)         ← registers callback
+TIME 5ms:   child.on("close", ...)         ← registers callback
+
+TIME 6ms:   child.stdin.write(JSON.stringify(input))  ← NOW we send the repoPath
+TIME 7ms:   child.stdin.end()              ← tells extractor "done sending"
+
+            ← runSubprocessExtractor returns the Promise here
+            ← Node.js moves on to other work
+
+            ... child process is running, parsing the repo ...
+
+TIME 5000ms: child finishes, writes JSON to stdout
+             → "data" handler fires, collects stdout
+
+TIME 5001ms: child exits with code 0
+             → "close" handler fires
+             → parseResult(stdout) runs
+             → resolve(result) — the Promise resolves
+*/
+import { spawn } from "child_process";
+import fs from "fs";
+import path from "path";
+import { analyzeFingerprint } from "../fingerprint/index.js";
+import { analyzeFilesystem } from "../filesystem/index.js";
+import { routesToCodeNodes } from "../pipeline/index.js";
+import { parseRepo } from "../parser/index.js";
+import { buildThirdPartyNodes } from "../graph/thirdPartyLibs.js";
+import { detectEdges } from "../graph/index.js";
+import { detectLanguage } from "./detectLanguage.js";
+import { getExtractor, INLINE_LANGUAGES, commandExists } from "./index.js";
+// 1. Subprocess Extractor
+// What it does: It will start a process for the given extractor, send it the input json, and wait for the output json. if the process times out, it will kill the process and return error result. If the process exits with non-zero code, it will return error result. If the process exits with zero code, it will parse the output json and return the result.
+export async function runSubprocessExtractor(extractor, input, timeoutMs = 10 * 60 * 1000) {
+    return new Promise((resolve, reject) => {
+        // Friendly guard for artifact-based extractors (java -jar ...):
+        // the jar path is resolved from the package location, not repoPath.
+        const jarArg = extractor.args.find((a) => a.endsWith(".jar"));
+        if (jarArg && !fs.existsSync(jarArg)) {
+            reject(new Error(`${extractor.language} extractor artifact not found: ${jarArg}. ` +
+                `Build it first: node extractors/${extractor.language}/build.mjs`));
+            return;
+        }
+        // Runtime prerequisites on the installing machine — friendly errors
+        // instead of a raw spawn ENOENT.
+        if (extractor.language === "java" && !commandExists("java")) {
+            reject(new Error("java extractor requires a Java 17+ runtime (JVM) on PATH — " +
+                "install a JDK (e.g. Adoptium Temurin) and retry."));
+            return;
+        }
+        if (extractor.language === "python" && !commandExists(extractor.command)) {
+            reject(new Error("python extractor unavailable: no Python 3.11+ found. Install Python, " +
+                "or bootstrap the extractor venv: node extractors/python/setup.mjs"));
+            return;
+        }
+        if (extractor.language === "go" && !fs.existsSync(extractor.command)) {
+            reject(new Error(`go extractor binary not found: ${extractor.command}. ` +
+                `Build it first: node extractors/go/build.mjs (requires the Go toolchain)`));
+            return;
+        }
+        if (extractor.language === "rust" && !fs.existsSync(extractor.command)) {
+            reject(new Error(`rust extractor binary not found: ${extractor.command}. ` +
+                `Build it first: node extractors/rust/build.mjs (requires the Rust toolchain)`));
+            return;
+        }
+        const child = spawn(extractor.command, extractor.args, { stdio: ["pipe", "pipe", "pipe"], cwd: input.repoPath });
+        let stdout = "";
+        let stderr = "";
+        child.stdout.on("data", (data) => {
+            stdout += data.toString();
+        });
+        child.stderr.on("data", (data) => {
+            stderr += data.toString();
+        });
+        const timer = setTimeout(() => {
+            child.kill("SIGTERM");
+            reject(new Error(`${extractor.language} extractor timed out after ${timeoutMs}ms`));
+        }, timeoutMs);
+        child.on("close", (code) => {
+            clearTimeout(timer);
+            if (code !== 0) {
+                // Log stderr for debugging but don't crash — return error result
+                reject(new Error(`${extractor.language} extractor exited with code ${code}.\n${stderr}`));
+                return;
+            }
+            try {
+                const result = extractor.parseResult(stdout);
+                resolve(result);
+            }
+            catch (err) {
+                reject(new Error(`Failed to parse ${extractor.language} extractor output: ` +
+                    `${err instanceof Error ? err.message : String(err)}`));
+            }
+        });
+        // Send input as JSON on stdin, then close stdin to signal "done sending"
+        child.stdin.write(JSON.stringify(input));
+        child.stdin.end();
+    });
+}
+;
+// 2. Inline Extractor
+// calls existing ts-morph code directly (no subprocess). This is used for TypeScript/JavaScript projects. It returns the result directly without spawning a child process.
+// This replicates steps 1-6 of the current analyzePipeline and wraps them
+// into an ExtractorResult.
+export async function runInlineExtractor(input, onStep) {
+    const { repoPath, options } = input;
+    const absoluteRepoPath = path.resolve(repoPath);
+    // Step 1: fingerprint
+    onStep?.("fingerprint");
+    const fingerprint = analyzeFingerprint(absoluteRepoPath);
+    // Step 2: Filesystem / routes
+    onStep?.("filesystem");
+    const routes = analyzeFilesystem(absoluteRepoPath, fingerprint);
+    // Step 3: Convert routes -> CodeNodes (so that they can join the graph)
+    // (no onStep here — it's part of the filesystem step)
+    let routeNodes = routesToCodeNodes(routes, absoluteRepoPath);
+    // Step 4: Parse source files into nodes 
+    onStep?.("parse");
+    const parserResult = parseRepo(absoluteRepoPath);
+    // Step 5: Build Third party nodes (if options is provided)
+    const thirdPartyNodes = buildThirdPartyNodes(absoluteRepoPath, options.includeThirdPartyLibs || []);
+    //Step 6: We have all the Nodes, now build the edges.
+    onStep?.("edges");
+    const edgeResult = detectEdges([...parserResult.nodes, ...routeNodes, ...thirdPartyNodes], routes, absoluteRepoPath, fingerprint);
+    // Step 7: Filter API route nodes without handlers (JS-specific cleanup)
+    routeNodes = routeNodes.filter(routeNode => {
+        if (routeNode.metadata.routeNodeType === "API_ROUTE") {
+            return edgeResult.edges.some((edge) => edge.type === "HANDLES" && edge.from === routeNode.id);
+        }
+        return true;
+    });
+    // Step 8: Assemble final nodes and edges
+    const allNodes = [
+        ...parserResult.nodes,
+        ...routeNodes,
+        ...thirdPartyNodes,
+        ...edgeResult.ghostNodes
+    ];
+    const allEdges = edgeResult.edges;
+    const stats = {
+        totalFiles: parserResult.stats.totalFiles,
+        totalNodes: allNodes.length,
+        skippedFiles: parserResult.stats.skippedFiles,
+    };
+    return {
+        fingerprint,
+        nodes: allNodes,
+        edges: allEdges,
+        routes,
+        stats,
+        errors: [],
+    };
+}
+// 3. Dispatch - auto detect other langauges (apart from JS/TS) and routes
+export async function runExtractor(repoPath, options) {
+    const language = detectLanguage(repoPath);
+    const input = {
+        repoPath,
+        options: {
+            includeThirdPartyLibs: options?.includeThirdPartyLibs || []
+        }
+    };
+    if (INLINE_LANGUAGES.has(language)) {
+        return runInlineExtractor(input, options?.onStep);
+    }
+    // For other languages (python, Go, Rust, Java), we need to spawn a subprocess
+    const extractor = getExtractor(language);
+    if (!extractor) {
+        throw new Error(`No extractor registered for language: "${language}". ` +
+            `This language may not be supported yet.`);
+    }
+    ;
+    return runSubprocessExtractor(extractor, input);
+}

package/dist/extractors/types.d.ts

@@ -0,0 +1,37 @@
+import { Language, BackendRouteNode, CodeEdge, CodeNode, ProjectFingerprint, RouteNode } from "../types.js";
+export interface ExtractorOptions {
+    includeThirdPartyLibs?: string[];
+}
+export interface ExtractorInput {
+    repoPath: string;
+    options: ExtractorOptions;
+}
+export interface ExtractorStats {
+    totalFiles: number;
+    totalNodes: number;
+    skippedFiles: number;
+}
+export interface ExtractorError {
+    file: string;
+    error: string;
+}
+export interface ExtractorResult {
+    fingerprint: ProjectFingerprint;
+    nodes: CodeNode[];
+    edges: CodeEdge[];
+    routes?: (RouteNode | BackendRouteNode)[];
+    stats: ExtractorStats;
+    errors: ExtractorError[];
+}
+export interface LanguageExtractor {
+    language: Language;
+    command: string;
+    args: string[];
+    /**
+     * Parse the raw stdout from the extractor process into ExtractorResult.
+     * The orchestrator calls this after the process exits successfully.
+     *
+     * Throwing here marks the extraction as failed.
+     */
+    parseResult: (stdout: string) => ExtractorResult;
+}

package/dist/extractors/types.js

@@ -0,0 +1,8 @@
+// src/extractors/types.ts
+//
+// The Extractor Contract — defines the interface between the Node.js
+// orchestrator and the native-language extractors (Python, Java, Go, Rust).
+//
+// Communication is JSON over stdin/stdout. See expansion-tracker/contract.html
+// for the full spec.
+export {};

package/dist/graph/buildLookup.d.ts

@@ -6,5 +6,6 @@ export interface LookupMaps {
     storeNodes: CodeNode[];
     thirdPartyNodesByName: Map<string, CodeNode>;
     thirdPartyImportAliases: Map<string, Map<string, string>>;
+    localImportSymbols: Map<string, Map<string, string>>;
 }
 export declare function buildLookupMaps(codeNodes: CodeNode[]): LookupMaps;

package/dist/graph/buildLookup.js

@@ -5,6 +5,7 @@ export function buildLookupMaps(codeNodes) {
     const storeNodes = [];
     const thirdPartyNodesByName = new Map();
     const thirdPartyImportAliases = new Map();
+    const localImportSymbols = new Map();
     for (const node of codeNodes) {
         if (node.type === "THIRD_PARTY") {
             thirdPartyNodesByName.set(node.name, node);
@@ -28,5 +29,5 @@ export function buildLookupMaps(codeNodes) {
             storeNodes.push(node);
         }
     }
-    return { nodesByName, nodesByFile, fileNodesByPath, storeNodes, thirdPartyNodesByName, thirdPartyImportAliases };
+    return { nodesByName, nodesByFile, fileNodesByPath, storeNodes, thirdPartyNodesByName, thirdPartyImportAliases, localImportSymbols };
 }