pi-lens 3.6.5 → 3.6.7

package/CHANGELOG.md

@@ -2,6 +2,93 @@
 
 All notable changes to pi-lens will be documented in this file.
 
+## [3.6.7] - 2026-04-04
+
+### Fixed
+- **LSP `ERR_STREAM_DESTROYED` crash** — When an LSP process (e.g. rust-analyzer) exits, Node.js emits
+  `'error'` events on the destroyed stdio streams. Without listeners these became uncaught exceptions
+  that crashed the extension. Added persistent `error` listeners to `stdin`, `stdout`, and `stderr`
+  before handing them to `vscode-jsonrpc`, covering the post-`connection.dispose()` window.
+  Same guard added to `NativeRustCoreClient` stdin writes.
+
+### Added
+- **Rust performance core (`pi-lens-core`)** — Optional Rust binary for CPU-intensive operations.
+  All features fall back to TypeScript automatically if the binary is not available (it is **not**
+  built automatically on `npm install` — run `npm run rust:build` once if you have Rust installed).
+  - **File scanning** — ripgrep’s `ignore` crate for `.gitignore`-aware project scanning
+  - **Similarity detection** — parallel 57×72 state-matrix index, persisted to
+    `.pi-lens/rust-index.json` between invocations (fixes in-memory cache that reset on every
+    process spawn)
+  - **Tree-sitter queries** — TypeScript and Rust AST queries via the binary
+  - **`NativeRustCoreClient`** — TypeScript wrapper with `isBinaryStale()` freshness detection,
+    JSON-IPC over stdin/stdout
+  - **Integration tests** — `npm run rust:test:integration` (37 assertions across all commands)
+
+- **Rust similarity fast-path in dispatch runner** — `similarity.ts` now tries the Rust binary
+  first (scan → build index → query), falls through to the TypeScript implementation on any
+  failure. Feature flag `USE_RUST = true` at top of file.
+
+### Changed
+- **Similarity threshold raised from 0.75 → 0.90** — Empirical evaluation showed that below 0.90
+  false positives (structurally similar but semantically unrelated functions) outnumber true
+  positives with the current 57×72 matrix resolution. Applies to both the dispatch runner and
+  `/lens-booboo`.
+
+- **Rust `kind_id` mapping improved** — Replaced `kind % dim` modulo (caused up to 4 unrelated
+  node types to share one matrix slot) with even-distribution across named slots plus a dedicated
+  last slot for anonymous punctuation tokens. Max named-slot collisions reduced from 4 to 3;
+  unnamed tokens no longer pollute named slots.
+
+### Fixed (Rust)
+- `tree_sitter_rust::language_rust()` → `language()` (correct API for tree-sitter-rust 0.21)
+- `FunctionInfo` missing `#[derive(Clone)]` — caused compile error in `find_similar_to`
+- `export function foo()` was missed by the index builder — TypeScript wraps exported functions
+  in `export_statement`; replaced flat top-level walk with recursive `collect_functions()`
+- `find_similar_to` returned only the first function in a file — changed `find` to `filter`
+- `tempfile` moved from `[dependencies]` to `[dev-dependencies]`
+- Deleted orphan `test_lsp.rs` (intentional type errors caused rust-analyzer to crash the LSP stream)
+
+### Repository
+- Rust source (`rust/src/`, `rust/Cargo.toml`) added to npm `files` whitelist so users can build
+  the binary from an npm-installed package
+- Removed stale `src/main.rs` rule from root `.gitignore` (no such file at repo root)
+- Untracked `docs/plans/2025-04-03-auto-install-logging.md` (committed before `*.md` exclusion rule)
+
+---
+
+## [3.6.3] - 2026-04-03
+
+### Removed (Dead Code Cleanup)
+- **Deleted unused interviewer tool** — Browser-based interview with diff confirmation was never used:
+  - Removed `clients/interviewer.ts` (290 lines)
+  - Removed `clients/interviewer-templates.ts` (240 lines)
+  - Removed initialization from `index.ts`
+  
+- **Deleted deprecated commands** — All were superseded by `/lens-booboo`:
+  - `/lens-booboo-fix` command (fix-from-booboo.ts, 430 lines) — showed warning to use `/lens-booboo`
+  - `/lens-fix-simplified` command (fix-simplified.ts, 770 lines) — never registered, unused
+  - `/lens-rate` command (rate.ts, 340 lines) — showed warning to use `/lens-booboo`
+  - `/lens-booboo-refactor` command (refactor.ts, 207 lines) — depended on removed interviewer tool
+
+- **Deleted duplicate safe-spawn module**:
+  - Removed `clients/safe-spawn-async.ts` (220 lines) — 100% duplicate of functions in `safe-spawn.ts`
+  - All imports already used `safe-spawn.ts`, making `safe-spawn-async.ts` pure dead code
+
+### Test Suite Overhaul
+- **Removed ~85 wasteful/broken test files**:
+  - "Is tool available" tests (8 files) — just checked if external CLIs installed
+  - Heavy integration tests (2 files) — 5s timeouts, full codebase scans
+  - Broken LSP tests (7 files) — import path errors
+  - Broken runner tests (7 files) — thin CLI wrappers with wrong imports
+  - Trivial utility tests (5 files) — file extension parsing, string sanitization
+  
+- **Added meaningful integration tests**:
+  - `tests/clients/dispatch/dispatcher-flow.test.ts` — Runner registration, execution, delta mode, conditional runners
+  - `tests/extension-hooks.test.ts` — pi API: tool/command/flag registration, event handlers
+  - `tests/mocks/runner-factory.ts` — Mock runners for testing without real CLI tools
+
+- **Results:** 22 tests passing in 1.2s (was 104 tests in ~18s with 48 failures)
+
 ## [3.6.2] - 2026-04-02
 
 ### Added

package/README.md

@@ -128,7 +128,7 @@ Enable full Language Server Protocol support with `--lens-lsp`:
 | **Config** | YAML, JSON, Prisma |
 | **Web** | Vue, Svelte, CSS/SCSS/Sass/Less |
 
-**Auto-installation (8 tools):** TypeScript, Python, Biome, Ruff, and analysis tools (Madge, jscpd, ast-grep, Knip) auto-install on first use to `.pi-lens/tools/`. Other LSP servers are launched via `npx` when available or require manual installation.
+**Auto-installation (8 tools):** TypeScript, Python, Biome, Ruff, and analysis tools (Madge, jscpd, ast-grep, Knip) auto-install on first use to `.pi-lens/tools/`. Other LSP servers require manual installation or are launched via `npx` when available.
 
 **Usage:**
 ```bash
@@ -196,7 +196,7 @@ pi-lens uses a **dispatcher-runner architecture** for extensible multi-language
 | **shellcheck** | Shell | 20 | Warning | Bash/sh/zsh/fish linting |
 | **python-slop** | Python | 25 | Warning | AI slop detection (~40 patterns) |
 | **spellcheck** | Markdown | 30 | Warning | Typo detection in docs |
-| **similarity** | TS | 35 | Warning | Semantic duplicate detection (structural similarity) |
+| **similarity** | TS | 35 | Warning | Semantic duplicate detection (≥90% structural similarity, Rust-accelerated when available) |
 | **architect** | All | 40 | Warning | Architectural rule violations |
 | **go-vet** | Go | 50 | Warning | Go static analysis |
 | **rust-clippy** | Rust | 50 | Warning | Rust linting |
@@ -327,7 +327,7 @@ Full codebase analysis with **10 tracked runners** producing a comprehensive rep
 |---|--------|---------------|
 | 1 | **ast-grep (design smells)** | Structural issues (empty catch, no-debugger, etc.) |
 | 2 | **ast-grep (similar functions)** | Duplicate function patterns across files |
-| 3 | **semantic similarity (Amain)** | 57×72 matrix semantic clones (>75% similarity) |
+| 3 | **semantic similarity (Amain)** | 57×72 matrix semantic clones (≥90% similarity) |
 | 4 | **complexity metrics** | Low MI, high cognitive complexity, AI slop indicators |
 | 5 | **TODO scanner** | TODO/FIXME annotations and tech debt markers |
 | 6 | **dead code (Knip)** | Unused exports, files, dependencies |
@@ -501,6 +501,9 @@ pi-lens/
 ├── commands/         # /lens-booboo, /lens-format commands
 ├── docs/             # Documentation
 ├── rules/            # AST-grep rules
+├── rust/             # Optional Rust core for performance acceleration
+│   ├── src/          # Rust source (pi-lens-core binary)
+│   └── Cargo.toml
 ├── skills/           # Built-in pi skills
 ├── index.ts          # Main extension entry point
 └── package.json
@@ -510,6 +513,40 @@ See source for detailed structure.
 
 ---
 
+## Rust Core (Optional)
+
+pi-lens includes a **Rust performance core** (`pi-lens-core`) for CPU-intensive operations. It is entirely optional — all features fall back to the TypeScript implementation automatically if the binary is not available.
+
+**What it accelerates:**
+- **File scanning** — Uses ripgrep's `ignore` crate for fast, `.gitignore`-aware project scanning (~10× faster than glob)
+- **Similarity detection** — Parallel 57×72 state-matrix computation and index querying
+- **Tree-sitter queries** — Runs TypeScript and Rust AST queries directly from the binary
+
+**Status:** Does not work out of the box after `npm install`. The source is included in the package so you can build it yourself if you have Rust installed.
+
+**Build the binary (one-time):**
+```bash
+# Requires Rust toolchain — https://rustup.rs
+npm run rust:build          # release build (recommended)
+npm run rust:build:debug    # debug build
+```
+
+Once built, pi-lens will automatically use the Rust binary and fall back to TypeScript if it is absent, outdated, or fails.
+
+**Verify the binary is being used:**
+```bash
+node -e "import('./clients/native-rust-client.js').then(m => console.log('available:', m.getNativeRustCoreClient(true).isAvailable()))"
+```
+
+**Run integration tests** (requires debug binary):
+```bash
+npm run rust:build:debug
+npm run rust:test:integration   # 37 assertions
+npm run rust:test               # Rust unit tests
+```
+
+---
+
 ## Skills
 
 pi-lens includes two built-in skills that guide the LLM on when to use specific tools:
@@ -575,13 +612,6 @@ pattern: "TODO"  // Use grep instead
 
 See [CHANGELOG.md](CHANGELOG.md) for full history.
 
-### Latest Highlights
-
-- **Tree-sitter Query Cache:** Compiled query cache with mtime-based invalidation — 10× faster structural analysis startup
-- **LSP Support:** 31 Language Server Protocol clients (4 core auto-installed, others via npx or manual)
-- **NAPI Runner:** 100x faster TypeScript/JavaScript structural analysis (~9ms vs ~1200ms) — security rules fire inline
-- **Slop Detection:** 33+ TypeScript and 40+ Python patterns for AI-generated code quality issues
-
 ---
 
 ## License

package/clients/dispatch/runners/similarity.ts

@@ -9,6 +9,7 @@ import * as fs from "node:fs/promises";
 import * as path from "node:path";
 import * as ts from "typescript";
 import { EXCLUDED_DIRS } from "../../file-utils.js";
+import { NativeRustCoreClient } from "../../native-rust-client.js";
 import {
 	buildProjectIndex,
 	findSimilarFunctions,
@@ -23,12 +24,18 @@ import type {
 	RunnerResult,
 } from "../types.js";
 
+// Singleton Rust client — initialised once, reused across runner invocations.
+const rustClient = new NativeRustCoreClient();
+
+/** Feature flag: set to false to force the pure-TypeScript path. */
+const USE_RUST = true;
+
 // ============================================================================
 // Configuration
 // ============================================================================
 
 const CONFIG = {
-	SIMILARITY_THRESHOLD: 0.75, // 75% minimum similarity
+	SIMILARITY_THRESHOLD: 0.9, // 90% minimum similarity — below this false positives dominate
 	MIN_TRANSITIONS: 20, // Skip functions with <20 AST transitions
 	MAX_SUGGESTIONS: 3, // Max 3 suggestions per file
 	USAGE_THRESHOLD: 2, // Only suggest utilities with 2+ uses (placeholder)
@@ -64,6 +71,24 @@ const similarityRunner: RunnerDefinition = {
 			return { status: "skipped", diagnostics: [], semantic: "none" };
 		}
 
+		// ── Rust fast-path ─────────────────────────────────────────────────────
+		// Try Rust for file scanning + similarity detection. If the Rust binary
+		// is available, use it. On any failure, fall through to the pure-TS path.
+		if (USE_RUST && rustClient.isAvailable()) {
+			try {
+				const rustResult = await runWithRust(
+					filePath,
+					projectRoot,
+					CONFIG.SIMILARITY_THRESHOLD,
+					CONFIG.MAX_SUGGESTIONS,
+				);
+				if (rustResult !== null) return rustResult;
+			} catch {
+				// Fall through to TypeScript implementation.
+			}
+		}
+		// ── TypeScript fallback ─────────────────────────────────────────────────
+
 		const index = await loadOrBuildIndex(projectRoot);
 		if (!index || index.entries.size === 0) {
 			return { status: "skipped", diagnostics: [], semantic: "none" };
@@ -261,6 +286,80 @@ function buildSuggestionMessage(
 	return `Function '${func.name}' has ${similarityPct}% similarity to existing utility '${name}()' in ${location}. Consider reusing the existing utility.`;
 }
 
+// ============================================================================
+// Rust fast-path
+// ============================================================================
+
+/**
+ * Run similarity detection via the Rust binary.
+ *
+ * Flow:
+ * 1. Scan project files with Rust (respects .gitignore, much faster than glob).
+ * 2. Build the Rust index (persisted to .pi-lens/rust-index.json).
+ * 3. Query similarity for the current file.
+ * 4. Convert matches to Diagnostics.
+ *
+ * Returns `null` if the Rust path cannot produce results (no matches is still
+ * a valid result — returned as an empty-diagnostic RunnerResult).
+ */
+async function runWithRust(
+	filePath: string,
+	projectRoot: string,
+	threshold: number,
+	maxSuggestions: number,
+): Promise<RunnerResult | null> {
+	// 1. Scan project files.
+	const scanned = await rustClient.scanProject(projectRoot, [".ts", ".tsx"]);
+	if (scanned.length === 0) return null;
+
+	const relativeFiles = scanned.map((e) =>
+		path.relative(projectRoot, e.path).replace(/\\/g, "/"),
+	);
+
+	// 2. Build index (saves to .pi-lens/rust-index.json).
+	await rustClient.buildIndex(projectRoot, relativeFiles);
+
+	// 3. Find similarities for the current file.
+	const matches = await rustClient.findSimilarities(
+		projectRoot,
+		filePath,
+		threshold,
+	);
+
+	if (matches.length === 0) {
+		return { status: "succeeded", diagnostics: [], semantic: "none" };
+	}
+
+	// 4. Convert to Diagnostics.
+	const diagnostics: Diagnostic[] = matches
+		.slice(0, maxSuggestions)
+		.map((m) => {
+			const similarityPct = Math.round(m.similarity * 100);
+			// source_id format: "path/to/file.ts::funcName@line"
+			const [srcFile, srcFunc] = m.source_id.split("::");
+			const [targetFile, targetFunc] = m.target_id.split("::");
+			const funcName = srcFunc?.split("@")[0] ?? "?";
+			const targetName = targetFunc?.split("@")[0] ?? "?";
+			void srcFile; // file is implicit (it's the current file)
+			return {
+				id: `similarity-rust-${m.source_id}-${m.target_id}`,
+				tool: "similarity",
+				filePath,
+				line: 1, // Rust gives us the function source_id; line resolution is TODO
+				column: 1,
+				message: `Function '${funcName}' has ${similarityPct}% similarity to '${targetName}()' in ${targetFile}. Consider reusing the existing utility.`,
+				severity: "warning" as const,
+				semantic: "warning" as const,
+			};
+		});
+
+	return {
+		status: "succeeded",
+		diagnostics,
+		semantic: diagnostics.length > 0 ? "warning" : "none",
+	};
+}
+
 // ============================================================================
 // Index Management
 // ============================================================================

package/clients/installer/index.ts

@@ -39,6 +39,19 @@ import path from "node:path";
 // Global installation directory for pi-lens tools
 const TOOLS_DIR = path.join(process.cwd(), ".pi-lens", "tools");
 
+// Debug flag - set via PI_LENS_DEBUG=1 or --debug
+const DEBUG =
+	process.env.PI_LENS_DEBUG === "1" || process.argv.includes("--debug");
+
+/**
+ * Log debug messages only when DEBUG is enabled
+ */
+function debugLog(...args: unknown[]): void {
+	if (DEBUG) {
+		console.error("[auto-install:debug]", ...args);
+	}
+}
+
 // --- Tool Definitions ---
 
 interface ToolDefinition {
@@ -237,22 +250,18 @@ async function verifyToolBinary(binPath: string): Promise<boolean> {
 
 		proc.on("exit", (code) => {
 			if (code === 0) {
-				console.error(
-					`[auto-install] Verified: ${binPath} (version: ${stdout.trim()})`,
-				);
+				debugLog(`Verified: ${binPath} (version: ${stdout.trim()})`);
 				resolve(true);
 			} else {
-				console.error(
-					`[auto-install] Verification failed for ${binPath}: exit code ${code}, stderr: ${stderr}`,
-				);
+				console.error(`[auto-install] Verification failed for ${binPath}`);
+				debugLog("Exit code:", code, "stderr:", stderr);
 				resolve(false);
 			}
 		});
 
 		proc.on("error", (err) => {
-			console.error(
-				`[auto-install] Verification failed for ${binPath}: ${err.message}`,
-			);
+			console.error(`[auto-install] Verification failed for ${binPath}`);
+			debugLog("Error:", err.message);
 			resolve(false);
 		});
 	});
@@ -322,11 +331,11 @@ async function installNpmTool(
 					}
 
 					// NEW: Verify the binary actually works before returning
-					console.error(`[auto-install] Verifying ${binaryName}...`);
+					debugLog(`Verifying ${binaryName}...`);
 					const isValid = await verifyToolBinary(binPath);
 					if (!isValid) {
 						console.error(
-							`[auto-install] ${packageName} installed but verification failed. The binary may be corrupted.`,
+							`[auto-install] ${packageName} installed but verification failed (binary may be corrupted)`,
 						);
 						// Clean up the broken installation
 						try {
@@ -357,10 +366,8 @@ async function installNpmTool(
 			proc.on("error", (err) => reject(err));
 		});
 	} catch (err) {
-		console.error(
-			`[auto-install] Failed to install npm tool ${packageName}:`,
-			err,
-		);
+		console.error(`[auto-install] Failed to install ${packageName}: ${(err as Error).message}`);
+		debugLog("Full error:", err);
 		return undefined;
 	}
 }
@@ -393,10 +400,8 @@ async function installPipTool(
 			proc.on("error", (err) => reject(err));
 		});
 	} catch (err) {
-		console.error(
-			`[auto-install] Failed to install pip tool ${packageName}:`,
-			err,
-		);
+		console.error(`[auto-install] Failed to install ${packageName}: ${(err as Error).message}`);
+		debugLog("Full error:", err);
 		return undefined;
 	}
 }
@@ -434,7 +439,8 @@ export async function installTool(toolId: string): Promise<boolean> {
 				return false;
 		}
 	} catch (err) {
-		console.error(`[auto-install] Failed to install ${tool.name}:`, err);
+		console.error(`[auto-install] Failed to install ${tool.name}: ${(err as Error).message}`);
+		debugLog("Full error:", err);
 		return false;
 	}
 }