token-pilot 0.14.2 → 0.16.0

package/CHANGELOG.md

@@ -5,6 +5,40 @@ All notable changes to Token Pilot will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.16.0] - 2026-03-21
+
+### Added
+- **`read_symbols` tool** — batch read multiple symbols from one file in a single call (max 10). File is read once, AST resolved once. Saves N-1 round-trips vs calling `read_symbol` N times.
+- **`read_for_edit` batch mode** — new `symbols` array parameter reads multiple symbol edit contexts in one call. Each symbol returns raw code ready for Edit tool's `old_string`.
+- **`find_usages` context_lines** — new `context_lines` parameter (0-10) shows surrounding source code for each match. Eliminates follow-up `read_symbol` calls after finding usages.
+- **`smart_diff` affected symbols summary** — consolidated "AFFECTED SYMBOLS" section at the top of diff output, grouped by MODIFIED/ADDED/REMOVED. See all changed functions/classes at a glance.
+
+### Changed
+- **19 tools** (was 18) — added `read_symbols`.
+- **MCP instructions** — added batch read_symbols to decision rules and refactor workflow.
+- **427 tests** (unchanged — all pass with new features).
+
+## [0.15.0] - 2026-03-19
+
+### Added
+- **Regex fallback parser (TS/JS)** — `smart_read` now works for TypeScript/JavaScript files even without ast-index binary. Parses classes, functions, interfaces, types, enums, and class methods via regex. Zero dependencies, 130 lines. Covers ~80% of new users who fail to download ast-index.
+- **Regex fallback parser (Python)** — `smart_read` now works for Python files without ast-index. Parses classes, functions, async functions, decorators (`@dataclass`, `@app.route`), module constants (`UPPER_CASE`), methods with visibility detection (`_private`, `__dunder__`). 150 lines.
+- **Benchmark script** — `scripts/benchmark.ts` measures real token savings on public repos (express, fastify, flask). 92% average savings across 97 files ≥50 lines. Run: `npx tsx scripts/benchmark.ts`.
+- **Guide skill** — `/guide` command shows a quick-reference table of all Token Pilot tools with usage examples and recommended workflow.
+- **`hooks.denyThreshold` config** — hook deny threshold is now configurable in `.token-pilot.json` (default: 300, was hardcoded 500). Intercepts ~2x more native Read calls.
+
+### Changed
+- **Compact session analytics** — `session_analytics` report reduced from ~30 lines to ~5 lines. Shows calls, tokens saved, top 5 tools, top 3 files, cache hit rate on a single screen. Verbose mode (`verbose=true`) restores full breakdown.
+- **`server.ts` refactor** — extracted tool definitions to `server/tool-definitions.ts` and token estimate helpers to `server/token-estimates.ts` (−500 lines from server.ts).
+- **`find_usages` output** — results grouped by file with compact rendering. Single match per file on one line, multiple matches indented under file header.
+- **Stale references** — all `grep_search` hints updated to `Grep` (code-audit, find-unused, find-usages).
+- **README** — benchmark table with real data from 4 public repos. Updated savings claims from 80% to 90% (backed by benchmark).
+- **427 tests** (was 393).
+
+### Fixed
+- **`npx token-pilot` CLI** — symlink path resolution in `isDirectRun` check. All CLI commands now work correctly via npx.
+- **Regex fallback was dead code** — parsers existed but weren't wired into `client.ts` `outline()` method. Now properly called as fallback when ast-index unavailable.
+
 ## [0.14.1] - 2026-03-14
 
 ### Fixed

package/README.md

@@ -1,6 +1,6 @@
 # Token Pilot
 
-MCP server that reduces token consumption in AI coding assistants by **up to 80%** via AST-aware lazy file reading.
+MCP server that reduces token consumption in AI coding assistants by **up to 90%** via AST-aware lazy file reading.
 
 Instead of dumping entire files into the LLM context, Token Pilot returns structural overviews (classes, functions, signatures, line ranges) and lets the AI load only the specific symbols it needs.
 
@@ -13,7 +13,23 @@ Token Pilot:  smart_read("user-service.ts")  →  15-line outline  →  ~200 tok
               After edit: read_diff("user-service.ts")  →  ~20 tokens
 ```
 
-**Up to 80% reduction** on large files. Files under 200 lines are returned in full automatically (zero overhead for small files). Typical sessions with a mix of file sizes see **30-50% savings**, scaling higher with repeated reads (session cache, compact reminders) and targeted symbol loading.
+**Up to 90% reduction** on large files. Files under 200 lines are returned in full automatically (zero overhead for small files).
+
+### Benchmarks (real data)
+
+Measured on public open-source repos using the regex fallback parser (no ast-index binary). Files ≥50 lines only:
+
+| Repo | Files | Raw Tokens | Outline Tokens | Savings |
+|------|------:|----------:|--------------:|--------:|
+| [token-pilot](https://github.com/Digital-Threads/token-pilot) (TS) | 48 | 88,920 | 8,123 | **91%** |
+| [express](https://github.com/expressjs/express) (JS) | 6 | 14,421 | 193 | **99%** |
+| [fastify](https://github.com/fastify/fastify) (JS) | 23 | 50,000 | 3,161 | **94%** |
+| [flask](https://github.com/pallets/flask) (Python) | 20 | 78,236 | 7,418 | **91%** |
+| **Total** | **97** | **231,577** | **18,895** | **92%** |
+
+> This measures `smart_read` structural outline savings only. Real sessions also benefit from session cache, dedup reminders, `read_symbol` targeted loading, and `read_for_edit` minimal context.
+>
+> Run the benchmark yourself: `npx tsx scripts/benchmark.ts`
 
 ## Installation
 
@@ -152,15 +168,16 @@ For more control, you can add rules to your project:
 - **Cursor** → `.cursorrules` in project root
 - **Codex** → `AGENTS.md` in project root
 
-## MCP Tools (18)
+## MCP Tools (19)
 
 ### Core Reading
 
 | Tool | Instead of | Description |
 |------|-----------|-------------|
-| `smart_read` | `Read` | AST structural overview: classes, functions, methods with signatures. Up to 80% savings on large files. Framework-aware: shows HTTP routes, column types, validation rules. |
+| `smart_read` | `Read` | AST structural overview: classes, functions, methods with signatures. Up to 90% savings on large files. Framework-aware: shows HTTP routes, column types, validation rules. |
 | `read_symbol` | `Read` + scroll | Load source of a specific symbol. Supports `Class.method`. `show` param: full/head/tail/outline. |
-| `read_for_edit` | `Read` before `Edit` | Minimal RAW code around a symbol — copy directly as `old_string` for Edit tool. |
+| `read_symbols` | N x `read_symbol` | Batch read multiple symbols from one file in a single call (max 10). One round-trip instead of N. |
+| `read_for_edit` | `Read` before `Edit` | Minimal RAW code around a symbol — copy directly as `old_string` for Edit tool. Batch mode: pass `symbols` array for multiple edit contexts. |
 | `read_range` | `Read` offset | Read a specific line range from a file. |
 | `read_diff` | re-`Read` | Show only changed hunks since last smart_read. Requires smart_read before editing (for baseline). Works with any edit tool. |
 | `smart_read_many` | multiple `Read` | Batch smart_read for up to 20 files in one call. |
@@ -169,14 +186,14 @@ For more control, you can add rules to your project:
 
 | Tool | Instead of | Description |
 |------|-----------|-------------|
-| `find_usages` | `Grep` (refs) | All usages of a symbol: definitions, imports, references. Filters: `scope` (path prefix), `kind` (definitions/imports/usages), `lang`, `limit`. |
+| `find_usages` | `Grep` (refs) | All usages of a symbol: definitions, imports, references. Filters: `scope`, `kind`, `lang`, `limit`. Use `context_lines` to include surrounding source code. |
 | `project_overview` | `ls` + explore | Dual-detection (ast-index + config files) with confidence scoring. Project type, frameworks, quality tools, CI, architecture, directory map. Filter sections with `include`. |
 | `related_files` | manual explore | Import graph: what a file imports, what imports it, test files. |
 | `outline` | multiple `smart_read` | Compact overview of all code files in a directory. One call instead of 5-6. Supports `recursive` mode with `max_depth` for deep exploration. |
 | `find_unused` | manual | Detect dead code — unused functions, classes, variables. |
 | `code_audit` | multiple `Grep` | Code quality issues in one call: TODO/FIXME comments, deprecated symbols, structural code patterns (via ast-grep), decorator search. |
 | `module_info` | manual analysis | Module dependency analysis: dependencies, dependents, public API, unused deps. Use for architecture understanding and dependency cleanup. |
-| `smart_diff` | raw `git diff` | Structural diff with AST symbol mapping — shows which functions/classes changed instead of raw patch. Scopes: unstaged, staged, commit, branch. |
+| `smart_diff` | raw `git diff` | Structural diff with AST symbol mapping — shows which functions/classes changed instead of raw patch. Affected symbols summary at top. Scopes: unstaged, staged, commit, branch. |
 | `explore_area` | outline + related_files + git log | One-call directory exploration: structure, imports, tests, recent changes. Replaces 3-5 separate calls. |
 | `smart_log` | raw `git log` | Structured commit history with category detection (feat/fix/refactor/docs), file stats, author breakdown. Filters by path and ref. |
 | `test_summary` | raw test output | Run tests and get structured summary: total/passed/failed + failure details. Supports vitest, jest, pytest, phpunit, go, cargo, rspec, mocha. |
@@ -280,7 +297,7 @@ When both are configured, Token Pilot automatically:
 - Suggests context-mode for large non-code files
 - Shows combined architecture info in `session_analytics`
 
-**Combined savings: up to 80%** in a typical coding session.
+**Combined savings: up to 90%** in a typical coding session.
 
 ## Supported Languages
 

package/dist/ast-index/client.d.ts

@@ -20,21 +20,10 @@ export declare class AstIndexClient {
     init(): Promise<void>;
     ensureIndex(): Promise<void>;
     private buildIndex;
-    /** Mark index as oversized — disables index-dependent tools, outline still works */
     private handleOversizedIndex;
-    /** Extract file count from stats output (JSON or text) */
-    private parseFileCount;
     outline(filePath: string): Promise<FileStructure | null>;
-    /**
-     * Parse text output from `ast-index outline`:
-     *   Outline of src/file.ts:
-     *     :10 ClassName [class]
-     *     :11 propName [property]
-     *     :14 methodName [function]
-     */
-    private parseOutlineText;
-    /** Compute end_line from sequential start positions */
-    private computeEndLines;
+    /** Regex-based fallback for TS/JS/Python when ast-index binary is unavailable. */
+    private regexFallback;
     symbol(name: string): Promise<AstIndexSymbolDetail | null>;
     search(query: string, options?: {
         inFile?: string;
@@ -44,119 +33,43 @@ export declare class AstIndexClient {
     usages(symbolName: string): Promise<AstIndexUsageResult[]>;
     implementations(name: string): Promise<AstIndexImplementation[]>;
     hierarchy(name: string): Promise<AstIndexHierarchyNode | null>;
-    private parseImplementationsText;
-    private parseHierarchyText;
     stats(): Promise<string | null>;
-    /**
-     * List all files known to the ast-index.
-     * Parses the `files` command output which lists one file per line.
-     */
     listFiles(): Promise<string[]>;
-    /**
-     * Cross-references: definitions + imports + usages in one call.
-     * Replaces separate symbol() + usages() calls.
-     */
     refs(symbolName: string, limit?: number): Promise<AstIndexRefsResponse>;
-    /**
-     * Project map: directory structure with file counts and symbol kinds.
-     */
     map(options?: {
         module?: string;
         limit?: number;
     }): Promise<AstIndexMapResponse | null>;
-    /**
-     * Detect project conventions: architecture, frameworks, naming patterns.
-     */
     conventions(): Promise<AstIndexConventionsResponse | null>;
-    /**
-     * Find callers of a function.
-     */
     callers(functionName: string, limit?: number): Promise<AstIndexCallerEntry[]>;
-    /**
-     * Show call hierarchy tree (callers tree up).
-     */
     callTree(functionName: string, depth?: number): Promise<AstIndexCallTreeNode | null>;
-    /**
-     * Show changed symbols since base branch (git diff).
-     */
     changed(base?: string): Promise<AstIndexChangedEntry[]>;
-    /**
-     * Find potentially unused symbols.
-     */
     unusedSymbols(options?: {
         module?: string;
         exportOnly?: boolean;
         limit?: number;
     }): Promise<AstIndexUnusedSymbol[]>;
-    /**
-     * Get imports for a specific file.
-     * Parses text output: "  { X, Y } from 'source';"
-     */
     fileImports(filePath: string): Promise<AstIndexImportEntry[]>;
-    private parseImportsText;
-    /** Check if ast-grep (sg) is available for structural pattern search */
     private checkAstGrep;
-    /** Structural pattern search via ast-grep. Requires ast-grep (sg) installed. */
     agrep(pattern: string, options?: {
         lang?: string;
         limit?: number;
     }): Promise<AstIndexAgrepMatch[]>;
-    private parseAgrepText;
-    /** Find TODO/FIXME/HACK comments in the project */
     todo(): Promise<AstIndexTodoEntry[]>;
-    private parseTodoText;
-    /** Find @Deprecated symbols in the project */
     deprecated(): Promise<AstIndexDeprecatedEntry[]>;
-    private parseDeprecatedText;
-    /** Find symbols with a specific annotation/decorator */
     annotations(name: string): Promise<AstIndexAnnotationEntry[]>;
-    private parseAnnotationsText;
-    /** Trigger incremental index update (called by file watcher after edits) */
     incrementalUpdate(): Promise<void>;
-    /** List project modules matching optional pattern */
     modules(pattern?: string): Promise<AstIndexModuleEntry[]>;
-    /** Get dependencies of a module */
     moduleDeps(module: string): Promise<AstIndexModuleDep[]>;
-    /** Get modules that depend on this module */
     moduleDependents(module: string): Promise<AstIndexModuleDep[]>;
-    /** Find unused dependencies of a module */
     unusedDeps(module: string): Promise<AstIndexUnusedDep[]>;
-    /** Get public API of a module */
     moduleApi(module: string): Promise<AstIndexModuleApi[]>;
-    private parseModuleListText;
-    private parseModuleDepText;
-    private parseUnusedDepsText;
-    private parseModuleApiText;
     isAvailable(): boolean;
-    /** Returns true if the index was built but found >50k files (node_modules leak) */
     isOversized(): boolean;
-    /** Returns true if index building is disabled (dangerous root like /) */
     isDisabled(): boolean;
-    /** Disable index building (e.g. project root is / or home dir) */
     disableIndex(): void;
-    /** Re-enable index building after auto-detecting a valid project root */
     enableIndex(): void;
-    /** Update project root (e.g. after auto-detecting from file path) */
     updateProjectRoot(newRoot: string): void;
     private exec;
-    private buildFileStructure;
-    /**
-     * Python: ast-index doesn't return methods inside classes.
-     * Parse file content to extract `def` methods for classes without children.
-     */
-    private enrichPythonClassMethods;
-    /**
-     * PHP: ast-index doesn't return methods inside classes.
-     * Parse file content to extract `function` methods for classes without children.
-     */
-    private enrichPHPClassMethods;
-    /** Fix the last entry's end_line to use actual file line count */
-    private fixLastEndLine;
-    /** Read actual signature lines from file content */
-    private enrichSignatures;
-    private mapOutlineEntry;
-    private mapKind;
-    private mapVisibility;
-    private detectLanguage;
 }
 //# sourceMappingURL=client.d.ts.map