ast-search 0.2.3 → 0.3.0

package/README.md

@@ -2,7 +2,7 @@
 
 A CLI tool for searching source files using AST patterns, designed to facilitate large-scale refactors.
 
-Accepts an [esquery](https://github.com/estools/esquery) CSS-selector-style query and searches all supported files under a directory, printing each match with its file path, line, and column.
+Accepts a query and searches all supported files under a directory, printing each match with its file path, line, and column. Language support is provided by plugins — the core handles JS/TS/Vue; additional languages are opt-in.
 
 ## Example
 
@@ -31,17 +31,28 @@ src/components/Foo.vue:5:13: return this.testValue
 src/components/Bar.vue:9:18: return this.otherProp
 ```
 
+## Installation
+
+```bash
+npm install -g ast-search
+
+# Optional: add Python support
+npm install -g ast-search-python
+```
+
 ## Usage
 
 ```
-ast-search <query> [--dir <path>] [--format <fmt>]
+ast-search <query> [--dir <path>] [--format <fmt>] [--lang <id>] [--plugin <pkg>]
 ```
 
-| Argument         | Description                                              | Default      |
-| ---------------- | -------------------------------------------------------- | ------------ |
-| `<query>`        | esquery selector string (see below)                      | required     |
-| `-d, --dir`      | root directory to search                                 | current dir  |
-| `-f, --format`   | output format: `text`, `json`, or `files`                | `text`       |
+| Argument           | Description                                                   | Default      |
+| ------------------ | ------------------------------------------------------------- | ------------ |
+| `<query>`          | Query string (see Query Syntax below)                         | required     |
+| `-d, --dir`        | Root directory to search                                      | current dir  |
+| `-f, --format`     | Output format: `text`, `json`, or `files`                     | `text`       |
+| `-l, --lang`       | Restrict search to one language backend (e.g. `js`, `python`) | all languages |
+| `-p, --plugin`     | Load a language plugin package (repeatable)                   | none         |
 
 ### Output formats
 
@@ -51,6 +62,8 @@ ast-search <query> [--dir <path>] [--format <fmt>]
 
 ## Query syntax
 
+### JavaScript / TypeScript / Vue
+
 Queries use [esquery](https://github.com/estools/esquery) CSS selector syntax over Babel AST node types. A few examples:
 
 ```bash
@@ -64,7 +77,7 @@ ast-search 'AwaitExpression'
 ast-search 'CatchClause AssignmentExpression'
 ```
 
-### Shorthands
+#### Shorthands
 
 Common node types can be written as short keywords:
 
@@ -92,7 +105,7 @@ The original Vue `this` example using shorthands:
 ast-search 'ObjectMethod[key.name="setup"] this'
 ```
 
-### Optional chaining
+#### Optional chaining
 
 Optional chains (`?.`) are normalized transparently — `CallExpression` and `MemberExpression` selectors match both regular and optional-chain variants:
 
@@ -107,6 +120,81 @@ The `optional` flag is preserved on matched nodes, so you can still narrow to st
 ast-search 'CallExpression[optional=true]'
 ```
 
+### Python (via `ast-search-python`)
+
+Python queries use [tree-sitter](https://tree-sitter.github.io/tree-sitter/) S-expression syntax. Install the plugin first:
+
+```bash
+npm install -g ast-search-python
+```
+
+Then pass `--plugin ast-search-python` to enable `.py` / `.pyw` file support:
+
+```bash
+# Find all function definitions (shorthand)
+ast-search 'fn' --plugin ast-search-python
+
+# Find all class definitions (raw S-expression)
+ast-search '(class_definition) @cls' --plugin ast-search-python
+
+# Find all calls to a specific function by name
+ast-search '(call function: (identifier) @n (#eq? @n "my_func")) @c' --plugin ast-search-python
+
+# Restrict to only Python files in a mixed-language repo
+ast-search 'fn' --lang python --plugin ast-search-python
+```
+
+#### Python shorthands
+
+| Shorthand   | Expands to                          |
+| ----------- | ----------------------------------- |
+| `fn`        | `(function_definition) @_`          |
+| `call`      | `(call) @_`                         |
+| `class`     | `(class_definition) @_`             |
+| `assign`    | `(assignment) @_`                   |
+| `return`    | `(return_statement) @_`             |
+| `await`     | `(await) @_`                        |
+| `yield`     | `(yield) @_`                        |
+| `import`    | `(import_statement) @_`             |
+| `from`      | `(import_from_statement) @_`        |
+| `if`        | `(if_statement) @_`                 |
+| `for`       | `(for_statement) @_`                |
+| `while`     | `(while_statement) @_`              |
+| `raise`     | `(raise_statement) @_`              |
+| `with`      | `(with_statement) @_`               |
+| `lambda`    | `(lambda) @_`                       |
+| `decorator` | `(decorator) @_`                    |
+| `augassign` | `(augmented_assignment) @_`         |
+
+For raw S-expression queries, include at least one `@capture_name` — tree-sitter's `captures()` requires it to return results.
+
+> **Note:** In tree-sitter-python 0.21+, `async def` functions are still typed as `function_definition` (no separate `async_function_definition` node). To match only async functions, use a full predicate query.
+
 ## Supported file types
 
-`.js`, `.ts`, `.jsx`, `.tsx`, `.mjs`, `.cjs`, `.vue`
+**Core:** `.js` `.ts` `.jsx` `.tsx` `.mjs` `.cjs` `.vue`
+
+**With `ast-search-python`:** `.py` `.pyw`
+
+## Plugin API
+
+To write a language plugin, implement the `LanguageBackend` interface exported from `ast-search/plugin` and export a `register` function:
+
+```typescript
+import type { LanguageBackend, LanguageRegistry } from 'ast-search/plugin';
+
+class MyLanguageBackend implements LanguageBackend {
+  readonly langId = 'mylang';
+  readonly name = 'My Language';
+  readonly extensions = new Set(['.ml']);
+  parse(source: string, filePath: string) { /* return opaque AST */ }
+  query(ast: unknown, selector: string, source: string, filePath: string) { /* return Match[] */ }
+  validateSelector(selector: string) { /* throw on invalid */ }
+}
+
+export function register(registry: LanguageRegistry) {
+  registry.register(new MyLanguageBackend());
+}
+```
+
+Name your package `ast-search-<lang>` and users load it with `--plugin ast-search-<lang>`.

package/build/backends/js/index.js

@@ -0,0 +1,30 @@
+import { extname } from "node:path";
+import { getAst, parseVueSFC } from "../../file.js";
+import { runQuery, validateSelector as validate, SHORTHANDS, expandShorthands } from "../../search.js";
+export class JSLanguageBackend {
+    constructor() {
+        this.langId = "js";
+        this.name = "JavaScript/TypeScript";
+        this.extensions = new Set([
+            ".js",
+            ".ts",
+            ".jsx",
+            ".tsx",
+            ".mjs",
+            ".cjs",
+            ".vue",
+        ]);
+    }
+    parse(source, filePath) {
+        const ext = extname(filePath);
+        const content = ext === ".vue" ? parseVueSFC(Buffer.from(source)) : source;
+        return getAst(content);
+    }
+    query(ast, selector, source, filePath) {
+        return runQuery(selector, ast, source, filePath);
+    }
+    validateSelector(selector) {
+        validate(selector);
+    }
+}
+export { SHORTHANDS, expandShorthands };

package/build/file.d.ts

@@ -0,0 +1,22 @@
+import { FileHandle } from "node:fs/promises";
+import * as parser from "@babel/parser";
+import { File } from "@babel/types";
+import type { LanguageBackend } from "./language.js";
+import type { LanguageRegistry } from "./registry.js";
+export declare function getAst(contents: string): parser.ParseResult<File>;
+export declare const SCRIPT_OPEN: RegExp;
+export declare const SCRIPT_CLOSE: RegExp;
+export declare function parseVueSFC(lines: Buffer): string;
+interface ParseReturn {
+    ast: File;
+    file: FileHandle;
+    source: string;
+}
+export declare function getAstFromPath(path: string): Promise<ParseReturn>;
+export interface ParsedFile {
+    ast: unknown;
+    source: string;
+    backend: LanguageBackend;
+}
+export declare function parseFile(path: string, registry: LanguageRegistry): Promise<ParsedFile>;
+export {};

package/build/file.js

@@ -56,3 +56,21 @@ export function getAstFromPath(path) {
         return { ast, file, source: fileContents };
     });
 }
+export function parseFile(path, registry) {
+    return __awaiter(this, void 0, void 0, function* () {
+        const ext = extname(path);
+        const backend = registry.getByExtension(ext);
+        if (!backend) {
+            throw new Error(`No backend registered for extension "${ext}"`);
+        }
+        const file = yield open(path);
+        try {
+            const source = (yield file.readFile()).toString();
+            const ast = yield backend.parse(source, path);
+            return { ast, source, backend };
+        }
+        finally {
+            yield file.close();
+        }
+    });
+}

package/build/language.d.ts

@@ -0,0 +1,23 @@
+import type { Match } from "./types.js";
+export type { Match };
+export interface LanguageBackend {
+    /** Short identifier used with --lang flag, e.g. "js", "python" */
+    readonly langId: string;
+    /** File extensions this backend handles, e.g. new Set([".py"]) */
+    readonly extensions: ReadonlySet<string>;
+    /** Human-readable name for error messages */
+    readonly name: string;
+    /** Parse source text into an opaque AST. Throws on unrecoverable parse error. */
+    parse(source: string, filePath: string): Promise<unknown> | unknown;
+    /**
+     * Run a selector query against a parsed AST. The selector is in the
+     * backend's native query syntax (after shorthand expansion). Returns matches
+     * with file/line/col/source fields.
+     */
+    query(ast: unknown, selector: string, source: string, filePath: string): Match[];
+    /**
+     * Validate a selector string. Expands shorthands internally, then checks
+     * syntax. Throws with a descriptive message on invalid syntax.
+     */
+    validateSelector(selector: string): void;
+}

package/build/language.js

@@ -0,0 +1 @@
+export {};

package/build/main.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+import type { Match } from "./types.js";
+export declare function searchRepo(selector: string, dir: string, registry?: import("./registry.js").LanguageRegistry): Promise<Match[]>;

package/build/main.js

@@ -16,35 +16,34 @@ var __asyncValues = (this && this.__asyncValues) || function (o) {
     function settle(resolve, reject, d, v) { Promise.resolve(v).then(function(v) { resolve({ value: v, done: d }); }, reject); }
 };
 import yargs from "yargs/yargs";
-import { createRequire } from "module";
 import { walkRepoFiles } from "./walk.js";
-import { getAstFromPath } from "./file.js";
-import { runQuery, validateSelector } from "./search.js";
+import { parseFile } from "./file.js";
 import { formatMatches } from "./output.js";
-const require = createRequire(import.meta.url);
-const { version } = require("../package.json");
-export function searchRepo(selector, dir) {
-    return __awaiter(this, void 0, void 0, function* () {
+import { defaultRegistry } from "./registry.js";
+import { JSLanguageBackend } from "./backends/js/index.js";
+import { VERSION } from "./version.js";
+// Register built-in JS/TS/Vue backend
+defaultRegistry.register(new JSLanguageBackend());
+export function searchRepo(selector_1, dir_1) {
+    return __awaiter(this, arguments, void 0, function* (selector, dir, registry = defaultRegistry) {
         var _a, e_1, _b, _c;
-        validateSelector(selector); // throws early on invalid selector syntax
+        // Early validation when only one backend is registered (common JS-only case)
+        if (registry.allBackends.length === 1) {
+            registry.allBackends[0].validateSelector(selector);
+        }
         const results = [];
         try {
-            for (var _d = true, _e = __asyncValues(walkRepoFiles(dir)), _f; _f = yield _e.next(), _a = _f.done, !_a; _d = true) {
+            for (var _d = true, _e = __asyncValues(walkRepoFiles(dir, registry.allExtensions)), _f; _f = yield _e.next(), _a = _f.done, !_a; _d = true) {
                 _c = _f.value;
                 _d = false;
                 const filePath = _c;
-                let file;
                 try {
-                    const result = yield getAstFromPath(filePath);
-                    file = result.file;
-                    const matches = runQuery(selector, result.ast, result.source, filePath);
+                    const { ast, source, backend } = yield parseFile(filePath, registry);
+                    const matches = backend.query(ast, selector, source, filePath);
                     results.push(...matches);
                 }
                 catch (_g) {
-                    // skip unparseable files
-                }
-                finally {
-                    yield (file === null || file === void 0 ? void 0 : file.close());
+                    // skip unparseable files / unsupported extensions
                 }
             }
         }
@@ -64,7 +63,7 @@ const y = yargs(process.argv.slice(2))
     .command("$0 <query>", "Search a repo for AST patterns using CSS selector syntax", (yargs) => yargs
     .positional("query", {
     type: "string",
-    describe: "esquery CSS selector string",
+    describe: "Query string (esquery CSS selector for JS/TS; tree-sitter S-expression for Python)",
     demandOption: true,
 })
     .option("dir", {
@@ -79,12 +78,47 @@ const y = yargs(process.argv.slice(2))
     describe: "output format: text (default), json, files",
     default: "text",
     choices: ["text", "json", "files"],
+})
+    .option("lang", {
+    alias: "l",
+    type: "string",
+    describe: "restrict search to a specific language backend by langId (e.g. js, python)",
+})
+    .option("plugin", {
+    alias: "p",
+    type: "string",
+    array: true,
+    describe: "load a language plugin package (e.g. ast-search-python)",
 }), (argv) => __awaiter(void 0, void 0, void 0, function* () {
-    var _a;
-    const { query, dir, format } = argv;
+    var _a, _b, _c;
+    const { query, dir, format, lang, plugin } = argv;
     try {
-        const matches = yield searchRepo(query, dir);
-        const isTTY = (_a = process.stdout.isTTY) !== null && _a !== void 0 ? _a : false;
+        // Load plugins before searching
+        for (const pkg of plugin !== null && plugin !== void 0 ? plugin : []) {
+            const mod = yield import(pkg);
+            const reg = (_a = mod.register) !== null && _a !== void 0 ? _a : (_b = mod.default) === null || _b === void 0 ? void 0 : _b.register;
+            if (typeof reg !== "function") {
+                throw new Error(`Plugin "${pkg}" does not export a register() function`);
+            }
+            reg(defaultRegistry);
+        }
+        // Build a scoped registry if --lang is specified
+        let registry = defaultRegistry;
+        if (lang) {
+            const backend = defaultRegistry.getByLangId(lang);
+            if (!backend) {
+                const available = defaultRegistry.allBackends.map((b) => b.langId).join(", ");
+                throw new Error(`Unknown language "${lang}". Available: ${available}`);
+            }
+            const { LanguageRegistry } = yield import("./registry.js");
+            const scoped = new LanguageRegistry();
+            scoped.register(backend);
+            registry = scoped;
+            // Always validate early when --lang restricts to a single backend
+            backend.validateSelector(query);
+        }
+        const matches = yield searchRepo(query, dir, registry);
+        const isTTY = (_c = process.stdout.isTTY) !== null && _c !== void 0 ? _c : false;
         for (const line of formatMatches(matches, isTTY, format)) {
             console.log(line);
         }
@@ -117,8 +151,16 @@ const y = yargs(process.argv.slice(2))
         "$0 'FunctionDeclaration[async=true]' --format files | xargs prettier --write",
         "reformat all files containing async functions",
     ],
+    [
+        "$0 'fn' --dir src --plugin ast-search-python",
+        "find all function definitions in Python files",
+    ],
+    [
+        "$0 '(class_definition)' --lang python --plugin ast-search-python",
+        "find all Python classes (tree-sitter S-expression syntax)",
+    ],
 ])
-    .version(version)
+    .version(VERSION)
     .alias("version", "V")
     .help();
 if (process.env.NODE_ENV !== "test") {

package/build/output.d.ts

@@ -0,0 +1,3 @@
+import type { Match } from "./search.js";
+export type OutputFormat = "text" | "json" | "files";
+export declare function formatMatches(matches: Match[], isTTY: boolean, format?: OutputFormat): string[];

package/build/plugin.d.ts

@@ -0,0 +1,3 @@
+export type { Match } from "./types.js";
+export type { LanguageBackend } from "./language.js";
+export { LanguageRegistry } from "./registry.js";

package/build/plugin.js

@@ -0,0 +1 @@
+export { LanguageRegistry } from "./registry.js";

package/build/registry.d.ts

@@ -0,0 +1,11 @@
+import type { LanguageBackend } from "./language.js";
+export declare class LanguageRegistry {
+    private readonly byExt;
+    private readonly byId;
+    register(backend: LanguageBackend): void;
+    getByExtension(ext: string): LanguageBackend | undefined;
+    getByLangId(langId: string): LanguageBackend | undefined;
+    get allExtensions(): ReadonlySet<string>;
+    get allBackends(): LanguageBackend[];
+}
+export declare const defaultRegistry: LanguageRegistry;

package/build/registry.js

@@ -0,0 +1,25 @@
+export class LanguageRegistry {
+    constructor() {
+        this.byExt = new Map();
+        this.byId = new Map();
+    }
+    register(backend) {
+        this.byId.set(backend.langId, backend);
+        for (const ext of backend.extensions) {
+            this.byExt.set(ext, backend);
+        }
+    }
+    getByExtension(ext) {
+        return this.byExt.get(ext);
+    }
+    getByLangId(langId) {
+        return this.byId.get(langId);
+    }
+    get allExtensions() {
+        return new Set(this.byExt.keys());
+    }
+    get allBackends() {
+        return [...this.byId.values()];
+    }
+}
+export const defaultRegistry = new LanguageRegistry();

package/build/search.d.ts

@@ -0,0 +1,8 @@
+import type { File } from "@babel/types";
+import type { Match } from "./types.js";
+export type { Match };
+export declare const SHORTHANDS: Record<string, string>;
+export declare function expandShorthands(selector: string): string;
+export declare function validateSelector(selector: string): void;
+export declare function normalizeOptionalChaining(node: any): void;
+export declare function runQuery(selector: string, ast: File, source?: string, filename?: string): Match[];

package/build/types.d.ts

@@ -0,0 +1,6 @@
+export interface Match {
+    file: string;
+    line: number;
+    col: number;
+    source: string;
+}

package/build/types.js

@@ -0,0 +1 @@
+export {};

package/build/version.d.ts

@@ -0,0 +1 @@
+export declare const VERSION = "0.3.0";

package/build/version.js

@@ -0,0 +1,2 @@
+// Auto-updated by semantic-release via @semantic-release/exec — do not edit manually.
+export const VERSION = "0.3.0";

package/build/walk.d.ts

@@ -0,0 +1 @@
+export declare function walkRepoFiles(dir: string, extensions: ReadonlySet<string>): AsyncGenerator<string>;

package/build/walk.js

@@ -25,16 +25,7 @@ var __asyncGenerator = (this && this.__asyncGenerator) || function (thisArg, _ar
 };
 import { readdir } from "node:fs/promises";
 import { extname, join } from "node:path";
-const SUPPORTED_EXTENSIONS = new Set([
-    ".js",
-    ".ts",
-    ".jsx",
-    ".tsx",
-    ".mjs",
-    ".cjs",
-    ".vue",
-]);
-export function walkRepoFiles(dir) {
+export function walkRepoFiles(dir, extensions) {
     return __asyncGenerator(this, arguments, function* walkRepoFiles_1() {
         const entries = yield __await(readdir(dir, { withFileTypes: true }));
         for (const entry of entries) {
@@ -44,9 +35,9 @@ export function walkRepoFiles(dir) {
                 continue;
             const fullPath = join(dir, entry.name);
             if (entry.isDirectory()) {
-                yield __await(yield* __asyncDelegator(__asyncValues(walkRepoFiles(fullPath))));
+                yield __await(yield* __asyncDelegator(__asyncValues(walkRepoFiles(fullPath, extensions))));
             }
-            else if (entry.isFile() && SUPPORTED_EXTENSIONS.has(extname(entry.name))) {
+            else if (entry.isFile() && extensions.has(extname(entry.name))) {
                 yield yield __await(fullPath);
             }
         }

package/package.json

@@ -1,15 +1,25 @@
 {
   "name": "ast-search",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "description": "",
-  "main": "main.js",
+  "main": "build/main.js",
+  "exports": {
+    ".": {
+      "import": "./build/main.js",
+      "types": "./build/main.d.ts"
+    },
+    "./plugin": {
+      "import": "./build/plugin.js",
+      "types": "./build/plugin.d.ts"
+    }
+  },
   "scripts": {
     "dev": "npx tsc --watch",
     "debug:vue": "npx tsc && node ./build/main.js 'setup > this' --dir src/__tests__/dummyFiles",
     "debug:react": "npx tsc && node ./build/main.js 'useState' --dir src/__tests__/dummyFiles",
     "debug:empty": "npx tsc && node ./build/main.js 'ValidationsTab > this' --dir src/__tests__/dummyFiles",
     "debug:basics": "npx tsc && node ./build/main.js 'arrowFunction' --dir src/__tests__/dummyFiles",
-    "test": "jest",
+    "test": "jest --testPathPattern='src/__tests__'",
     "build": "npx tsc && cp build/main.js ast-search",
     "prepublishOnly": "npx tsc"
   },
@@ -25,6 +35,12 @@
     "url": "https://github.com/willey-shiplet/ast-search.git"
   },
   "packageManager": "pnpm@10.33.0",
+  "pnpm": {
+    "onlyBuiltDependencies": [
+      "tree-sitter",
+      "tree-sitter-python"
+    ]
+  },
   "dependencies": {
     "@babel/parser": "^7.24.4",
     "@babel/types": "^7.24.0",
@@ -36,7 +52,9 @@
   },
   "files": [
     "build/*.js",
+    "build/*.d.ts",
     "build/helpers/*.js",
+    "build/backends/**/*.js",
     "README.md"
   ],
   "pkg": {
@@ -49,6 +67,7 @@
     "@babel/preset-typescript": "^7.24.1",
     "@jest/globals": "^29.7.0",
     "@semantic-release/changelog": "^6.0.3",
+    "@semantic-release/exec": "^6.0.3",
     "@semantic-release/git": "^10.0.1",
     "@semantic-release/github": "^10.0.0",
     "@semantic-release/npm": "^13.1.3",