@appthreat/atom-parsetools 1.2.2 → 1.4.0

package/README.md

@@ -4,9 +4,13 @@ This package hosts a collection of parsing tools that complement the `@appthreat
 
 - astgen - Generates AST for JavaScript and TypeScript projects in JSON format
 - phpastgen - Generates AST for PHP projects using `php-parse` command from `nikic/php-parser`
-- rbastgen - Generates AST for Ruby projects using the AppThreat's `ruby_ast_gen` gem
+- rbastgen - Generates AST for Ruby projects using AppThreat's [`ruby_ast_gen`](https://github.com/AppThreat/ruby_ast_gen) gem (2.0.1)
 - scalasem - Generates a custom semantics slice for Scala Projects by utilising scalac command.
 
+## Runtime support
+
+These tools run on both [Node.js](https://nodejs.org) (>= 22, required by `@babel/parser` 8) and [Bun](https://bun.sh). All commands and the accompanying regression test-suite are exercised under both runtimes in CI, so the commands below can be invoked with either `node` or `bun` interchangeably (for example `bun astgen.js -i .`).
+
 ## Command usages
 
 ### astgen
@@ -25,6 +29,16 @@ Options:
   -h             Show help                                             [boolean]
 ```
 
+#### Environment variables
+
+| Variable                              | Default   | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| ------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASTGEN_TYPE_WORKERS`                 | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
+| `ASTGEN_INCLUDE_TEST_FILES`           | `false`   | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation.                                                                                                                                                                                                                                                                                                                              |
+| `ASTGEN_CONCURRENCY`                  | `10`      | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false`   | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely.                                                                                                                                                                                                                                                                                                                                                                                          |
+| `ASTGEN_IGNORE_DIRS`                  | unset     | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`).                                                                                                                                                                                                                                                                                                                                           |
+
 ### phpastgen
 
 ```text
@@ -50,17 +64,90 @@ Operations is a list of the following options (--dump by default):
 
 ### rbastgen
 
+Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and
+its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be
+gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed
+to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which
+resolved its paths from the ABI of the Ruby that built it.
+
+`prism` and `racc` are deliberately **not** bundled. Both carry C extensions, which are built per
+ABI and per platform, and both are default gems in every supported Ruby, so the runtime's own copies
+are used. One consequence is visible: the newest grammar available follows the interpreter's prism,
+so Ruby 3.4 tops out lower than Ruby 4.0 (grammar 3.5 versus 4.1 at the time of writing). Installing
+a newer `prism` gem on the machine raises it, since the caller's `GEM_PATH` is preserved.
+
 ```text
 node rbastgen.js --help
 Usage:
-  -i, --input      The input file or directory (required)
-  -o, --output     The output directory (default: '.ast')
-  -e, --exclude    The exclusion regex (default: '^(tests?|vendor|spec)')
-  -d, --debug      Enable debug logging
-      --version    Print the version
-      --help       Print usage
+  -i, --input <path>       The input file or directory (required)
+  -o, --output <dir>       The output directory (default: '.ast')
+  -e, --exclude <regex>    The exclusion regex (default: '^(tests?|vendor|spec)')
+  -l, --log <level>        The logging level: debug, info, warn or error (default: info)
+  -d, --debug              Enable debug logging (same as --log debug)
+      --parser-target <x.y>
+                           Parse with a specific Ruby grammar (e.g. 3.4, 4.0) instead
+                           of the newest available
+      --max-depth <n>      Maximum AST depth before truncation (default: 250)
+      --threads <n>        Worker threads for a directory run (default: 10)
+      --fail-on-error      Exit non-zero when any input file failed to parse
+      --parser-info        Print parser/runtime capability information
+      --version            Print the version
+      --help               Print usage
+```
+
+Problems with individual files are reported and skipped, never fatal; only usage errors (a missing
+`-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that
+`rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the
+log but the wrapper still exits 0 — call the gem directly if a CI job needs to fail on a parse
+error.
+
+#### What the bundled generator emits (ruby_ast_gen 2.0.1)
+
+- **Parsing is decoupled from the running Ruby.** When `prism` is available the newest grammar its
+  translation layer supports is used, so a 3.4 runtime parses Ruby 4.0/4.1 syntax. `--parser-target
+x.y` pins a grammar instead (down to 1.8 through the `parser` gem), and a file that fails under
+  the selected grammar is retried once with the newest one. Every JSON file records the backend
+  that produced it in `parser_backend`, alongside `generator_version` and `ruby_version`.
+- **Ruby DSL files are discovered, not just `.rb`.** `Rakefile`, `Gemfile`, `Capfile`,
+  `Vagrantfile`, `Fastfile` and friends are matched by basename, plus the `.gemspec`, `.rake`,
+  `.ru`, `.rbi`, `.thor`, `.jbuilder`, `.axlsx` and `.rabl` extensions. Vendor and tool
+  directories (`.git`, `.bundle`, `.venv`, …) are skipped.
+- **Non-UTF-8 sources survive.** `# coding:` magic comments are honoured and undecodable bytes are
+  scrubbed rather than dropping the file, marked with `encoding_scrubbed: true`.
+- **Deeply nested source is truncated, not dropped**, with `truncated: true` on the boundary node
+  and a `truncated_nodes` count at the top level (`--max-depth`).
+- **Semantic metadata for consumers**: a `magic_comments` array (Sorbet `typed:` levels,
+  `frozen_string_literal`, …), `call_operator`/`has_parentheses` on calls, `percent_array` and
+  regexp `options`, heredoc body offsets, and `has_sig: true` on a `def` preceded by a Sorbet `sig`
+  block.
+- **Two side-records per run, written inside the output directory**, both ending in `.jsonl` so a
+  consumer globbing `*.json` for ASTs never mistakes them for one:
+  `ruby_ast_gen_manifest.jsonl` (one object: input/output, backend, counts of parsed, failed,
+  skipped, excluded and truncated files, threads, max depth) and
+  `ruby_ast_gen_diagnostics.jsonl` (one object per failed file with message, line, column), the
+  latter written only when something failed and removed when a later run is clean.
+
+To check which backend a machine will use:
+
+```shell
+rbastgen --parser-info
 ```
 
+`Parser gem` and `Prism gem` name the versions actually loaded — the vendored `parser` and the
+runtime's `prism` — so they are the quickest way to tell which copy of each library a machine is
+parsing with. They read `unavailable` only when a library genuinely is not loaded.
+
+#### Environment variables
+
+| Variable                          | Default                | Purpose                                                                                                                                      |
+| --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ATOM_RUBY_HOME`                  | unset                  | Ruby install directory to use when a suitable `ruby` is not on the `PATH`; its `bin` is prepended to `PATH` for the child process.           |
+| `RUBY_CMD`                        | `ruby`                 | Ruby interpreter to invoke. Set this (or `ATOM_RUBY_HOME`) when the detected version is not 3.4.x/4.0.x.                                     |
+| `RUBY_ASTGEN_BIN`                 | bundled `ruby_ast_gen` | The generator script to run. Point it at a checkout's `exe/ruby_ast_gen` to test an unreleased `ruby_ast_gen` without touching this package. |
+| `ATOM_CWD`                        | `process.cwd()`        | Working directory for the generator, which is what relative `-i`/`-o` paths resolve against.                                                 |
+| `ATOM_TIMEOUT` / `ASTGEN_TIMEOUT` | unset (no timeout)     | Milliseconds before the generator process is killed. `ATOM_TIMEOUT` wins; a non-numeric value is ignored. Also honoured by `scalasem`.       |
+| `GEM_PATH`                        | unset                  | Preserved and appended to the vendored bundle, so gems installed on the machine (a newer `prism`, for instance) stay reachable.              |
+
 ### scalasem
 
 ```text
@@ -73,6 +160,30 @@ Example:
 scalasem $(pwd) slices.json
 ```
 
+## Testing
+
+`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby workflow tests
+run the `rbastgen` command end to end against `test-fixtures/projects/ruby-parsing`, so they need a
+built plugin bundle and a supported Ruby, and they skip themselves cleanly when either is missing:
+
+```shell
+bash build.sh --ruby-only
+npm run test:ruby
+```
+
+`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch
+project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle
+is present and free of compiled extensions. CI runs it, and it runs the same way locally.
+
+They assert what a consumer depends on rather than that the command ran: which files are discovered
+(including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`,
+`percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a
+failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no
+stale record behind, and that the bundle contains no compiled extension. CI runs them under both
+Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball
+and parses with that, which is the check that catches a bundle usable only on the Ruby that built
+it.
+
 ## License
 
 MIT

package/astgen.js

@@ -4,8 +4,15 @@ import { join, dirname, relative, resolve, basename } from "path";
 import { fileURLToPath } from "url";
 import { parse } from "@babel/parser";
 import { parse as parseHermes } from "hermes-parser";
-import tsc from "typescript";
-import { tmpdir } from "os";
+// TypeScript 7.0 (the native Go port) ships without a programmatic compiler
+// API, which astgen relies on heavily (createProgram, getTypeChecker,
+// forEachChild, SyntaxKind, TypeFormatFlags, ...). The officially supported
+// bridge until the API returns in TS 7.1 is the @typescript/typescript6
+// package, which is the same TS 6 engine and therefore keeps AST shapes and
+// type-inference accuracy identical.
+import tsc from "@typescript/typescript6";
+import { tmpdir, cpus, availableParallelism } from "os";
+import { Worker, isMainThread, parentPort, workerData } from "worker_threads";
 import {
   readFileSync,
   mkdirSync,
@@ -18,7 +25,11 @@ import {
 } from "fs";
 import { getAllFiles } from "@appthreat/atom-common";
 
-const ASTGEN_VERSION = "4.0.0";
+// Printed by `astgen --version`. Downstream frontends (e.g. chen's jssrc2cpg)
+// fold this into their parse-cache fingerprint, so it MUST be bumped whenever
+// the emitted AST/type shape changes — otherwise stale cached parses from an
+// older astgen are silently reused. Bumped for the Babel 8 AST-shape change.
+const ASTGEN_VERSION = "4.1.0";
 
 const HELP_TEXT = `Options:
   -i, --src      Source directory                                 [default: "."]
@@ -381,6 +392,25 @@ const babelSafeFlowParserOptions = {
   ]
 };
 
+// Test files (e.g. *.poku.js, *.test.ts, *.spec.js, __tests__/*) are
+// test-runner artifacts that are typically the heaviest, lowest-value inputs
+// for type generation (each is often a full twin of a source module wrapped in
+// test scaffolding). They are excluded by default to keep the type-generation
+// phase scalable. Set ASTGEN_INCLUDE_TEST_FILES=true to restore them (e.g. when
+// a downstream consumer wants test files analysed).
+const shouldIncludeTestFiles =
+  process.env?.ASTGEN_INCLUDE_TEST_FILES === "true";
+
+const TEST_FILE_EXT = "(?:js|jsx|cjs|mjs|ts|tsx|mts|cts)";
+const TEST_FILE_PATTERN = new RegExp(
+  `(?:\\.(?:poku|test|spec|e2e|integration|it)\\.${TEST_FILE_EXT}$` +
+    `|[\\\\/]__(?:tests|mocks)__[\\\\/])`,
+  "i"
+);
+
+const isExcludedTestFile = (file) =>
+  !shouldIncludeTestFiles && TEST_FILE_PATTERN.test(file);
+
 const shouldIncludeNodeModulesBundles =
   process.env?.ASTGEN_INCLUDE_NODE_MODULES_BUNDLES === "true" ||
   (process.env?.ASTGEN_IGNORE_DIRS &&
@@ -426,7 +456,9 @@ const getAllSrcJSAndTSFiles = (src) => {
   // Step 2: Combine both lists
   return Promise.all([allFilesPromise, bundledFilesPromise]).then(
     ([allFiles, bundledFiles]) =>
-      [...new Set([...allFiles, ...bundledFiles])].sort()
+      [...new Set([...allFiles, ...bundledFiles])]
+        .filter((file) => !isExcludedTestFile(file))
+        .sort()
   );
 };
 
@@ -1087,6 +1119,33 @@ function createTsc(srcFiles, src) {
     const typeChecker = program.getTypeChecker();
     const seenTypes = new Map();
 
+    // Return-type inference below can re-walk the body of the *same* function
+    // declaration once per referencing call site / initializer (e.g. a helper
+    // called 50 times triggers 50 identical body walks). Since the checker
+    // state is fixed for the lifetime of this program, the inferred string for
+    // a given node is deterministic, so we memoize by node to remove the
+    // redundant re-walks without changing any output. A sentinel distinguishes
+    // "computed and produced undefined" from "not yet computed".
+    const RETURN_TYPE_NOT_COMPUTED = Symbol("returnTypeNotComputed");
+    const memoizeByNode = (fn) => {
+      const cache = new WeakMap();
+      return (node, ...rest) => {
+        if (!node || typeof node !== "object" || rest.length > 0) {
+          return fn(node, ...rest);
+        }
+        const cached = cache.get(node);
+        if (cached !== undefined) {
+          return cached === RETURN_TYPE_NOT_COMPUTED ? undefined : cached;
+        }
+        const result = fn(node);
+        cache.set(
+          node,
+          result === undefined ? RETURN_TYPE_NOT_COMPUTED : result
+        );
+        return result;
+      };
+    };
+
     const safeTypeToString = (type, context) => {
       try {
         return normalizeTypeString(
@@ -1218,7 +1277,7 @@ function createTsc(srcFiles, src) {
       return collectedTypes;
     };
 
-    const inferAsyncReturnTypeFromBody = (node) => {
+    const inferAsyncReturnTypeFromBodyImpl = (node) => {
       if (!node.body) {
         return undefined;
       }
@@ -1240,6 +1299,9 @@ function createTsc(srcFiles, src) {
         ? unionType
         : `Promise<${unionType}>`;
     };
+    const inferAsyncReturnTypeFromBody = memoizeByNode(
+      inferAsyncReturnTypeFromBodyImpl
+    );
 
     const buildFunctionSignatureType = (node, returnTypeStr) => {
       if (!node.parameters) {
@@ -1295,7 +1357,7 @@ function createTsc(srcFiles, src) {
       return undefined;
     };
 
-    const inferFunctionDeclarationReturnType = (declaration) => {
+    const inferFunctionDeclarationReturnTypeImpl = (declaration) => {
       const signature = typeChecker.getSignatureFromDeclaration(declaration);
       if (!signature) {
         return undefined;
@@ -1317,8 +1379,11 @@ function createTsc(srcFiles, src) {
       }
       return inferredType;
     };
+    const inferFunctionDeclarationReturnType = memoizeByNode(
+      inferFunctionDeclarationReturnTypeImpl
+    );
 
-    const inferAsyncReturnTypeFromSyntaxBody = (node) => {
+    const inferAsyncReturnTypeFromSyntaxBodyImpl = (node) => {
       if (!node?.body) {
         return undefined;
       }
@@ -1361,6 +1426,9 @@ function createTsc(srcFiles, src) {
         ? unionType
         : `Promise<${unionType}>`;
     };
+    const inferAsyncReturnTypeFromSyntaxBody = memoizeByNode(
+      inferAsyncReturnTypeFromSyntaxBodyImpl
+    );
 
     const addType = (node, currentSeenTypes = seenTypes) => {
       // STRUCTURAL/CONTAINER NODES
@@ -1761,64 +1829,209 @@ function createTsc(srcFiles, src) {
   }
 }
 
+/**
+ * Expand the set of output files to include the source files that tsconfig
+ * pulls in (that live under the source root), mirroring the program root names.
+ * Computed from the parsed tsconfig alone so no TypeScript program has to be
+ * built on the main thread.
+ */
+const expandSrcFilesWithRootNames = (srcFiles, rootNames, src) => {
+  if (!rootNames) {
+    return srcFiles;
+  }
+  const srcRoot = resolve(src);
+  const srcFileByResolvedPath = new Map(
+    srcFiles.map((file) => [resolve(file), file])
+  );
+  for (const file of rootNames) {
+    const resolvedFile = resolve(file);
+    if (
+      resolvedFile.startsWith(srcRoot) &&
+      /\.(?:js|jsx|cjs|mjs|ts|tsx|mts|cts)$/.test(file) &&
+      !isExcludedTestFile(file) &&
+      !srcFileByResolvedPath.has(resolvedFile)
+    ) {
+      srcFileByResolvedPath.set(
+        resolvedFile,
+        join(src, relative(srcRoot, resolvedFile))
+      );
+    }
+  }
+  return [...srcFileByResolvedPath.values()].sort();
+};
+
+const runGc = () => {
+  if (typeof globalThis.gc === "function") {
+    try {
+      globalThis.gc();
+    } catch (e) {
+      // ignore
+    }
+  } else if (typeof Bun !== "undefined" && typeof Bun.gc === "function") {
+    try {
+      Bun.gc(true);
+    } catch (e) {
+      // ignore
+    }
+  }
+};
+
+/**
+ * Process a list of files (AST + optional type generation) on the current
+ * thread, in memory-bounded chunks. `ts` is the shared TypeScript program
+ * instance (or undefined when type generation is disabled).
+ */
+const processFilesInline = async (srcFiles, options, ts) => {
+  const CONCURRENCY_LIMIT = Math.max(
+    1,
+    Number.parseInt(process.env.ASTGEN_CONCURRENCY || "10", 10) || 10
+  );
+  for (let i = 0; i < srcFiles.length; i += CONCURRENCY_LIMIT) {
+    const chunk = srcFiles.slice(i, i + CONCURRENCY_LIMIT);
+    await Promise.all(chunk.map((file) => processFile(file, options, ts)));
+    runGc();
+  }
+};
+
+/**
+ * Decide how many worker threads to use for the type-generation phase.
+ * The TypeScript checker is single-threaded and CPU-bound, so real speedups
+ * only come from running independent programs on separate threads. Each worker
+ * builds its own full program (needed for cross-file type resolution).
+ *
+ * IMPORTANT: parallelism is OPT-IN. When files are sharded across workers,
+ * TypeScript's per-program type-id assignment order changes, which reorders the
+ * members of a small number of inferred union types (e.g. `A | B` -> `B | A`;
+ * semantically identical, textually different). To keep the default output
+ * byte-identical for downstream consumers, workers are only used when
+ * ASTGEN_TYPE_WORKERS is set explicitly. Set it to the desired worker count
+ * (e.g. number of cores) to trade that cosmetic reordering for speed; "auto"
+ * derives the count from the available CPUs. Unset or 1 => single-threaded.
+ */
+const resolveTypeWorkerCount = (fileCount) => {
+  const envValue = process.env.ASTGEN_TYPE_WORKERS;
+  if (envValue === undefined || envValue === "") {
+    return 1;
+  }
+  if (envValue.toLowerCase() === "auto") {
+    let cores = 1;
+    try {
+      cores =
+        typeof availableParallelism === "function"
+          ? availableParallelism()
+          : cpus().length;
+    } catch (e) {
+      cores = 1;
+    }
+    const autoCount = Math.max(cores - 1, 1);
+    return Math.min(autoCount, Math.max(1, fileCount));
+  }
+  const requested = Number.parseInt(envValue, 10);
+  if (!Number.isFinite(requested) || requested < 1) {
+    return 1;
+  }
+  return Math.min(requested, Math.max(1, fileCount));
+};
+
+/**
+ * Run the type-generation phase across worker threads. Files are sharded
+ * round-robin so heavy files spread across workers. Each worker builds its own
+ * program over `projectFiles`; the per-file AST and type set are identical to
+ * the single-threaded path, except that a small number of inferred union types
+ * may have their members printed in a different order (see resolveTypeWorkerCount).
+ * Returns true only if every worker completed cleanly; on any failure the
+ * caller falls back to inline processing (writes are idempotent, so
+ * re-processing is safe).
+ */
+const runTypeGenerationInWorkers = (
+  srcFiles,
+  projectFiles,
+  options,
+  workerCount
+) => {
+  const shards = Array.from({ length: workerCount }, () => []);
+  srcFiles.forEach((file, index) => shards[index % workerCount].push(file));
+  const workerEntry = fileURLToPath(import.meta.url);
+  return Promise.all(
+    shards.map((shard, index) => {
+      if (shard.length === 0) {
+        return Promise.resolve(true);
+      }
+      return new Promise((resolvePromise) => {
+        let settled = false;
+        const settle = (value) => {
+          if (!settled) {
+            settled = true;
+            resolvePromise(value);
+          }
+        };
+        try {
+          const worker = new Worker(workerEntry, {
+            workerData: {
+              kind: "astgen-typegen",
+              shard,
+              projectFiles,
+              options,
+              index
+            }
+          });
+          worker.on("error", (err) => {
+            console.error("astgen type worker failed:", err?.message || err);
+            settle(false);
+          });
+          worker.on("exit", (code) => settle(code === 0));
+        } catch (err) {
+          console.error("Unable to start astgen type worker:", err?.message);
+          settle(false);
+        }
+      });
+    })
+  ).then((results) => results.every(Boolean));
+};
+
 /**
  * Generate AST for JavaScript or TypeScript
  */
 const createJSAst = async (options) => {
   try {
-    const promiseMap = await getAllSrcJSAndTSFiles(options.src);
-    let srcFiles = promiseMap.flatMap((d) => d).sort();
-    let ts;
-    if (options.tsTypes) {
-      const projectFiles = !shouldIncludeNodeModulesBundles
-        ? srcFiles.filter((file) => !file.includes("node_modules"))
-        : srcFiles;
-      ts = createTsc(projectFiles, options.src);
-      if (ts?.rootNames) {
-        const srcRoot = resolve(options.src);
-        const srcFileByResolvedPath = new Map(
-          srcFiles.map((file) => [resolve(file), file])
-        );
-        for (const file of ts.rootNames) {
-          const resolvedFile = resolve(file);
-          if (
-            resolvedFile.startsWith(srcRoot) &&
-            /\.(?:js|jsx|cjs|mjs|ts|tsx|mts|cts)$/.test(file) &&
-            !srcFileByResolvedPath.has(resolvedFile)
-          ) {
-            srcFileByResolvedPath.set(
-              resolvedFile,
-              join(options.src, relative(srcRoot, resolvedFile))
-            );
-          }
-        }
-        srcFiles = [...srcFileByResolvedPath.values()].sort();
-      }
+    const discovered = await getAllSrcJSAndTSFiles(options.src);
+    let srcFiles = [...discovered].sort();
+
+    if (!options.tsTypes) {
+      await processFilesInline(srcFiles, options, undefined);
+      return;
     }
-    const CONCURRENCY_LIMIT = Math.max(
-      1,
-      Number.parseInt(process.env.ASTGEN_CONCURRENCY || "10", 10) || 10
+
+    const projectFiles = !shouldIncludeNodeModulesBundles
+      ? srcFiles.filter((file) => !file.includes("node_modules"))
+      : srcFiles;
+    // Compute the program root names from the parsed tsconfig without building
+    // a program on the main thread, then expand the output file set to match.
+    const tscConfig = createTscProgramConfig(projectFiles, options.src);
+    srcFiles = expandSrcFilesWithRootNames(
+      srcFiles,
+      tscConfig?.rootNames,
+      options.src
     );
-    const chunks = [];
-    for (let i = 0; i < srcFiles.length; i += CONCURRENCY_LIMIT) {
-      chunks.push(srcFiles.slice(i, i + CONCURRENCY_LIMIT));
-    }
-    for (const chunk of chunks) {
-      await Promise.all(chunk.map((file) => processFile(file, options, ts)));
-      if (typeof globalThis.gc === "function") {
-        try {
-          globalThis.gc();
-        } catch (e) {
-          // ignore
-        }
-      } else if (typeof Bun !== "undefined" && typeof Bun.gc === "function") {
-        try {
-          Bun.gc(true);
-        } catch (e) {
-          // ignore
-        }
+
+    const workerCount = resolveTypeWorkerCount(srcFiles.length);
+    if (workerCount > 1) {
+      const ranInParallel = await runTypeGenerationInWorkers(
+        srcFiles,
+        projectFiles,
+        options,
+        workerCount
+      );
+      if (ranInParallel) {
+        return;
       }
+      console.error(
+        "Falling back to single-threaded type generation after worker failure."
+      );
     }
+
+    const ts = createTsc(projectFiles, options.src);
+    await processFilesInline(srcFiles, options, ts);
   } catch (err) {
     console.error(err);
   }
@@ -1861,6 +2074,12 @@ const createVueAst = async (options) => {
 const getCircularReplacer = () => {
   const seen = new WeakSet();
   return (key, value) => {
+    // Babel 8 emits BigIntLiteral/`extra.value` as a native bigint, which
+    // JSON.stringify cannot serialize. Emit it as a string, which also matches
+    // the Babel 7 shape (BigIntLiteral.value was already a string there).
+    if (typeof value === "bigint") {
+      return value.toString();
+    }
     if (typeof value === "object" && value !== null) {
       if (seen.has(value)) {
         return;
@@ -1969,4 +2188,25 @@ async function main(argvs) {
   }
 }
 
-main(process.argv);
+/**
+ * Type-generation worker entry point. Runs in a worker thread spawned by
+ * runTypeGenerationInWorkers: builds its own TypeScript program over the full
+ * project file set and processes only its assigned shard, so cross-file type
+ * resolution is identical to the single-threaded path.
+ */
+const runTypeGenWorker = async ({ shard, projectFiles, options }) => {
+  try {
+    const ts = createTsc(projectFiles, options.src);
+    await processFilesInline(shard, options, ts);
+    parentPort?.postMessage({ done: true });
+  } catch (err) {
+    console.error(err);
+    process.exit(1);
+  }
+};
+
+if (isMainThread) {
+  main(process.argv);
+} else if (workerData?.kind === "astgen-typegen") {
+  runTypeGenWorker(workerData);
+}

package/package.json

@@ -1,22 +1,25 @@
 {
   "name": "@appthreat/atom-parsetools",
-  "version": "1.2.2",
+  "version": "1.4.0",
   "description": "Parsing tools that complement the @appthreat/atom project.",
   "main": "./index.js",
   "type": "module",
   "scripts": {
     "pretty": "prettier --write *.js --trailing-comma=none",
-    "test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/evaluate-astgen.js",
+    "test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
     "test:evaluate": "node test-fixtures/evaluate-astgen.js",
+    "test:ruby": "node test-fixtures/rbastgen-regression.js",
+    "test:shape": "node test-fixtures/astgen-shape-snapshot.js",
+    "test:shape:update": "UPDATE_SHAPE_SNAPSHOT=1 node test-fixtures/astgen-shape-snapshot.js",
     "test:json": "node test-fixtures/astgen-json-regression.js",
     "test:fixtures": "node test-fixtures/test-suite.js",
     "test:vue": "node test-fixtures/astgen-vue-regression.js"
   },
   "dependencies": {
     "@appthreat/atom-common": "^1.1.0",
-    "@babel/parser": "^7.29.3",
-    "hermes-parser": "^0.36.1",
-    "typescript": "^6.0.3"
+    "@babel/parser": "^8.0.4",
+    "@typescript/typescript6": "^6.0.2",
+    "hermes-parser": "^0.37.0"
   },
   "bin": {
     "astgen": "astgen.js",
@@ -25,7 +28,7 @@
     "scalasem": "scalasem.js"
   },
   "engines": {
-    "node": ">=16.0.0"
+    "node": ">=22.0.0"
   },
   "repository": {
     "type": "git",