yuku-analyzer 0.5.32

package/README.md

@@ -0,0 +1,164 @@
+# yuku-analyzer
+
+Full semantic analysis for JavaScript and TypeScript: scopes, symbols, resolved references, closures, and cross-file module linking. Powered by [Yuku](https://github.com/yuku-toolchain/yuku), written in Zig.
+
+The usual options are a hand-rolled scope tracker (fragile, per-file, re-bugged in every tool) or the TypeScript compiler (correct, but hundreds of milliseconds per file and a compiler-sized dependency). This is the fast path between them. One native call per file, then every query is plain JavaScript with zero per-query FFI cost, sub-millisecond on a typical file.
+
+```bash
+npm install yuku-analyzer yuku-parser
+```
+
+`yuku-parser` is a peer dependency.
+
+## Quick start
+
+```js
+import { Analyzer, SymbolFlags } from "yuku-analyzer";
+
+const analyzer = new Analyzer();
+
+analyzer.addFile("lib.ts", `export const helper = (x: number) => x * 2;`);
+const main = analyzer.addFile(
+  "main.ts",
+  `import { helper } from "./lib.ts";
+   export const out = helper(21);`,
+);
+
+// per-file semantics
+const helperSym = main.rootScope.find("helper");
+console.log(helperSym.has(SymbolFlags.Import)); // true
+console.log(helperSym.references.length); // 1, the call site
+
+// cross-file: follow the import to where helper is actually defined
+const def = helperSym.definition();
+console.log(def.module.path); // "lib.ts"
+console.log(def.symbol.has(SymbolFlags.Const)); // true
+```
+
+## Why this exists
+
+JavaScript tooling that needs real semantics has had two options: track scopes by hand during a walk (fragile, incomplete, and re-implemented in every tool), or embed a full language service (heavy). `yuku-analyzer` is the missing middle: the exact scope and binding model of a production compiler, exposed to JavaScript as a small, fast object graph.
+
+Nothing semantic is reimplemented in JavaScript. The binder, scope tree, reference resolution, and module records are computed by the same well-tested native analyzer that powers the rest of Yuku, then shipped to JavaScript as one compact buffer that the JS side only decodes, through lazy zero-copy views. That handoff is usually where native tooling stalls: either every query pays an FFI round trip, or the semantics get a hand-written JS twin that slowly drifts. Here there is one implementation and one crossing, so it cannot drift, and the corner cases arrive already correct: catch-clause scope sharing, named function expression scopes, `var` hoist targets, TS declaration merging, value space versus type space, and write detection through destructuring patterns.
+
+## What one `addFile` gives you
+
+```js
+const module = analyzer.addFile("app.tsx", source);
+
+module.ast                  // ESTree / TS-ESTree program
+module.scopes               // every lexical scope, as a tree
+module.symbols              // every declared binding
+module.references           // every identifier use, resolved to its symbol
+module.unresolvedReferences // free names and globals
+module.imports              // spec-true import records
+module.exports              // spec-true export records
+module.diagnostics          // syntax + semantic errors
+```
+
+And node-level queries that work directly on AST nodes:
+
+```js
+module.symbolOf(node)    // the symbol a node declares or references
+module.referenceOf(node) // the reference recorded for an identifier
+module.scopeOf(node)     // the innermost scope containing the node
+module.resolve("name")   // scope-chain lookup, like the engine does at runtime
+```
+
+Node identity is exact: the node you reach by walking `module.ast` and the node a semantic query returns are the same JavaScript object, so `===` always works.
+
+## Walking with semantic context
+
+`module.walk` is a typed visitor walk where every handler also receives the current scope, symbol, and reference. No manual scope tracking, ever:
+
+```js
+module.walk({
+  Identifier(node, ctx) {
+    if (ctx.reference?.isWrite && ctx.symbol?.has(SymbolFlags.Import)) {
+      console.log(`${node.name} assigns to an import`);
+    }
+  },
+  FunctionDeclaration: {
+    enter(node, ctx) {
+      console.log(node.id.name, "declared in a", ctx.scope.kind, "scope");
+    },
+  },
+});
+```
+
+The walk mutates in place: `ctx.replace(node)`, `ctx.remove()`, `ctx.insertBefore(node)`, `ctx.insertAfter(node)`, plus `ctx.skip()` and `ctx.stop()`. Transform the AST during the walk and print it with [`yuku-codegen`](https://www.npmjs.com/package/yuku-codegen).
+
+## Scanning without building the AST
+
+`module.scan` visits the parsed node records directly in the binary buffer. Nothing is materialized unless you ask for it, and symbols and references resolve in index space:
+
+```js
+const writes = [];
+module.scan({
+  Identifier(cursor) {
+    if (cursor.reference?.isWrite) writes.push(cursor.reference.name);
+  },
+});
+```
+
+This answers questions like "every write to an imported binding across 10,000 files" without constructing a single AST object. For semantic queries like this, scanning runs about 4x faster than the equivalent walk, since it never builds the AST. Scan to find, walk to change.
+
+## Closure analysis
+
+`capturesOf` reports the free variables of any function: every outer binding it closes over, with the capturing reference sites and whether the function writes to the binding. Shadowing and aliasing are handled by the resolved reference table, not by name matching:
+
+```js
+const [fn] = main.findAll("FunctionDeclaration");
+for (const capture of main.capturesOf(fn)) {
+  console.log(capture.symbol.name, capture.isWritten ? "(written)" : "");
+}
+```
+
+## Cross-file analysis
+
+The analyzer joins imports to exports across every added file with the spec's ResolveExport semantics: re-export and `export *` chains are followed per name, `default` never travels through `export *`, and a name supplied by multiple `export *` declarations through different bindings is reported as ambiguous.
+
+```js
+// where is this binding actually defined?
+analyzer.definitionOf(symbol); // { module, symbol } or null for external modules
+
+// every use across the whole graph, imports followed back
+analyzer.referencesOf(symbol); // [{ module, reference }, ...]
+
+module.exportedNames(); // every exported name, `export *` chains included
+module.dependencies;    // modules this file imports from
+module.dependents;      // modules that import this file
+analyzer.diagnostics;   // e.g. "Module './lib.ts' has no export 'helpr'"
+```
+
+Linking is automatic: every cross-file surface relinks on demand after files change. Call `analyzer.link()` explicitly if you want to control when the work happens.
+
+Module resolution is pluggable. The default resolves relative specifiers among added files with standard extension and index probing. Pass your own resolver for anything else:
+
+```js
+const analyzer = new Analyzer({
+  resolve: (specifier, importerPath) => myResolver(specifier, importerPath),
+});
+```
+
+## Performance
+
+Analysis runs in the native parser pass, so full semantics cost roughly half of parsing time on top of the parse itself. Validated against 55,000+ real-world files.
+
+Concretely, on an Apple M-series machine: parsing plus complete semantic analysis of a typical source file lands well under a millisecond, walking sustains tens of millions of nodes per second, and linking a 2,000-module graph takes about a millisecond.
+
+## TypeScript
+
+Everything is fully typed. Visitor handlers receive exact node types, AST types come from `yuku-parser`, and the semantic surface (`Module`, `Scope`, `Symbol`, `Reference`, `Import`, `Export`, `Capture`) is exported:
+
+```ts
+import type { Module, Symbol, Capture } from "yuku-analyzer";
+```
+
+## Documentation
+
+The full documentation, including the architecture, every type, and the design decisions: [yuku.fyi/analyzer](https://yuku.fyi/analyzer).
+
+## License
+
+MIT

package/analyzer.js

@@ -0,0 +1,290 @@
+import { Module, SymbolFlags } from "./module.js";
+
+const RESOLVE_EXTENSIONS = [".tsx", ".ts", ".jsx", ".js", ".mts", ".mjs", ".cts", ".cjs"];
+
+// the third resolution outcome of ResolveExport (16.2.1.7.2.2): more
+// than one `export *` declaration supplies the name through different
+// bindings. distinct from null (not found / circular).
+const AMBIGUOUS = Symbol("ambiguous");
+
+export class Analyzer {
+  #modules = new Map();
+  #resolve;
+  #diagnostics = [];
+  #dirty = false;
+  #linking = false;
+
+  constructor(options = {}) {
+    this.#resolve = options.resolve ?? defaultResolve(this.#modules);
+  }
+
+  addFile(path, source, options) {
+    const module = new Module(this, path, source, options);
+    this.#modules.set(path, module);
+    this.#dirty = true;
+    return module;
+  }
+
+  removeFile(path) {
+    const removed = this.#modules.delete(path);
+    if (removed) this.#dirty = true;
+    return removed;
+  }
+
+  module(path) {
+    return this.#modules.get(path);
+  }
+
+  get modules() {
+    return this.#modules;
+  }
+
+  get diagnostics() {
+    this._ensureLinked();
+    return this.#diagnostics;
+  }
+
+  // the static half of spec linking. resolves every module request to an
+  // added module, wires the dependency graph, and reports binding
+  // validation failures as diagnostics rather than throwing. unresolvable
+  // specifiers are skipped so a partial graph still links.
+  link() {
+    this.#dirty = false;
+    this.#linking = true;
+    this.#diagnostics = [];
+    for (const module of this.#modules.values()) {
+      module._deps = [];
+      module._dependents = [];
+      // drop earlier resolutions so removed modules cannot leak through
+      for (const record of module.imports) record._resolved = null;
+      for (const record of module.exports) record._resolved = null;
+    }
+    for (const module of this.#modules.values()) {
+      for (const record of module.imports) {
+        record._resolved = this.#resolveModule(record.specifier, module, false);
+        if (record._resolved === null) continue;
+        this.#wire(module, record._resolved);
+        // namespace and side-effect imports name nothing to validate
+        if (record.name !== null) {
+          this.#validate(module, record, record.name, "Import");
+        }
+      }
+      for (const record of module.exports) {
+        if (record.specifier === null) continue;
+        record._resolved = this.#resolveModule(record.specifier, module, false);
+        if (record._resolved === null) continue;
+        this.#wire(module, record._resolved);
+        // `export * as ns` (fromName null) trivially resolves and bare
+        // `export *` conflicts are deferred to use sites, per spec
+        if (record.fromName !== null) {
+          this.#validate(module, record, record.fromName, "Re-export");
+        }
+      }
+    }
+    this.#linking = false;
+  }
+
+  // reports a record whose name does not resolve, or resolves
+  // ambiguously, against its (already resolved) source module
+  #validate(module, record, name, what) {
+    const resolution = this.#resolveExport(record._resolved, name, []);
+    if (resolution !== null && resolution !== AMBIGUOUS) return;
+    const message =
+      resolution === null
+        ? `Module '${record.specifier}' has no export '${name}'`
+        : `${what} '${name}' of module '${record.specifier}' is ambiguous: multiple 'export *' declarations supply it`;
+    this.#diagnostics.push({
+      severity: "error",
+      message,
+      module: module.path,
+      start: record.node.start,
+      end: record.node.end,
+    });
+  }
+
+  // follows an import binding to its defining module and symbol via
+  // ResolveExport (InitializeEnvironment step 7.c). namespace
+  // resolutions yield { module, symbol: null }; unresolved or ambiguous
+  // chains yield null.
+  definitionOf(symbol) {
+    this._ensureLinked();
+    let module = symbol.module;
+    let current = symbol;
+    const seen = new Set();
+    while ((current.flags & SymbolFlags.Import) !== 0) {
+      const key = `${module.path}:${current.id}`;
+      if (seen.has(key)) return null;
+      seen.add(key);
+      const record = module._importOfSymbol(current.id);
+      if (record === undefined || record._resolved === null) return null;
+      if (record.isNamespace || record.name === null) {
+        return { module: record._resolved, symbol: null };
+      }
+      const resolution = this.#resolveExport(record._resolved, record.name, []);
+      if (resolution === null || resolution === AMBIGUOUS) return null;
+      if (resolution.namespace) return { module: resolution.module, symbol: null };
+      if (resolution.symbol === null) return null;
+      module = resolution.module;
+      current = resolution.symbol;
+    }
+    return { module, symbol: current };
+  }
+
+  referencesOf(symbol) {
+    this._ensureLinked();
+    const origin = this.definitionOf(symbol) ?? { module: symbol.module, symbol };
+    if (origin.symbol === null) return [];
+    const out = [];
+    for (const reference of origin.symbol.references) {
+      out.push({ module: origin.module, reference });
+    }
+    for (const module of this.#modules.values()) {
+      if (module === origin.module) continue;
+      for (const record of module.imports) {
+        if (record.local === null) continue;
+        const definition = this.definitionOf(record.local);
+        if (
+          definition !== null &&
+          definition.symbol === origin.symbol &&
+          definition.module === origin.module
+        ) {
+          for (const reference of record.local.references) {
+            out.push({ module, reference });
+          }
+        }
+      }
+    }
+    return out;
+  }
+
+  _ensureLinked() {
+    if (this.#dirty && !this.#linking) this.link();
+  }
+
+  #resolveModule(specifier, importer, quiet) {
+    const resolved = this.#resolve(specifier, importer.path);
+    if (resolved === null || resolved === undefined) return null;
+    const module = this.#modules.get(resolved);
+    if (module === undefined) {
+      if (!quiet) {
+        this.#diagnostics.push({
+          severity: "warning",
+          message: `Resolver returned '${resolved}' for '${specifier}' but no such file was added`,
+          module: importer.path,
+          start: 0,
+          end: 0,
+        });
+      }
+      return null;
+    }
+    return module;
+  }
+
+  #wire(from, to) {
+    if (!from._deps.includes(to)) from._deps.push(to);
+    if (!to._dependents.includes(from)) to._dependents.push(from);
+  }
+
+  // ResolveExport(exportName, resolveSet), 16.2.1.7.2.2: resolves a
+  // name to the binding that defines it, following named re-export and
+  // `export *` chains. returns { module, symbol, namespace } (the
+  // spec's ResolvedBinding, with namespace: true for its namespace
+  // [[BindingName]]), null when unresolvable or circular, or AMBIGUOUS.
+  //
+  // local and indirect entries live in one name-keyed map (_exportMap);
+  // duplicate export names are an early error, so the spec's step 5/6
+  // partition cannot change the outcome. ParseModule step 10.a.ii
+  // (rewriting an export of an imported binding through its original
+  // module) is applied at resolution time instead of collection time.
+  #resolveExport(module, exportName, resolveSet) {
+    // step 3: a repeated (module, exportName) pair is a circular import
+    // request. the pair key still permits re-entering a module for a
+    // different name, which renaming re-export chains legitimately do.
+    for (const entry of resolveSet) {
+      if (entry.module === module && entry.exportName === exportName) return null;
+    }
+    resolveSet.push({ module, exportName });
+
+    const direct = module._exportMap().get(exportName);
+    if (direct !== undefined) {
+      if (direct.specifier === null) {
+        const local = direct.local;
+        if (local !== null && (local.flags & SymbolFlags.Import) !== 0) {
+          // export of an imported binding: resolve through the original
+          // module (the ParseModule rewrite)
+          const record = module._importOfSymbol(local.id);
+          if (record !== undefined) {
+            const imported =
+              record._resolved ?? this.#resolveModule(record.specifier, module, true);
+            if (imported === null) return null;
+            if (record.isNamespace) return { module: imported, symbol: null, namespace: true };
+            return this.#resolveExport(imported, record.name, resolveSet);
+          }
+        }
+        // step 5: the module provides the direct binding
+        return { module, symbol: local, namespace: false };
+      }
+      const next = direct._resolved ?? this.#resolveModule(direct.specifier, module, true);
+      if (next === null) return null;
+      // step 6: `export * as ns` binds the namespace, named re-exports
+      // follow the chain under the source name
+      if (direct.isNamespaceReexport) return { module: next, symbol: null, namespace: true };
+      return this.#resolveExport(next, direct.fromName, resolveSet);
+    }
+
+    // step 7: "default" is never satisfied through `export *`
+    if (exportName === "default") return null;
+
+    // steps 8-9: identical star bindings collapse, different ones are
+    // ambiguous
+    let starResolution = null;
+    for (const star of module._starExports()) {
+      const next = star._resolved ?? this.#resolveModule(star.specifier, module, true);
+      if (next === null) continue;
+      const resolution = this.#resolveExport(next, exportName, resolveSet);
+      if (resolution === AMBIGUOUS) return AMBIGUOUS;
+      if (resolution === null) continue;
+      if (starResolution === null) {
+        starResolution = resolution;
+      } else if (
+        resolution.module !== starResolution.module ||
+        resolution.namespace !== starResolution.namespace ||
+        resolution.symbol !== starResolution.symbol
+      ) {
+        return AMBIGUOUS;
+      }
+    }
+    return starResolution;
+  }
+}
+
+function defaultResolve(modules) {
+  return (specifier, importer) => {
+    if (!specifier.startsWith(".")) return null;
+    const base = joinPath(dirnameOf(importer), specifier);
+    if (modules.has(base)) return base;
+    for (const ext of RESOLVE_EXTENSIONS) {
+      if (modules.has(base + ext)) return base + ext;
+    }
+    for (const ext of RESOLVE_EXTENSIONS) {
+      const index = `${base}/index${ext}`;
+      if (modules.has(index)) return index;
+    }
+    return null;
+  };
+}
+
+function dirnameOf(path) {
+  const i = path.lastIndexOf("/");
+  return i === -1 ? "" : path.slice(0, i);
+}
+
+function joinPath(dir, relative) {
+  const parts = dir === "" ? [] : dir.split("/");
+  for (const part of relative.split("/")) {
+    if (part === "" || part === ".") continue;
+    if (part === "..") parts.pop();
+    else parts.push(part);
+  }
+  return parts.join("/");
+}

package/binding.js

@@ -0,0 +1,63 @@
+import { createRequire } from 'node:module';
+import { readFileSync } from 'node:fs';
+import { execSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { dirname, join } from 'node:path';
+
+const require = createRequire(import.meta.url);
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const { platform, arch } = process;
+
+const isFileMusl = (f) => f.includes('libc.musl-') || f.includes('ld-musl-');
+
+function isMusl() {
+  if (platform !== 'linux') return false;
+
+  try {
+    if (readFileSync('/usr/bin/ldd', 'utf-8').includes('musl')) return true;
+  } catch {}
+
+  try {
+    const report = typeof process.report?.getReport === 'function'
+      ? process.report.getReport()
+      : null;
+    if (report) {
+      const header = typeof report === 'string' ? JSON.parse(report).header : report.header;
+      if (header?.glibcVersionRuntime) return false;
+      if (Array.isArray(report.sharedObjects) && report.sharedObjects.some(isFileMusl)) return true;
+    }
+  } catch {}
+
+  try {
+    return execSync('ldd --version', { encoding: 'utf8' }).includes('musl');
+  } catch {}
+
+  return false;
+}
+
+function loadBinding() {
+  const errors = [];
+  const libc = platform === 'linux' ? (isMusl() ? '-musl' : '-gnu') : '';
+  const suffix = `${platform}-${arch}${libc}`;
+
+  try {
+    return require(join(__dirname, '@yuku-analyzer', 'binding-' + suffix, 'yuku-analyzer.node'));
+  } catch (e) {
+    errors.push(e);
+  }
+
+  try {
+    return require('@yuku-analyzer/binding-' + suffix + '/yuku-analyzer.node');
+  } catch (e) {
+    errors.push(e);
+  }
+
+  throw new Error(
+    `Failed to load native binding for ${platform}-${arch}.\n` +
+    `If this persists, try removing node_modules and reinstalling.\n` +
+    errors.map(e => `  - ${e.message}`).join('\n'),
+    { cause: errors[errors.length - 1] }
+  );
+}
+
+export default loadBinding();