@appthreat/atom-parsetools 1.5.0 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,12 +1,16 @@
1
- # Introduction
1
+ # atom-parsetools
2
2
 
3
3
  This package hosts a collection of parsing tools that complement the `@appthreat/atom` project. These tools offer parsing and analysis-related functionalities such as generating AST and semantics information in JSON format. The full list of tools and bin commands exposed by this package is below:
4
4
 
5
- - astgen - Generates AST for JavaScript and TypeScript projects in JSON format
5
+ - astgen - Generates AST for JavaScript, TypeScript, Vue and Svelte projects in JSON format
6
6
  - phpastgen - Generates AST for PHP projects using `php-parse` command from `nikic/php-parser`
7
7
  - rbastgen - Generates AST for Ruby projects using AppThreat's [`ruby_ast_gen`](https://github.com/AppThreat/ruby_ast_gen) gem (2.0.1)
8
8
  - scalasem - Generates a custom semantics slice for Scala Projects by utilising scalac command.
9
9
 
10
+ ## Documentation
11
+
12
+ The full documentation lives at [https://appthreat.github.io/atom-parsetools/](https://appthreat.github.io/atom-parsetools/): per-tool guides, the output format specification, environment variable reference, packaging notes, and ten hands-on tutorials. The pages are rendered from the [`docs`](docs/) directory of this repository.
13
+
10
14
  ## Runtime support
11
15
 
12
16
  These tools run on both [Node.js](https://nodejs.org) (>= 22, required by `@babel/parser` 8) and [Bun](https://bun.sh). All commands and the accompanying regression test-suite are exercised under both runtimes in CI, so the commands below can be invoked with either `node` or `bun` interchangeably (for example `bun astgen.js -i .`).
@@ -29,52 +33,37 @@ Options:
29
33
  -h Show help [boolean]
30
34
  ```
31
35
 
32
- #### Environment variables
36
+ Each source file becomes an AST JSON document plus a `.typemap` of inferred types keyed by node offsets. Test files and `node_modules` are excluded by default; the [astgen guide](docs/ASTGEN.md) covers every option and env variable (`ASTGEN_TYPE_WORKERS`, `ASTGEN_INCLUDE_TEST_FILES`, and friends).
33
37
 
34
- | Variable | Default | Purpose |
35
- | ------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
36
- | `ASTGEN_TYPE_WORKERS` | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
37
- | `ASTGEN_INCLUDE_TEST_FILES` | `false` | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation. |
38
- | `ASTGEN_CONCURRENCY` | `10` | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes). |
39
- | `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false` | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely. |
40
- | `ASTGEN_IGNORE_DIRS` | unset | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`). |
38
+ `.vue` and `.svelte` single-file components are flattened into the same Babel shape as any other file: script statements at the top level of the `Program`, and the template as standard JSX nodes. Svelte components are segmented with `svelte/compiler` and every offset is an absolute byte position into the component source, so consumers that slice the original file for `code` fields get the real text back. Control flow maps onto its JSX equivalent — `{#if}` to a `ConditionalExpression`, `{#each}` to `.map()` with an arrow closure, `{#await}` to `.then()` — which means no framework-specific node types are emitted. See the [astgen guide](docs/ASTGEN.md) for the full mapping and the list of accepted losses.
41
39
 
42
40
  ### phpastgen
43
41
 
44
42
  ```text
45
43
  node phpastgen.js --help
46
44
 
47
- Usage: phpastgen [operations] file1.php [file2.php ...]
48
- or: phpastgen [operations] "<?php code"
49
- Turn PHP source code into an abstract syntax tree.
50
-
51
- Operations is a list of the following options (--dump by default):
52
-
53
- -d, --dump Dump nodes using NodeDumper
54
- -p, --pretty-print Pretty print file using PrettyPrinter\Standard
55
- -j, --json-dump Print json_encode() result
56
- --var-dump var_dump() nodes (for exact structure)
57
- -N, --resolve-names Resolve names using NodeVisitor\NameResolver
58
- -c, --with-column-info Show column numbers for errors (if available)
59
- -P, --with-positions Show positions in node dumps
60
- -r, --with-recovery Use parsing with error recovery
61
- -h, --help Display this page
45
+ Usage: phpastgen [options] [-- <legacy php-parse args>]
62
46
 
47
+ Options:
48
+ -i, --input <path> input file or directory (batch mode)
49
+ -o, --output <dir> output directory (default: '.ast')
50
+ -e, --exclude <regex> exclusion regex (default: '^(tests?|vendor|Tests?)')
51
+ -l, --log <level> debug | info | warn | error (default: info)
52
+ -d, --debug same as --log debug
53
+ --target-version <x.y> pin PHP grammar (alias: --parser-target)
54
+ --max-depth <n> depth cap before truncation (default: 250)
55
+ --threads <n> worker processes for directory runs (default: 10)
56
+ --fail-on-error exit non-zero if any file failed
57
+ --parser-info print parser/runtime capability report and exit 0
58
+ --version print generator version and exit 0
59
+ --help print usage
63
60
  ```
64
61
 
65
- ### rbastgen
62
+ The PHP parser (nikic/php-parser 5.8.0, grammars 8.0 to 8.5) is vendored under `plugins/`, so only a PHP runtime on the machine is required. Batch runs write one JSON per file plus `phpastgen_manifest.jsonl` and, on failure, `phpastgen_diagnostics.jsonl`. Details in the [phpastgen guide](docs/PHPASTGEN.md).
66
63
 
67
- Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and
68
- its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be
69
- gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed
70
- to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which
71
- resolved its paths from the ABI of the Ruby that built it.
64
+ ### rbastgen
72
65
 
73
- `prism` and `racc` are deliberately **not** bundled. Both carry C extensions, which are built per
74
- ABI and per platform, and both are default gems in every supported Ruby, so the runtime's own copies
75
- are used. One consequence is visible: the newest grammar available follows the interpreter's prism,
76
- so Ruby 3.4 tops out lower than Ruby 4.0 (grammar 3.5 versus 4.1 at the time of writing). Installing
77
- a newer `prism` gem on the machine raises it, since the caller's `GEM_PATH` is preserved.
66
+ Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which resolved its paths from the ABI of the Ruby that built it.
78
67
 
79
68
  ```text
80
69
  node rbastgen.js --help
@@ -95,58 +84,7 @@ Usage:
95
84
  --help Print usage
96
85
  ```
97
86
 
98
- Problems with individual files are reported and skipped, never fatal; only usage errors (a missing
99
- `-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that
100
- `rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the
101
- log but the wrapper still exits 0 — call the gem directly if a CI job needs to fail on a parse
102
- error.
103
-
104
- #### What the bundled generator emits (ruby_ast_gen 2.0.1)
105
-
106
- - **Parsing is decoupled from the running Ruby.** When `prism` is available the newest grammar its
107
- translation layer supports is used, so a 3.4 runtime parses Ruby 4.0/4.1 syntax. `--parser-target
108
- x.y` pins a grammar instead (down to 1.8 through the `parser` gem), and a file that fails under
109
- the selected grammar is retried once with the newest one. Every JSON file records the backend
110
- that produced it in `parser_backend`, alongside `generator_version` and `ruby_version`.
111
- - **Ruby DSL files are discovered, not just `.rb`.** `Rakefile`, `Gemfile`, `Capfile`,
112
- `Vagrantfile`, `Fastfile` and friends are matched by basename, plus the `.gemspec`, `.rake`,
113
- `.ru`, `.rbi`, `.thor`, `.jbuilder`, `.axlsx` and `.rabl` extensions. Vendor and tool
114
- directories (`.git`, `.bundle`, `.venv`, …) are skipped.
115
- - **Non-UTF-8 sources survive.** `# coding:` magic comments are honoured and undecodable bytes are
116
- scrubbed rather than dropping the file, marked with `encoding_scrubbed: true`.
117
- - **Deeply nested source is truncated, not dropped**, with `truncated: true` on the boundary node
118
- and a `truncated_nodes` count at the top level (`--max-depth`).
119
- - **Semantic metadata for consumers**: a `magic_comments` array (Sorbet `typed:` levels,
120
- `frozen_string_literal`, …), `call_operator`/`has_parentheses` on calls, `percent_array` and
121
- regexp `options`, heredoc body offsets, and `has_sig: true` on a `def` preceded by a Sorbet `sig`
122
- block.
123
- - **Two side-records per run, written inside the output directory**, both ending in `.jsonl` so a
124
- consumer globbing `*.json` for ASTs never mistakes them for one:
125
- `ruby_ast_gen_manifest.jsonl` (one object: input/output, backend, counts of parsed, failed,
126
- skipped, excluded and truncated files, threads, max depth) and
127
- `ruby_ast_gen_diagnostics.jsonl` (one object per failed file with message, line, column), the
128
- latter written only when something failed and removed when a later run is clean.
129
-
130
- To check which backend a machine will use:
131
-
132
- ```shell
133
- rbastgen --parser-info
134
- ```
135
-
136
- `Parser gem` and `Prism gem` name the versions actually loaded — the vendored `parser` and the
137
- runtime's `prism` — so they are the quickest way to tell which copy of each library a machine is
138
- parsing with. They read `unavailable` only when a library genuinely is not loaded.
139
-
140
- #### Environment variables
141
-
142
- | Variable | Default | Purpose |
143
- | --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
144
- | `ATOM_RUBY_HOME` | unset | Ruby install directory to use when a suitable `ruby` is not on the `PATH`; its `bin` is prepended to `PATH` for the child process. |
145
- | `RUBY_CMD` | `ruby` | Ruby interpreter to invoke. Set this (or `ATOM_RUBY_HOME`) when the detected version is not 3.4.x/4.0.x. |
146
- | `RUBY_ASTGEN_BIN` | bundled `ruby_ast_gen` | The generator script to run. Point it at a checkout's `exe/ruby_ast_gen` to test an unreleased `ruby_ast_gen` without touching this package. |
147
- | `ATOM_CWD` | `process.cwd()` | Working directory for the generator, which is what relative `-i`/`-o` paths resolve against. |
148
- | `ATOM_TIMEOUT` / `ASTGEN_TIMEOUT` | unset (no timeout) | Milliseconds before the generator process is killed. `ATOM_TIMEOUT` wins; a non-numeric value is ignored. Also honoured by `scalasem`. |
149
- | `GEM_PATH` | unset | Preserved and appended to the vendored bundle, so gems installed on the machine (a newer `prism`, for instance) stay reachable. |
87
+ Problems with individual files are reported and skipped, never fatal; only usage errors (a missing `-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that `rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the log but the wrapper still exits 0; call the gem directly if a CI job needs to fail on a parse error. Discovery covers Ruby DSL files (`Gemfile`, `Rakefile`, `.gemspec`, `.rake`, and more), and each run writes `ruby_ast_gen_manifest.jsonl` and, on failure, `ruby_ast_gen_diagnostics.jsonl`. Details in the [rbastgen guide](docs/RBASTGEN.md).
150
88
 
151
89
  ### scalasem
152
90
 
@@ -160,29 +98,20 @@ Example:
160
98
  scalasem $(pwd) slices.json
161
99
  ```
162
100
 
101
+ Compiles the project with sbt or mill if no `.tasty` files exist, then extracts literals, used types, and Play framework tags into a semantic slice. The [scalasem guide](docs/SCALASEM.md) covers the pipeline.
102
+
163
103
  ## Testing
164
104
 
165
- `npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby workflow tests
166
- run the `rbastgen` command end to end against `test-fixtures/projects/ruby-parsing`, so they need a
167
- built plugin bundle and a supported Ruby, and they skip themselves cleanly when either is missing:
105
+ `npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby and PHP workflows need their runtimes and a built plugin bundle, and they skip themselves cleanly when either is missing:
168
106
 
169
107
  ```shell
170
108
  bash build.sh --ruby-only
171
109
  npm run test:ruby
172
110
  ```
173
111
 
174
- `ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch
175
- project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle
176
- is present and free of compiled extensions. CI runs it, and it runs the same way locally.
177
-
178
- They assert what a consumer depends on rather than that the command ran: which files are discovered
179
- (including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`,
180
- `percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a
181
- failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no
182
- stale record behind, and that the bundle contains no compiled extension. CI runs them under both
183
- Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball
184
- and parses with that, which is the check that catches a bundle usable only on the Ruby that built
185
- it.
112
+ `ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle is present and free of compiled extensions. CI runs it, and it runs the same way locally.
113
+
114
+ They assert what a consumer depends on rather than that the command ran: which files are discovered (including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`, `percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no stale record behind, and that the bundle contains no compiled extension. CI runs them under both Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball and parses with that, which is the check that catches a bundle usable only on the Ruby that built it. The [testing guide](docs/TESTING.md) maps the suites.
186
115
 
187
116
  ## License
188
117
 
package/astgen.js CHANGED
@@ -24,12 +24,16 @@ import {
24
24
  rmSync
25
25
  } from "fs";
26
26
  import { getAllFiles } from "@appthreat/atom-common";
27
+ import { parseSvelteFile, parseSvelteScriptBuffer } from "./svelteAst.js";
27
28
 
28
29
  // Printed by `astgen --version`. Downstream frontends (e.g. chen's jssrc2cpg)
29
30
  // fold this into their parse-cache fingerprint, so it MUST be bumped whenever
30
31
  // the emitted AST/type shape changes — otherwise stale cached parses from an
31
- // older astgen are silently reused. Bumped for the Babel 8 AST-shape change.
32
- const ASTGEN_VERSION = "4.1.0";
32
+ // older astgen are silently reused. Bumped to 4.3.0 for Vue directive
33
+ // expression values: `v-html="x"` / `:prop="x"` / `@event="x"` attribute
34
+ // values in `.vue` templates now emit JSX expression containers (real
35
+ // expression ASTs) instead of string literals.
36
+ const ASTGEN_VERSION = "4.3.0";
33
37
 
34
38
  const HELP_TEXT = `Options:
35
39
  -i, --src Source directory [default: "."]
@@ -466,15 +470,42 @@ const getAllSrcJSAndTSFiles = (src) => {
466
470
  * Convert a single JS/TS file to AST
467
471
  */
468
472
  const fileToJsAst = (file, projectType, tsInstance) => {
469
- if (file.endsWith(".vue") || file.endsWith(".svelte")) {
473
+ if (file.endsWith(".vue")) {
470
474
  return toVueAst(file, tsInstance);
471
475
  }
476
+ if (file.endsWith(".svelte")) {
477
+ return toSvelteAst(file);
478
+ }
472
479
  if (file.endsWith(".ejs")) {
473
480
  return toEjsAst(file);
474
481
  }
475
482
  return codeToJsAst(file, readFileSync(file, "utf-8"), projectType);
476
483
  };
477
484
 
485
+ /**
486
+ * Convert a single Svelte file to AST. Svelte's own compiler segments the
487
+ * single-file component; the Babel options are built here and passed in, so
488
+ * every Svelte sub-parse shares the exact configuration used for regular
489
+ * JS/TS files. If `svelte/compiler` rejects the whole file (a genuinely
490
+ * broken template), the script blocks are still parsed over a
491
+ * position-preserving masked buffer - absolute offsets, so line numbers stay
492
+ * correct - and the failure is recorded on the emitted AST's `errors` array.
493
+ */
494
+ const toSvelteAst = (file) => {
495
+ const code = readFileSync(file, "utf-8");
496
+ const options = makeBabelOptions(babelParserOptions, file);
497
+ try {
498
+ return parseSvelteFile(file, code, options);
499
+ } catch (err) {
500
+ console.error(
501
+ `Svelte parse failed for ${file}, falling back to script-only parsing:`,
502
+ err?.message
503
+ );
504
+ const { source: maskedSource } = createVirtualTypeSource(code);
505
+ return parseSvelteScriptBuffer(file, code, maskedSource, options, err?.message);
506
+ }
507
+ };
508
+
478
509
  /**
479
510
  * Convert a single JS/TS code snippet to AST
480
511
  */
@@ -590,8 +621,168 @@ declare module "vue" {
590
621
  }
591
622
  `;
592
623
 
624
+ // Ambient declarations for the Svelte 5 runes and the most common `svelte` /
625
+ // `svelte/store` imports, so the TypeScript checker sees real declarations
626
+ // when it type-checks a virtual `.svelte.ts` source. This is a pragmatic
627
+ // starting set rather than a mirror of svelte's own types; grow it when a
628
+ // fixture needs more.
629
+ const SVELTE_RUNE_SHIMS = `
630
+ declare function $state<T>(initial?: T): T;
631
+ declare namespace $state { function raw<T>(initial?: T): T; function snapshot<T>(v: T): T; }
632
+ declare function $derived<T>(expression: T): T;
633
+ declare namespace $derived { function by<T>(fn: () => T): T; }
634
+ declare function $effect(fn: () => void | (() => void)): void;
635
+ declare namespace $effect {
636
+ function pre(fn: () => void | (() => void)): void;
637
+ function tracking(): boolean;
638
+ function root(fn: () => void | (() => void)): () => void;
639
+ function pending(): number;
640
+ }
641
+ declare function $props<T = any>(): T;
642
+ declare namespace $props { function id(): string; }
643
+ declare function $bindable<T>(fallback?: T): T;
644
+ declare function $inspect<T extends any[]>(...values: T): { with: (fn: (...args: any[]) => void) => void };
645
+ declare function $host<T = HTMLElement>(): T;
646
+
647
+ declare module "svelte" {
648
+ export function onMount(fn: () => void | (() => void)): void;
649
+ export function onDestroy(fn: () => void): void;
650
+ export function tick(): Promise<void>;
651
+ export function untrack<T>(fn: () => T): T;
652
+ export function getContext<T>(key: any): T;
653
+ export function setContext<T>(key: any, value: T): T;
654
+ export function hasContext(key: any): boolean;
655
+ export function createEventDispatcher<T = any>(): (type: string, detail?: any) => void;
656
+ export function mount(component: any, options: any): any;
657
+ export function unmount(component: any): Promise<void>;
658
+ export type Component<P = any> = (...args: any[]) => any;
659
+ export type Snippet<P extends any[] = any[]> = (...args: P) => any;
660
+ }
661
+ declare module "svelte/store" {
662
+ export type Readable<T> = { subscribe(run: (value: T) => void): () => void };
663
+ export type Writable<T> = Readable<T> & { set(value: T): void; update(fn: (value: T) => T): void };
664
+ export function writable<T>(value?: T): Writable<T>;
665
+ export function readable<T>(value?: T): Readable<T>;
666
+ export function derived<T>(stores: any, fn: any, initial?: T): Readable<T>;
667
+ export function get<T>(store: Readable<T>): T;
668
+ }
669
+ `;
670
+
593
671
  const maskNonNewlineChars = (value) => value.replace(/[^\r\n]/g, " ");
594
672
 
673
+ // Vue directive attributes - `v-html="expr"`, `:prop="expr"`, `@event="expr"`
674
+ // - carry a JavaScript expression in a quoted string. Babel parses that value
675
+ // as a StringLiteral, which severs the data flow from the script binding into
676
+ // the template: `v-html="rawContent"` has no reference to the `rawContent`
677
+ // binding, so no source-to-sink path can ever be found for the most
678
+ // security-relevant Vue shape. Replacing the value's quotes with braces turns
679
+ // it into a JSX expression container - `v-html={rawContent}` - which parses to
680
+ // a real expression AST. The swap is length-preserving (`"x"` -> `{x}`), so
681
+ // every offset in the emitted AST still maps onto the original file.
682
+ //
683
+ // Only values that parse as an expression are converted, and `v-for`/`v-slot`
684
+ // directives are skipped by name before the parse is even attempted:
685
+ // `v-for="item in items"` DOES parse (`in` is a relational operator), but the
686
+ // resulting expression is meaningless for analysis and would synthesise a read
687
+ // of the loop variable `item` that the script never declares - a spurious
688
+ // reference. Those directives keep their string value.
689
+ //
690
+ // Scoped between the root `<template>` block's opening tag and the last
691
+ // `</template>` that lies OUTSIDE any `<script>` block: a script string could
692
+ // legitimately contain the literal text `</template>`, and rewriting inside
693
+ // `<script>` would change script semantics while still parsing - a silent
694
+ // corruption. That still narrows, not eliminates, the risk - a `</template>`
695
+ // inside a template-side attribute value could in principle swallow markup up
696
+ // to a later close - which is why the whole-file candidates remain as
697
+ // fallbacks for any file the converted candidate fails to parse.
698
+ const VUE_DIRECTIVE_ATTR_REGEX =
699
+ /(\sv-[\w.:-]+|\s[:@.][\w.:-]*)=("(?:[^"\\\r\n]|\\.)*"|'(?:[^'\\\r\n]|\\.)*')/g;
700
+ const VUE_TEMPLATE_OPEN_REGEX = /<template\b[^>]*>/i;
701
+ const VUE_TEMPLATE_CLOSE = "</template>";
702
+
703
+ // `v-for` and `v-slot` (incl. `v-slot:name`) values are not plain expressions.
704
+ const VUE_NON_EXPRESSION_DIRECTIVES = /^v-(for|slot)\b/i;
705
+
706
+ const DIRECTIVE_VALUE_PARSE_OPTIONS = {
707
+ sourceType: "unambiguous",
708
+ allowImportExportEverywhere: true,
709
+ allowAwaitOutsideFunction: true,
710
+ allowReturnOutsideFunction: true,
711
+ allowSuperOutsideMethod: true,
712
+ allowUndeclaredExports: true,
713
+ errorRecovery: false,
714
+ plugins: babelSyntaxPlugins
715
+ };
716
+
717
+ // Ranges of the file covered by `<script ...>...</script>` blocks, so the
718
+ // template-region search can ignore `</template>` text that only appears
719
+ // inside a script string.
720
+ const vueScriptRanges = (code) => {
721
+ const ranges = [];
722
+ let scriptMatch;
723
+ vueScriptTagRegex.lastIndex = 0;
724
+ while ((scriptMatch = vueScriptTagRegex.exec(code)) !== null) {
725
+ ranges.push([scriptMatch.index, scriptMatch.index + scriptMatch[0].length]);
726
+ }
727
+ return ranges;
728
+ };
729
+
730
+ const vueDirectiveValueToExpression = (code) => {
731
+ const templateOpen = code.match(VUE_TEMPLATE_OPEN_REGEX);
732
+ if (!templateOpen) {
733
+ return { code, changed: false };
734
+ }
735
+ const templateStart = templateOpen.index + templateOpen[0].length;
736
+ const scriptRanges = vueScriptRanges(code);
737
+ const inScript = (index) =>
738
+ scriptRanges.some(([start, end]) => index >= start && index < end);
739
+ let templateEnd = -1;
740
+ let searchFrom = templateStart;
741
+ let closeIndex;
742
+ while (
743
+ (closeIndex = code.toLowerCase().indexOf(VUE_TEMPLATE_CLOSE, searchFrom)) !==
744
+ -1
745
+ ) {
746
+ if (!inScript(closeIndex)) {
747
+ templateEnd = closeIndex;
748
+ }
749
+ searchFrom = closeIndex + 1;
750
+ }
751
+ if (templateEnd < templateStart) {
752
+ return { code, changed: false };
753
+ }
754
+
755
+ let changed = false;
756
+ const convert = (templateSlice) =>
757
+ templateSlice.replace(
758
+ VUE_DIRECTIVE_ATTR_REGEX,
759
+ (match, namePart, quoted) => {
760
+ if (VUE_NON_EXPRESSION_DIRECTIVES.test(namePart.trim())) {
761
+ return match;
762
+ }
763
+ const inner = quoted.slice(1, -1);
764
+ // An empty value, or a `{{` opening (Vue ignores interpolations in
765
+ // attribute values), is left alone.
766
+ if (!inner.trim() || inner.includes("{{")) {
767
+ return match;
768
+ }
769
+ try {
770
+ parse(`(${inner})`, DIRECTIVE_VALUE_PARSE_OPTIONS);
771
+ } catch {
772
+ return match;
773
+ }
774
+ changed = true;
775
+ return `${namePart}={${inner}}`;
776
+ }
777
+ );
778
+
779
+ const converted =
780
+ code.slice(0, templateStart) +
781
+ convert(code.slice(templateStart, templateEnd)) +
782
+ code.slice(templateEnd);
783
+ return { code: converted, changed };
784
+ };
785
+
595
786
  const cleanVueCodeForParsing = (code, { includeScripts = true } = {}) => {
596
787
  let cleanedCode = code
597
788
  .replace(vueCommentRegex, (match) => maskNonNewlineChars(match))
@@ -653,7 +844,32 @@ const buildVueParseCandidates = (code) => {
653
844
  ? `${scriptOnlyCandidate}\n${templateOnlyCandidate}`
654
845
  : templateOnlyCandidate;
655
846
 
847
+ // Directive-expression candidates first: same masking, but `v-html="x"`
848
+ // style values are expression containers, so template expressions keep
849
+ // their references. The plain candidates below remain as fallbacks for the
850
+ // (checked-per-attribute, but defensive) case of a converted file not
851
+ // parsing as a whole.
852
+ const directiveCandidates = (() => {
853
+ const { code: directiveCode, changed } = vueDirectiveValueToExpression(
854
+ code
855
+ );
856
+ if (!changed) {
857
+ return [];
858
+ }
859
+ const directiveFull = cleanVueCodeForParsing(directiveCode, {
860
+ includeScripts: true
861
+ });
862
+ const directiveTemplateOnly = cleanVueCodeForParsing(directiveCode, {
863
+ includeScripts: false
864
+ });
865
+ return [
866
+ { name: "directive-full", code: directiveFull },
867
+ { name: "directive-template-only", code: directiveTemplateOnly }
868
+ ];
869
+ })();
870
+
656
871
  const candidates = [
872
+ ...directiveCandidates,
657
873
  { name: "full", code: fullCandidate },
658
874
  { name: "combined", code: combinedCandidate },
659
875
  { name: "template-only", code: templateOnlyCandidate },
@@ -686,7 +902,14 @@ const parseVueAstWithFallback = (file, code) => {
686
902
  throw lastError || new Error(`Unable to parse Vue file: ${file}`);
687
903
  };
688
904
 
689
- const createVueVirtualTypeSource = (code) => {
905
+ /**
906
+ * Build the virtual type source for a single-file component (`.vue` or
907
+ * `.svelte`): the file masked to spaces/newlines with only the `<script>`
908
+ * contents left verbatim. Positions are preserved exactly, so type offsets
909
+ * map back onto the original file. Works for both frameworks because it is
910
+ * driven purely by the `<script>...</script>` regex.
911
+ */
912
+ const createVirtualTypeSource = (code) => {
690
913
  const output = maskNonNewlineChars(code).split("");
691
914
  let hasScriptContent = false;
692
915
  let scriptMatch;
@@ -708,13 +931,22 @@ const createVueVirtualTypeSource = (code) => {
708
931
  };
709
932
  };
710
933
 
711
- const collectVueTypesWithVirtualProgram = (file, virtualSource) => {
712
- const tempDir = mkdtempSync(join(tmpdir(), "atom-parsetools-vue-"));
934
+ /**
935
+ * Type-check the virtual source with a throwaway program: the virtual file
936
+ * plus the framework's shim declarations, both under a temp directory.
937
+ */
938
+ const collectTypesWithVirtualProgram = (
939
+ file,
940
+ virtualSource,
941
+ shimFileName,
942
+ shimSource
943
+ ) => {
944
+ const tempDir = mkdtempSync(join(tmpdir(), "atom-parsetools-sfc-"));
713
945
  const virtualFile = join(tempDir, `${basename(file)}.ts`);
714
- const shimFile = join(tempDir, "vue-shims.d.ts");
946
+ const shimFile = join(tempDir, shimFileName);
715
947
  try {
716
948
  writeFileSync(virtualFile, virtualSource, "utf8");
717
- writeFileSync(shimFile, VUE_COMPILER_MACRO_SHIMS, "utf8");
949
+ writeFileSync(shimFile, shimSource, "utf8");
718
950
  const virtualTs = createTsc([virtualFile, shimFile], tempDir);
719
951
  const sourceFile = virtualTs?.program?.getSourceFile(virtualFile);
720
952
  if (!virtualTs || !sourceFile) {
@@ -728,7 +960,24 @@ const collectVueTypesWithVirtualProgram = (file, virtualSource) => {
728
960
  }
729
961
  };
730
962
 
731
- const collectVueSeenTypes = (file, code, tsInstance) => {
963
+ /**
964
+ * Collect types for a single-file component. The project program is tried
965
+ * first: a `.vue`/`.svelte` path is normally absent from it, but a tsconfig
966
+ * that maps the extension can make it resolvable, and that mapping is more
967
+ * accurate than the virtual source. Otherwise fall back to type-checking the
968
+ * position-preserving virtual source against the framework's shims.
969
+ *
970
+ * `.vue` and `.svelte` differ only in which shim declarations the checker
971
+ * needs, so both go through here rather than through near-identical copies
972
+ * that would drift apart.
973
+ */
974
+ const collectSfcSeenTypes = (
975
+ file,
976
+ code,
977
+ tsInstance,
978
+ shimFileName,
979
+ shimSource
980
+ ) => {
732
981
  let seenTypes;
733
982
  if (tsInstance?.program) {
734
983
  try {
@@ -742,15 +991,38 @@ const collectVueSeenTypes = (file, code, tsInstance) => {
742
991
  }
743
992
 
744
993
  if (!seenTypes || seenTypes.size === 0) {
745
- const virtualSource = createVueVirtualTypeSource(code);
994
+ const virtualSource = createVirtualTypeSource(code);
746
995
  if (virtualSource.hasScriptContent) {
747
- seenTypes = collectVueTypesWithVirtualProgram(file, virtualSource.source);
996
+ seenTypes = collectTypesWithVirtualProgram(
997
+ file,
998
+ virtualSource.source,
999
+ shimFileName,
1000
+ shimSource
1001
+ );
748
1002
  }
749
1003
  }
750
1004
 
751
1005
  return seenTypes;
752
1006
  };
753
1007
 
1008
+ const collectVueSeenTypes = (file, code, tsInstance) =>
1009
+ collectSfcSeenTypes(
1010
+ file,
1011
+ code,
1012
+ tsInstance,
1013
+ "vue-shims.d.ts",
1014
+ VUE_COMPILER_MACRO_SHIMS
1015
+ );
1016
+
1017
+ const collectSvelteSeenTypes = (file, code, tsInstance) =>
1018
+ collectSfcSeenTypes(
1019
+ file,
1020
+ code,
1021
+ tsInstance,
1022
+ "svelte-shims.d.ts",
1023
+ SVELTE_RUNE_SHIMS
1024
+ );
1025
+
754
1026
  const TSC_FLAGS =
755
1027
  tsc.TypeFormatFlags.NoTruncation |
756
1028
  tsc.TypeFormatFlags.InTypeAlias |
@@ -778,6 +1050,10 @@ const collectSeenTypesForFile = (file, ts, options) => {
778
1050
  return collectVueSeenTypes(file, readFileSync(file, "utf-8"), ts);
779
1051
  }
780
1052
 
1053
+ if (file.endsWith(".svelte")) {
1054
+ return collectSvelteSeenTypes(file, readFileSync(file, "utf-8"), ts);
1055
+ }
1056
+
781
1057
  if (!ts?.program) {
782
1058
  return undefined;
783
1059
  }
package/package.json CHANGED
@@ -1,12 +1,12 @@
1
1
  {
2
2
  "name": "@appthreat/atom-parsetools",
3
- "version": "1.5.0",
3
+ "version": "1.7.0",
4
4
  "description": "Parsing tools that complement the @appthreat/atom project.",
5
5
  "main": "./index.js",
6
6
  "type": "module",
7
7
  "scripts": {
8
8
  "pretty": "prettier --write *.js --trailing-comma=none",
9
- "test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
9
+ "test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-svelte-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
10
10
  "test:evaluate": "node test-fixtures/evaluate-astgen.js",
11
11
  "test:ruby": "node test-fixtures/rbastgen-regression.js",
12
12
  "test:php": "node test-fixtures/phpastgen-cli.js && node test-fixtures/phpastgen-parser-info.js && node test-fixtures/phpastgen-discovery.js && node test-fixtures/phpastgen-provenance.js && node test-fixtures/phpastgen-framework-facts.js && node test-fixtures/phpastgen-legacy-regression.js && node test-fixtures/phpastgen-concurrency.js && node test-fixtures/phpastgen-regression.js && node test-fixtures/phpastgen-contract-snapshot.js && npm run test:php:pbt",
@@ -19,13 +19,15 @@
19
19
  "test:shape:update": "UPDATE_SHAPE_SNAPSHOT=1 node test-fixtures/astgen-shape-snapshot.js",
20
20
  "test:json": "node test-fixtures/astgen-json-regression.js",
21
21
  "test:fixtures": "node test-fixtures/test-suite.js",
22
- "test:vue": "node test-fixtures/astgen-vue-regression.js"
22
+ "test:vue": "node test-fixtures/astgen-vue-regression.js",
23
+ "test:svelte": "node test-fixtures/astgen-svelte-regression.js"
23
24
  },
24
25
  "dependencies": {
25
26
  "@appthreat/atom-common": "^1.1.0",
26
27
  "@babel/parser": "^8.0.4",
27
28
  "@typescript/typescript6": "^6.0.2",
28
- "hermes-parser": "^0.37.0"
29
+ "hermes-parser": "^0.37.0",
30
+ "svelte": "5.57.0"
29
31
  },
30
32
  "bin": {
31
33
  "astgen": "astgen.js",
@@ -1,9 +1,9 @@
1
1
  <?php return array(
2
2
  'root' => array(
3
3
  'name' => '__root__',
4
- 'pretty_version' => 'v1.5.0',
5
- 'version' => '1.5.0.0',
6
- 'reference' => '21c39993dc05f74643f5047de71f4bb65f0ead4a',
4
+ 'pretty_version' => 'v1.7.0',
5
+ 'version' => '1.7.0.0',
6
+ 'reference' => '26f7c3eeab1661fdf7ee1a5a7b40f0fd034235eb',
7
7
  'type' => 'library',
8
8
  'install_path' => __DIR__ . '/../../',
9
9
  'aliases' => array(),
@@ -11,9 +11,9 @@
11
11
  ),
12
12
  'versions' => array(
13
13
  '__root__' => array(
14
- 'pretty_version' => 'v1.5.0',
15
- 'version' => '1.5.0.0',
16
- 'reference' => '21c39993dc05f74643f5047de71f4bb65f0ead4a',
14
+ 'pretty_version' => 'v1.7.0',
15
+ 'version' => '1.7.0.0',
16
+ 'reference' => '26f7c3eeab1661fdf7ee1a5a7b40f0fd034235eb',
17
17
  'type' => 'library',
18
18
  'install_path' => __DIR__ . '/../../',
19
19
  'aliases' => array(),