@appthreat/atom-parsetools 1.5.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -103
- package/astgen.js +287 -11
- package/package.json +6 -4
- package/plugins/composer/installed.php +6 -6
- package/svelteAst.js +1291 -0
package/README.md
CHANGED
|
@@ -1,12 +1,16 @@
|
|
|
1
|
-
#
|
|
1
|
+
# atom-parsetools
|
|
2
2
|
|
|
3
3
|
This package hosts a collection of parsing tools that complement the `@appthreat/atom` project. These tools offer parsing and analysis-related functionalities such as generating AST and semantics information in JSON format. The full list of tools and bin commands exposed by this package is below:
|
|
4
4
|
|
|
5
|
-
- astgen - Generates AST for JavaScript and
|
|
5
|
+
- astgen - Generates AST for JavaScript, TypeScript, Vue and Svelte projects in JSON format
|
|
6
6
|
- phpastgen - Generates AST for PHP projects using `php-parse` command from `nikic/php-parser`
|
|
7
7
|
- rbastgen - Generates AST for Ruby projects using AppThreat's [`ruby_ast_gen`](https://github.com/AppThreat/ruby_ast_gen) gem (2.0.1)
|
|
8
8
|
- scalasem - Generates a custom semantics slice for Scala Projects by utilising scalac command.
|
|
9
9
|
|
|
10
|
+
## Documentation
|
|
11
|
+
|
|
12
|
+
The full documentation lives at [https://appthreat.github.io/atom-parsetools/](https://appthreat.github.io/atom-parsetools/): per-tool guides, the output format specification, environment variable reference, packaging notes, and ten hands-on tutorials. The pages are rendered from the [`docs`](docs/) directory of this repository.
|
|
13
|
+
|
|
10
14
|
## Runtime support
|
|
11
15
|
|
|
12
16
|
These tools run on both [Node.js](https://nodejs.org) (>= 22, required by `@babel/parser` 8) and [Bun](https://bun.sh). All commands and the accompanying regression test-suite are exercised under both runtimes in CI, so the commands below can be invoked with either `node` or `bun` interchangeably (for example `bun astgen.js -i .`).
|
|
@@ -29,52 +33,37 @@ Options:
|
|
|
29
33
|
-h Show help [boolean]
|
|
30
34
|
```
|
|
31
35
|
|
|
32
|
-
|
|
36
|
+
Each source file becomes an AST JSON document plus a `.typemap` of inferred types keyed by node offsets. Test files and `node_modules` are excluded by default; the [astgen guide](docs/ASTGEN.md) covers every option and env variable (`ASTGEN_TYPE_WORKERS`, `ASTGEN_INCLUDE_TEST_FILES`, and friends).
|
|
33
37
|
|
|
34
|
-
|
|
35
|
-
| ------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
36
|
-
| `ASTGEN_TYPE_WORKERS` | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
|
|
37
|
-
| `ASTGEN_INCLUDE_TEST_FILES` | `false` | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation. |
|
|
38
|
-
| `ASTGEN_CONCURRENCY` | `10` | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes). |
|
|
39
|
-
| `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false` | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely. |
|
|
40
|
-
| `ASTGEN_IGNORE_DIRS` | unset | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`). |
|
|
38
|
+
`.vue` and `.svelte` single-file components are flattened into the same Babel shape as any other file: script statements at the top level of the `Program`, and the template as standard JSX nodes. Svelte components are segmented with `svelte/compiler` and every offset is an absolute byte position into the component source, so consumers that slice the original file for `code` fields get the real text back. Control flow maps onto its JSX equivalent — `{#if}` to a `ConditionalExpression`, `{#each}` to `.map()` with an arrow closure, `{#await}` to `.then()` — which means no framework-specific node types are emitted. See the [astgen guide](docs/ASTGEN.md) for the full mapping and the list of accepted losses.
|
|
41
39
|
|
|
42
40
|
### phpastgen
|
|
43
41
|
|
|
44
42
|
```text
|
|
45
43
|
node phpastgen.js --help
|
|
46
44
|
|
|
47
|
-
Usage: phpastgen [
|
|
48
|
-
or: phpastgen [operations] "<?php code"
|
|
49
|
-
Turn PHP source code into an abstract syntax tree.
|
|
50
|
-
|
|
51
|
-
Operations is a list of the following options (--dump by default):
|
|
52
|
-
|
|
53
|
-
-d, --dump Dump nodes using NodeDumper
|
|
54
|
-
-p, --pretty-print Pretty print file using PrettyPrinter\Standard
|
|
55
|
-
-j, --json-dump Print json_encode() result
|
|
56
|
-
--var-dump var_dump() nodes (for exact structure)
|
|
57
|
-
-N, --resolve-names Resolve names using NodeVisitor\NameResolver
|
|
58
|
-
-c, --with-column-info Show column numbers for errors (if available)
|
|
59
|
-
-P, --with-positions Show positions in node dumps
|
|
60
|
-
-r, --with-recovery Use parsing with error recovery
|
|
61
|
-
-h, --help Display this page
|
|
45
|
+
Usage: phpastgen [options] [-- <legacy php-parse args>]
|
|
62
46
|
|
|
47
|
+
Options:
|
|
48
|
+
-i, --input <path> input file or directory (batch mode)
|
|
49
|
+
-o, --output <dir> output directory (default: '.ast')
|
|
50
|
+
-e, --exclude <regex> exclusion regex (default: '^(tests?|vendor|Tests?)')
|
|
51
|
+
-l, --log <level> debug | info | warn | error (default: info)
|
|
52
|
+
-d, --debug same as --log debug
|
|
53
|
+
--target-version <x.y> pin PHP grammar (alias: --parser-target)
|
|
54
|
+
--max-depth <n> depth cap before truncation (default: 250)
|
|
55
|
+
--threads <n> worker processes for directory runs (default: 10)
|
|
56
|
+
--fail-on-error exit non-zero if any file failed
|
|
57
|
+
--parser-info print parser/runtime capability report and exit 0
|
|
58
|
+
--version print generator version and exit 0
|
|
59
|
+
--help print usage
|
|
63
60
|
```
|
|
64
61
|
|
|
65
|
-
|
|
62
|
+
The PHP parser (nikic/php-parser 5.8.0, grammars 8.0 to 8.5) is vendored under `plugins/`, so only a PHP runtime on the machine is required. Batch runs write one JSON per file plus `phpastgen_manifest.jsonl` and, on failure, `phpastgen_diagnostics.jsonl`. Details in the [phpastgen guide](docs/PHPASTGEN.md).
|
|
66
63
|
|
|
67
|
-
|
|
68
|
-
its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be
|
|
69
|
-
gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed
|
|
70
|
-
to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which
|
|
71
|
-
resolved its paths from the ABI of the Ruby that built it.
|
|
64
|
+
### rbastgen
|
|
72
65
|
|
|
73
|
-
`
|
|
74
|
-
ABI and per platform, and both are default gems in every supported Ruby, so the runtime's own copies
|
|
75
|
-
are used. One consequence is visible: the newest grammar available follows the interpreter's prism,
|
|
76
|
-
so Ruby 3.4 tops out lower than Ruby 4.0 (grammar 3.5 versus 4.1 at the time of writing). Installing
|
|
77
|
-
a newer `prism` gem on the machine raises it, since the caller's `GEM_PATH` is preserved.
|
|
66
|
+
Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which resolved its paths from the ABI of the Ruby that built it.
|
|
78
67
|
|
|
79
68
|
```text
|
|
80
69
|
node rbastgen.js --help
|
|
@@ -95,58 +84,7 @@ Usage:
|
|
|
95
84
|
--help Print usage
|
|
96
85
|
```
|
|
97
86
|
|
|
98
|
-
Problems with individual files are reported and skipped, never fatal; only usage errors (a missing
|
|
99
|
-
`-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that
|
|
100
|
-
`rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the
|
|
101
|
-
log but the wrapper still exits 0 — call the gem directly if a CI job needs to fail on a parse
|
|
102
|
-
error.
|
|
103
|
-
|
|
104
|
-
#### What the bundled generator emits (ruby_ast_gen 2.0.1)
|
|
105
|
-
|
|
106
|
-
- **Parsing is decoupled from the running Ruby.** When `prism` is available the newest grammar its
|
|
107
|
-
translation layer supports is used, so a 3.4 runtime parses Ruby 4.0/4.1 syntax. `--parser-target
|
|
108
|
-
x.y` pins a grammar instead (down to 1.8 through the `parser` gem), and a file that fails under
|
|
109
|
-
the selected grammar is retried once with the newest one. Every JSON file records the backend
|
|
110
|
-
that produced it in `parser_backend`, alongside `generator_version` and `ruby_version`.
|
|
111
|
-
- **Ruby DSL files are discovered, not just `.rb`.** `Rakefile`, `Gemfile`, `Capfile`,
|
|
112
|
-
`Vagrantfile`, `Fastfile` and friends are matched by basename, plus the `.gemspec`, `.rake`,
|
|
113
|
-
`.ru`, `.rbi`, `.thor`, `.jbuilder`, `.axlsx` and `.rabl` extensions. Vendor and tool
|
|
114
|
-
directories (`.git`, `.bundle`, `.venv`, …) are skipped.
|
|
115
|
-
- **Non-UTF-8 sources survive.** `# coding:` magic comments are honoured and undecodable bytes are
|
|
116
|
-
scrubbed rather than dropping the file, marked with `encoding_scrubbed: true`.
|
|
117
|
-
- **Deeply nested source is truncated, not dropped**, with `truncated: true` on the boundary node
|
|
118
|
-
and a `truncated_nodes` count at the top level (`--max-depth`).
|
|
119
|
-
- **Semantic metadata for consumers**: a `magic_comments` array (Sorbet `typed:` levels,
|
|
120
|
-
`frozen_string_literal`, …), `call_operator`/`has_parentheses` on calls, `percent_array` and
|
|
121
|
-
regexp `options`, heredoc body offsets, and `has_sig: true` on a `def` preceded by a Sorbet `sig`
|
|
122
|
-
block.
|
|
123
|
-
- **Two side-records per run, written inside the output directory**, both ending in `.jsonl` so a
|
|
124
|
-
consumer globbing `*.json` for ASTs never mistakes them for one:
|
|
125
|
-
`ruby_ast_gen_manifest.jsonl` (one object: input/output, backend, counts of parsed, failed,
|
|
126
|
-
skipped, excluded and truncated files, threads, max depth) and
|
|
127
|
-
`ruby_ast_gen_diagnostics.jsonl` (one object per failed file with message, line, column), the
|
|
128
|
-
latter written only when something failed and removed when a later run is clean.
|
|
129
|
-
|
|
130
|
-
To check which backend a machine will use:
|
|
131
|
-
|
|
132
|
-
```shell
|
|
133
|
-
rbastgen --parser-info
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
`Parser gem` and `Prism gem` name the versions actually loaded — the vendored `parser` and the
|
|
137
|
-
runtime's `prism` — so they are the quickest way to tell which copy of each library a machine is
|
|
138
|
-
parsing with. They read `unavailable` only when a library genuinely is not loaded.
|
|
139
|
-
|
|
140
|
-
#### Environment variables
|
|
141
|
-
|
|
142
|
-
| Variable | Default | Purpose |
|
|
143
|
-
| --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
144
|
-
| `ATOM_RUBY_HOME` | unset | Ruby install directory to use when a suitable `ruby` is not on the `PATH`; its `bin` is prepended to `PATH` for the child process. |
|
|
145
|
-
| `RUBY_CMD` | `ruby` | Ruby interpreter to invoke. Set this (or `ATOM_RUBY_HOME`) when the detected version is not 3.4.x/4.0.x. |
|
|
146
|
-
| `RUBY_ASTGEN_BIN` | bundled `ruby_ast_gen` | The generator script to run. Point it at a checkout's `exe/ruby_ast_gen` to test an unreleased `ruby_ast_gen` without touching this package. |
|
|
147
|
-
| `ATOM_CWD` | `process.cwd()` | Working directory for the generator, which is what relative `-i`/`-o` paths resolve against. |
|
|
148
|
-
| `ATOM_TIMEOUT` / `ASTGEN_TIMEOUT` | unset (no timeout) | Milliseconds before the generator process is killed. `ATOM_TIMEOUT` wins; a non-numeric value is ignored. Also honoured by `scalasem`. |
|
|
149
|
-
| `GEM_PATH` | unset | Preserved and appended to the vendored bundle, so gems installed on the machine (a newer `prism`, for instance) stay reachable. |
|
|
87
|
+
Problems with individual files are reported and skipped, never fatal; only usage errors (a missing `-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that `rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the log but the wrapper still exits 0; call the gem directly if a CI job needs to fail on a parse error. Discovery covers Ruby DSL files (`Gemfile`, `Rakefile`, `.gemspec`, `.rake`, and more), and each run writes `ruby_ast_gen_manifest.jsonl` and, on failure, `ruby_ast_gen_diagnostics.jsonl`. Details in the [rbastgen guide](docs/RBASTGEN.md).
|
|
150
88
|
|
|
151
89
|
### scalasem
|
|
152
90
|
|
|
@@ -160,29 +98,20 @@ Example:
|
|
|
160
98
|
scalasem $(pwd) slices.json
|
|
161
99
|
```
|
|
162
100
|
|
|
101
|
+
Compiles the project with sbt or mill if no `.tasty` files exist, then extracts literals, used types, and Play framework tags into a semantic slice. The [scalasem guide](docs/SCALASEM.md) covers the pipeline.
|
|
102
|
+
|
|
163
103
|
## Testing
|
|
164
104
|
|
|
165
|
-
`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby
|
|
166
|
-
run the `rbastgen` command end to end against `test-fixtures/projects/ruby-parsing`, so they need a
|
|
167
|
-
built plugin bundle and a supported Ruby, and they skip themselves cleanly when either is missing:
|
|
105
|
+
`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby and PHP workflows need their runtimes and a built plugin bundle, and they skip themselves cleanly when either is missing:
|
|
168
106
|
|
|
169
107
|
```shell
|
|
170
108
|
bash build.sh --ruby-only
|
|
171
109
|
npm run test:ruby
|
|
172
110
|
```
|
|
173
111
|
|
|
174
|
-
`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
They assert what a consumer depends on rather than that the command ran: which files are discovered
|
|
179
|
-
(including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`,
|
|
180
|
-
`percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a
|
|
181
|
-
failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no
|
|
182
|
-
stale record behind, and that the bundle contains no compiled extension. CI runs them under both
|
|
183
|
-
Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball
|
|
184
|
-
and parses with that, which is the check that catches a bundle usable only on the Ruby that built
|
|
185
|
-
it.
|
|
112
|
+
`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle is present and free of compiled extensions. CI runs it, and it runs the same way locally.
|
|
113
|
+
|
|
114
|
+
They assert what a consumer depends on rather than that the command ran: which files are discovered (including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`, `percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no stale record behind, and that the bundle contains no compiled extension. CI runs them under both Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball and parses with that, which is the check that catches a bundle usable only on the Ruby that built it. The [testing guide](docs/TESTING.md) maps the suites.
|
|
186
115
|
|
|
187
116
|
## License
|
|
188
117
|
|
package/astgen.js
CHANGED
|
@@ -24,12 +24,16 @@ import {
|
|
|
24
24
|
rmSync
|
|
25
25
|
} from "fs";
|
|
26
26
|
import { getAllFiles } from "@appthreat/atom-common";
|
|
27
|
+
import { parseSvelteFile, parseSvelteScriptBuffer } from "./svelteAst.js";
|
|
27
28
|
|
|
28
29
|
// Printed by `astgen --version`. Downstream frontends (e.g. chen's jssrc2cpg)
|
|
29
30
|
// fold this into their parse-cache fingerprint, so it MUST be bumped whenever
|
|
30
31
|
// the emitted AST/type shape changes — otherwise stale cached parses from an
|
|
31
|
-
// older astgen are silently reused. Bumped
|
|
32
|
-
|
|
32
|
+
// older astgen are silently reused. Bumped to 4.3.0 for Vue directive
|
|
33
|
+
// expression values: `v-html="x"` / `:prop="x"` / `@event="x"` attribute
|
|
34
|
+
// values in `.vue` templates now emit JSX expression containers (real
|
|
35
|
+
// expression ASTs) instead of string literals.
|
|
36
|
+
const ASTGEN_VERSION = "4.3.0";
|
|
33
37
|
|
|
34
38
|
const HELP_TEXT = `Options:
|
|
35
39
|
-i, --src Source directory [default: "."]
|
|
@@ -466,15 +470,42 @@ const getAllSrcJSAndTSFiles = (src) => {
|
|
|
466
470
|
* Convert a single JS/TS file to AST
|
|
467
471
|
*/
|
|
468
472
|
const fileToJsAst = (file, projectType, tsInstance) => {
|
|
469
|
-
if (file.endsWith(".vue")
|
|
473
|
+
if (file.endsWith(".vue")) {
|
|
470
474
|
return toVueAst(file, tsInstance);
|
|
471
475
|
}
|
|
476
|
+
if (file.endsWith(".svelte")) {
|
|
477
|
+
return toSvelteAst(file);
|
|
478
|
+
}
|
|
472
479
|
if (file.endsWith(".ejs")) {
|
|
473
480
|
return toEjsAst(file);
|
|
474
481
|
}
|
|
475
482
|
return codeToJsAst(file, readFileSync(file, "utf-8"), projectType);
|
|
476
483
|
};
|
|
477
484
|
|
|
485
|
+
/**
|
|
486
|
+
* Convert a single Svelte file to AST. Svelte's own compiler segments the
|
|
487
|
+
* single-file component; the Babel options are built here and passed in, so
|
|
488
|
+
* every Svelte sub-parse shares the exact configuration used for regular
|
|
489
|
+
* JS/TS files. If `svelte/compiler` rejects the whole file (a genuinely
|
|
490
|
+
* broken template), the script blocks are still parsed over a
|
|
491
|
+
* position-preserving masked buffer - absolute offsets, so line numbers stay
|
|
492
|
+
* correct - and the failure is recorded on the emitted AST's `errors` array.
|
|
493
|
+
*/
|
|
494
|
+
const toSvelteAst = (file) => {
|
|
495
|
+
const code = readFileSync(file, "utf-8");
|
|
496
|
+
const options = makeBabelOptions(babelParserOptions, file);
|
|
497
|
+
try {
|
|
498
|
+
return parseSvelteFile(file, code, options);
|
|
499
|
+
} catch (err) {
|
|
500
|
+
console.error(
|
|
501
|
+
`Svelte parse failed for ${file}, falling back to script-only parsing:`,
|
|
502
|
+
err?.message
|
|
503
|
+
);
|
|
504
|
+
const { source: maskedSource } = createVirtualTypeSource(code);
|
|
505
|
+
return parseSvelteScriptBuffer(file, code, maskedSource, options, err?.message);
|
|
506
|
+
}
|
|
507
|
+
};
|
|
508
|
+
|
|
478
509
|
/**
|
|
479
510
|
* Convert a single JS/TS code snippet to AST
|
|
480
511
|
*/
|
|
@@ -590,8 +621,168 @@ declare module "vue" {
|
|
|
590
621
|
}
|
|
591
622
|
`;
|
|
592
623
|
|
|
624
|
+
// Ambient declarations for the Svelte 5 runes and the most common `svelte` /
|
|
625
|
+
// `svelte/store` imports, so the TypeScript checker sees real declarations
|
|
626
|
+
// when it type-checks a virtual `.svelte.ts` source. This is a pragmatic
|
|
627
|
+
// starting set rather than a mirror of svelte's own types; grow it when a
|
|
628
|
+
// fixture needs more.
|
|
629
|
+
const SVELTE_RUNE_SHIMS = `
|
|
630
|
+
declare function $state<T>(initial?: T): T;
|
|
631
|
+
declare namespace $state { function raw<T>(initial?: T): T; function snapshot<T>(v: T): T; }
|
|
632
|
+
declare function $derived<T>(expression: T): T;
|
|
633
|
+
declare namespace $derived { function by<T>(fn: () => T): T; }
|
|
634
|
+
declare function $effect(fn: () => void | (() => void)): void;
|
|
635
|
+
declare namespace $effect {
|
|
636
|
+
function pre(fn: () => void | (() => void)): void;
|
|
637
|
+
function tracking(): boolean;
|
|
638
|
+
function root(fn: () => void | (() => void)): () => void;
|
|
639
|
+
function pending(): number;
|
|
640
|
+
}
|
|
641
|
+
declare function $props<T = any>(): T;
|
|
642
|
+
declare namespace $props { function id(): string; }
|
|
643
|
+
declare function $bindable<T>(fallback?: T): T;
|
|
644
|
+
declare function $inspect<T extends any[]>(...values: T): { with: (fn: (...args: any[]) => void) => void };
|
|
645
|
+
declare function $host<T = HTMLElement>(): T;
|
|
646
|
+
|
|
647
|
+
declare module "svelte" {
|
|
648
|
+
export function onMount(fn: () => void | (() => void)): void;
|
|
649
|
+
export function onDestroy(fn: () => void): void;
|
|
650
|
+
export function tick(): Promise<void>;
|
|
651
|
+
export function untrack<T>(fn: () => T): T;
|
|
652
|
+
export function getContext<T>(key: any): T;
|
|
653
|
+
export function setContext<T>(key: any, value: T): T;
|
|
654
|
+
export function hasContext(key: any): boolean;
|
|
655
|
+
export function createEventDispatcher<T = any>(): (type: string, detail?: any) => void;
|
|
656
|
+
export function mount(component: any, options: any): any;
|
|
657
|
+
export function unmount(component: any): Promise<void>;
|
|
658
|
+
export type Component<P = any> = (...args: any[]) => any;
|
|
659
|
+
export type Snippet<P extends any[] = any[]> = (...args: P) => any;
|
|
660
|
+
}
|
|
661
|
+
declare module "svelte/store" {
|
|
662
|
+
export type Readable<T> = { subscribe(run: (value: T) => void): () => void };
|
|
663
|
+
export type Writable<T> = Readable<T> & { set(value: T): void; update(fn: (value: T) => T): void };
|
|
664
|
+
export function writable<T>(value?: T): Writable<T>;
|
|
665
|
+
export function readable<T>(value?: T): Readable<T>;
|
|
666
|
+
export function derived<T>(stores: any, fn: any, initial?: T): Readable<T>;
|
|
667
|
+
export function get<T>(store: Readable<T>): T;
|
|
668
|
+
}
|
|
669
|
+
`;
|
|
670
|
+
|
|
593
671
|
const maskNonNewlineChars = (value) => value.replace(/[^\r\n]/g, " ");
|
|
594
672
|
|
|
673
|
+
// Vue directive attributes - `v-html="expr"`, `:prop="expr"`, `@event="expr"`
|
|
674
|
+
// - carry a JavaScript expression in a quoted string. Babel parses that value
|
|
675
|
+
// as a StringLiteral, which severs the data flow from the script binding into
|
|
676
|
+
// the template: `v-html="rawContent"` has no reference to the `rawContent`
|
|
677
|
+
// binding, so no source-to-sink path can ever be found for the most
|
|
678
|
+
// security-relevant Vue shape. Replacing the value's quotes with braces turns
|
|
679
|
+
// it into a JSX expression container - `v-html={rawContent}` - which parses to
|
|
680
|
+
// a real expression AST. The swap is length-preserving (`"x"` -> `{x}`), so
|
|
681
|
+
// every offset in the emitted AST still maps onto the original file.
|
|
682
|
+
//
|
|
683
|
+
// Only values that parse as an expression are converted, and `v-for`/`v-slot`
|
|
684
|
+
// directives are skipped by name before the parse is even attempted:
|
|
685
|
+
// `v-for="item in items"` DOES parse (`in` is a relational operator), but the
|
|
686
|
+
// resulting expression is meaningless for analysis and would synthesise a read
|
|
687
|
+
// of the loop variable `item` that the script never declares - a spurious
|
|
688
|
+
// reference. Those directives keep their string value.
|
|
689
|
+
//
|
|
690
|
+
// Scoped between the root `<template>` block's opening tag and the last
|
|
691
|
+
// `</template>` that lies OUTSIDE any `<script>` block: a script string could
|
|
692
|
+
// legitimately contain the literal text `</template>`, and rewriting inside
|
|
693
|
+
// `<script>` would change script semantics while still parsing - a silent
|
|
694
|
+
// corruption. That still narrows, not eliminates, the risk - a `</template>`
|
|
695
|
+
// inside a template-side attribute value could in principle swallow markup up
|
|
696
|
+
// to a later close - which is why the whole-file candidates remain as
|
|
697
|
+
// fallbacks for any file the converted candidate fails to parse.
|
|
698
|
+
const VUE_DIRECTIVE_ATTR_REGEX =
|
|
699
|
+
/(\sv-[\w.:-]+|\s[:@.][\w.:-]*)=("(?:[^"\\\r\n]|\\.)*"|'(?:[^'\\\r\n]|\\.)*')/g;
|
|
700
|
+
const VUE_TEMPLATE_OPEN_REGEX = /<template\b[^>]*>/i;
|
|
701
|
+
const VUE_TEMPLATE_CLOSE = "</template>";
|
|
702
|
+
|
|
703
|
+
// `v-for` and `v-slot` (incl. `v-slot:name`) values are not plain expressions.
|
|
704
|
+
const VUE_NON_EXPRESSION_DIRECTIVES = /^v-(for|slot)\b/i;
|
|
705
|
+
|
|
706
|
+
const DIRECTIVE_VALUE_PARSE_OPTIONS = {
|
|
707
|
+
sourceType: "unambiguous",
|
|
708
|
+
allowImportExportEverywhere: true,
|
|
709
|
+
allowAwaitOutsideFunction: true,
|
|
710
|
+
allowReturnOutsideFunction: true,
|
|
711
|
+
allowSuperOutsideMethod: true,
|
|
712
|
+
allowUndeclaredExports: true,
|
|
713
|
+
errorRecovery: false,
|
|
714
|
+
plugins: babelSyntaxPlugins
|
|
715
|
+
};
|
|
716
|
+
|
|
717
|
+
// Ranges of the file covered by `<script ...>...</script>` blocks, so the
|
|
718
|
+
// template-region search can ignore `</template>` text that only appears
|
|
719
|
+
// inside a script string.
|
|
720
|
+
const vueScriptRanges = (code) => {
|
|
721
|
+
const ranges = [];
|
|
722
|
+
let scriptMatch;
|
|
723
|
+
vueScriptTagRegex.lastIndex = 0;
|
|
724
|
+
while ((scriptMatch = vueScriptTagRegex.exec(code)) !== null) {
|
|
725
|
+
ranges.push([scriptMatch.index, scriptMatch.index + scriptMatch[0].length]);
|
|
726
|
+
}
|
|
727
|
+
return ranges;
|
|
728
|
+
};
|
|
729
|
+
|
|
730
|
+
const vueDirectiveValueToExpression = (code) => {
|
|
731
|
+
const templateOpen = code.match(VUE_TEMPLATE_OPEN_REGEX);
|
|
732
|
+
if (!templateOpen) {
|
|
733
|
+
return { code, changed: false };
|
|
734
|
+
}
|
|
735
|
+
const templateStart = templateOpen.index + templateOpen[0].length;
|
|
736
|
+
const scriptRanges = vueScriptRanges(code);
|
|
737
|
+
const inScript = (index) =>
|
|
738
|
+
scriptRanges.some(([start, end]) => index >= start && index < end);
|
|
739
|
+
let templateEnd = -1;
|
|
740
|
+
let searchFrom = templateStart;
|
|
741
|
+
let closeIndex;
|
|
742
|
+
while (
|
|
743
|
+
(closeIndex = code.toLowerCase().indexOf(VUE_TEMPLATE_CLOSE, searchFrom)) !==
|
|
744
|
+
-1
|
|
745
|
+
) {
|
|
746
|
+
if (!inScript(closeIndex)) {
|
|
747
|
+
templateEnd = closeIndex;
|
|
748
|
+
}
|
|
749
|
+
searchFrom = closeIndex + 1;
|
|
750
|
+
}
|
|
751
|
+
if (templateEnd < templateStart) {
|
|
752
|
+
return { code, changed: false };
|
|
753
|
+
}
|
|
754
|
+
|
|
755
|
+
let changed = false;
|
|
756
|
+
const convert = (templateSlice) =>
|
|
757
|
+
templateSlice.replace(
|
|
758
|
+
VUE_DIRECTIVE_ATTR_REGEX,
|
|
759
|
+
(match, namePart, quoted) => {
|
|
760
|
+
if (VUE_NON_EXPRESSION_DIRECTIVES.test(namePart.trim())) {
|
|
761
|
+
return match;
|
|
762
|
+
}
|
|
763
|
+
const inner = quoted.slice(1, -1);
|
|
764
|
+
// An empty value, or a `{{` opening (Vue ignores interpolations in
|
|
765
|
+
// attribute values), is left alone.
|
|
766
|
+
if (!inner.trim() || inner.includes("{{")) {
|
|
767
|
+
return match;
|
|
768
|
+
}
|
|
769
|
+
try {
|
|
770
|
+
parse(`(${inner})`, DIRECTIVE_VALUE_PARSE_OPTIONS);
|
|
771
|
+
} catch {
|
|
772
|
+
return match;
|
|
773
|
+
}
|
|
774
|
+
changed = true;
|
|
775
|
+
return `${namePart}={${inner}}`;
|
|
776
|
+
}
|
|
777
|
+
);
|
|
778
|
+
|
|
779
|
+
const converted =
|
|
780
|
+
code.slice(0, templateStart) +
|
|
781
|
+
convert(code.slice(templateStart, templateEnd)) +
|
|
782
|
+
code.slice(templateEnd);
|
|
783
|
+
return { code: converted, changed };
|
|
784
|
+
};
|
|
785
|
+
|
|
595
786
|
const cleanVueCodeForParsing = (code, { includeScripts = true } = {}) => {
|
|
596
787
|
let cleanedCode = code
|
|
597
788
|
.replace(vueCommentRegex, (match) => maskNonNewlineChars(match))
|
|
@@ -653,7 +844,32 @@ const buildVueParseCandidates = (code) => {
|
|
|
653
844
|
? `${scriptOnlyCandidate}\n${templateOnlyCandidate}`
|
|
654
845
|
: templateOnlyCandidate;
|
|
655
846
|
|
|
847
|
+
// Directive-expression candidates first: same masking, but `v-html="x"`
|
|
848
|
+
// style values are expression containers, so template expressions keep
|
|
849
|
+
// their references. The plain candidates below remain as fallbacks for the
|
|
850
|
+
// (checked-per-attribute, but defensive) case of a converted file not
|
|
851
|
+
// parsing as a whole.
|
|
852
|
+
const directiveCandidates = (() => {
|
|
853
|
+
const { code: directiveCode, changed } = vueDirectiveValueToExpression(
|
|
854
|
+
code
|
|
855
|
+
);
|
|
856
|
+
if (!changed) {
|
|
857
|
+
return [];
|
|
858
|
+
}
|
|
859
|
+
const directiveFull = cleanVueCodeForParsing(directiveCode, {
|
|
860
|
+
includeScripts: true
|
|
861
|
+
});
|
|
862
|
+
const directiveTemplateOnly = cleanVueCodeForParsing(directiveCode, {
|
|
863
|
+
includeScripts: false
|
|
864
|
+
});
|
|
865
|
+
return [
|
|
866
|
+
{ name: "directive-full", code: directiveFull },
|
|
867
|
+
{ name: "directive-template-only", code: directiveTemplateOnly }
|
|
868
|
+
];
|
|
869
|
+
})();
|
|
870
|
+
|
|
656
871
|
const candidates = [
|
|
872
|
+
...directiveCandidates,
|
|
657
873
|
{ name: "full", code: fullCandidate },
|
|
658
874
|
{ name: "combined", code: combinedCandidate },
|
|
659
875
|
{ name: "template-only", code: templateOnlyCandidate },
|
|
@@ -686,7 +902,14 @@ const parseVueAstWithFallback = (file, code) => {
|
|
|
686
902
|
throw lastError || new Error(`Unable to parse Vue file: ${file}`);
|
|
687
903
|
};
|
|
688
904
|
|
|
689
|
-
|
|
905
|
+
/**
|
|
906
|
+
* Build the virtual type source for a single-file component (`.vue` or
|
|
907
|
+
* `.svelte`): the file masked to spaces/newlines with only the `<script>`
|
|
908
|
+
* contents left verbatim. Positions are preserved exactly, so type offsets
|
|
909
|
+
* map back onto the original file. Works for both frameworks because it is
|
|
910
|
+
* driven purely by the `<script>...</script>` regex.
|
|
911
|
+
*/
|
|
912
|
+
const createVirtualTypeSource = (code) => {
|
|
690
913
|
const output = maskNonNewlineChars(code).split("");
|
|
691
914
|
let hasScriptContent = false;
|
|
692
915
|
let scriptMatch;
|
|
@@ -708,13 +931,22 @@ const createVueVirtualTypeSource = (code) => {
|
|
|
708
931
|
};
|
|
709
932
|
};
|
|
710
933
|
|
|
711
|
-
|
|
712
|
-
|
|
934
|
+
/**
|
|
935
|
+
* Type-check the virtual source with a throwaway program: the virtual file
|
|
936
|
+
* plus the framework's shim declarations, both under a temp directory.
|
|
937
|
+
*/
|
|
938
|
+
const collectTypesWithVirtualProgram = (
|
|
939
|
+
file,
|
|
940
|
+
virtualSource,
|
|
941
|
+
shimFileName,
|
|
942
|
+
shimSource
|
|
943
|
+
) => {
|
|
944
|
+
const tempDir = mkdtempSync(join(tmpdir(), "atom-parsetools-sfc-"));
|
|
713
945
|
const virtualFile = join(tempDir, `${basename(file)}.ts`);
|
|
714
|
-
const shimFile = join(tempDir,
|
|
946
|
+
const shimFile = join(tempDir, shimFileName);
|
|
715
947
|
try {
|
|
716
948
|
writeFileSync(virtualFile, virtualSource, "utf8");
|
|
717
|
-
writeFileSync(shimFile,
|
|
949
|
+
writeFileSync(shimFile, shimSource, "utf8");
|
|
718
950
|
const virtualTs = createTsc([virtualFile, shimFile], tempDir);
|
|
719
951
|
const sourceFile = virtualTs?.program?.getSourceFile(virtualFile);
|
|
720
952
|
if (!virtualTs || !sourceFile) {
|
|
@@ -728,7 +960,24 @@ const collectVueTypesWithVirtualProgram = (file, virtualSource) => {
|
|
|
728
960
|
}
|
|
729
961
|
};
|
|
730
962
|
|
|
731
|
-
|
|
963
|
+
/**
|
|
964
|
+
* Collect types for a single-file component. The project program is tried
|
|
965
|
+
* first: a `.vue`/`.svelte` path is normally absent from it, but a tsconfig
|
|
966
|
+
* that maps the extension can make it resolvable, and that mapping is more
|
|
967
|
+
* accurate than the virtual source. Otherwise fall back to type-checking the
|
|
968
|
+
* position-preserving virtual source against the framework's shims.
|
|
969
|
+
*
|
|
970
|
+
* `.vue` and `.svelte` differ only in which shim declarations the checker
|
|
971
|
+
* needs, so both go through here rather than through near-identical copies
|
|
972
|
+
* that would drift apart.
|
|
973
|
+
*/
|
|
974
|
+
const collectSfcSeenTypes = (
|
|
975
|
+
file,
|
|
976
|
+
code,
|
|
977
|
+
tsInstance,
|
|
978
|
+
shimFileName,
|
|
979
|
+
shimSource
|
|
980
|
+
) => {
|
|
732
981
|
let seenTypes;
|
|
733
982
|
if (tsInstance?.program) {
|
|
734
983
|
try {
|
|
@@ -742,15 +991,38 @@ const collectVueSeenTypes = (file, code, tsInstance) => {
|
|
|
742
991
|
}
|
|
743
992
|
|
|
744
993
|
if (!seenTypes || seenTypes.size === 0) {
|
|
745
|
-
const virtualSource =
|
|
994
|
+
const virtualSource = createVirtualTypeSource(code);
|
|
746
995
|
if (virtualSource.hasScriptContent) {
|
|
747
|
-
seenTypes =
|
|
996
|
+
seenTypes = collectTypesWithVirtualProgram(
|
|
997
|
+
file,
|
|
998
|
+
virtualSource.source,
|
|
999
|
+
shimFileName,
|
|
1000
|
+
shimSource
|
|
1001
|
+
);
|
|
748
1002
|
}
|
|
749
1003
|
}
|
|
750
1004
|
|
|
751
1005
|
return seenTypes;
|
|
752
1006
|
};
|
|
753
1007
|
|
|
1008
|
+
const collectVueSeenTypes = (file, code, tsInstance) =>
|
|
1009
|
+
collectSfcSeenTypes(
|
|
1010
|
+
file,
|
|
1011
|
+
code,
|
|
1012
|
+
tsInstance,
|
|
1013
|
+
"vue-shims.d.ts",
|
|
1014
|
+
VUE_COMPILER_MACRO_SHIMS
|
|
1015
|
+
);
|
|
1016
|
+
|
|
1017
|
+
const collectSvelteSeenTypes = (file, code, tsInstance) =>
|
|
1018
|
+
collectSfcSeenTypes(
|
|
1019
|
+
file,
|
|
1020
|
+
code,
|
|
1021
|
+
tsInstance,
|
|
1022
|
+
"svelte-shims.d.ts",
|
|
1023
|
+
SVELTE_RUNE_SHIMS
|
|
1024
|
+
);
|
|
1025
|
+
|
|
754
1026
|
const TSC_FLAGS =
|
|
755
1027
|
tsc.TypeFormatFlags.NoTruncation |
|
|
756
1028
|
tsc.TypeFormatFlags.InTypeAlias |
|
|
@@ -778,6 +1050,10 @@ const collectSeenTypesForFile = (file, ts, options) => {
|
|
|
778
1050
|
return collectVueSeenTypes(file, readFileSync(file, "utf-8"), ts);
|
|
779
1051
|
}
|
|
780
1052
|
|
|
1053
|
+
if (file.endsWith(".svelte")) {
|
|
1054
|
+
return collectSvelteSeenTypes(file, readFileSync(file, "utf-8"), ts);
|
|
1055
|
+
}
|
|
1056
|
+
|
|
781
1057
|
if (!ts?.program) {
|
|
782
1058
|
return undefined;
|
|
783
1059
|
}
|
package/package.json
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@appthreat/atom-parsetools",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.7.0",
|
|
4
4
|
"description": "Parsing tools that complement the @appthreat/atom project.",
|
|
5
5
|
"main": "./index.js",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"scripts": {
|
|
8
8
|
"pretty": "prettier --write *.js --trailing-comma=none",
|
|
9
|
-
"test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
|
|
9
|
+
"test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-svelte-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
|
|
10
10
|
"test:evaluate": "node test-fixtures/evaluate-astgen.js",
|
|
11
11
|
"test:ruby": "node test-fixtures/rbastgen-regression.js",
|
|
12
12
|
"test:php": "node test-fixtures/phpastgen-cli.js && node test-fixtures/phpastgen-parser-info.js && node test-fixtures/phpastgen-discovery.js && node test-fixtures/phpastgen-provenance.js && node test-fixtures/phpastgen-framework-facts.js && node test-fixtures/phpastgen-legacy-regression.js && node test-fixtures/phpastgen-concurrency.js && node test-fixtures/phpastgen-regression.js && node test-fixtures/phpastgen-contract-snapshot.js && npm run test:php:pbt",
|
|
@@ -19,13 +19,15 @@
|
|
|
19
19
|
"test:shape:update": "UPDATE_SHAPE_SNAPSHOT=1 node test-fixtures/astgen-shape-snapshot.js",
|
|
20
20
|
"test:json": "node test-fixtures/astgen-json-regression.js",
|
|
21
21
|
"test:fixtures": "node test-fixtures/test-suite.js",
|
|
22
|
-
"test:vue": "node test-fixtures/astgen-vue-regression.js"
|
|
22
|
+
"test:vue": "node test-fixtures/astgen-vue-regression.js",
|
|
23
|
+
"test:svelte": "node test-fixtures/astgen-svelte-regression.js"
|
|
23
24
|
},
|
|
24
25
|
"dependencies": {
|
|
25
26
|
"@appthreat/atom-common": "^1.1.0",
|
|
26
27
|
"@babel/parser": "^8.0.4",
|
|
27
28
|
"@typescript/typescript6": "^6.0.2",
|
|
28
|
-
"hermes-parser": "^0.37.0"
|
|
29
|
+
"hermes-parser": "^0.37.0",
|
|
30
|
+
"svelte": "5.57.0"
|
|
29
31
|
},
|
|
30
32
|
"bin": {
|
|
31
33
|
"astgen": "astgen.js",
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
<?php return array(
|
|
2
2
|
'root' => array(
|
|
3
3
|
'name' => '__root__',
|
|
4
|
-
'pretty_version' => 'v1.
|
|
5
|
-
'version' => '1.
|
|
6
|
-
'reference' => '
|
|
4
|
+
'pretty_version' => 'v1.7.0',
|
|
5
|
+
'version' => '1.7.0.0',
|
|
6
|
+
'reference' => '26f7c3eeab1661fdf7ee1a5a7b40f0fd034235eb',
|
|
7
7
|
'type' => 'library',
|
|
8
8
|
'install_path' => __DIR__ . '/../../',
|
|
9
9
|
'aliases' => array(),
|
|
@@ -11,9 +11,9 @@
|
|
|
11
11
|
),
|
|
12
12
|
'versions' => array(
|
|
13
13
|
'__root__' => array(
|
|
14
|
-
'pretty_version' => 'v1.
|
|
15
|
-
'version' => '1.
|
|
16
|
-
'reference' => '
|
|
14
|
+
'pretty_version' => 'v1.7.0',
|
|
15
|
+
'version' => '1.7.0.0',
|
|
16
|
+
'reference' => '26f7c3eeab1661fdf7ee1a5a7b40f0fd034235eb',
|
|
17
17
|
'type' => 'library',
|
|
18
18
|
'install_path' => __DIR__ . '/../../',
|
|
19
19
|
'aliases' => array(),
|