@appthreat/atom-parsetools 1.4.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -103
- package/astgen.js +148 -11
- package/package.json +12 -4
- package/phpastgen.js +1764 -18
- package/plugins/autoload.php +1 -1
- package/plugins/composer/autoload_classmap.php +1 -0
- package/plugins/composer/autoload_real.php +4 -4
- package/plugins/composer/autoload_static.php +5 -4
- package/plugins/composer/installed.json +7 -8
- package/plugins/composer/installed.php +9 -9
- package/plugins/composer.json +1 -1
- package/plugins/composer.lock +7 -8
- package/plugins/nikic/php-parser/composer.json +1 -2
- package/plugins/nikic/php-parser/lib/PhpParser/ConstExprEvaluator.php +1 -3
- package/plugins/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php +2 -0
- package/plugins/nikic/php-parser/lib/PhpParser/Lexer/TokenEmulator/FnTokenEmulator.php +20 -0
- package/plugins/nikic/php-parser/lib/PhpParser/Lexer/TokenEmulator/KeywordEmulator.php +5 -4
- package/plugins/nikic/php-parser/lib/PhpParser/ParserAbstract.php +7 -0
- package/plugins/nikic/php-parser/lib/PhpParser/PrettyPrinterAbstract.php +20 -4
- package/svelteAst.js +1291 -0
package/README.md
CHANGED
|
@@ -1,12 +1,16 @@
|
|
|
1
|
-
#
|
|
1
|
+
# atom-parsetools
|
|
2
2
|
|
|
3
3
|
This package hosts a collection of parsing tools that complement the `@appthreat/atom` project. These tools offer parsing and analysis-related functionalities such as generating AST and semantics information in JSON format. The full list of tools and bin commands exposed by this package is below:
|
|
4
4
|
|
|
5
|
-
- astgen - Generates AST for JavaScript and
|
|
5
|
+
- astgen - Generates AST for JavaScript, TypeScript, Vue and Svelte projects in JSON format
|
|
6
6
|
- phpastgen - Generates AST for PHP projects using `php-parse` command from `nikic/php-parser`
|
|
7
7
|
- rbastgen - Generates AST for Ruby projects using AppThreat's [`ruby_ast_gen`](https://github.com/AppThreat/ruby_ast_gen) gem (2.0.1)
|
|
8
8
|
- scalasem - Generates a custom semantics slice for Scala Projects by utilising scalac command.
|
|
9
9
|
|
|
10
|
+
## Documentation
|
|
11
|
+
|
|
12
|
+
The full documentation lives at [https://appthreat.github.io/atom-parsetools/](https://appthreat.github.io/atom-parsetools/): per-tool guides, the output format specification, environment variable reference, packaging notes, and ten hands-on tutorials. The pages are rendered from the [`docs`](docs/) directory of this repository.
|
|
13
|
+
|
|
10
14
|
## Runtime support
|
|
11
15
|
|
|
12
16
|
These tools run on both [Node.js](https://nodejs.org) (>= 22, required by `@babel/parser` 8) and [Bun](https://bun.sh). All commands and the accompanying regression test-suite are exercised under both runtimes in CI, so the commands below can be invoked with either `node` or `bun` interchangeably (for example `bun astgen.js -i .`).
|
|
@@ -29,52 +33,37 @@ Options:
|
|
|
29
33
|
-h Show help [boolean]
|
|
30
34
|
```
|
|
31
35
|
|
|
32
|
-
|
|
36
|
+
Each source file becomes an AST JSON document plus a `.typemap` of inferred types keyed by node offsets. Test files and `node_modules` are excluded by default; the [astgen guide](docs/ASTGEN.md) covers every option and env variable (`ASTGEN_TYPE_WORKERS`, `ASTGEN_INCLUDE_TEST_FILES`, and friends).
|
|
33
37
|
|
|
34
|
-
|
|
35
|
-
| ------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
36
|
-
| `ASTGEN_TYPE_WORKERS` | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
|
|
37
|
-
| `ASTGEN_INCLUDE_TEST_FILES` | `false` | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation. |
|
|
38
|
-
| `ASTGEN_CONCURRENCY` | `10` | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes). |
|
|
39
|
-
| `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false` | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely. |
|
|
40
|
-
| `ASTGEN_IGNORE_DIRS` | unset | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`). |
|
|
38
|
+
`.vue` and `.svelte` single-file components are flattened into the same Babel shape as any other file: script statements at the top level of the `Program`, and the template as standard JSX nodes. Svelte components are segmented with `svelte/compiler` and every offset is an absolute byte position into the component source, so consumers that slice the original file for `code` fields get the real text back. Control flow maps onto its JSX equivalent — `{#if}` to a `ConditionalExpression`, `{#each}` to `.map()` with an arrow closure, `{#await}` to `.then()` — which means no framework-specific node types are emitted. See the [astgen guide](docs/ASTGEN.md) for the full mapping and the list of accepted losses.
|
|
41
39
|
|
|
42
40
|
### phpastgen
|
|
43
41
|
|
|
44
42
|
```text
|
|
45
43
|
node phpastgen.js --help
|
|
46
44
|
|
|
47
|
-
Usage: phpastgen [
|
|
48
|
-
or: phpastgen [operations] "<?php code"
|
|
49
|
-
Turn PHP source code into an abstract syntax tree.
|
|
50
|
-
|
|
51
|
-
Operations is a list of the following options (--dump by default):
|
|
52
|
-
|
|
53
|
-
-d, --dump Dump nodes using NodeDumper
|
|
54
|
-
-p, --pretty-print Pretty print file using PrettyPrinter\Standard
|
|
55
|
-
-j, --json-dump Print json_encode() result
|
|
56
|
-
--var-dump var_dump() nodes (for exact structure)
|
|
57
|
-
-N, --resolve-names Resolve names using NodeVisitor\NameResolver
|
|
58
|
-
-c, --with-column-info Show column numbers for errors (if available)
|
|
59
|
-
-P, --with-positions Show positions in node dumps
|
|
60
|
-
-r, --with-recovery Use parsing with error recovery
|
|
61
|
-
-h, --help Display this page
|
|
45
|
+
Usage: phpastgen [options] [-- <legacy php-parse args>]
|
|
62
46
|
|
|
47
|
+
Options:
|
|
48
|
+
-i, --input <path> input file or directory (batch mode)
|
|
49
|
+
-o, --output <dir> output directory (default: '.ast')
|
|
50
|
+
-e, --exclude <regex> exclusion regex (default: '^(tests?|vendor|Tests?)')
|
|
51
|
+
-l, --log <level> debug | info | warn | error (default: info)
|
|
52
|
+
-d, --debug same as --log debug
|
|
53
|
+
--target-version <x.y> pin PHP grammar (alias: --parser-target)
|
|
54
|
+
--max-depth <n> depth cap before truncation (default: 250)
|
|
55
|
+
--threads <n> worker processes for directory runs (default: 10)
|
|
56
|
+
--fail-on-error exit non-zero if any file failed
|
|
57
|
+
--parser-info print parser/runtime capability report and exit 0
|
|
58
|
+
--version print generator version and exit 0
|
|
59
|
+
--help print usage
|
|
63
60
|
```
|
|
64
61
|
|
|
65
|
-
|
|
62
|
+
The PHP parser (nikic/php-parser 5.8.0, grammars 8.0 to 8.5) is vendored under `plugins/`, so only a PHP runtime on the machine is required. Batch runs write one JSON per file plus `phpastgen_manifest.jsonl` and, on failure, `phpastgen_diagnostics.jsonl`. Details in the [phpastgen guide](docs/PHPASTGEN.md).
|
|
66
63
|
|
|
67
|
-
|
|
68
|
-
its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be
|
|
69
|
-
gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed
|
|
70
|
-
to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which
|
|
71
|
-
resolved its paths from the ABI of the Ruby that built it.
|
|
64
|
+
### rbastgen
|
|
72
65
|
|
|
73
|
-
`
|
|
74
|
-
ABI and per platform, and both are default gems in every supported Ruby, so the runtime's own copies
|
|
75
|
-
are used. One consequence is visible: the newest grammar available follows the interpreter's prism,
|
|
76
|
-
so Ruby 3.4 tops out lower than Ruby 4.0 (grammar 3.5 versus 4.1 at the time of writing). Installing
|
|
77
|
-
a newer `prism` gem on the machine raises it, since the caller's `GEM_PATH` is preserved.
|
|
66
|
+
Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which resolved its paths from the ABI of the Ruby that built it.
|
|
78
67
|
|
|
79
68
|
```text
|
|
80
69
|
node rbastgen.js --help
|
|
@@ -95,58 +84,7 @@ Usage:
|
|
|
95
84
|
--help Print usage
|
|
96
85
|
```
|
|
97
86
|
|
|
98
|
-
Problems with individual files are reported and skipped, never fatal; only usage errors (a missing
|
|
99
|
-
`-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that
|
|
100
|
-
`rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the
|
|
101
|
-
log but the wrapper still exits 0 — call the gem directly if a CI job needs to fail on a parse
|
|
102
|
-
error.
|
|
103
|
-
|
|
104
|
-
#### What the bundled generator emits (ruby_ast_gen 2.0.1)
|
|
105
|
-
|
|
106
|
-
- **Parsing is decoupled from the running Ruby.** When `prism` is available the newest grammar its
|
|
107
|
-
translation layer supports is used, so a 3.4 runtime parses Ruby 4.0/4.1 syntax. `--parser-target
|
|
108
|
-
x.y` pins a grammar instead (down to 1.8 through the `parser` gem), and a file that fails under
|
|
109
|
-
the selected grammar is retried once with the newest one. Every JSON file records the backend
|
|
110
|
-
that produced it in `parser_backend`, alongside `generator_version` and `ruby_version`.
|
|
111
|
-
- **Ruby DSL files are discovered, not just `.rb`.** `Rakefile`, `Gemfile`, `Capfile`,
|
|
112
|
-
`Vagrantfile`, `Fastfile` and friends are matched by basename, plus the `.gemspec`, `.rake`,
|
|
113
|
-
`.ru`, `.rbi`, `.thor`, `.jbuilder`, `.axlsx` and `.rabl` extensions. Vendor and tool
|
|
114
|
-
directories (`.git`, `.bundle`, `.venv`, …) are skipped.
|
|
115
|
-
- **Non-UTF-8 sources survive.** `# coding:` magic comments are honoured and undecodable bytes are
|
|
116
|
-
scrubbed rather than dropping the file, marked with `encoding_scrubbed: true`.
|
|
117
|
-
- **Deeply nested source is truncated, not dropped**, with `truncated: true` on the boundary node
|
|
118
|
-
and a `truncated_nodes` count at the top level (`--max-depth`).
|
|
119
|
-
- **Semantic metadata for consumers**: a `magic_comments` array (Sorbet `typed:` levels,
|
|
120
|
-
`frozen_string_literal`, …), `call_operator`/`has_parentheses` on calls, `percent_array` and
|
|
121
|
-
regexp `options`, heredoc body offsets, and `has_sig: true` on a `def` preceded by a Sorbet `sig`
|
|
122
|
-
block.
|
|
123
|
-
- **Two side-records per run, written inside the output directory**, both ending in `.jsonl` so a
|
|
124
|
-
consumer globbing `*.json` for ASTs never mistakes them for one:
|
|
125
|
-
`ruby_ast_gen_manifest.jsonl` (one object: input/output, backend, counts of parsed, failed,
|
|
126
|
-
skipped, excluded and truncated files, threads, max depth) and
|
|
127
|
-
`ruby_ast_gen_diagnostics.jsonl` (one object per failed file with message, line, column), the
|
|
128
|
-
latter written only when something failed and removed when a later run is clean.
|
|
129
|
-
|
|
130
|
-
To check which backend a machine will use:
|
|
131
|
-
|
|
132
|
-
```shell
|
|
133
|
-
rbastgen --parser-info
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
`Parser gem` and `Prism gem` name the versions actually loaded — the vendored `parser` and the
|
|
137
|
-
runtime's `prism` — so they are the quickest way to tell which copy of each library a machine is
|
|
138
|
-
parsing with. They read `unavailable` only when a library genuinely is not loaded.
|
|
139
|
-
|
|
140
|
-
#### Environment variables
|
|
141
|
-
|
|
142
|
-
| Variable | Default | Purpose |
|
|
143
|
-
| --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
144
|
-
| `ATOM_RUBY_HOME` | unset | Ruby install directory to use when a suitable `ruby` is not on the `PATH`; its `bin` is prepended to `PATH` for the child process. |
|
|
145
|
-
| `RUBY_CMD` | `ruby` | Ruby interpreter to invoke. Set this (or `ATOM_RUBY_HOME`) when the detected version is not 3.4.x/4.0.x. |
|
|
146
|
-
| `RUBY_ASTGEN_BIN` | bundled `ruby_ast_gen` | The generator script to run. Point it at a checkout's `exe/ruby_ast_gen` to test an unreleased `ruby_ast_gen` without touching this package. |
|
|
147
|
-
| `ATOM_CWD` | `process.cwd()` | Working directory for the generator, which is what relative `-i`/`-o` paths resolve against. |
|
|
148
|
-
| `ATOM_TIMEOUT` / `ASTGEN_TIMEOUT` | unset (no timeout) | Milliseconds before the generator process is killed. `ATOM_TIMEOUT` wins; a non-numeric value is ignored. Also honoured by `scalasem`. |
|
|
149
|
-
| `GEM_PATH` | unset | Preserved and appended to the vendored bundle, so gems installed on the machine (a newer `prism`, for instance) stay reachable. |
|
|
87
|
+
Problems with individual files are reported and skipped, never fatal; only usage errors (a missing `-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that `rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the log but the wrapper still exits 0; call the gem directly if a CI job needs to fail on a parse error. Discovery covers Ruby DSL files (`Gemfile`, `Rakefile`, `.gemspec`, `.rake`, and more), and each run writes `ruby_ast_gen_manifest.jsonl` and, on failure, `ruby_ast_gen_diagnostics.jsonl`. Details in the [rbastgen guide](docs/RBASTGEN.md).
|
|
150
88
|
|
|
151
89
|
### scalasem
|
|
152
90
|
|
|
@@ -160,29 +98,20 @@ Example:
|
|
|
160
98
|
scalasem $(pwd) slices.json
|
|
161
99
|
```
|
|
162
100
|
|
|
101
|
+
Compiles the project with sbt or mill if no `.tasty` files exist, then extracts literals, used types, and Play framework tags into a semantic slice. The [scalasem guide](docs/SCALASEM.md) covers the pipeline.
|
|
102
|
+
|
|
163
103
|
## Testing
|
|
164
104
|
|
|
165
|
-
`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby
|
|
166
|
-
run the `rbastgen` command end to end against `test-fixtures/projects/ruby-parsing`, so they need a
|
|
167
|
-
built plugin bundle and a supported Ruby, and they skip themselves cleanly when either is missing:
|
|
105
|
+
`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby and PHP workflows need their runtimes and a built plugin bundle, and they skip themselves cleanly when either is missing:
|
|
168
106
|
|
|
169
107
|
```shell
|
|
170
108
|
bash build.sh --ruby-only
|
|
171
109
|
npm run test:ruby
|
|
172
110
|
```
|
|
173
111
|
|
|
174
|
-
`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
They assert what a consumer depends on rather than that the command ran: which files are discovered
|
|
179
|
-
(including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`,
|
|
180
|
-
`percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a
|
|
181
|
-
failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no
|
|
182
|
-
stale record behind, and that the bundle contains no compiled extension. CI runs them under both
|
|
183
|
-
Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball
|
|
184
|
-
and parses with that, which is the check that catches a bundle usable only on the Ruby that built
|
|
185
|
-
it.
|
|
112
|
+
`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle is present and free of compiled extensions. CI runs it, and it runs the same way locally.
|
|
113
|
+
|
|
114
|
+
They assert what a consumer depends on rather than that the command ran: which files are discovered (including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`, `percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no stale record behind, and that the bundle contains no compiled extension. CI runs them under both Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball and parses with that, which is the check that catches a bundle usable only on the Ruby that built it. The [testing guide](docs/TESTING.md) maps the suites.
|
|
186
115
|
|
|
187
116
|
## License
|
|
188
117
|
|
package/astgen.js
CHANGED
|
@@ -24,12 +24,15 @@ import {
|
|
|
24
24
|
rmSync
|
|
25
25
|
} from "fs";
|
|
26
26
|
import { getAllFiles } from "@appthreat/atom-common";
|
|
27
|
+
import { parseSvelteFile, parseSvelteScriptBuffer } from "./svelteAst.js";
|
|
27
28
|
|
|
28
29
|
// Printed by `astgen --version`. Downstream frontends (e.g. chen's jssrc2cpg)
|
|
29
30
|
// fold this into their parse-cache fingerprint, so it MUST be bumped whenever
|
|
30
31
|
// the emitted AST/type shape changes — otherwise stale cached parses from an
|
|
31
|
-
// older astgen are silently reused. Bumped
|
|
32
|
-
|
|
32
|
+
// older astgen are silently reused. Bumped to 4.2.0 for first-class Svelte
|
|
33
|
+
// support: `.svelte` files now emit absolute-offset Babel JSX ASTs instead of
|
|
34
|
+
// the offset-shifted script-only output of the legacy masking path.
|
|
35
|
+
const ASTGEN_VERSION = "4.2.0";
|
|
33
36
|
|
|
34
37
|
const HELP_TEXT = `Options:
|
|
35
38
|
-i, --src Source directory [default: "."]
|
|
@@ -466,15 +469,42 @@ const getAllSrcJSAndTSFiles = (src) => {
|
|
|
466
469
|
* Convert a single JS/TS file to AST
|
|
467
470
|
*/
|
|
468
471
|
const fileToJsAst = (file, projectType, tsInstance) => {
|
|
469
|
-
if (file.endsWith(".vue")
|
|
472
|
+
if (file.endsWith(".vue")) {
|
|
470
473
|
return toVueAst(file, tsInstance);
|
|
471
474
|
}
|
|
475
|
+
if (file.endsWith(".svelte")) {
|
|
476
|
+
return toSvelteAst(file);
|
|
477
|
+
}
|
|
472
478
|
if (file.endsWith(".ejs")) {
|
|
473
479
|
return toEjsAst(file);
|
|
474
480
|
}
|
|
475
481
|
return codeToJsAst(file, readFileSync(file, "utf-8"), projectType);
|
|
476
482
|
};
|
|
477
483
|
|
|
484
|
+
/**
|
|
485
|
+
* Convert a single Svelte file to AST. Svelte's own compiler segments the
|
|
486
|
+
* single-file component; the Babel options are built here and passed in, so
|
|
487
|
+
* every Svelte sub-parse shares the exact configuration used for regular
|
|
488
|
+
* JS/TS files. If `svelte/compiler` rejects the whole file (a genuinely
|
|
489
|
+
* broken template), the script blocks are still parsed over a
|
|
490
|
+
* position-preserving masked buffer - absolute offsets, so line numbers stay
|
|
491
|
+
* correct - and the failure is recorded on the emitted AST's `errors` array.
|
|
492
|
+
*/
|
|
493
|
+
const toSvelteAst = (file) => {
|
|
494
|
+
const code = readFileSync(file, "utf-8");
|
|
495
|
+
const options = makeBabelOptions(babelParserOptions, file);
|
|
496
|
+
try {
|
|
497
|
+
return parseSvelteFile(file, code, options);
|
|
498
|
+
} catch (err) {
|
|
499
|
+
console.error(
|
|
500
|
+
`Svelte parse failed for ${file}, falling back to script-only parsing:`,
|
|
501
|
+
err?.message
|
|
502
|
+
);
|
|
503
|
+
const { source: maskedSource } = createVirtualTypeSource(code);
|
|
504
|
+
return parseSvelteScriptBuffer(file, code, maskedSource, options, err?.message);
|
|
505
|
+
}
|
|
506
|
+
};
|
|
507
|
+
|
|
478
508
|
/**
|
|
479
509
|
* Convert a single JS/TS code snippet to AST
|
|
480
510
|
*/
|
|
@@ -590,6 +620,53 @@ declare module "vue" {
|
|
|
590
620
|
}
|
|
591
621
|
`;
|
|
592
622
|
|
|
623
|
+
// Ambient declarations for the Svelte 5 runes and the most common `svelte` /
|
|
624
|
+
// `svelte/store` imports, so the TypeScript checker sees real declarations
|
|
625
|
+
// when it type-checks a virtual `.svelte.ts` source. This is a pragmatic
|
|
626
|
+
// starting set rather than a mirror of svelte's own types; grow it when a
|
|
627
|
+
// fixture needs more.
|
|
628
|
+
const SVELTE_RUNE_SHIMS = `
|
|
629
|
+
declare function $state<T>(initial?: T): T;
|
|
630
|
+
declare namespace $state { function raw<T>(initial?: T): T; function snapshot<T>(v: T): T; }
|
|
631
|
+
declare function $derived<T>(expression: T): T;
|
|
632
|
+
declare namespace $derived { function by<T>(fn: () => T): T; }
|
|
633
|
+
declare function $effect(fn: () => void | (() => void)): void;
|
|
634
|
+
declare namespace $effect {
|
|
635
|
+
function pre(fn: () => void | (() => void)): void;
|
|
636
|
+
function tracking(): boolean;
|
|
637
|
+
function root(fn: () => void | (() => void)): () => void;
|
|
638
|
+
function pending(): number;
|
|
639
|
+
}
|
|
640
|
+
declare function $props<T = any>(): T;
|
|
641
|
+
declare namespace $props { function id(): string; }
|
|
642
|
+
declare function $bindable<T>(fallback?: T): T;
|
|
643
|
+
declare function $inspect<T extends any[]>(...values: T): { with: (fn: (...args: any[]) => void) => void };
|
|
644
|
+
declare function $host<T = HTMLElement>(): T;
|
|
645
|
+
|
|
646
|
+
declare module "svelte" {
|
|
647
|
+
export function onMount(fn: () => void | (() => void)): void;
|
|
648
|
+
export function onDestroy(fn: () => void): void;
|
|
649
|
+
export function tick(): Promise<void>;
|
|
650
|
+
export function untrack<T>(fn: () => T): T;
|
|
651
|
+
export function getContext<T>(key: any): T;
|
|
652
|
+
export function setContext<T>(key: any, value: T): T;
|
|
653
|
+
export function hasContext(key: any): boolean;
|
|
654
|
+
export function createEventDispatcher<T = any>(): (type: string, detail?: any) => void;
|
|
655
|
+
export function mount(component: any, options: any): any;
|
|
656
|
+
export function unmount(component: any): Promise<void>;
|
|
657
|
+
export type Component<P = any> = (...args: any[]) => any;
|
|
658
|
+
export type Snippet<P extends any[] = any[]> = (...args: P) => any;
|
|
659
|
+
}
|
|
660
|
+
declare module "svelte/store" {
|
|
661
|
+
export type Readable<T> = { subscribe(run: (value: T) => void): () => void };
|
|
662
|
+
export type Writable<T> = Readable<T> & { set(value: T): void; update(fn: (value: T) => T): void };
|
|
663
|
+
export function writable<T>(value?: T): Writable<T>;
|
|
664
|
+
export function readable<T>(value?: T): Readable<T>;
|
|
665
|
+
export function derived<T>(stores: any, fn: any, initial?: T): Readable<T>;
|
|
666
|
+
export function get<T>(store: Readable<T>): T;
|
|
667
|
+
}
|
|
668
|
+
`;
|
|
669
|
+
|
|
593
670
|
const maskNonNewlineChars = (value) => value.replace(/[^\r\n]/g, " ");
|
|
594
671
|
|
|
595
672
|
const cleanVueCodeForParsing = (code, { includeScripts = true } = {}) => {
|
|
@@ -686,7 +763,14 @@ const parseVueAstWithFallback = (file, code) => {
|
|
|
686
763
|
throw lastError || new Error(`Unable to parse Vue file: ${file}`);
|
|
687
764
|
};
|
|
688
765
|
|
|
689
|
-
|
|
766
|
+
/**
|
|
767
|
+
* Build the virtual type source for a single-file component (`.vue` or
|
|
768
|
+
* `.svelte`): the file masked to spaces/newlines with only the `<script>`
|
|
769
|
+
* contents left verbatim. Positions are preserved exactly, so type offsets
|
|
770
|
+
* map back onto the original file. Works for both frameworks because it is
|
|
771
|
+
* driven purely by the `<script>...</script>` regex.
|
|
772
|
+
*/
|
|
773
|
+
const createVirtualTypeSource = (code) => {
|
|
690
774
|
const output = maskNonNewlineChars(code).split("");
|
|
691
775
|
let hasScriptContent = false;
|
|
692
776
|
let scriptMatch;
|
|
@@ -708,13 +792,22 @@ const createVueVirtualTypeSource = (code) => {
|
|
|
708
792
|
};
|
|
709
793
|
};
|
|
710
794
|
|
|
711
|
-
|
|
712
|
-
|
|
795
|
+
/**
|
|
796
|
+
* Type-check the virtual source with a throwaway program: the virtual file
|
|
797
|
+
* plus the framework's shim declarations, both under a temp directory.
|
|
798
|
+
*/
|
|
799
|
+
const collectTypesWithVirtualProgram = (
|
|
800
|
+
file,
|
|
801
|
+
virtualSource,
|
|
802
|
+
shimFileName,
|
|
803
|
+
shimSource
|
|
804
|
+
) => {
|
|
805
|
+
const tempDir = mkdtempSync(join(tmpdir(), "atom-parsetools-sfc-"));
|
|
713
806
|
const virtualFile = join(tempDir, `${basename(file)}.ts`);
|
|
714
|
-
const shimFile = join(tempDir,
|
|
807
|
+
const shimFile = join(tempDir, shimFileName);
|
|
715
808
|
try {
|
|
716
809
|
writeFileSync(virtualFile, virtualSource, "utf8");
|
|
717
|
-
writeFileSync(shimFile,
|
|
810
|
+
writeFileSync(shimFile, shimSource, "utf8");
|
|
718
811
|
const virtualTs = createTsc([virtualFile, shimFile], tempDir);
|
|
719
812
|
const sourceFile = virtualTs?.program?.getSourceFile(virtualFile);
|
|
720
813
|
if (!virtualTs || !sourceFile) {
|
|
@@ -728,7 +821,24 @@ const collectVueTypesWithVirtualProgram = (file, virtualSource) => {
|
|
|
728
821
|
}
|
|
729
822
|
};
|
|
730
823
|
|
|
731
|
-
|
|
824
|
+
/**
|
|
825
|
+
* Collect types for a single-file component. The project program is tried
|
|
826
|
+
* first: a `.vue`/`.svelte` path is normally absent from it, but a tsconfig
|
|
827
|
+
* that maps the extension can make it resolvable, and that mapping is more
|
|
828
|
+
* accurate than the virtual source. Otherwise fall back to type-checking the
|
|
829
|
+
* position-preserving virtual source against the framework's shims.
|
|
830
|
+
*
|
|
831
|
+
* `.vue` and `.svelte` differ only in which shim declarations the checker
|
|
832
|
+
* needs, so both go through here rather than through near-identical copies
|
|
833
|
+
* that would drift apart.
|
|
834
|
+
*/
|
|
835
|
+
const collectSfcSeenTypes = (
|
|
836
|
+
file,
|
|
837
|
+
code,
|
|
838
|
+
tsInstance,
|
|
839
|
+
shimFileName,
|
|
840
|
+
shimSource
|
|
841
|
+
) => {
|
|
732
842
|
let seenTypes;
|
|
733
843
|
if (tsInstance?.program) {
|
|
734
844
|
try {
|
|
@@ -742,15 +852,38 @@ const collectVueSeenTypes = (file, code, tsInstance) => {
|
|
|
742
852
|
}
|
|
743
853
|
|
|
744
854
|
if (!seenTypes || seenTypes.size === 0) {
|
|
745
|
-
const virtualSource =
|
|
855
|
+
const virtualSource = createVirtualTypeSource(code);
|
|
746
856
|
if (virtualSource.hasScriptContent) {
|
|
747
|
-
seenTypes =
|
|
857
|
+
seenTypes = collectTypesWithVirtualProgram(
|
|
858
|
+
file,
|
|
859
|
+
virtualSource.source,
|
|
860
|
+
shimFileName,
|
|
861
|
+
shimSource
|
|
862
|
+
);
|
|
748
863
|
}
|
|
749
864
|
}
|
|
750
865
|
|
|
751
866
|
return seenTypes;
|
|
752
867
|
};
|
|
753
868
|
|
|
869
|
+
const collectVueSeenTypes = (file, code, tsInstance) =>
|
|
870
|
+
collectSfcSeenTypes(
|
|
871
|
+
file,
|
|
872
|
+
code,
|
|
873
|
+
tsInstance,
|
|
874
|
+
"vue-shims.d.ts",
|
|
875
|
+
VUE_COMPILER_MACRO_SHIMS
|
|
876
|
+
);
|
|
877
|
+
|
|
878
|
+
const collectSvelteSeenTypes = (file, code, tsInstance) =>
|
|
879
|
+
collectSfcSeenTypes(
|
|
880
|
+
file,
|
|
881
|
+
code,
|
|
882
|
+
tsInstance,
|
|
883
|
+
"svelte-shims.d.ts",
|
|
884
|
+
SVELTE_RUNE_SHIMS
|
|
885
|
+
);
|
|
886
|
+
|
|
754
887
|
const TSC_FLAGS =
|
|
755
888
|
tsc.TypeFormatFlags.NoTruncation |
|
|
756
889
|
tsc.TypeFormatFlags.InTypeAlias |
|
|
@@ -778,6 +911,10 @@ const collectSeenTypesForFile = (file, ts, options) => {
|
|
|
778
911
|
return collectVueSeenTypes(file, readFileSync(file, "utf-8"), ts);
|
|
779
912
|
}
|
|
780
913
|
|
|
914
|
+
if (file.endsWith(".svelte")) {
|
|
915
|
+
return collectSvelteSeenTypes(file, readFileSync(file, "utf-8"), ts);
|
|
916
|
+
}
|
|
917
|
+
|
|
781
918
|
if (!ts?.program) {
|
|
782
919
|
return undefined;
|
|
783
920
|
}
|
package/package.json
CHANGED
|
@@ -1,25 +1,33 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@appthreat/atom-parsetools",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.6.0",
|
|
4
4
|
"description": "Parsing tools that complement the @appthreat/atom project.",
|
|
5
5
|
"main": "./index.js",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"scripts": {
|
|
8
8
|
"pretty": "prettier --write *.js --trailing-comma=none",
|
|
9
|
-
"test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
|
|
9
|
+
"test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-svelte-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
|
|
10
10
|
"test:evaluate": "node test-fixtures/evaluate-astgen.js",
|
|
11
11
|
"test:ruby": "node test-fixtures/rbastgen-regression.js",
|
|
12
|
+
"test:php": "node test-fixtures/phpastgen-cli.js && node test-fixtures/phpastgen-parser-info.js && node test-fixtures/phpastgen-discovery.js && node test-fixtures/phpastgen-provenance.js && node test-fixtures/phpastgen-framework-facts.js && node test-fixtures/phpastgen-legacy-regression.js && node test-fixtures/phpastgen-concurrency.js && node test-fixtures/phpastgen-regression.js && node test-fixtures/phpastgen-contract-snapshot.js && npm run test:php:pbt",
|
|
13
|
+
"test:php:concurrency": "node test-fixtures/phpastgen-concurrency.js",
|
|
14
|
+
"test:php:contract": "node test-fixtures/phpastgen-contract-snapshot.js",
|
|
15
|
+
"test:php:contract:update": "UPDATE_PHP_CONTRACT_SNAPSHOT=1 node test-fixtures/phpastgen-contract-snapshot.js",
|
|
16
|
+
"test:php:cli": "node test-fixtures/phpastgen-cli.js",
|
|
17
|
+
"test:php:pbt": "node test-fixtures/phpastgen-encoding-pbt.js && node test-fixtures/phpastgen-truncation-pbt.js && node test-fixtures/phpastgen-failure-isolation-pbt.js && node test-fixtures/phpastgen-manifest-pbt.js && node test-fixtures/phpastgen-sidecar-naming-pbt.js && node test-fixtures/phpastgen-stale-diagnostics-pbt.js && node test-fixtures/phpastgen-framework-facts-pbt.js",
|
|
12
18
|
"test:shape": "node test-fixtures/astgen-shape-snapshot.js",
|
|
13
19
|
"test:shape:update": "UPDATE_SHAPE_SNAPSHOT=1 node test-fixtures/astgen-shape-snapshot.js",
|
|
14
20
|
"test:json": "node test-fixtures/astgen-json-regression.js",
|
|
15
21
|
"test:fixtures": "node test-fixtures/test-suite.js",
|
|
16
|
-
"test:vue": "node test-fixtures/astgen-vue-regression.js"
|
|
22
|
+
"test:vue": "node test-fixtures/astgen-vue-regression.js",
|
|
23
|
+
"test:svelte": "node test-fixtures/astgen-svelte-regression.js"
|
|
17
24
|
},
|
|
18
25
|
"dependencies": {
|
|
19
26
|
"@appthreat/atom-common": "^1.1.0",
|
|
20
27
|
"@babel/parser": "^8.0.4",
|
|
21
28
|
"@typescript/typescript6": "^6.0.2",
|
|
22
|
-
"hermes-parser": "^0.37.0"
|
|
29
|
+
"hermes-parser": "^0.37.0",
|
|
30
|
+
"svelte": "5.57.0"
|
|
23
31
|
},
|
|
24
32
|
"bin": {
|
|
25
33
|
"astgen": "astgen.js",
|