prism 1.9.0 → 2.0.0.rc2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +88 -0
- data/CODE_OF_CONDUCT.md +1 -1
- data/LICENSE.md +1 -1
- data/Makefile +41 -12
- data/README.md +35 -97
- data/config.yml +174 -161
- data/docs/build_system.md +2 -2
- data/docs/configuration.md +5 -4
- data/docs/encoding.md +2 -2
- data/docs/fuzzing.md +4 -11
- data/docs/javascript.md +19 -4
- data/docs/mapping.md +1 -1
- data/docs/releasing.md +77 -5
- data/docs/relocation.md +2 -4
- data/docs/serialization.md +22 -24
- data/docs/testing.md +46 -29
- data/ext/prism/api_node.c +1461 -1401
- data/ext/prism/extconf.rb +3 -4
- data/ext/prism/extension.c +796 -422
- data/ext/prism/extension.h +3 -3
- data/include/prism/arena.h +37 -0
- data/include/prism/ast.h +2727 -610
- data/include/prism/buffer.h +52 -0
- data/include/prism/comments.h +55 -0
- data/include/prism/compiler/accel.h +26 -0
- data/include/prism/compiler/align.h +36 -0
- data/include/prism/compiler/assume.h +25 -0
- data/include/prism/compiler/bswap.h +46 -0
- data/include/prism/compiler/endianness.h +34 -0
- data/include/prism/compiler/exported.h +24 -0
- data/include/prism/compiler/fallthrough.h +22 -0
- data/include/prism/compiler/filesystem.h +32 -0
- data/include/prism/compiler/flex_array.h +19 -0
- data/include/prism/compiler/force_inline.h +21 -0
- data/include/prism/compiler/format.h +25 -0
- data/include/prism/compiler/inline.h +17 -0
- data/include/prism/compiler/nodiscard.h +22 -0
- data/include/prism/compiler/nonnull.h +18 -0
- data/include/prism/compiler/unused.h +18 -0
- data/include/prism/constant_pool.h +81 -0
- data/include/prism/diagnostic.h +38 -403
- data/include/prism/errors_format.h +53 -0
- data/include/prism/excludes.h +29 -0
- data/include/prism/integer.h +41 -0
- data/include/prism/internal/allocator.h +68 -0
- data/include/prism/internal/allocator_debug.h +88 -0
- data/include/prism/internal/arena.h +108 -0
- data/include/prism/internal/bit.h +42 -0
- data/include/prism/internal/buffer.h +94 -0
- data/include/prism/internal/char.h +145 -0
- data/include/prism/internal/comments.h +53 -0
- data/include/prism/internal/constant_pool.h +152 -0
- data/include/prism/internal/diagnostic.h +389 -0
- data/include/prism/{encoding.h → internal/encoding.h} +27 -68
- data/include/prism/internal/integer.h +68 -0
- data/include/prism/internal/isinf.h +16 -0
- data/include/prism/internal/line_offset_list.h +34 -0
- data/include/prism/{util/pm_list.h → internal/list.h} +11 -52
- data/include/prism/internal/magic_comments.h +23 -0
- data/include/prism/internal/memchr.h +15 -0
- data/include/prism/internal/node.h +38 -0
- data/include/prism/internal/options.h +232 -0
- data/include/prism/internal/parser.h +1004 -0
- data/include/prism/internal/regexp.h +26 -0
- data/include/prism/internal/serialize.h +39 -0
- data/include/prism/internal/source.h +72 -0
- data/include/prism/{static_literals.h → internal/static_literals.h} +26 -47
- data/include/prism/internal/stringy.h +35 -0
- data/include/prism/{util/pm_strncasecmp.h → internal/strncasecmp.h} +3 -17
- data/include/prism/{util/pm_strpbrk.h → internal/strpbrk.h} +7 -20
- data/include/prism/internal/tokens.h +11 -0
- data/include/prism/internal/xxh3.h +16 -0
- data/include/prism/json.h +32 -0
- data/include/prism/line_offset_list.h +61 -0
- data/include/prism/magic_comments.h +35 -0
- data/include/prism/node.h +18 -53
- data/include/prism/options.h +146 -280
- data/include/prism/parser.h +356 -867
- data/include/prism/prettyprint.h +6 -9
- data/include/prism/serialize.h +115 -0
- data/include/prism/source.h +155 -0
- data/include/prism/stream.h +28 -0
- data/include/prism/string_query.h +74 -0
- data/include/prism/stringy.h +72 -0
- data/include/prism/version.h +12 -3
- data/include/prism.h +43 -320
- data/lib/prism/compiler.rb +319 -305
- data/lib/prism/desugar_compiler.rb +105 -34
- data/lib/prism/dispatcher.rb +646 -473
- data/lib/prism/dot_visitor.rb +592 -511
- data/lib/prism/dsl.rb +391 -25
- data/lib/prism/ffi.rb +258 -90
- data/lib/prism/inspect_visitor.rb +357 -325
- data/lib/prism/lex_compat.rb +254 -212
- data/lib/prism/mutation_compiler.rb +313 -305
- data/lib/prism/node.rb +20966 -6992
- data/lib/prism/node_ext.rb +180 -187
- data/lib/prism/node_find.rb +215 -0
- data/lib/prism/parse_result/errors.rb +7 -1
- data/lib/prism/parse_result/newlines.rb +276 -8
- data/lib/prism/parse_result.rb +468 -139
- data/lib/prism/pattern.rb +65 -20
- data/lib/prism/reflection.rb +23 -12
- data/lib/prism/relocation.rb +149 -87
- data/lib/prism/serialize.rb +3460 -539
- data/lib/prism/string_query.rb +24 -1
- data/lib/prism/translation/parser/builder.rb +10 -2
- data/lib/prism/translation/parser/compiler.rb +44 -52
- data/lib/prism/translation/parser/lexer.rb +41 -92
- data/lib/prism/translation/parser.rb +1 -1
- data/lib/prism/translation/parser_current.rb +1 -1
- data/lib/prism/translation/ripper/lexer.rb +8 -10
- data/lib/prism/translation/ripper/sexp.rb +2 -10
- data/lib/prism/translation/ripper/shim.rb +2 -0
- data/lib/prism/translation/ripper.rb +967 -209
- data/lib/prism/translation/ruby_parser.rb +73 -316
- data/lib/prism/translation.rb +2 -0
- data/lib/prism/visitor.rb +632 -5
- data/lib/prism.rb +79 -9
- data/prism.gemspec +128 -62
- data/rbi/generated/prism/compiler.rbi +489 -0
- data/rbi/generated/prism/desugar_compiler.rbi +306 -0
- data/rbi/generated/prism/dispatcher.rbi +1138 -0
- data/rbi/generated/prism/dot_visitor.rbi +618 -0
- data/rbi/generated/prism/dsl.rbi +751 -0
- data/rbi/generated/prism/inspect_visitor.rbi +503 -0
- data/rbi/generated/prism/lex_compat.rbi +162 -0
- data/rbi/generated/prism/mutation_compiler.rbi +464 -0
- data/rbi/generated/prism/node.rbi +14407 -0
- data/rbi/generated/prism/node_ext.rbi +259 -0
- data/rbi/generated/prism/node_find.rbi +79 -0
- data/rbi/generated/prism/parse_result/errors.rbi +21 -0
- data/rbi/generated/prism/parse_result/newlines.rbi +208 -0
- data/rbi/generated/prism/parse_result.rbi +776 -0
- data/rbi/generated/prism/pattern.rbi +134 -0
- data/rbi/generated/prism/reflection.rbi +89 -0
- data/rbi/generated/prism/relocation.rbi +329 -0
- data/rbi/generated/prism/serialize.rbi +184 -0
- data/rbi/generated/prism/string_query.rbi +43 -0
- data/rbi/generated/prism/translation.rbi +7 -0
- data/rbi/generated/prism/visitor.rbi +650 -0
- data/rbi/generated/prism.rbi +86 -0
- data/rbi/prism/translation/parser_versions.rbi +5 -0
- data/rbi/rubyvm/node_find.rbi +14 -0
- data/sig/generated/prism/compiler.rbs +492 -0
- data/sig/generated/prism/desugar_compiler.rbs +305 -0
- data/sig/generated/prism/dispatcher.rbs +1292 -0
- data/sig/generated/prism/dot_visitor.rbs +635 -0
- data/sig/generated/prism/dsl.rbs +928 -0
- data/sig/generated/prism/inspect_visitor.rbs +509 -0
- data/sig/generated/prism/lex_compat.rbs +166 -0
- data/sig/{prism → generated/prism}/mutation_compiler.rbs +309 -4
- data/sig/generated/prism/node.rbs +20896 -0
- data/sig/generated/prism/node_ext.rbs +290 -0
- data/sig/generated/prism/node_find.rbs +90 -0
- data/sig/generated/prism/parse_result/errors.rbs +22 -0
- data/sig/generated/prism/parse_result/newlines.rbs +223 -0
- data/sig/generated/prism/parse_result.rbs +871 -0
- data/sig/generated/prism/pattern.rbs +156 -0
- data/sig/generated/prism/reflection.rbs +101 -0
- data/sig/generated/prism/relocation.rbs +428 -0
- data/sig/generated/prism/serialize.rbs +198 -0
- data/sig/generated/prism/string_query.rbs +43 -0
- data/sig/generated/prism/translation.rbs +8 -0
- data/sig/generated/prism/visitor.rbs +1111 -0
- data/sig/generated/prism.rbs +84 -0
- data/sig/manifest.yaml +2 -0
- data/src/arena.c +117 -0
- data/src/{util/pm_buffer.c → buffer.c} +53 -21
- data/src/{util/pm_char.c → char.c} +28 -57
- data/src/comments.c +454 -0
- data/src/constant_pool.c +406 -0
- data/src/diagnostic.c +97 -65
- data/src/encoding.c +8 -3
- data/src/errors_format.c +519 -0
- data/src/{util/pm_integer.c → integer.c} +33 -22
- data/src/json.c +5733 -0
- data/src/line_offset_list.c +100 -0
- data/src/list.c +24 -0
- data/src/{util/pm_memchr.c → memchr.c} +8 -6
- data/src/node.c +3504 -7674
- data/src/options.c +162 -49
- data/src/parser.c +367 -0
- data/src/prettyprint.c +421 -354
- data/src/prism.c +8402 -7161
- data/src/regexp.c +961 -45
- data/src/serialize.c +460 -424
- data/src/source.c +518 -0
- data/src/static_literals.c +111 -43
- data/src/string_query.c +180 -0
- data/src/stringy.c +91 -0
- data/src/{util/pm_strncasecmp.c → strncasecmp.c} +5 -4
- data/src/strpbrk.c +447 -0
- data/src/{token_type.c → tokens.c} +39 -18
- data/src/xxh3.c +292 -0
- metadata +128 -63
- data/ext/prism/api_pack.c +0 -276
- data/include/prism/defines.h +0 -260
- data/include/prism/pack.h +0 -163
- data/include/prism/regexp.h +0 -43
- data/include/prism/util/pm_buffer.h +0 -236
- data/include/prism/util/pm_char.h +0 -204
- data/include/prism/util/pm_constant_pool.h +0 -218
- data/include/prism/util/pm_integer.h +0 -130
- data/include/prism/util/pm_memchr.h +0 -29
- data/include/prism/util/pm_newline_list.h +0 -113
- data/include/prism/util/pm_string.h +0 -200
- data/lib/prism/pack.rb +0 -230
- data/lib/prism/parse_result/comments.rb +0 -188
- data/rbi/prism/compiler.rbi +0 -12
- data/rbi/prism/dsl.rbi +0 -524
- data/rbi/prism/inspect_visitor.rbi +0 -12
- data/rbi/prism/node.rbi +0 -8750
- data/rbi/prism/node_ext.rbi +0 -107
- data/rbi/prism/parse_result.rbi +0 -404
- data/rbi/prism/reflection.rbi +0 -58
- data/rbi/prism/string_query.rbi +0 -12
- data/rbi/prism/visitor.rbi +0 -473
- data/rbi/prism.rbi +0 -63
- data/sig/prism/compiler.rbs +0 -9
- data/sig/prism/dispatcher.rbs +0 -19
- data/sig/prism/dot_visitor.rbs +0 -6
- data/sig/prism/dsl.rbs +0 -351
- data/sig/prism/inspect_visitor.rbs +0 -22
- data/sig/prism/lex_compat.rbs +0 -10
- data/sig/prism/node.rbs +0 -4042
- data/sig/prism/node_ext.rbs +0 -149
- data/sig/prism/pack.rbs +0 -43
- data/sig/prism/parse_result/comments.rbs +0 -38
- data/sig/prism/parse_result.rbs +0 -197
- data/sig/prism/pattern.rbs +0 -13
- data/sig/prism/reflection.rbs +0 -50
- data/sig/prism/relocation.rbs +0 -185
- data/sig/prism/serialize.rbs +0 -8
- data/sig/prism/string_query.rbs +0 -11
- data/sig/prism/visitor.rbs +0 -169
- data/sig/prism.rbs +0 -272
- data/src/pack.c +0 -509
- data/src/util/pm_constant_pool.c +0 -342
- data/src/util/pm_list.c +0 -49
- data/src/util/pm_newline_list.c +0 -125
- data/src/util/pm_string.c +0 -381
- data/src/util/pm_strpbrk.c +0 -206
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e07eb7cf91add89de3aa71e9707336a2a561c977b98f5a71ab31c9f0083623e4
|
|
4
|
+
data.tar.gz: 684ec87f807ac1e17a3b251746745751707cf300b4d2f6429cdc852102261204
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 86a9128a3c56bffa1b9c6684f83e2e320fc58113f0a778ff1139d1393573ed5c42db80d7b9665b558aef9af9fbe72d84d929c997ce2ebdbfe07eafaa00864a86
|
|
7
|
+
data.tar.gz: 02c7498a1119b1106aa3a75bd07ca4d2aca13ba133e8c716693bbb33cca1d3e6004ccc6811155a4adff65a6758088066e817d7084727b80f7b6437cc280b6040
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,84 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
### Highlights
|
|
10
|
+
|
|
11
|
+
- Significant performance and memory usage improvements.
|
|
12
|
+
- Precompiled platform-specific variants on rubygems.org.
|
|
13
|
+
- Many documentation improvements, including a new playground.
|
|
14
|
+
- Prism now ships accurate and complete RBS and RBI definitions.
|
|
15
|
+
- The license is now owned by the Ruby developers.
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- Support for parsing Ruby 4.1 syntax:
|
|
20
|
+
- Add support for `&nil` to declare that a method accepts no block.
|
|
21
|
+
- Thoroughly implement void value expression check.
|
|
22
|
+
- Allow trailing commas in method signatures like `def foo(a,); end`.
|
|
23
|
+
- Reject `END { break }` and `END { next }`.
|
|
24
|
+
- `a /b/` no longer warns about ambiguous `/`.
|
|
25
|
+
- `version: "nearest"` is now supported. It behaves similarly to `version: "current"` but will not raise when the version is unsupported, instead falling back to the nearest syntax version prism currently supports.
|
|
26
|
+
- `raise_error` was added as a valid option. Prism will raise a `SyntaxError` when given syntax-invalid code. When `true` is given, prism decides the formatting (`tty?`, `NO_COLOR`, etc.). If you need a specific style, `:plain`, `:style`, and `:color` are also supported values.
|
|
27
|
+
- Added `Prism::ParseResult#continuable` to check if the parsed source can become valid by appending more code.
|
|
28
|
+
- `Prism.find` takes a `Method`, `UnboundMethod`, `Proc`, or `Thread::Backtrace::Location` and returns the associated prism AST node.
|
|
29
|
+
- `ForwardingSuperNode` now has `keyword_loc`.
|
|
30
|
+
- Allow pattern matching with prism nodes to match on location slices, like `node in Prism::StringNode[content: "foo"]`.
|
|
31
|
+
- `Prism.parse_file` and similar now accept anything responding to `to_path` (like `Pathname`) directly.
|
|
32
|
+
- `attach_comments` was added as a valid option. Each comment is attached to the node that ends latest on the comment's line before it, otherwise to the innermost node that contains it (or to that node's statements, if only whitespace and comments come between the comment and the statements), otherwise to the top-level statements. The attached comments are available through `Node#comments` and through `pm_node_comments_size` and `pm_node_comment` in the C API, as well as `pm_parser_comment_attachment`, `pm_parser_comment_ranges_size`, and `pm_parser_comment_range`, which expose the attachments grouped by node. When the option is set, the comments on the parse result are sorted by start offset.
|
|
33
|
+
- `StringQuery#keyword?` was added to check if the string query matches a keyword.
|
|
34
|
+
- `Source#checksum` was added to retrieve the checksum of the source content.
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
Note that for this major version, almost every change listed below is breaking.
|
|
39
|
+
|
|
40
|
+
- Prism now defaults to `version: "nearest"` instead of `version: "latest"`, so code is parsed with the syntax of the running Ruby (or the closest version prism supports). Pass `version: "latest"` to keep the previous behavior.
|
|
41
|
+
- `MissingNode` has been replaced by `ErrorRecoveryNode`. `ErrorRecoveryNode` wraps unexpected nodes, so when walking syntax-invalid code you will now only need to handle this one specific node type.
|
|
42
|
+
- Some fields were renamed for consistency. Previous names are aliased and will continue to work in the Ruby layer.
|
|
43
|
+
- `InNode#{in,then}_loc` -> `InNode#{in,then}_keyword_loc`
|
|
44
|
+
- `MatchPredicateNode#operator_loc` -> `MatchPredicateNode#keyword_loc`
|
|
45
|
+
- `UnlessNode#keyword_loc` -> `UnlessNode#unless_keyword_loc`
|
|
46
|
+
- `UntilNode#{keyword,closing}_loc` -> `UntilNode#{until,end}_keyword_loc`
|
|
47
|
+
- `WhenNode#keyword_loc` -> `WhenNode#when_keyword_loc`
|
|
48
|
+
- `WhileNode#{keyword,closing}_loc` -> `WhileNode#{while,end}_keyword_loc`
|
|
49
|
+
- Optional locations like `StringNode#closing_loc` in `'foo` that were empty for syntax-invalid code will now be `nil` instead.
|
|
50
|
+
- `Prism.lex` now returns an array of tokens instead of an array of `[token, internal_parse_state]`. Previous access like `token[0]` or `token.first` to extract the actual token will continue to work. Additionally the following tokens were changed:
|
|
51
|
+
- `PARENTHESIS_LEFT` was split up into `PARENTHESIS_LEFT` and `PARENTHESIS_LEFT_GROUPING`.
|
|
52
|
+
- `BRACE_LEFT` was split up into `BRACE_LEFT`, `BRACE_LEFT_ARGUMENT`, and `BRACE_LEFT_HASH`.
|
|
53
|
+
- `BACKTICK` was split up into `BACKTICK` and `XSTRING_BACKTICK`.
|
|
54
|
+
- `NEWLINE` was split up into `NEWLINE` and `NEWLINE_TERMINATOR`.
|
|
55
|
+
- `WORDS_SEP` was split up into `WORDS_SEP` and `WORDS_SEP_IMPLICIT`.
|
|
56
|
+
|
|
57
|
+
#### Ruby
|
|
58
|
+
|
|
59
|
+
- Require arguments for `Prism::Source.for`. The second and third positional parameters are no longer optional.
|
|
60
|
+
- Comments now live on the source. `Prism::Source#comments` returns them and `Result#comments` delegates to it, so `ParseResult.new`, `LexResult.new`, and `ParseLexResult.new` no longer take a `comments` argument.
|
|
61
|
+
- Comment attachment now happens during parsing with the `attach_comments` option instead of afterward with `ParseResult#attach_comments!`, which was removed. `Node#comments` now returns the comments attached to the node by that option. `Location#comments`, `Location#leading_comments`, `Location#trailing_comments`, `Node#leading_comments`, `Node#trailing_comments`, and the comment fields of `Prism::Relocation` were removed.
|
|
62
|
+
|
|
63
|
+
#### C
|
|
64
|
+
|
|
65
|
+
- Most structs are now opaque (except for nodes), and many internal functions have been hidden.
|
|
66
|
+
- Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
|
|
67
|
+
- Most allocations are now performed through an arena.
|
|
68
|
+
- You may now build side tables of comment attachments through the `attach_comments` option.
|
|
69
|
+
- `newline_list` was renamed to `list_offset_list`.
|
|
70
|
+
|
|
71
|
+
#### Rust
|
|
72
|
+
|
|
73
|
+
- Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
|
|
74
|
+
- Most allocations are now performed through an arena.
|
|
75
|
+
- `newline_list` was renamed to `list_offset_list`.
|
|
76
|
+
- `Diagnostic::message` now returns `&[u8]` instead of `&str`.
|
|
77
|
+
- `StringQuery` is now exposed.
|
|
78
|
+
|
|
79
|
+
#### JavaScript
|
|
80
|
+
|
|
81
|
+
- The parse method now only accepts `Uint8Array`, instead of forcing a transcoding to UTF-16.
|
|
82
|
+
|
|
83
|
+
### Removed
|
|
84
|
+
|
|
85
|
+
- Removed `Prism::Pack`.
|
|
86
|
+
|
|
9
87
|
## [1.9.0] - 2026-01-27
|
|
10
88
|
|
|
11
89
|
### Added
|
|
@@ -19,6 +97,15 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
|
19
97
|
- Fixed location of opening tokens when invalid syntax is parsed.
|
|
20
98
|
- Fix RBI for parsing options.
|
|
21
99
|
|
|
100
|
+
## [1.8.1] - 2026-03-16 (Ruby 4.0.2 maintenance release)
|
|
101
|
+
|
|
102
|
+
### Changed
|
|
103
|
+
|
|
104
|
+
- Fix `not` binding power in endless methods.
|
|
105
|
+
- Correctly handle `and?` and similar on Ruby 4.0.
|
|
106
|
+
- Fix error message for block/lambda with `...` argument.
|
|
107
|
+
- Fix `in` handling.
|
|
108
|
+
|
|
22
109
|
## [1.8.0] - 2026-01-12
|
|
23
110
|
|
|
24
111
|
### Added
|
|
@@ -746,6 +833,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
|
746
833
|
|
|
747
834
|
[unreleased]: https://github.com/ruby/prism/compare/v1.9.0...HEAD
|
|
748
835
|
[1.9.0]: https://github.com/ruby/prism/compare/v1.8.0...v1.9.0
|
|
836
|
+
[1.8.1]: https://github.com/ruby/prism/compare/v1.8.0...v1.8.1
|
|
749
837
|
[1.8.0]: https://github.com/ruby/prism/compare/v1.7.0...v1.8.0
|
|
750
838
|
[1.7.0]: https://github.com/ruby/prism/compare/v1.6.0...v1.7.0
|
|
751
839
|
[1.6.0]: https://github.com/ruby/prism/compare/v1.5.2...v1.6.0
|
data/CODE_OF_CONDUCT.md
CHANGED
|
@@ -55,7 +55,7 @@ further defined and clarified by project maintainers.
|
|
|
55
55
|
## Enforcement
|
|
56
56
|
|
|
57
57
|
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
|
58
|
-
reported by contacting the project team at
|
|
58
|
+
reported by contacting the project team at ruby-core@ruby-lang.org. All
|
|
59
59
|
complaints will be reviewed and investigated and will result in a response that
|
|
60
60
|
is deemed necessary and appropriate to the circumstances. The project team is
|
|
61
61
|
obligated to maintain confidentiality with regard to the reporter of an incident.
|
data/LICENSE.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
Copyright 2022-present,
|
|
1
|
+
Copyright 2022-present, Ruby developers
|
|
2
2
|
|
|
3
3
|
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
4
4
|
|
data/Makefile
CHANGED
|
@@ -12,16 +12,30 @@ SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
|
|
|
12
12
|
|
|
13
13
|
CPPFLAGS := -Iinclude $(CPPFLAGS)
|
|
14
14
|
CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(CFLAGS)
|
|
15
|
-
JAVA_WASM_CFLAGS := -
|
|
15
|
+
JAVA_WASM_CFLAGS := -g0 -O2 -Oz -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fno-sanitize=all -fno-stack-protector -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(JAVA_WASM_CFLAGS)
|
|
16
16
|
CC ?= cc
|
|
17
17
|
AR ?= ar
|
|
18
18
|
ARFLAGS ?= -r$(V0:1=v)
|
|
19
19
|
WASI_SDK_PATH := /opt/wasi-sdk
|
|
20
20
|
|
|
21
|
+
# Exporting a symbol makes it a GC root, so the JavaScript module exports only
|
|
22
|
+
# the symbols javascript/src/parsePrism.js calls. That lets --gc-sections drop
|
|
23
|
+
# the rest of the C API along with the wasi-libc code reachable from it.
|
|
24
|
+
WASM_EXPORTS := -Wl,--export=malloc,--export=free,--export=pm_buffer_new,--export=pm_buffer_value,--export=pm_buffer_length,--export=pm_buffer_free,--export=pm_serialize_parse
|
|
25
|
+
|
|
26
|
+
# Bind the library's own references to its own definitions at link time, so
|
|
27
|
+
# that the functions exported for FFI cannot be interposed by another copy of
|
|
28
|
+
# libprism loaded in the same process (for example the copy built into the
|
|
29
|
+
# interpreter next to the one built by a prism gem, when either copy was loaded
|
|
30
|
+
# with RTLD_GLOBAL). Mach-O and PE do this by default; only ELF needs the flag.
|
|
31
|
+
ifeq ($(SOEXT),so)
|
|
32
|
+
SHARED_LDFLAGS := -Wl,-Bsymbolic
|
|
33
|
+
endif
|
|
34
|
+
|
|
21
35
|
MAKEDIRS ?= mkdir -p
|
|
22
36
|
RMALL ?= rm -f -r
|
|
23
37
|
|
|
24
|
-
HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h
|
|
38
|
+
HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h)
|
|
25
39
|
SOURCES := $(wildcard src/*.c src/*/*.c)
|
|
26
40
|
SHARED_OBJECTS := $(subst src/,build/shared/,$(SOURCES:.c=.o))
|
|
27
41
|
STATIC_OBJECTS := $(subst src/,build/static/,$(SOURCES:.c=.o))
|
|
@@ -31,11 +45,12 @@ all: shared static
|
|
|
31
45
|
shared: build/libprism.$(SOEXT)
|
|
32
46
|
static: build/libprism.a
|
|
33
47
|
wasm: javascript/src/prism.wasm
|
|
34
|
-
java-wasm: java
|
|
48
|
+
java-wasm: java/wasm/src/main/wasm/prism.wasm
|
|
49
|
+
playground: doc/playground
|
|
35
50
|
|
|
36
51
|
build/libprism.$(SOEXT): $(SHARED_OBJECTS)
|
|
37
52
|
$(ECHO) "linking $@ with $(CC)"
|
|
38
|
-
$(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared -o $@ $(SHARED_OBJECTS)
|
|
53
|
+
$(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared $(SHARED_LDFLAGS) -o $@ $(SHARED_OBJECTS)
|
|
39
54
|
|
|
40
55
|
build/libprism.a: $(STATIC_OBJECTS)
|
|
41
56
|
$(ECHO) "building $@ with $(AR)"
|
|
@@ -45,15 +60,29 @@ javascript/src/prism.wasm: Makefile $(SOURCES) $(HEADERS)
|
|
|
45
60
|
$(ECHO) "building $@"
|
|
46
61
|
$(Q) $(WASI_SDK_PATH)/bin/clang --sysroot=$(WASI_SDK_PATH)/share/wasi-sysroot/ \
|
|
47
62
|
$(DEBUG_FLAGS) \
|
|
48
|
-
-DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON
|
|
63
|
+
-DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON \
|
|
49
64
|
-D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(CFLAGS) \
|
|
50
|
-
|
|
65
|
+
$(WASM_EXPORTS) -Wl,--gc-sections -Wl,--strip-all -Wl,--lto-O3 -Wl,--no-entry -mexec-model=reactor \
|
|
51
66
|
-Oz -g0 -flto -fdata-sections -ffunction-sections \
|
|
52
67
|
-o $@ $(SOURCES)
|
|
53
68
|
|
|
54
|
-
|
|
69
|
+
# The playground parses with the JavaScript package built from this checkout, so
|
|
70
|
+
# the page is served the parser the rest of the site was generated from. The
|
|
71
|
+
# JavaScript half of the package is generated by `rake templates`.
|
|
72
|
+
doc/playground: javascript/src/prism.wasm
|
|
73
|
+
$(ECHO) "building $@"
|
|
74
|
+
$(Q) $(MAKEDIRS) $@
|
|
75
|
+
$(Q) cp javascript/src/*.js javascript/src/prism.wasm javascript/package.json $@
|
|
76
|
+
|
|
77
|
+
java/wasm/src/main/wasm/prism.wasm: Makefile $(SOURCES) $(HEADERS)
|
|
55
78
|
$(ECHO) "building $@"
|
|
56
|
-
$(Q) $(
|
|
79
|
+
$(Q) $(MAKEDIRS) $(@D)
|
|
80
|
+
$(Q) $(WASI_SDK_PATH)/bin/clang \
|
|
81
|
+
$(DEBUG_FLAGS) \
|
|
82
|
+
-DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN \
|
|
83
|
+
-lwasi-emulated-mman $(CPPFLAGS) $(JAVA_WASM_CFLAGS) \
|
|
84
|
+
-Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi \
|
|
85
|
+
-o $@ $(SOURCES)
|
|
57
86
|
|
|
58
87
|
build/shared/%.o: src/%.c Makefile $(HEADERS)
|
|
59
88
|
$(ECHO) "compiling $@"
|
|
@@ -69,12 +98,12 @@ build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
|
|
|
69
98
|
$(ECHO) "building $* fuzzer"
|
|
70
99
|
$(Q) $(MAKEDIRS) $(@D)
|
|
71
100
|
$(ECHO) "building main fuzz binary"
|
|
72
|
-
$(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize
|
|
101
|
+
$(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
|
|
73
102
|
$(ECHO) "building cmplog binary"
|
|
74
|
-
$(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize
|
|
103
|
+
$(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
|
|
75
104
|
|
|
76
105
|
build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
|
|
77
|
-
$(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize
|
|
106
|
+
$(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
|
|
78
107
|
|
|
79
108
|
fuzz-debug:
|
|
80
109
|
$(ECHO) "entering debug shell"
|
|
@@ -100,7 +129,7 @@ fuzz-clean:
|
|
|
100
129
|
clean:
|
|
101
130
|
$(Q) $(RMALL) build
|
|
102
131
|
|
|
103
|
-
.PHONY: clean fuzz-clean
|
|
132
|
+
.PHONY: clean doc/playground fuzz-clean
|
|
104
133
|
|
|
105
134
|
all-no-debug: DEBUG_FLAGS := -DNDEBUG=1
|
|
106
135
|
all-no-debug: OPTFLAGS := -O3
|
data/README.md
CHANGED
|
@@ -1,10 +1,35 @@
|
|
|
1
|
-
<h1 align="center">Prism
|
|
2
|
-
<div align="center">
|
|
3
|
-
<img alt="Prism Ruby parser" height="256px" src="https://github.com/ruby/prism/blob/main/doc/images/prism.png?raw=true">
|
|
4
|
-
</div>
|
|
1
|
+
<h1 align="center">Prism</h1>
|
|
5
2
|
|
|
6
3
|
This is a parser for the Ruby programming language. It is designed to be portable, error tolerant, and maintainable. It is written in C99 and has no dependencies.
|
|
7
4
|
|
|
5
|
+
## Getting started
|
|
6
|
+
|
|
7
|
+
To compile the parser, you will need:
|
|
8
|
+
|
|
9
|
+
* C99 compiler
|
|
10
|
+
* GNU make
|
|
11
|
+
* Ruby 2.7.0 or later
|
|
12
|
+
|
|
13
|
+
Once you have these dependencies, run:
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
bundle install
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
to fetch the Ruby dependencies. Finally, run:
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
bundle exec rake compile
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
to compile the parser. It will be built in the `build` directory. To test that everything is working, run:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
bin/parse -e "1 + 2"
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
to see the syntax tree for the expression `1 + 2`.
|
|
32
|
+
|
|
8
33
|
## Overview
|
|
9
34
|
|
|
10
35
|
The repository contains the infrastructure for both a shared library (libprism) and a native CRuby extension. The shared library has no bindings to CRuby itself, and so can be used by other projects. The native CRuby extension links against `ruby.h`, and so is suitable in the context of CRuby.
|
|
@@ -12,9 +37,9 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
12
37
|
```
|
|
13
38
|
.
|
|
14
39
|
├── Makefile configuration to compile the shared library and native tests
|
|
15
|
-
├── Rakefile configuration to compile the native extension and
|
|
40
|
+
├── Rakefile configuration to compile the native extension and Ruby tests
|
|
16
41
|
├── bin
|
|
17
|
-
│ ├── lex runs the lexer on a file or string
|
|
42
|
+
│ ├── lex runs the lexer on a file or string and prints the tokens
|
|
18
43
|
│ ├── parse runs the parse on a file or string and prints the AST
|
|
19
44
|
│ └── prism a CLI for development and debugging
|
|
20
45
|
├── config.yml specification for tokens and nodes in the tree
|
|
@@ -29,8 +54,10 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
29
54
|
├── include
|
|
30
55
|
│ ├── prism header files for the shared library
|
|
31
56
|
│ └── prism.h main header file for the shared library
|
|
32
|
-
├── java
|
|
33
|
-
├──
|
|
57
|
+
├── java
|
|
58
|
+
| ├── api Java API for Prism
|
|
59
|
+
| ├── native Java native bindings for the shared library
|
|
60
|
+
| └── wasm Java WASM bindings for the shared library
|
|
34
61
|
├── javascript JavaScript WASM bindings for the shared library
|
|
35
62
|
├── lib
|
|
36
63
|
│ ├── prism Ruby library files
|
|
@@ -44,7 +71,6 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
44
71
|
│ └── prism Sample code that uses the Ruby API for documentation purposes
|
|
45
72
|
├── sig RBS type signatures for the Ruby library
|
|
46
73
|
├── src
|
|
47
|
-
│ ├── util various utility files
|
|
48
74
|
│ └── prism.c main entrypoint for the shared library
|
|
49
75
|
├── templates contains ERB templates generated by templates/template.rb
|
|
50
76
|
│ └── template.rb generates code from the nodes and tokens configured by config.yml
|
|
@@ -53,91 +79,3 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
53
79
|
├── fixtures Ruby code used for testing
|
|
54
80
|
└── snapshots snapshots of generated syntax trees corresponding to fixtures
|
|
55
81
|
```
|
|
56
|
-
|
|
57
|
-
## Getting started
|
|
58
|
-
|
|
59
|
-
To compile the shared library, you will need:
|
|
60
|
-
|
|
61
|
-
* C99 compiler
|
|
62
|
-
* GNU make
|
|
63
|
-
* Ruby 2.7.0 or later
|
|
64
|
-
|
|
65
|
-
Once you have these dependencies, run:
|
|
66
|
-
|
|
67
|
-
```
|
|
68
|
-
bundle install
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
to fetch the Ruby dependencies. Finally, run:
|
|
72
|
-
|
|
73
|
-
```
|
|
74
|
-
bundle exec rake compile
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
to compile the shared library. It will be built in the `build` directory. To test that everything is working, run:
|
|
78
|
-
|
|
79
|
-
```
|
|
80
|
-
bin/parse -e "1 + 2"
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
to see the syntax tree for the expression `1 + 2`.
|
|
84
|
-
|
|
85
|
-
## Contributing
|
|
86
|
-
|
|
87
|
-
See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additionally have documentation about the overall design of the project as well as various subtopics.
|
|
88
|
-
|
|
89
|
-
* [Build system](docs/build_system.md)
|
|
90
|
-
* [Configuration](docs/configuration.md)
|
|
91
|
-
* [CRuby compilation](docs/cruby_compilation.md)
|
|
92
|
-
* [Design](docs/design.md)
|
|
93
|
-
* [Encoding](docs/encoding.md)
|
|
94
|
-
* [Fuzzing](docs/fuzzing.md)
|
|
95
|
-
* [Heredocs](docs/heredocs.md)
|
|
96
|
-
* [JavaScript](docs/javascript.md)
|
|
97
|
-
* [Local variable depth](docs/local_variable_depth.md)
|
|
98
|
-
* [Mapping](docs/mapping.md)
|
|
99
|
-
* [Parser translation](docs/parser_translation.md)
|
|
100
|
-
* [Parsing rules](docs/parsing_rules.md)
|
|
101
|
-
* [Releasing](docs/releasing.md)
|
|
102
|
-
* [Ripper translation](docs/ripper_translation.md)
|
|
103
|
-
* [Ruby API](docs/ruby_api.md)
|
|
104
|
-
* [RubyParser translation](docs/ruby_parser_translation.md)
|
|
105
|
-
* [Serialization](docs/serialization.md)
|
|
106
|
-
* [Testing](docs/testing.md)
|
|
107
|
-
|
|
108
|
-
## Examples
|
|
109
|
-
|
|
110
|
-
Prism has been integrated into the majority of Ruby runtimes, many libraries, and some applications. Below is a list of some of the projects that use Prism:
|
|
111
|
-
|
|
112
|
-
### Runtimes
|
|
113
|
-
|
|
114
|
-
* [CRuby](https://github.com/ruby/ruby/pull/7964) (via C)
|
|
115
|
-
* [Garnet](https://github.com/camertron/garnet-js) (via WASM)
|
|
116
|
-
* [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java)
|
|
117
|
-
* [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby)
|
|
118
|
-
* [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM)
|
|
119
|
-
* [TruffleRuby](https://github.com/truffleruby/truffleruby/issues/3117) (via Java)
|
|
120
|
-
|
|
121
|
-
### Libraries
|
|
122
|
-
|
|
123
|
-
* [dispersion](https://github.com/joeldrapper/dispersion)
|
|
124
|
-
* [minifyrb](https://github.com/koic/minifyrb)
|
|
125
|
-
* [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator)
|
|
126
|
-
* [rbi](https://github.com/Shopify/rbi)
|
|
127
|
-
* [rails](https://github.com/rails/rails)
|
|
128
|
-
* [parsing renders](https://github.com/rails/rails/pull/49438)
|
|
129
|
-
* [parsing rdoc](https://github.com/rails/rails/pull/50870)
|
|
130
|
-
* [parsing tests](https://github.com/rails/rails/pull/51006)
|
|
131
|
-
* [repl_type_completor](https://github.com/ruby/repl_type_completor)
|
|
132
|
-
* [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine) (via parser translator)
|
|
133
|
-
* [ruby-lsp](https://github.com/Shopify/ruby-lsp)
|
|
134
|
-
* [smart_todo](https://github.com/Shopify/smart_todo/pull/69)
|
|
135
|
-
* [sorbet-eraser](https://github.com/kddnewton/sorbet-eraser/pull/25)
|
|
136
|
-
* [synvert](https://github.com/xinminlabs/synvert-core-ruby)
|
|
137
|
-
* [typeprof](https://github.com/ruby/typeprof)
|
|
138
|
-
* [unparser](https://github.com/mbj/unparser) (via parser translator)
|
|
139
|
-
|
|
140
|
-
### Applications
|
|
141
|
-
|
|
142
|
-
* [gem.sh](https://github.com/marcoroth/gem.sh/pull/96)
|
|
143
|
-
* [Sorbet](https://github.com/sorbet/sorbet)
|