prism 1.9.0 → 2.0.0.rc2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (244) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +88 -0
  3. data/CODE_OF_CONDUCT.md +1 -1
  4. data/LICENSE.md +1 -1
  5. data/Makefile +41 -12
  6. data/README.md +35 -97
  7. data/config.yml +174 -161
  8. data/docs/build_system.md +2 -2
  9. data/docs/configuration.md +5 -4
  10. data/docs/encoding.md +2 -2
  11. data/docs/fuzzing.md +4 -11
  12. data/docs/javascript.md +19 -4
  13. data/docs/mapping.md +1 -1
  14. data/docs/releasing.md +77 -5
  15. data/docs/relocation.md +2 -4
  16. data/docs/serialization.md +22 -24
  17. data/docs/testing.md +46 -29
  18. data/ext/prism/api_node.c +1461 -1401
  19. data/ext/prism/extconf.rb +3 -4
  20. data/ext/prism/extension.c +796 -422
  21. data/ext/prism/extension.h +3 -3
  22. data/include/prism/arena.h +37 -0
  23. data/include/prism/ast.h +2727 -610
  24. data/include/prism/buffer.h +52 -0
  25. data/include/prism/comments.h +55 -0
  26. data/include/prism/compiler/accel.h +26 -0
  27. data/include/prism/compiler/align.h +36 -0
  28. data/include/prism/compiler/assume.h +25 -0
  29. data/include/prism/compiler/bswap.h +46 -0
  30. data/include/prism/compiler/endianness.h +34 -0
  31. data/include/prism/compiler/exported.h +24 -0
  32. data/include/prism/compiler/fallthrough.h +22 -0
  33. data/include/prism/compiler/filesystem.h +32 -0
  34. data/include/prism/compiler/flex_array.h +19 -0
  35. data/include/prism/compiler/force_inline.h +21 -0
  36. data/include/prism/compiler/format.h +25 -0
  37. data/include/prism/compiler/inline.h +17 -0
  38. data/include/prism/compiler/nodiscard.h +22 -0
  39. data/include/prism/compiler/nonnull.h +18 -0
  40. data/include/prism/compiler/unused.h +18 -0
  41. data/include/prism/constant_pool.h +81 -0
  42. data/include/prism/diagnostic.h +38 -403
  43. data/include/prism/errors_format.h +53 -0
  44. data/include/prism/excludes.h +29 -0
  45. data/include/prism/integer.h +41 -0
  46. data/include/prism/internal/allocator.h +68 -0
  47. data/include/prism/internal/allocator_debug.h +88 -0
  48. data/include/prism/internal/arena.h +108 -0
  49. data/include/prism/internal/bit.h +42 -0
  50. data/include/prism/internal/buffer.h +94 -0
  51. data/include/prism/internal/char.h +145 -0
  52. data/include/prism/internal/comments.h +53 -0
  53. data/include/prism/internal/constant_pool.h +152 -0
  54. data/include/prism/internal/diagnostic.h +389 -0
  55. data/include/prism/{encoding.h → internal/encoding.h} +27 -68
  56. data/include/prism/internal/integer.h +68 -0
  57. data/include/prism/internal/isinf.h +16 -0
  58. data/include/prism/internal/line_offset_list.h +34 -0
  59. data/include/prism/{util/pm_list.h → internal/list.h} +11 -52
  60. data/include/prism/internal/magic_comments.h +23 -0
  61. data/include/prism/internal/memchr.h +15 -0
  62. data/include/prism/internal/node.h +38 -0
  63. data/include/prism/internal/options.h +232 -0
  64. data/include/prism/internal/parser.h +1004 -0
  65. data/include/prism/internal/regexp.h +26 -0
  66. data/include/prism/internal/serialize.h +39 -0
  67. data/include/prism/internal/source.h +72 -0
  68. data/include/prism/{static_literals.h → internal/static_literals.h} +26 -47
  69. data/include/prism/internal/stringy.h +35 -0
  70. data/include/prism/{util/pm_strncasecmp.h → internal/strncasecmp.h} +3 -17
  71. data/include/prism/{util/pm_strpbrk.h → internal/strpbrk.h} +7 -20
  72. data/include/prism/internal/tokens.h +11 -0
  73. data/include/prism/internal/xxh3.h +16 -0
  74. data/include/prism/json.h +32 -0
  75. data/include/prism/line_offset_list.h +61 -0
  76. data/include/prism/magic_comments.h +35 -0
  77. data/include/prism/node.h +18 -53
  78. data/include/prism/options.h +146 -280
  79. data/include/prism/parser.h +356 -867
  80. data/include/prism/prettyprint.h +6 -9
  81. data/include/prism/serialize.h +115 -0
  82. data/include/prism/source.h +155 -0
  83. data/include/prism/stream.h +28 -0
  84. data/include/prism/string_query.h +74 -0
  85. data/include/prism/stringy.h +72 -0
  86. data/include/prism/version.h +12 -3
  87. data/include/prism.h +43 -320
  88. data/lib/prism/compiler.rb +319 -305
  89. data/lib/prism/desugar_compiler.rb +105 -34
  90. data/lib/prism/dispatcher.rb +646 -473
  91. data/lib/prism/dot_visitor.rb +592 -511
  92. data/lib/prism/dsl.rb +391 -25
  93. data/lib/prism/ffi.rb +258 -90
  94. data/lib/prism/inspect_visitor.rb +357 -325
  95. data/lib/prism/lex_compat.rb +254 -212
  96. data/lib/prism/mutation_compiler.rb +313 -305
  97. data/lib/prism/node.rb +20966 -6992
  98. data/lib/prism/node_ext.rb +180 -187
  99. data/lib/prism/node_find.rb +215 -0
  100. data/lib/prism/parse_result/errors.rb +7 -1
  101. data/lib/prism/parse_result/newlines.rb +276 -8
  102. data/lib/prism/parse_result.rb +468 -139
  103. data/lib/prism/pattern.rb +65 -20
  104. data/lib/prism/reflection.rb +23 -12
  105. data/lib/prism/relocation.rb +149 -87
  106. data/lib/prism/serialize.rb +3460 -539
  107. data/lib/prism/string_query.rb +24 -1
  108. data/lib/prism/translation/parser/builder.rb +10 -2
  109. data/lib/prism/translation/parser/compiler.rb +44 -52
  110. data/lib/prism/translation/parser/lexer.rb +41 -92
  111. data/lib/prism/translation/parser.rb +1 -1
  112. data/lib/prism/translation/parser_current.rb +1 -1
  113. data/lib/prism/translation/ripper/lexer.rb +8 -10
  114. data/lib/prism/translation/ripper/sexp.rb +2 -10
  115. data/lib/prism/translation/ripper/shim.rb +2 -0
  116. data/lib/prism/translation/ripper.rb +967 -209
  117. data/lib/prism/translation/ruby_parser.rb +73 -316
  118. data/lib/prism/translation.rb +2 -0
  119. data/lib/prism/visitor.rb +632 -5
  120. data/lib/prism.rb +79 -9
  121. data/prism.gemspec +128 -62
  122. data/rbi/generated/prism/compiler.rbi +489 -0
  123. data/rbi/generated/prism/desugar_compiler.rbi +306 -0
  124. data/rbi/generated/prism/dispatcher.rbi +1138 -0
  125. data/rbi/generated/prism/dot_visitor.rbi +618 -0
  126. data/rbi/generated/prism/dsl.rbi +751 -0
  127. data/rbi/generated/prism/inspect_visitor.rbi +503 -0
  128. data/rbi/generated/prism/lex_compat.rbi +162 -0
  129. data/rbi/generated/prism/mutation_compiler.rbi +464 -0
  130. data/rbi/generated/prism/node.rbi +14407 -0
  131. data/rbi/generated/prism/node_ext.rbi +259 -0
  132. data/rbi/generated/prism/node_find.rbi +79 -0
  133. data/rbi/generated/prism/parse_result/errors.rbi +21 -0
  134. data/rbi/generated/prism/parse_result/newlines.rbi +208 -0
  135. data/rbi/generated/prism/parse_result.rbi +776 -0
  136. data/rbi/generated/prism/pattern.rbi +134 -0
  137. data/rbi/generated/prism/reflection.rbi +89 -0
  138. data/rbi/generated/prism/relocation.rbi +329 -0
  139. data/rbi/generated/prism/serialize.rbi +184 -0
  140. data/rbi/generated/prism/string_query.rbi +43 -0
  141. data/rbi/generated/prism/translation.rbi +7 -0
  142. data/rbi/generated/prism/visitor.rbi +650 -0
  143. data/rbi/generated/prism.rbi +86 -0
  144. data/rbi/prism/translation/parser_versions.rbi +5 -0
  145. data/rbi/rubyvm/node_find.rbi +14 -0
  146. data/sig/generated/prism/compiler.rbs +492 -0
  147. data/sig/generated/prism/desugar_compiler.rbs +305 -0
  148. data/sig/generated/prism/dispatcher.rbs +1292 -0
  149. data/sig/generated/prism/dot_visitor.rbs +635 -0
  150. data/sig/generated/prism/dsl.rbs +928 -0
  151. data/sig/generated/prism/inspect_visitor.rbs +509 -0
  152. data/sig/generated/prism/lex_compat.rbs +166 -0
  153. data/sig/{prism → generated/prism}/mutation_compiler.rbs +309 -4
  154. data/sig/generated/prism/node.rbs +20896 -0
  155. data/sig/generated/prism/node_ext.rbs +290 -0
  156. data/sig/generated/prism/node_find.rbs +90 -0
  157. data/sig/generated/prism/parse_result/errors.rbs +22 -0
  158. data/sig/generated/prism/parse_result/newlines.rbs +223 -0
  159. data/sig/generated/prism/parse_result.rbs +871 -0
  160. data/sig/generated/prism/pattern.rbs +156 -0
  161. data/sig/generated/prism/reflection.rbs +101 -0
  162. data/sig/generated/prism/relocation.rbs +428 -0
  163. data/sig/generated/prism/serialize.rbs +198 -0
  164. data/sig/generated/prism/string_query.rbs +43 -0
  165. data/sig/generated/prism/translation.rbs +8 -0
  166. data/sig/generated/prism/visitor.rbs +1111 -0
  167. data/sig/generated/prism.rbs +84 -0
  168. data/sig/manifest.yaml +2 -0
  169. data/src/arena.c +117 -0
  170. data/src/{util/pm_buffer.c → buffer.c} +53 -21
  171. data/src/{util/pm_char.c → char.c} +28 -57
  172. data/src/comments.c +454 -0
  173. data/src/constant_pool.c +406 -0
  174. data/src/diagnostic.c +97 -65
  175. data/src/encoding.c +8 -3
  176. data/src/errors_format.c +519 -0
  177. data/src/{util/pm_integer.c → integer.c} +33 -22
  178. data/src/json.c +5733 -0
  179. data/src/line_offset_list.c +100 -0
  180. data/src/list.c +24 -0
  181. data/src/{util/pm_memchr.c → memchr.c} +8 -6
  182. data/src/node.c +3504 -7674
  183. data/src/options.c +162 -49
  184. data/src/parser.c +367 -0
  185. data/src/prettyprint.c +421 -354
  186. data/src/prism.c +8402 -7161
  187. data/src/regexp.c +961 -45
  188. data/src/serialize.c +460 -424
  189. data/src/source.c +518 -0
  190. data/src/static_literals.c +111 -43
  191. data/src/string_query.c +180 -0
  192. data/src/stringy.c +91 -0
  193. data/src/{util/pm_strncasecmp.c → strncasecmp.c} +5 -4
  194. data/src/strpbrk.c +447 -0
  195. data/src/{token_type.c → tokens.c} +39 -18
  196. data/src/xxh3.c +292 -0
  197. metadata +128 -63
  198. data/ext/prism/api_pack.c +0 -276
  199. data/include/prism/defines.h +0 -260
  200. data/include/prism/pack.h +0 -163
  201. data/include/prism/regexp.h +0 -43
  202. data/include/prism/util/pm_buffer.h +0 -236
  203. data/include/prism/util/pm_char.h +0 -204
  204. data/include/prism/util/pm_constant_pool.h +0 -218
  205. data/include/prism/util/pm_integer.h +0 -130
  206. data/include/prism/util/pm_memchr.h +0 -29
  207. data/include/prism/util/pm_newline_list.h +0 -113
  208. data/include/prism/util/pm_string.h +0 -200
  209. data/lib/prism/pack.rb +0 -230
  210. data/lib/prism/parse_result/comments.rb +0 -188
  211. data/rbi/prism/compiler.rbi +0 -12
  212. data/rbi/prism/dsl.rbi +0 -524
  213. data/rbi/prism/inspect_visitor.rbi +0 -12
  214. data/rbi/prism/node.rbi +0 -8750
  215. data/rbi/prism/node_ext.rbi +0 -107
  216. data/rbi/prism/parse_result.rbi +0 -404
  217. data/rbi/prism/reflection.rbi +0 -58
  218. data/rbi/prism/string_query.rbi +0 -12
  219. data/rbi/prism/visitor.rbi +0 -473
  220. data/rbi/prism.rbi +0 -63
  221. data/sig/prism/compiler.rbs +0 -9
  222. data/sig/prism/dispatcher.rbs +0 -19
  223. data/sig/prism/dot_visitor.rbs +0 -6
  224. data/sig/prism/dsl.rbs +0 -351
  225. data/sig/prism/inspect_visitor.rbs +0 -22
  226. data/sig/prism/lex_compat.rbs +0 -10
  227. data/sig/prism/node.rbs +0 -4042
  228. data/sig/prism/node_ext.rbs +0 -149
  229. data/sig/prism/pack.rbs +0 -43
  230. data/sig/prism/parse_result/comments.rbs +0 -38
  231. data/sig/prism/parse_result.rbs +0 -197
  232. data/sig/prism/pattern.rbs +0 -13
  233. data/sig/prism/reflection.rbs +0 -50
  234. data/sig/prism/relocation.rbs +0 -185
  235. data/sig/prism/serialize.rbs +0 -8
  236. data/sig/prism/string_query.rbs +0 -11
  237. data/sig/prism/visitor.rbs +0 -169
  238. data/sig/prism.rbs +0 -272
  239. data/src/pack.c +0 -509
  240. data/src/util/pm_constant_pool.c +0 -342
  241. data/src/util/pm_list.c +0 -49
  242. data/src/util/pm_newline_list.c +0 -125
  243. data/src/util/pm_string.c +0 -381
  244. data/src/util/pm_strpbrk.c +0 -206
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1f7205a73da36d10903c1495fd475e014a30c875568c932568cac22351eb3059
4
- data.tar.gz: d59eac5f6e8e8955ea9b259b63eec423a0af06c79a748edaa9f64f24463dc874
3
+ metadata.gz: e07eb7cf91add89de3aa71e9707336a2a561c977b98f5a71ab31c9f0083623e4
4
+ data.tar.gz: 684ec87f807ac1e17a3b251746745751707cf300b4d2f6429cdc852102261204
5
5
  SHA512:
6
- metadata.gz: 59571b78dba19ec01a6f52094cc6d30e5648e633cd24f3abfbbc765145bed778f88fea75eada071d780ecb3dbe77a1bebab563d676289fed0ea6d21e50056290
7
- data.tar.gz: 3c5b40536f98839fefb0afcefd49d5e78a3a31f9a55bd33b78b80665b2ca3bd1c2262c1836f9c6f1d3ddc7ad2d9b7dd12f06b3dff2b4aea9475a7171340d95b4
6
+ metadata.gz: 86a9128a3c56bffa1b9c6684f83e2e320fc58113f0a778ff1139d1393573ed5c42db80d7b9665b558aef9af9fbe72d84d929c997ce2ebdbfe07eafaa00864a86
7
+ data.tar.gz: 02c7498a1119b1106aa3a75bd07ca4d2aca13ba133e8c716693bbb33cca1d3e6004ccc6811155a4adff65a6758088066e817d7084727b80f7b6437cc280b6040
data/CHANGELOG.md CHANGED
@@ -6,6 +6,84 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ### Highlights
10
+
11
+ - Significant performance and memory usage improvements.
12
+ - Precompiled platform-specific variants on rubygems.org.
13
+ - Many documentation improvements, including a new playground.
14
+ - Prism now ships accurate and complete RBS and RBI definitions.
15
+ - The license is now owned by the Ruby developers.
16
+
17
+ ### Added
18
+
19
+ - Support for parsing Ruby 4.1 syntax:
20
+ - Add support for `&nil` to declare that a method accepts no block.
21
+ - Thoroughly implement void value expression check.
22
+ - Allow trailing commas in method signatures like `def foo(a,); end`.
23
+ - Reject `END { break }` and `END { next }`.
24
+ - `a /b/` no longer warns about ambiguous `/`.
25
+ - `version: "nearest"` is now supported. It behaves similarly to `version: "current"` but will not raise when the version is unsupported, instead falling back to the nearest syntax version prism currently supports.
26
+ - `raise_error` was added as a valid option. Prism will raise a `SyntaxError` when given syntax-invalid code. When `true` is given, prism decides the formatting (`tty?`, `NO_COLOR`, etc.). If you need a specific style, `:plain`, `:style`, and `:color` are also supported values.
27
+ - Added `Prism::ParseResult#continuable` to check if the parsed source can become valid by appending more code.
28
+ - `Prism.find` takes a `Method`, `UnboundMethod`, `Proc`, or `Thread::Backtrace::Location` and returns the associated prism AST node.
29
+ - `ForwardingSuperNode` now has `keyword_loc`.
30
+ - Allow pattern matching with prism nodes to match on location slices, like `node in Prism::StringNode[content: "foo"]`.
31
+ - `Prism.parse_file` and similar now accept anything responding to `to_path` (like `Pathname`) directly.
32
+ - `attach_comments` was added as a valid option. Each comment is attached to the node that ends latest on the comment's line before it, otherwise to the innermost node that contains it (or to that node's statements, if only whitespace and comments come between the comment and the statements), otherwise to the top-level statements. The attached comments are available through `Node#comments` and through `pm_node_comments_size` and `pm_node_comment` in the C API, as well as `pm_parser_comment_attachment`, `pm_parser_comment_ranges_size`, and `pm_parser_comment_range`, which expose the attachments grouped by node. When the option is set, the comments on the parse result are sorted by start offset.
33
+ - `StringQuery#keyword?` was added to check if the string query matches a keyword.
34
+ - `Source#checksum` was added to retrieve the checksum of the source content.
35
+
36
+ ### Changed
37
+
38
+ Note that for this major version, almost every change listed below is breaking.
39
+
40
+ - Prism now defaults to `version: "nearest"` instead of `version: "latest"`, so code is parsed with the syntax of the running Ruby (or the closest version prism supports). Pass `version: "latest"` to keep the previous behavior.
41
+ - `MissingNode` has been replaced by `ErrorRecoveryNode`. `ErrorRecoveryNode` wraps unexpected nodes, so when walking syntax-invalid code you will now only need to handle this one specific node type.
42
+ - Some fields were renamed for consistency. Previous names are aliased and will continue to work in the Ruby layer.
43
+ - `InNode#{in,then}_loc` -> `InNode#{in,then}_keyword_loc`
44
+ - `MatchPredicateNode#operator_loc` -> `MatchPredicateNode#keyword_loc`
45
+ - `UnlessNode#keyword_loc` -> `UnlessNode#unless_keyword_loc`
46
+ - `UntilNode#{keyword,closing}_loc` -> `UntilNode#{until,end}_keyword_loc`
47
+ - `WhenNode#keyword_loc` -> `WhenNode#when_keyword_loc`
48
+ - `WhileNode#{keyword,closing}_loc` -> `WhileNode#{while,end}_keyword_loc`
49
+ - Optional locations like `StringNode#closing_loc` in `'foo` that were empty for syntax-invalid code will now be `nil` instead.
50
+ - `Prism.lex` now returns an array of tokens instead of an array of `[token, internal_parse_state]`. Previous access like `token[0]` or `token.first` to extract the actual token will continue to work. Additionally the following tokens were changed:
51
+ - `PARENTHESIS_LEFT` was split up into `PARENTHESIS_LEFT` and `PARENTHESIS_LEFT_GROUPING`.
52
+ - `BRACE_LEFT` was split up into `BRACE_LEFT`, `BRACE_LEFT_ARGUMENT`, and `BRACE_LEFT_HASH`.
53
+ - `BACKTICK` was split up into `BACKTICK` and `XSTRING_BACKTICK`.
54
+ - `NEWLINE` was split up into `NEWLINE` and `NEWLINE_TERMINATOR`.
55
+ - `WORDS_SEP` was split up into `WORDS_SEP` and `WORDS_SEP_IMPLICIT`.
56
+
57
+ #### Ruby
58
+
59
+ - Require arguments for `Prism::Source.for`. The second and third positional parameters are no longer optional.
60
+ - Comments now live on the source. `Prism::Source#comments` returns them and `Result#comments` delegates to it, so `ParseResult.new`, `LexResult.new`, and `ParseLexResult.new` no longer take a `comments` argument.
61
+ - Comment attachment now happens during parsing with the `attach_comments` option instead of afterward with `ParseResult#attach_comments!`, which was removed. `Node#comments` now returns the comments attached to the node by that option. `Location#comments`, `Location#leading_comments`, `Location#trailing_comments`, `Node#leading_comments`, `Node#trailing_comments`, and the comment fields of `Prism::Relocation` were removed.
62
+
63
+ #### C
64
+
65
+ - Most structs are now opaque (except for nodes), and many internal functions have been hidden.
66
+ - Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
67
+ - Most allocations are now performed through an arena.
68
+ - You may now build side tables of comment attachments through the `attach_comments` option.
69
+ - `newline_list` was renamed to `list_offset_list`.
70
+
71
+ #### Rust
72
+
73
+ - Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
74
+ - Most allocations are now performed through an arena.
75
+ - `newline_list` was renamed to `list_offset_list`.
76
+ - `Diagnostic::message` now returns `&[u8]` instead of `&str`.
77
+ - `StringQuery` is now exposed.
78
+
79
+ #### JavaScript
80
+
81
+ - The parse method now only accepts `Uint8Array`, instead of forcing a transcoding to UTF-16.
82
+
83
+ ### Removed
84
+
85
+ - Removed `Prism::Pack`.
86
+
9
87
  ## [1.9.0] - 2026-01-27
10
88
 
11
89
  ### Added
@@ -19,6 +97,15 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
19
97
  - Fixed location of opening tokens when invalid syntax is parsed.
20
98
  - Fix RBI for parsing options.
21
99
 
100
+ ## [1.8.1] - 2026-03-16 (Ruby 4.0.2 maintenance release)
101
+
102
+ ### Changed
103
+
104
+ - Fix `not` binding power in endless methods.
105
+ - Correctly handle `and?` and similar on Ruby 4.0.
106
+ - Fix error message for block/lambda with `...` argument.
107
+ - Fix `in` handling.
108
+
22
109
  ## [1.8.0] - 2026-01-12
23
110
 
24
111
  ### Added
@@ -746,6 +833,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
746
833
 
747
834
  [unreleased]: https://github.com/ruby/prism/compare/v1.9.0...HEAD
748
835
  [1.9.0]: https://github.com/ruby/prism/compare/v1.8.0...v1.9.0
836
+ [1.8.1]: https://github.com/ruby/prism/compare/v1.8.0...v1.8.1
749
837
  [1.8.0]: https://github.com/ruby/prism/compare/v1.7.0...v1.8.0
750
838
  [1.7.0]: https://github.com/ruby/prism/compare/v1.6.0...v1.7.0
751
839
  [1.6.0]: https://github.com/ruby/prism/compare/v1.5.2...v1.6.0
data/CODE_OF_CONDUCT.md CHANGED
@@ -55,7 +55,7 @@ further defined and clarified by project maintainers.
55
55
  ## Enforcement
56
56
 
57
57
  Instances of abusive, harassing, or otherwise unacceptable behavior may be
58
- reported by contacting the project team at opensource@shopify.com. All
58
+ reported by contacting the project team at ruby-core@ruby-lang.org. All
59
59
  complaints will be reviewed and investigated and will result in a response that
60
60
  is deemed necessary and appropriate to the circumstances. The project team is
61
61
  obligated to maintain confidentiality with regard to the reporter of an incident.
data/LICENSE.md CHANGED
@@ -1,4 +1,4 @@
1
- Copyright 2022-present, Shopify Inc.
1
+ Copyright 2022-present, Ruby developers
2
2
 
3
3
  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
4
4
 
data/Makefile CHANGED
@@ -12,16 +12,30 @@ SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
12
12
 
13
13
  CPPFLAGS := -Iinclude $(CPPFLAGS)
14
14
  CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(CFLAGS)
15
- JAVA_WASM_CFLAGS := -g -Oz -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(JAVA_WASM_CFLAGS)
15
+ JAVA_WASM_CFLAGS := -g0 -O2 -Oz -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fno-sanitize=all -fno-stack-protector -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(JAVA_WASM_CFLAGS)
16
16
  CC ?= cc
17
17
  AR ?= ar
18
18
  ARFLAGS ?= -r$(V0:1=v)
19
19
  WASI_SDK_PATH := /opt/wasi-sdk
20
20
 
21
+ # Exporting a symbol makes it a GC root, so the JavaScript module exports only
22
+ # the symbols javascript/src/parsePrism.js calls. That lets --gc-sections drop
23
+ # the rest of the C API along with the wasi-libc code reachable from it.
24
+ WASM_EXPORTS := -Wl,--export=malloc,--export=free,--export=pm_buffer_new,--export=pm_buffer_value,--export=pm_buffer_length,--export=pm_buffer_free,--export=pm_serialize_parse
25
+
26
+ # Bind the library's own references to its own definitions at link time, so
27
+ # that the functions exported for FFI cannot be interposed by another copy of
28
+ # libprism loaded in the same process (for example the copy built into the
29
+ # interpreter next to the one built by a prism gem, when either copy was loaded
30
+ # with RTLD_GLOBAL). Mach-O and PE do this by default; only ELF needs the flag.
31
+ ifeq ($(SOEXT),so)
32
+ SHARED_LDFLAGS := -Wl,-Bsymbolic
33
+ endif
34
+
21
35
  MAKEDIRS ?= mkdir -p
22
36
  RMALL ?= rm -f -r
23
37
 
24
- HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h')
38
+ HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h)
25
39
  SOURCES := $(wildcard src/*.c src/*/*.c)
26
40
  SHARED_OBJECTS := $(subst src/,build/shared/,$(SOURCES:.c=.o))
27
41
  STATIC_OBJECTS := $(subst src/,build/static/,$(SOURCES:.c=.o))
@@ -31,11 +45,12 @@ all: shared static
31
45
  shared: build/libprism.$(SOEXT)
32
46
  static: build/libprism.a
33
47
  wasm: javascript/src/prism.wasm
34
- java-wasm: java-wasm/src/test/resources/prism.wasm
48
+ java-wasm: java/wasm/src/main/wasm/prism.wasm
49
+ playground: doc/playground
35
50
 
36
51
  build/libprism.$(SOEXT): $(SHARED_OBJECTS)
37
52
  $(ECHO) "linking $@ with $(CC)"
38
- $(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared -o $@ $(SHARED_OBJECTS)
53
+ $(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared $(SHARED_LDFLAGS) -o $@ $(SHARED_OBJECTS)
39
54
 
40
55
  build/libprism.a: $(STATIC_OBJECTS)
41
56
  $(ECHO) "building $@ with $(AR)"
@@ -45,15 +60,29 @@ javascript/src/prism.wasm: Makefile $(SOURCES) $(HEADERS)
45
60
  $(ECHO) "building $@"
46
61
  $(Q) $(WASI_SDK_PATH)/bin/clang --sysroot=$(WASI_SDK_PATH)/share/wasi-sysroot/ \
47
62
  $(DEBUG_FLAGS) \
48
- -DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON -DPRISM_EXCLUDE_PACK \
63
+ -DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON \
49
64
  -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(CFLAGS) \
50
- -Wl,--export-all -Wl,--gc-sections -Wl,--strip-all -Wl,--lto-O3 -Wl,--no-entry -mexec-model=reactor \
65
+ $(WASM_EXPORTS) -Wl,--gc-sections -Wl,--strip-all -Wl,--lto-O3 -Wl,--no-entry -mexec-model=reactor \
51
66
  -Oz -g0 -flto -fdata-sections -ffunction-sections \
52
67
  -o $@ $(SOURCES)
53
68
 
54
- java-wasm/src/test/resources/prism.wasm: Makefile $(SOURCES) $(HEADERS)
69
+ # The playground parses with the JavaScript package built from this checkout, so
70
+ # the page is served the parser the rest of the site was generated from. The
71
+ # JavaScript half of the package is generated by `rake templates`.
72
+ doc/playground: javascript/src/prism.wasm
73
+ $(ECHO) "building $@"
74
+ $(Q) $(MAKEDIRS) $@
75
+ $(Q) cp javascript/src/*.js javascript/src/prism.wasm javascript/package.json $@
76
+
77
+ java/wasm/src/main/wasm/prism.wasm: Makefile $(SOURCES) $(HEADERS)
55
78
  $(ECHO) "building $@"
56
- $(Q) $(WASI_SDK_PATH)/bin/clang $(DEBUG_FLAGS) -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(JAVA_WASM_CFLAGS) -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi -o $@ $(SOURCES)
79
+ $(Q) $(MAKEDIRS) $(@D)
80
+ $(Q) $(WASI_SDK_PATH)/bin/clang \
81
+ $(DEBUG_FLAGS) \
82
+ -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN \
83
+ -lwasi-emulated-mman $(CPPFLAGS) $(JAVA_WASM_CFLAGS) \
84
+ -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi \
85
+ -o $@ $(SOURCES)
57
86
 
58
87
  build/shared/%.o: src/%.c Makefile $(HEADERS)
59
88
  $(ECHO) "compiling $@"
@@ -69,12 +98,12 @@ build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
69
98
  $(ECHO) "building $* fuzzer"
70
99
  $(Q) $(MAKEDIRS) $(@D)
71
100
  $(ECHO) "building main fuzz binary"
72
- $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
101
+ $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
73
102
  $(ECHO) "building cmplog binary"
74
- $(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
103
+ $(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
75
104
 
76
105
  build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
77
- $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
106
+ $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
78
107
 
79
108
  fuzz-debug:
80
109
  $(ECHO) "entering debug shell"
@@ -100,7 +129,7 @@ fuzz-clean:
100
129
  clean:
101
130
  $(Q) $(RMALL) build
102
131
 
103
- .PHONY: clean fuzz-clean
132
+ .PHONY: clean doc/playground fuzz-clean
104
133
 
105
134
  all-no-debug: DEBUG_FLAGS := -DNDEBUG=1
106
135
  all-no-debug: OPTFLAGS := -O3
data/README.md CHANGED
@@ -1,10 +1,35 @@
1
- <h1 align="center">Prism Ruby parser</h1>
2
- <div align="center">
3
- <img alt="Prism Ruby parser" height="256px" src="https://github.com/ruby/prism/blob/main/doc/images/prism.png?raw=true">
4
- </div>
1
+ <h1 align="center">Prism</h1>
5
2
 
6
3
  This is a parser for the Ruby programming language. It is designed to be portable, error tolerant, and maintainable. It is written in C99 and has no dependencies.
7
4
 
5
+ ## Getting started
6
+
7
+ To compile the parser, you will need:
8
+
9
+ * C99 compiler
10
+ * GNU make
11
+ * Ruby 2.7.0 or later
12
+
13
+ Once you have these dependencies, run:
14
+
15
+ ```
16
+ bundle install
17
+ ```
18
+
19
+ to fetch the Ruby dependencies. Finally, run:
20
+
21
+ ```
22
+ bundle exec rake compile
23
+ ```
24
+
25
+ to compile the parser. It will be built in the `build` directory. To test that everything is working, run:
26
+
27
+ ```
28
+ bin/parse -e "1 + 2"
29
+ ```
30
+
31
+ to see the syntax tree for the expression `1 + 2`.
32
+
8
33
  ## Overview
9
34
 
10
35
  The repository contains the infrastructure for both a shared library (libprism) and a native CRuby extension. The shared library has no bindings to CRuby itself, and so can be used by other projects. The native CRuby extension links against `ruby.h`, and so is suitable in the context of CRuby.
@@ -12,9 +37,9 @@ The repository contains the infrastructure for both a shared library (libprism)
12
37
  ```
13
38
  .
14
39
  ├── Makefile configuration to compile the shared library and native tests
15
- ├── Rakefile configuration to compile the native extension and run the Ruby tests
40
+ ├── Rakefile configuration to compile the native extension and Ruby tests
16
41
  ├── bin
17
- │   ├── lex runs the lexer on a file or string, prints the tokens, and compares to ripper
42
+ │   ├── lex runs the lexer on a file or string and prints the tokens
18
43
  │   ├── parse runs the parse on a file or string and prints the AST
19
44
  │   └── prism a CLI for development and debugging
20
45
  ├── config.yml specification for tokens and nodes in the tree
@@ -29,8 +54,10 @@ The repository contains the infrastructure for both a shared library (libprism)
29
54
  ├── include
30
55
  │   ├── prism header files for the shared library
31
56
  │   └── prism.h main header file for the shared library
32
- ├── java Java bindings for the shared library
33
- ├── java-wasm Java WASM bindings for the shared library
57
+ ├── java
58
+ | ├── api Java API for Prism
59
+ | ├── native Java native bindings for the shared library
60
+ | └── wasm Java WASM bindings for the shared library
34
61
  ├── javascript JavaScript WASM bindings for the shared library
35
62
  ├── lib
36
63
  │   ├── prism Ruby library files
@@ -44,7 +71,6 @@ The repository contains the infrastructure for both a shared library (libprism)
44
71
  │ └── prism Sample code that uses the Ruby API for documentation purposes
45
72
  ├── sig RBS type signatures for the Ruby library
46
73
  ├── src
47
- │   ├── util various utility files
48
74
  │   └── prism.c main entrypoint for the shared library
49
75
  ├── templates contains ERB templates generated by templates/template.rb
50
76
  │   └── template.rb generates code from the nodes and tokens configured by config.yml
@@ -53,91 +79,3 @@ The repository contains the infrastructure for both a shared library (libprism)
53
79
  ├── fixtures Ruby code used for testing
54
80
  └── snapshots snapshots of generated syntax trees corresponding to fixtures
55
81
  ```
56
-
57
- ## Getting started
58
-
59
- To compile the shared library, you will need:
60
-
61
- * C99 compiler
62
- * GNU make
63
- * Ruby 2.7.0 or later
64
-
65
- Once you have these dependencies, run:
66
-
67
- ```
68
- bundle install
69
- ```
70
-
71
- to fetch the Ruby dependencies. Finally, run:
72
-
73
- ```
74
- bundle exec rake compile
75
- ```
76
-
77
- to compile the shared library. It will be built in the `build` directory. To test that everything is working, run:
78
-
79
- ```
80
- bin/parse -e "1 + 2"
81
- ```
82
-
83
- to see the syntax tree for the expression `1 + 2`.
84
-
85
- ## Contributing
86
-
87
- See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additionally have documentation about the overall design of the project as well as various subtopics.
88
-
89
- * [Build system](docs/build_system.md)
90
- * [Configuration](docs/configuration.md)
91
- * [CRuby compilation](docs/cruby_compilation.md)
92
- * [Design](docs/design.md)
93
- * [Encoding](docs/encoding.md)
94
- * [Fuzzing](docs/fuzzing.md)
95
- * [Heredocs](docs/heredocs.md)
96
- * [JavaScript](docs/javascript.md)
97
- * [Local variable depth](docs/local_variable_depth.md)
98
- * [Mapping](docs/mapping.md)
99
- * [Parser translation](docs/parser_translation.md)
100
- * [Parsing rules](docs/parsing_rules.md)
101
- * [Releasing](docs/releasing.md)
102
- * [Ripper translation](docs/ripper_translation.md)
103
- * [Ruby API](docs/ruby_api.md)
104
- * [RubyParser translation](docs/ruby_parser_translation.md)
105
- * [Serialization](docs/serialization.md)
106
- * [Testing](docs/testing.md)
107
-
108
- ## Examples
109
-
110
- Prism has been integrated into the majority of Ruby runtimes, many libraries, and some applications. Below is a list of some of the projects that use Prism:
111
-
112
- ### Runtimes
113
-
114
- * [CRuby](https://github.com/ruby/ruby/pull/7964) (via C)
115
- * [Garnet](https://github.com/camertron/garnet-js) (via WASM)
116
- * [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java)
117
- * [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby)
118
- * [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM)
119
- * [TruffleRuby](https://github.com/truffleruby/truffleruby/issues/3117) (via Java)
120
-
121
- ### Libraries
122
-
123
- * [dispersion](https://github.com/joeldrapper/dispersion)
124
- * [minifyrb](https://github.com/koic/minifyrb)
125
- * [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator)
126
- * [rbi](https://github.com/Shopify/rbi)
127
- * [rails](https://github.com/rails/rails)
128
- * [parsing renders](https://github.com/rails/rails/pull/49438)
129
- * [parsing rdoc](https://github.com/rails/rails/pull/50870)
130
- * [parsing tests](https://github.com/rails/rails/pull/51006)
131
- * [repl_type_completor](https://github.com/ruby/repl_type_completor)
132
- * [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine) (via parser translator)
133
- * [ruby-lsp](https://github.com/Shopify/ruby-lsp)
134
- * [smart_todo](https://github.com/Shopify/smart_todo/pull/69)
135
- * [sorbet-eraser](https://github.com/kddnewton/sorbet-eraser/pull/25)
136
- * [synvert](https://github.com/xinminlabs/synvert-core-ruby)
137
- * [typeprof](https://github.com/ruby/typeprof)
138
- * [unparser](https://github.com/mbj/unparser) (via parser translator)
139
-
140
- ### Applications
141
-
142
- * [gem.sh](https://github.com/marcoroth/gem.sh/pull/96)
143
- * [Sorbet](https://github.com/sorbet/sorbet)