prism 1.8.1 → 2.0.0.rc1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (241) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +90 -2
  3. data/Makefile +41 -12
  4. data/README.md +35 -97
  5. data/config.yml +174 -163
  6. data/docs/build_system.md +2 -2
  7. data/docs/configuration.md +5 -4
  8. data/docs/encoding.md +2 -2
  9. data/docs/fuzzing.md +4 -11
  10. data/docs/javascript.md +19 -4
  11. data/docs/mapping.md +1 -1
  12. data/docs/releasing.md +77 -5
  13. data/docs/relocation.md +2 -4
  14. data/docs/ripper_translation.md +8 -17
  15. data/docs/serialization.md +21 -24
  16. data/docs/testing.md +46 -29
  17. data/ext/prism/api_node.c +1461 -1401
  18. data/ext/prism/extconf.rb +3 -4
  19. data/ext/prism/extension.c +778 -422
  20. data/ext/prism/extension.h +3 -3
  21. data/include/prism/arena.h +37 -0
  22. data/include/prism/ast.h +2729 -612
  23. data/include/prism/buffer.h +52 -0
  24. data/include/prism/comments.h +55 -0
  25. data/include/prism/compiler/accel.h +24 -0
  26. data/include/prism/compiler/align.h +36 -0
  27. data/include/prism/compiler/assume.h +25 -0
  28. data/include/prism/compiler/exported.h +24 -0
  29. data/include/prism/compiler/fallthrough.h +22 -0
  30. data/include/prism/compiler/filesystem.h +32 -0
  31. data/include/prism/compiler/flex_array.h +19 -0
  32. data/include/prism/compiler/force_inline.h +21 -0
  33. data/include/prism/compiler/format.h +25 -0
  34. data/include/prism/compiler/inline.h +17 -0
  35. data/include/prism/compiler/nodiscard.h +22 -0
  36. data/include/prism/compiler/nonnull.h +18 -0
  37. data/include/prism/compiler/unused.h +18 -0
  38. data/include/prism/constant_pool.h +81 -0
  39. data/include/prism/diagnostic.h +38 -405
  40. data/include/prism/errors_format.h +53 -0
  41. data/include/prism/excludes.h +29 -0
  42. data/include/prism/integer.h +41 -0
  43. data/include/prism/internal/allocator.h +68 -0
  44. data/include/prism/internal/allocator_debug.h +88 -0
  45. data/include/prism/internal/arena.h +108 -0
  46. data/include/prism/internal/bit.h +42 -0
  47. data/include/prism/internal/buffer.h +91 -0
  48. data/include/prism/internal/char.h +145 -0
  49. data/include/prism/internal/comments.h +53 -0
  50. data/include/prism/internal/constant_pool.h +152 -0
  51. data/include/prism/internal/diagnostic.h +389 -0
  52. data/include/prism/{encoding.h → internal/encoding.h} +27 -68
  53. data/include/prism/internal/integer.h +68 -0
  54. data/include/prism/internal/isinf.h +16 -0
  55. data/include/prism/internal/line_offset_list.h +34 -0
  56. data/include/prism/{util/pm_list.h → internal/list.h} +11 -52
  57. data/include/prism/internal/magic_comments.h +23 -0
  58. data/include/prism/internal/memchr.h +15 -0
  59. data/include/prism/internal/node.h +38 -0
  60. data/include/prism/internal/options.h +232 -0
  61. data/include/prism/internal/parser.h +984 -0
  62. data/include/prism/internal/regexp.h +26 -0
  63. data/include/prism/internal/serialize.h +39 -0
  64. data/include/prism/internal/source.h +72 -0
  65. data/include/prism/{static_literals.h → internal/static_literals.h} +26 -47
  66. data/include/prism/internal/stringy.h +35 -0
  67. data/include/prism/{util/pm_strncasecmp.h → internal/strncasecmp.h} +3 -17
  68. data/include/prism/{util/pm_strpbrk.h → internal/strpbrk.h} +7 -20
  69. data/include/prism/internal/tokens.h +11 -0
  70. data/include/prism/json.h +32 -0
  71. data/include/prism/line_offset_list.h +61 -0
  72. data/include/prism/magic_comments.h +35 -0
  73. data/include/prism/node.h +18 -53
  74. data/include/prism/options.h +146 -280
  75. data/include/prism/parser.h +336 -868
  76. data/include/prism/prettyprint.h +6 -9
  77. data/include/prism/serialize.h +115 -0
  78. data/include/prism/source.h +155 -0
  79. data/include/prism/stream.h +28 -0
  80. data/include/prism/string_query.h +63 -0
  81. data/include/prism/stringy.h +72 -0
  82. data/include/prism/version.h +13 -4
  83. data/include/prism.h +43 -320
  84. data/lib/prism/compiler.rb +319 -305
  85. data/lib/prism/desugar_compiler.rb +105 -34
  86. data/lib/prism/dispatcher.rb +646 -473
  87. data/lib/prism/dot_visitor.rb +592 -511
  88. data/lib/prism/dsl.rb +391 -25
  89. data/lib/prism/ffi.rb +252 -90
  90. data/lib/prism/inspect_visitor.rb +357 -325
  91. data/lib/prism/lex_compat.rb +311 -228
  92. data/lib/prism/mutation_compiler.rb +313 -305
  93. data/lib/prism/node.rb +20284 -7038
  94. data/lib/prism/node_ext.rb +180 -187
  95. data/lib/prism/node_find.rb +215 -0
  96. data/lib/prism/parse_result/errors.rb +7 -1
  97. data/lib/prism/parse_result/newlines.rb +276 -8
  98. data/lib/prism/parse_result.rb +462 -139
  99. data/lib/prism/pattern.rb +65 -20
  100. data/lib/prism/reflection.rb +23 -12
  101. data/lib/prism/relocation.rb +149 -87
  102. data/lib/prism/serialize.rb +3442 -536
  103. data/lib/prism/string_query.rb +16 -1
  104. data/lib/prism/translation/parser/builder.rb +10 -2
  105. data/lib/prism/translation/parser/compiler.rb +44 -52
  106. data/lib/prism/translation/parser/lexer.rb +41 -92
  107. data/lib/prism/translation/parser.rb +1 -1
  108. data/lib/prism/translation/parser_current.rb +1 -1
  109. data/lib/prism/translation/ripper/filter.rb +53 -0
  110. data/lib/prism/translation/ripper/lexer.rb +92 -5
  111. data/lib/prism/translation/ripper/sexp.rb +2 -10
  112. data/lib/prism/translation/ripper/shim.rb +2 -0
  113. data/lib/prism/translation/ripper.rb +1005 -224
  114. data/lib/prism/translation/ruby_parser.rb +73 -316
  115. data/lib/prism/translation.rb +2 -0
  116. data/lib/prism/visitor.rb +632 -5
  117. data/lib/prism.rb +79 -22
  118. data/prism.gemspec +123 -61
  119. data/rbi/generated/prism/compiler.rbi +489 -0
  120. data/rbi/generated/prism/desugar_compiler.rbi +306 -0
  121. data/rbi/generated/prism/dispatcher.rbi +1138 -0
  122. data/rbi/generated/prism/dot_visitor.rbi +618 -0
  123. data/rbi/generated/prism/dsl.rbi +751 -0
  124. data/rbi/generated/prism/inspect_visitor.rbi +503 -0
  125. data/rbi/generated/prism/lex_compat.rbi +162 -0
  126. data/rbi/generated/prism/mutation_compiler.rbi +464 -0
  127. data/rbi/generated/prism/node.rbi +13681 -0
  128. data/rbi/generated/prism/node_ext.rbi +259 -0
  129. data/rbi/generated/prism/node_find.rbi +79 -0
  130. data/rbi/generated/prism/parse_result/errors.rbi +21 -0
  131. data/rbi/generated/prism/parse_result/newlines.rbi +208 -0
  132. data/rbi/generated/prism/parse_result.rbi +764 -0
  133. data/rbi/generated/prism/pattern.rbi +134 -0
  134. data/rbi/generated/prism/reflection.rbi +89 -0
  135. data/rbi/generated/prism/relocation.rbi +329 -0
  136. data/rbi/generated/prism/serialize.rbi +181 -0
  137. data/rbi/generated/prism/string_query.rbi +36 -0
  138. data/rbi/generated/prism/translation.rbi +7 -0
  139. data/rbi/generated/prism/visitor.rbi +650 -0
  140. data/rbi/generated/prism.rbi +86 -0
  141. data/rbi/prism/translation/parser_versions.rbi +5 -0
  142. data/rbi/rubyvm/node_find.rbi +14 -0
  143. data/sig/generated/prism/compiler.rbs +492 -0
  144. data/sig/generated/prism/desugar_compiler.rbs +305 -0
  145. data/sig/generated/prism/dispatcher.rbs +1292 -0
  146. data/sig/generated/prism/dot_visitor.rbs +635 -0
  147. data/sig/generated/prism/dsl.rbs +928 -0
  148. data/sig/generated/prism/inspect_visitor.rbs +509 -0
  149. data/sig/generated/prism/lex_compat.rbs +166 -0
  150. data/sig/{prism → generated/prism}/mutation_compiler.rbs +309 -4
  151. data/sig/generated/prism/node.rbs +20160 -0
  152. data/sig/generated/prism/node_ext.rbs +290 -0
  153. data/sig/generated/prism/node_find.rbs +90 -0
  154. data/sig/generated/prism/parse_result/errors.rbs +22 -0
  155. data/sig/generated/prism/parse_result/newlines.rbs +223 -0
  156. data/sig/generated/prism/parse_result.rbs +859 -0
  157. data/sig/generated/prism/pattern.rbs +156 -0
  158. data/sig/generated/prism/reflection.rbs +101 -0
  159. data/sig/generated/prism/relocation.rbs +428 -0
  160. data/sig/generated/prism/serialize.rbs +195 -0
  161. data/sig/generated/prism/string_query.rbs +36 -0
  162. data/sig/generated/prism/translation.rbs +8 -0
  163. data/sig/generated/prism/visitor.rbs +1111 -0
  164. data/sig/generated/prism.rbs +84 -0
  165. data/sig/manifest.yaml +2 -0
  166. data/src/arena.c +117 -0
  167. data/src/{util/pm_buffer.c → buffer.c} +44 -21
  168. data/src/{util/pm_char.c → char.c} +28 -57
  169. data/src/comments.c +454 -0
  170. data/src/constant_pool.c +406 -0
  171. data/src/diagnostic.c +93 -65
  172. data/src/encoding.c +8 -3
  173. data/src/errors_format.c +519 -0
  174. data/src/{util/pm_integer.c → integer.c} +33 -22
  175. data/src/json.c +5733 -0
  176. data/src/line_offset_list.c +100 -0
  177. data/src/list.c +24 -0
  178. data/src/{util/pm_memchr.c → memchr.c} +8 -6
  179. data/src/node.c +3504 -7674
  180. data/src/options.c +162 -49
  181. data/src/parser.c +349 -0
  182. data/src/prettyprint.c +421 -354
  183. data/src/prism.c +8035 -7019
  184. data/src/regexp.c +961 -45
  185. data/src/serialize.c +459 -424
  186. data/src/source.c +518 -0
  187. data/src/static_literals.c +111 -43
  188. data/src/string_query.c +166 -0
  189. data/src/stringy.c +91 -0
  190. data/src/{util/pm_strncasecmp.c → strncasecmp.c} +5 -4
  191. data/src/strpbrk.c +447 -0
  192. data/src/{token_type.c → tokens.c} +39 -18
  193. metadata +123 -62
  194. data/ext/prism/api_pack.c +0 -276
  195. data/include/prism/defines.h +0 -260
  196. data/include/prism/pack.h +0 -163
  197. data/include/prism/regexp.h +0 -43
  198. data/include/prism/util/pm_buffer.h +0 -236
  199. data/include/prism/util/pm_char.h +0 -204
  200. data/include/prism/util/pm_constant_pool.h +0 -218
  201. data/include/prism/util/pm_integer.h +0 -130
  202. data/include/prism/util/pm_memchr.h +0 -29
  203. data/include/prism/util/pm_newline_list.h +0 -113
  204. data/include/prism/util/pm_string.h +0 -200
  205. data/lib/prism/lex_ripper.rb +0 -64
  206. data/lib/prism/pack.rb +0 -230
  207. data/lib/prism/parse_result/comments.rb +0 -188
  208. data/rbi/prism/compiler.rbi +0 -12
  209. data/rbi/prism/dsl.rbi +0 -524
  210. data/rbi/prism/inspect_visitor.rbi +0 -12
  211. data/rbi/prism/node.rbi +0 -8747
  212. data/rbi/prism/node_ext.rbi +0 -107
  213. data/rbi/prism/parse_result.rbi +0 -404
  214. data/rbi/prism/reflection.rbi +0 -58
  215. data/rbi/prism/string_query.rbi +0 -12
  216. data/rbi/prism/visitor.rbi +0 -473
  217. data/rbi/prism.rbi +0 -66
  218. data/sig/prism/compiler.rbs +0 -9
  219. data/sig/prism/dispatcher.rbs +0 -19
  220. data/sig/prism/dot_visitor.rbs +0 -6
  221. data/sig/prism/dsl.rbs +0 -351
  222. data/sig/prism/inspect_visitor.rbs +0 -22
  223. data/sig/prism/lex_compat.rbs +0 -10
  224. data/sig/prism/node.rbs +0 -4039
  225. data/sig/prism/node_ext.rbs +0 -149
  226. data/sig/prism/pack.rbs +0 -43
  227. data/sig/prism/parse_result/comments.rbs +0 -38
  228. data/sig/prism/parse_result.rbs +0 -196
  229. data/sig/prism/pattern.rbs +0 -13
  230. data/sig/prism/reflection.rbs +0 -50
  231. data/sig/prism/relocation.rbs +0 -185
  232. data/sig/prism/serialize.rbs +0 -8
  233. data/sig/prism/string_query.rbs +0 -11
  234. data/sig/prism/visitor.rbs +0 -169
  235. data/sig/prism.rbs +0 -258
  236. data/src/pack.c +0 -509
  237. data/src/util/pm_constant_pool.c +0 -342
  238. data/src/util/pm_list.c +0 -49
  239. data/src/util/pm_newline_list.c +0 -125
  240. data/src/util/pm_string.c +0 -381
  241. data/src/util/pm_strpbrk.c +0 -206
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cc702a11a2b52196e74eee050fefa294ac3c2aef3dcd7d4ab2f347e075707286
4
- data.tar.gz: 32666fef05c64edc7a55ab517d6b582931f9b5b11ff22b5d9dfc5c7d00ee00b2
3
+ metadata.gz: 355edb5b4cad3553f53996ce8b1714e26c760aaf8631c5798d3c59458327d648
4
+ data.tar.gz: 96af4612ddaea02181ba46fd3a879b8367bb850e69a42fe77e81c8d03b403381
5
5
  SHA512:
6
- metadata.gz: 2507880056ab51a2aafb14b6927a5d009ff8f3be4eb9917705a56ac59ac6c21c4a687f984652f2f4abe813e24f239077ee5aa0cd7308b6f3240fa864b1b06b28
7
- data.tar.gz: 921a61d5bd263f70975250aca097c6ed36b747762098b2bc90d113d88f8c8f802e931bf7c2f78c365bce9390190b53bc1c243c1dfa5d3ea2a9c89a12b45659fb
6
+ metadata.gz: f0ee8cb3854031b8977dac87bf6f2e2b92cc0c7073453e0c7f14170b9606a0a9622a01cd28f25e21349036258199c5ad31b3b4156beff27071757487ac2d7a7d
7
+ data.tar.gz: 157b85dbe849a6ec8f341e61bffcf34c29da3c0ec4929d4068570a31b1eac39fb791c951c787404fabcb97f2bf6d86a391b1279df3c5038410e8c5abded18d02
data/CHANGELOG.md CHANGED
@@ -6,7 +6,94 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
- ## [1.8.1] - 2026-03-16
9
+ ### Highlights
10
+
11
+ - Significant performance and memory usage improvements.
12
+ - Precompiled platform-specific variants on rubygems.org.
13
+ - Many documentation improvements, including a new playground.
14
+ - Prism now ships accurate and complete RBS and RBI definitions.
15
+
16
+ ### Added
17
+
18
+ - Support for parsing Ruby 4.1 syntax:
19
+ - Add support for `&nil` to declare that a method accepts no block.
20
+ - Thoroughly implement void value expression check.
21
+ - Allow trailing commas in method signatures like `def foo(a,); end`.
22
+ - Reject `END { break }` and `END { next }`.
23
+ - `a /b/` no longer warns about ambiguous `/`.
24
+ - `version: "nearest"` is now supported. It behaves similarly to `version: "current"` but will not raise when the version is unsupported, instead falling back to the nearest syntax version prism currently supports.
25
+ - `raise_error` was added as a valid option. Prism will raise a `SyntaxError` when given syntax-invalid code. When `true` is given, prism decides the formatting (`tty?`, `NO_COLOR`, etc.). If you need a specific style, `:plain`, `:style`, and `:color` are also supported values.
26
+ - Added `Prism::ParseResult#continuable` to check if the parsed source can become valid by appending more code.
27
+ - `Prism.find` takes a `Method`, `UnboundMethod`, `Proc`, or `Thread::Backtrace::Location` and returns the associated prism AST node.
28
+ - `ForwardingSuperNode` now has `keyword_loc`.
29
+ - Allow pattern matching with prism nodes to match on location slices, like `node in Prism::StringNode[content: "foo"]`.
30
+ - `Prism.parse_file` and similar now accept anything responding to `to_path` (like `Pathname`) directly.
31
+ - `attach_comments` was added as a valid option. Each comment is attached to the node that ends latest on the comment's line before it, otherwise to the innermost node that contains it (or to that node's statements, if only whitespace and comments come between the comment and the statements), otherwise to the top-level statements. The attached comments are available through `Node#comments` and through `pm_node_comments_size` and `pm_node_comment` in the C API, as well as `pm_parser_comment_attachment`, `pm_parser_comment_ranges_size`, and `pm_parser_comment_range`, which expose the attachments grouped by node. When the option is set, the comments on the parse result are sorted by start offset.
32
+
33
+ ### Changed
34
+
35
+ Note that for this major version, almost every change listed below is breaking.
36
+
37
+ - Prism now defaults to `version: "nearest"` instead of `version: "latest"`, so code is parsed with the syntax of the running Ruby (or the closest version prism supports). Pass `version: "latest"` to keep the previous behavior.
38
+ - `MissingNode` has been replaced by `ErrorRecoveryNode`. `ErrorRecoveryNode` wraps unexpected nodes, so when walking syntax-invalid code you will now only need to handle this one specific node type.
39
+ - Some fields were renamed for consistency. Previous names are aliased and will continue to work in the Ruby layer.
40
+ - `InNode#{in,then}_loc` -> `InNode#{in,then}_keyword_loc`
41
+ - `MatchPredicateNode#operator_loc` -> `MatchPredicateNode#keyword_loc`
42
+ - `UnlessNode#keyword_loc` -> `UnlessNode#unless_keyword_loc`
43
+ - `UntilNode#{keyword,closing}_loc` -> `UntilNode#{until,end}_keyword_loc`
44
+ - `WhenNode#keyword_loc` -> `WhenNode#when_keyword_loc`
45
+ - `WhileNode#{keyword,closing}_loc` -> `WhileNode#{while,end}_keyword_loc`
46
+ - Optional locations like `StringNode#closing_loc` in `'foo` that were empty for syntax-invalid code will now be `nil` instead.
47
+ - `Prism.lex` now returns an array of tokens instead of an array of `[token, internal_parse_state]`. Previous access like `token[0]` or `token.first` to extract the actual token will continue to work. Additionally the following tokens were changed:
48
+ - `PARENTHESIS_LEFT` was split up into `PARENTHESIS_LEFT` and `PARENTHESIS_LEFT_GROUPING`.
49
+ - `BRACE_LEFT` was split up into `BRACE_LEFT`, `BRACE_LEFT_ARGUMENT`, and `BRACE_LEFT_HASH`.
50
+ - `BACKTICK` was split up into `BACKTICK` and `XSTRING_BACKTICK`.
51
+ - `NEWLINE` was split up into `NEWLINE` and `NEWLINE_TERMINATOR`.
52
+ - `WORDS_SEP` was split up into `WORDS_SEP` and `WORDS_SEP_IMPLICIT`.
53
+
54
+ #### Ruby
55
+
56
+ - Require arguments for `Prism::Source.for`. The second and third positional parameters are no longer optional.
57
+ - Comments now live on the source. `Prism::Source#comments` returns them and `Result#comments` delegates to it, so `ParseResult.new`, `LexResult.new`, and `ParseLexResult.new` no longer take a `comments` argument.
58
+ - Comment attachment now happens during parsing with the `attach_comments` option instead of afterward with `ParseResult#attach_comments!`, which was removed. `Node#comments` now returns the comments attached to the node by that option. `Location#comments`, `Location#leading_comments`, `Location#trailing_comments`, `Node#leading_comments`, `Node#trailing_comments`, and the comment fields of `Prism::Relocation` were removed.
59
+
60
+ #### C
61
+
62
+ - Most structs are now opaque (except for nodes), and many internal functions have been hidden.
63
+ - Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
64
+ - Most allocations are now performed through an arena.
65
+ - You may now build side tables of comment attachments through the `attach_comments` option.
66
+ - `newline_list` was renamed to `list_offset_list`.
67
+
68
+ #### Rust
69
+
70
+ - Locations are now represented using slices (pairs of 32-bit integers for start offset and length).
71
+ - Most allocations are now performed through an arena.
72
+ - `newline_list` was renamed to `list_offset_list`.
73
+ - `Diagnostic::message` now returns `&[u8]` instead of `&str`.
74
+
75
+ #### JavaScript
76
+
77
+ - The parse method now only accepts `Uint8Array`, instead of forcing a transcoding to UTF-16.
78
+
79
+ ### Removed
80
+
81
+ - Removed `Prism::Pack`.
82
+
83
+ ## [1.9.0] - 2026-01-27
84
+
85
+ ### Added
86
+
87
+ - Lots of work on the Ripper translation layer to make it more compatible and efficient.
88
+ - Alias `Prism::Node#breadth_first_search` to `Prism::Node#find`.
89
+ - Add `Prism::Node#breadth_first_search_all`/`Prism::Node#find_all` for finding all nodes matching a condition.
90
+
91
+ ### Changed
92
+
93
+ - Fixed location of opening tokens when invalid syntax is parsed.
94
+ - Fix RBI for parsing options.
95
+
96
+ ## [1.8.1] - 2026-03-16 (Ruby 4.0.2 maintenance release)
10
97
 
11
98
  ### Changed
12
99
 
@@ -740,7 +827,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
740
827
 
741
828
  - 🎉 Initial release! 🎉
742
829
 
743
- [unreleased]: https://github.com/ruby/prism/compare/v1.8.1...HEAD
830
+ [unreleased]: https://github.com/ruby/prism/compare/v1.9.0...HEAD
831
+ [1.9.0]: https://github.com/ruby/prism/compare/v1.8.0...v1.9.0
744
832
  [1.8.1]: https://github.com/ruby/prism/compare/v1.8.0...v1.8.1
745
833
  [1.8.0]: https://github.com/ruby/prism/compare/v1.7.0...v1.8.0
746
834
  [1.7.0]: https://github.com/ruby/prism/compare/v1.6.0...v1.7.0
data/Makefile CHANGED
@@ -12,16 +12,30 @@ SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
12
12
 
13
13
  CPPFLAGS := -Iinclude $(CPPFLAGS)
14
14
  CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(CFLAGS)
15
- JAVA_WASM_CFLAGS := -g -Oz -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(JAVA_WASM_CFLAGS)
15
+ JAVA_WASM_CFLAGS := -g0 -O2 -Oz -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fno-sanitize=all -fno-stack-protector -fPIC -fvisibility=hidden -Wimplicit-fallthrough $(JAVA_WASM_CFLAGS)
16
16
  CC ?= cc
17
17
  AR ?= ar
18
18
  ARFLAGS ?= -r$(V0:1=v)
19
19
  WASI_SDK_PATH := /opt/wasi-sdk
20
20
 
21
+ # Exporting a symbol makes it a GC root, so the JavaScript module exports only
22
+ # the symbols javascript/src/parsePrism.js calls. That lets --gc-sections drop
23
+ # the rest of the C API along with the wasi-libc code reachable from it.
24
+ WASM_EXPORTS := -Wl,--export=malloc,--export=free,--export=pm_buffer_new,--export=pm_buffer_value,--export=pm_buffer_length,--export=pm_buffer_free,--export=pm_serialize_parse
25
+
26
+ # Bind the library's own references to its own definitions at link time, so
27
+ # that the functions exported for FFI cannot be interposed by another copy of
28
+ # libprism loaded in the same process (for example the copy built into the
29
+ # interpreter next to the one built by a prism gem, when either copy was loaded
30
+ # with RTLD_GLOBAL). Mach-O and PE do this by default; only ELF needs the flag.
31
+ ifeq ($(SOEXT),so)
32
+ SHARED_LDFLAGS := -Wl,-Bsymbolic
33
+ endif
34
+
21
35
  MAKEDIRS ?= mkdir -p
22
36
  RMALL ?= rm -f -r
23
37
 
24
- HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h')
38
+ HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h)
25
39
  SOURCES := $(wildcard src/*.c src/*/*.c)
26
40
  SHARED_OBJECTS := $(subst src/,build/shared/,$(SOURCES:.c=.o))
27
41
  STATIC_OBJECTS := $(subst src/,build/static/,$(SOURCES:.c=.o))
@@ -31,11 +45,12 @@ all: shared static
31
45
  shared: build/libprism.$(SOEXT)
32
46
  static: build/libprism.a
33
47
  wasm: javascript/src/prism.wasm
34
- java-wasm: java-wasm/src/test/resources/prism.wasm
48
+ java-wasm: java/wasm/src/main/wasm/prism.wasm
49
+ playground: doc/playground
35
50
 
36
51
  build/libprism.$(SOEXT): $(SHARED_OBJECTS)
37
52
  $(ECHO) "linking $@ with $(CC)"
38
- $(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared -o $@ $(SHARED_OBJECTS)
53
+ $(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared $(SHARED_LDFLAGS) -o $@ $(SHARED_OBJECTS)
39
54
 
40
55
  build/libprism.a: $(STATIC_OBJECTS)
41
56
  $(ECHO) "building $@ with $(AR)"
@@ -45,15 +60,29 @@ javascript/src/prism.wasm: Makefile $(SOURCES) $(HEADERS)
45
60
  $(ECHO) "building $@"
46
61
  $(Q) $(WASI_SDK_PATH)/bin/clang --sysroot=$(WASI_SDK_PATH)/share/wasi-sysroot/ \
47
62
  $(DEBUG_FLAGS) \
48
- -DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON -DPRISM_EXCLUDE_PACK \
63
+ -DPRISM_EXPORT_SYMBOLS -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXCLUDE_JSON \
49
64
  -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(CFLAGS) \
50
- -Wl,--export-all -Wl,--gc-sections -Wl,--strip-all -Wl,--lto-O3 -Wl,--no-entry -mexec-model=reactor \
65
+ $(WASM_EXPORTS) -Wl,--gc-sections -Wl,--strip-all -Wl,--lto-O3 -Wl,--no-entry -mexec-model=reactor \
51
66
  -Oz -g0 -flto -fdata-sections -ffunction-sections \
52
67
  -o $@ $(SOURCES)
53
68
 
54
- java-wasm/src/test/resources/prism.wasm: Makefile $(SOURCES) $(HEADERS)
69
+ # The playground parses with the JavaScript package built from this checkout, so
70
+ # the page is served the parser the rest of the site was generated from. The
71
+ # JavaScript half of the package is generated by `rake templates`.
72
+ doc/playground: javascript/src/prism.wasm
73
+ $(ECHO) "building $@"
74
+ $(Q) $(MAKEDIRS) $@
75
+ $(Q) cp javascript/src/*.js javascript/src/prism.wasm javascript/package.json $@
76
+
77
+ java/wasm/src/main/wasm/prism.wasm: Makefile $(SOURCES) $(HEADERS)
55
78
  $(ECHO) "building $@"
56
- $(Q) $(WASI_SDK_PATH)/bin/clang $(DEBUG_FLAGS) -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(JAVA_WASM_CFLAGS) -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi -o $@ $(SOURCES)
79
+ $(Q) $(MAKEDIRS) $(@D)
80
+ $(Q) $(WASI_SDK_PATH)/bin/clang \
81
+ $(DEBUG_FLAGS) \
82
+ -DPRISM_EXCLUDE_PRETTYPRINT -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN \
83
+ -lwasi-emulated-mman $(CPPFLAGS) $(JAVA_WASM_CFLAGS) \
84
+ -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi \
85
+ -o $@ $(SOURCES)
57
86
 
58
87
  build/shared/%.o: src/%.c Makefile $(HEADERS)
59
88
  $(ECHO) "compiling $@"
@@ -69,12 +98,12 @@ build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
69
98
  $(ECHO) "building $* fuzzer"
70
99
  $(Q) $(MAKEDIRS) $(@D)
71
100
  $(ECHO) "building main fuzz binary"
72
- $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
101
+ $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
73
102
  $(ECHO) "building cmplog binary"
74
- $(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
103
+ $(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
75
104
 
76
105
  build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
77
- $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
106
+ $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
78
107
 
79
108
  fuzz-debug:
80
109
  $(ECHO) "entering debug shell"
@@ -100,7 +129,7 @@ fuzz-clean:
100
129
  clean:
101
130
  $(Q) $(RMALL) build
102
131
 
103
- .PHONY: clean fuzz-clean
132
+ .PHONY: clean doc/playground fuzz-clean
104
133
 
105
134
  all-no-debug: DEBUG_FLAGS := -DNDEBUG=1
106
135
  all-no-debug: OPTFLAGS := -O3
data/README.md CHANGED
@@ -1,10 +1,35 @@
1
- <h1 align="center">Prism Ruby parser</h1>
2
- <div align="center">
3
- <img alt="Prism Ruby parser" height="256px" src="https://github.com/ruby/prism/blob/main/doc/images/prism.png?raw=true">
4
- </div>
1
+ <h1 align="center">Prism</h1>
5
2
 
6
3
  This is a parser for the Ruby programming language. It is designed to be portable, error tolerant, and maintainable. It is written in C99 and has no dependencies.
7
4
 
5
+ ## Getting started
6
+
7
+ To compile the parser, you will need:
8
+
9
+ * C99 compiler
10
+ * GNU make
11
+ * Ruby 2.7.0 or later
12
+
13
+ Once you have these dependencies, run:
14
+
15
+ ```
16
+ bundle install
17
+ ```
18
+
19
+ to fetch the Ruby dependencies. Finally, run:
20
+
21
+ ```
22
+ bundle exec rake compile
23
+ ```
24
+
25
+ to compile the parser. It will be built in the `build` directory. To test that everything is working, run:
26
+
27
+ ```
28
+ bin/parse -e "1 + 2"
29
+ ```
30
+
31
+ to see the syntax tree for the expression `1 + 2`.
32
+
8
33
  ## Overview
9
34
 
10
35
  The repository contains the infrastructure for both a shared library (libprism) and a native CRuby extension. The shared library has no bindings to CRuby itself, and so can be used by other projects. The native CRuby extension links against `ruby.h`, and so is suitable in the context of CRuby.
@@ -12,9 +37,9 @@ The repository contains the infrastructure for both a shared library (libprism)
12
37
  ```
13
38
  .
14
39
  ├── Makefile configuration to compile the shared library and native tests
15
- ├── Rakefile configuration to compile the native extension and run the Ruby tests
40
+ ├── Rakefile configuration to compile the native extension and Ruby tests
16
41
  ├── bin
17
- │   ├── lex runs the lexer on a file or string, prints the tokens, and compares to ripper
42
+ │   ├── lex runs the lexer on a file or string and prints the tokens
18
43
  │   ├── parse runs the parse on a file or string and prints the AST
19
44
  │   └── prism a CLI for development and debugging
20
45
  ├── config.yml specification for tokens and nodes in the tree
@@ -29,8 +54,10 @@ The repository contains the infrastructure for both a shared library (libprism)
29
54
  ├── include
30
55
  │   ├── prism header files for the shared library
31
56
  │   └── prism.h main header file for the shared library
32
- ├── java Java bindings for the shared library
33
- ├── java-wasm Java WASM bindings for the shared library
57
+ ├── java
58
+ | ├── api Java API for Prism
59
+ | ├── native Java native bindings for the shared library
60
+ | └── wasm Java WASM bindings for the shared library
34
61
  ├── javascript JavaScript WASM bindings for the shared library
35
62
  ├── lib
36
63
  │   ├── prism Ruby library files
@@ -44,7 +71,6 @@ The repository contains the infrastructure for both a shared library (libprism)
44
71
  │ └── prism Sample code that uses the Ruby API for documentation purposes
45
72
  ├── sig RBS type signatures for the Ruby library
46
73
  ├── src
47
- │   ├── util various utility files
48
74
  │   └── prism.c main entrypoint for the shared library
49
75
  ├── templates contains ERB templates generated by templates/template.rb
50
76
  │   └── template.rb generates code from the nodes and tokens configured by config.yml
@@ -53,91 +79,3 @@ The repository contains the infrastructure for both a shared library (libprism)
53
79
  ├── fixtures Ruby code used for testing
54
80
  └── snapshots snapshots of generated syntax trees corresponding to fixtures
55
81
  ```
56
-
57
- ## Getting started
58
-
59
- To compile the shared library, you will need:
60
-
61
- * C99 compiler
62
- * GNU make
63
- * Ruby 2.7.0 or later
64
-
65
- Once you have these dependencies, run:
66
-
67
- ```
68
- bundle install
69
- ```
70
-
71
- to fetch the Ruby dependencies. Finally, run:
72
-
73
- ```
74
- bundle exec rake compile
75
- ```
76
-
77
- to compile the shared library. It will be built in the `build` directory. To test that everything is working, run:
78
-
79
- ```
80
- bin/parse -e "1 + 2"
81
- ```
82
-
83
- to see the syntax tree for the expression `1 + 2`.
84
-
85
- ## Contributing
86
-
87
- See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additionally have documentation about the overall design of the project as well as various subtopics.
88
-
89
- * [Build system](docs/build_system.md)
90
- * [Configuration](docs/configuration.md)
91
- * [CRuby compilation](docs/cruby_compilation.md)
92
- * [Design](docs/design.md)
93
- * [Encoding](docs/encoding.md)
94
- * [Fuzzing](docs/fuzzing.md)
95
- * [Heredocs](docs/heredocs.md)
96
- * [JavaScript](docs/javascript.md)
97
- * [Local variable depth](docs/local_variable_depth.md)
98
- * [Mapping](docs/mapping.md)
99
- * [Parser translation](docs/parser_translation.md)
100
- * [Parsing rules](docs/parsing_rules.md)
101
- * [Releasing](docs/releasing.md)
102
- * [Ripper translation](docs/ripper_translation.md)
103
- * [Ruby API](docs/ruby_api.md)
104
- * [RubyParser translation](docs/ruby_parser_translation.md)
105
- * [Serialization](docs/serialization.md)
106
- * [Testing](docs/testing.md)
107
-
108
- ## Examples
109
-
110
- Prism has been integrated into the majority of Ruby runtimes, many libraries, and some applications. Below is a list of some of the projects that use Prism:
111
-
112
- ### Runtimes
113
-
114
- * [CRuby](https://github.com/ruby/ruby/pull/7964) (via C)
115
- * [Garnet](https://github.com/camertron/garnet-js) (via WASM)
116
- * [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java)
117
- * [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby)
118
- * [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM)
119
- * [TruffleRuby](https://github.com/truffleruby/truffleruby/issues/3117) (via Java)
120
-
121
- ### Libraries
122
-
123
- * [dispersion](https://github.com/joeldrapper/dispersion)
124
- * [minifyrb](https://github.com/koic/minifyrb)
125
- * [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator)
126
- * [rbi](https://github.com/Shopify/rbi)
127
- * [rails](https://github.com/rails/rails)
128
- * [parsing renders](https://github.com/rails/rails/pull/49438)
129
- * [parsing rdoc](https://github.com/rails/rails/pull/50870)
130
- * [parsing tests](https://github.com/rails/rails/pull/51006)
131
- * [repl_type_completor](https://github.com/ruby/repl_type_completor)
132
- * [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine) (via parser translator)
133
- * [ruby-lsp](https://github.com/Shopify/ruby-lsp)
134
- * [smart_todo](https://github.com/Shopify/smart_todo/pull/69)
135
- * [sorbet-eraser](https://github.com/kddnewton/sorbet-eraser/pull/25)
136
- * [synvert](https://github.com/xinminlabs/synvert-core-ruby)
137
- * [typeprof](https://github.com/ruby/typeprof)
138
- * [unparser](https://github.com/mbj/unparser) (via parser translator)
139
-
140
- ### Applications
141
-
142
- * [gem.sh](https://github.com/marcoroth/gem.sh/pull/96)
143
- * [Sorbet](https://github.com/sorbet/sorbet)