prism 0.22.0 → 0.24.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +39 -1
  3. data/README.md +2 -1
  4. data/docs/releasing.md +67 -17
  5. data/docs/ruby_parser_translation.md +19 -0
  6. data/docs/serialization.md +2 -0
  7. data/ext/prism/api_node.c +1982 -1538
  8. data/ext/prism/extension.c +12 -7
  9. data/ext/prism/extension.h +2 -2
  10. data/include/prism/diagnostic.h +3 -4
  11. data/include/prism/encoding.h +7 -0
  12. data/include/prism/util/pm_constant_pool.h +1 -1
  13. data/include/prism/util/pm_newline_list.h +4 -3
  14. data/include/prism/util/pm_strpbrk.h +4 -1
  15. data/include/prism/version.h +2 -2
  16. data/lib/prism/desugar_compiler.rb +225 -80
  17. data/lib/prism/dsl.rb +302 -299
  18. data/lib/prism/ffi.rb +103 -77
  19. data/lib/prism/lex_compat.rb +1 -0
  20. data/lib/prism/node.rb +3624 -2114
  21. data/lib/prism/node_ext.rb +25 -2
  22. data/lib/prism/parse_result.rb +56 -19
  23. data/lib/prism/serialize.rb +605 -303
  24. data/lib/prism/translation/parser/compiler.rb +1 -1
  25. data/lib/prism/translation/parser/rubocop.rb +11 -3
  26. data/lib/prism/translation/parser.rb +25 -12
  27. data/lib/prism/translation/parser33.rb +12 -0
  28. data/lib/prism/translation/parser34.rb +12 -0
  29. data/lib/prism/translation/ripper.rb +696 -0
  30. data/lib/prism/translation/ruby_parser.rb +1521 -0
  31. data/lib/prism/translation.rb +3 -3
  32. data/lib/prism.rb +0 -1
  33. data/prism.gemspec +6 -2
  34. data/src/diagnostic.c +10 -11
  35. data/src/encoding.c +16 -17
  36. data/src/options.c +7 -2
  37. data/src/prettyprint.c +3 -3
  38. data/src/prism.c +172 -97
  39. data/src/serialize.c +24 -13
  40. data/src/token_type.c +3 -3
  41. data/src/util/pm_constant_pool.c +1 -1
  42. data/src/util/pm_newline_list.c +6 -3
  43. data/src/util/pm_strpbrk.c +122 -14
  44. metadata +8 -4
  45. data/lib/prism/ripper_compat.rb +0 -285
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 407353660e25fcada7fe3b7e7c79d58584bf3bc1c4d9ffad8f74d178a0fbe2b7
4
- data.tar.gz: a23b28dada65ba503b2d2c6cb74e9389386bb9c59a504820ea46bdeddcc16169
3
+ metadata.gz: 827194fd8b477f9f021677ca386e35fd8300daa86a09fc11fe66a85ee54da97d
4
+ data.tar.gz: 2a44a3cf5e31260a949965aaf2aa5fba0cd8b1ac910ae694d22fd8e42f4d6b9b
5
5
  SHA512:
6
- metadata.gz: aa848ed878d8fbc2b83b04789748d7d228226a5ca0963601c3dadfa901bc5b6c528667791d220e187399a91f6b1d8c2edcaee2c65721473683c349a7c8e4cebc
7
- data.tar.gz: 6c7c4671fc9f53ca898fdce09de14eb4a79bf2ace45f88efff8302f99d6ae871c0336730d143c5bd87a8a8698a558a3d008fa68e9797c4502c7515fc67cc3099
6
+ metadata.gz: 41dab6cd85f9f57bf2e2256e7b7fd7ac44c46883ad6d79c0056bff3b34500b2ecb28a1febeb1deed5a49b862c4f4382856ad2195639f86e770c006d1b6a41f8e
7
+ data.tar.gz: 0474d64c9bbd3385f65b00bf83e6b2d3c9543785c85b42d50bce59327e6af7a8fc13d54e5b485b6c930997dabe0fd81c43835243fc96fb28beb3523395ddde71
data/CHANGELOG.md CHANGED
@@ -6,6 +6,42 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.24.0] - 2024-02-15
10
+
11
+ ### Added
12
+
13
+ - More support for `Prism::Translation::Ripper` is added.
14
+ - Support multiple versions for `Prism::Translation::Parser`.
15
+ - Improved memory usage in the FFI backend.
16
+ - Very large speed and memory improvements for creating the Ruby AST.
17
+
18
+ ### Changed
19
+
20
+ - Fix location for empty symbol in hash key.
21
+ - Fix parsing a rescue modifier on the value of an assignment when the LHS is a method call with arguments and no parentheses.
22
+
23
+ ## [0.23.0] - 2024-02-14
24
+
25
+ ### Added
26
+
27
+ - More support for `Prism::RipperCompat` is added.
28
+ - A significantly faster offset cache for `Prism::Translation::Parser` is added for files with multibyte characters.
29
+ - `Prism::Translation::RubyParser` is added.
30
+ - `Prism::ConstantPathTarget#full_name` is added.
31
+ - `version: "3.4.0"` is added as an option that is an alias for `version: "latest"`.
32
+ - Four new APIs are added to `Prism::Location`:
33
+ - `Prism::Location#start_code_units_offset`
34
+ - `Prism::Location#end_code_units_offset`
35
+ - `Prism::Location#start_code_units_column`
36
+ - `Prism::Location#end_code_units_column`
37
+ - Invalid multibyte characters are now validated within strings, lists, and heredocs.
38
+
39
+ ### Changed
40
+
41
+ - When defining `def !@`, the `name_loc` was previously only pointing to `!`, but now includes the `@`. The `name` is the same.
42
+ - `Prism::RipperCompat` has been moved to `Prism::Translation::Ripper`.
43
+ - Many of the error messages that prism produces have been changed to match the error messages that CRuby produces.
44
+
9
45
  ## [0.22.0] - 2024-02-07
10
46
 
11
47
  ### Added
@@ -356,7 +392,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
356
392
 
357
393
  - 🎉 Initial release! 🎉
358
394
 
359
- [unreleased]: https://github.com/ruby/prism/compare/v0.22.0...HEAD
395
+ [unreleased]: https://github.com/ruby/prism/compare/v0.24.0...HEAD
396
+ [0.24.0]: https://github.com/ruby/prism/compare/v0.23.0...v0.24.0
397
+ [0.23.0]: https://github.com/ruby/prism/compare/v0.22.0...v0.23.0
360
398
  [0.22.0]: https://github.com/ruby/prism/compare/v0.21.0...v0.22.0
361
399
  [0.21.0]: https://github.com/ruby/prism/compare/v0.20.0...v0.21.0
362
400
  [0.20.0]: https://github.com/ruby/prism/compare/v0.19.0...v0.20.0
data/README.md CHANGED
@@ -52,7 +52,7 @@ To compile the shared library, you will need:
52
52
 
53
53
  * C99 compiler
54
54
  * make
55
- * Ruby 3.0.0 or later
55
+ * Ruby 2.7.0 or later
56
56
 
57
57
  Once you have these dependencies, run:
58
58
 
@@ -93,5 +93,6 @@ See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additio
93
93
  * [Releasing](docs/releasing.md)
94
94
  * [Ripper](docs/ripper.md)
95
95
  * [Ruby API](docs/ruby_api.md)
96
+ * [RubyParser translation](docs/ruby_parser_translation.md)
96
97
  * [Serialization](docs/serialization.md)
97
98
  * [Testing](docs/testing.md)
data/docs/releasing.md CHANGED
@@ -8,22 +8,43 @@ To release a new version of Prism, perform the following steps:
8
8
  * Add a new section for the new version at the top of the file.
9
9
  * Fill in the relevant changes — it may be easiest to click the link for the `Unreleased` heading to find the commits.
10
10
  * Update the links at the bottom of the file.
11
- * Update the version in the following files:
12
- * `prism.gemspec` in the `Gem::Specification#version=` method call
13
- * `ext/prism/extension.h` in the `EXPECTED_PRISM_VERSION` macro
14
- * `include/prism/version.h` in the version macros
15
- * `javascript/package.json` in the `version` field
16
- * `rust/ruby-prism-sys/tests/utils_tests.rs` in the `version_test` function
17
- * `templates/java/org/prism/Loader.java.erb` in the `load` function
18
- * `templates/javascript/src/deserialize.js.erb` in the version constants
19
- * `templates/lib/prism/serialize.rb.erb` in the version constants
20
- * Run `bundle install` to update the `Gemfile.lock` file.
11
+ * Update the version numbers in the various files that reference them:
21
12
 
22
13
  ```sh
14
+ export PRISM_MAJOR="x"
15
+ export PRISM_MINOR="y"
16
+ export PRISM_PATCH="z"
17
+ export PRISM_VERSION="$PRISM_MAJOR.$PRISM_MINOR.$PRISM_PATCH"
18
+ ruby -pi -e 'gsub(/spec\.version = ".+?"/, %Q{spec.version = "#{ENV["PRISM_VERSION"]}"})' prism.gemspec
19
+ ruby -pi -e 'gsub(/EXPECTED_PRISM_VERSION ".+?"/, %Q{EXPECTED_PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' ext/prism/extension.h
20
+ ruby -pi -e 'gsub(/PRISM_VERSION_MAJOR \d+/, %Q{PRISM_VERSION_MAJOR #{ENV["PRISM_MAJOR"]}})' include/prism/version.h
21
+ ruby -pi -e 'gsub(/PRISM_VERSION_MINOR \d+/, %Q{PRISM_VERSION_MINOR #{ENV["PRISM_MINOR"]}})' include/prism/version.h
22
+ ruby -pi -e 'gsub(/PRISM_VERSION_PATCH \d+/, %Q{PRISM_VERSION_PATCH #{ENV["PRISM_PATCH"]}})' include/prism/version.h
23
+ ruby -pi -e 'gsub(/PRISM_VERSION ".+?"/, %Q{PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' include/prism/version.h
24
+ ruby -pi -e 'gsub(/"version": ".+?"/, %Q{"version": "#{ENV["PRISM_VERSION"]}"})' javascript/package.json
25
+ ruby -pi -e 'gsub(/lossy\(\), ".+?"/, %Q{lossy(), "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/tests/utils_tests.rs
26
+ ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/org/prism/Loader.java.erb
27
+ ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/org/prism/Loader.java.erb
28
+ ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/org/prism/Loader.java.erb
29
+ ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/javascript/src/deserialize.js.erb
30
+ ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/javascript/src/deserialize.js.erb
31
+ ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/javascript/src/deserialize.js.erb
32
+ ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/lib/prism/serialize.rb.erb
33
+ ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/lib/prism/serialize.rb.erb
34
+ ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/lib/prism/serialize.rb.erb
35
+ ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/Cargo.toml
36
+ ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
37
+ ruby -pi -e 'gsub(/^ruby-prism-sys = \{ version = ".+?"/, %Q{ruby-prism-sys = \{ version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
38
+ ```
39
+
40
+ * Update the `Gemfile.lock` file:
41
+
42
+ ```sh
43
+ chruby ruby-3.4.0-dev
23
44
  bundle install
24
45
  ```
25
46
 
26
- * Run `bundle install` with the various other gemfiles:
47
+ * Update the version-specific lockfiles:
27
48
 
28
49
  ```sh
29
50
  chruby ruby-2.7.8 && BUNDLE_GEMFILE=gemfiles/2.7/Gemfile bundle install
@@ -34,15 +55,44 @@ chruby ruby-3.3.0 && BUNDLE_GEMFILE=gemfiles/3.3/Gemfile bundle install
34
55
  chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/3.4/Gemfile bundle install
35
56
  chruby jruby-9.4.5.0 && BUNDLE_GEMFILE=gemfiles/jruby/Gemfile bundle install
36
57
  chruby truffleruby-23.1.2 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
58
+ chruby ruby-3.4.0-dev
37
59
  ```
38
60
 
39
- * Update `rust/ruby-prism-sys/Cargo.toml` to match the new version and run `cargo build`
40
- * Update `rust/ruby-prism/Cargo.toml` to match the new version and run `cargo build`
41
- * Commit all of the updated files.
61
+ * Update the cargo lockfiles:
62
+
63
+ ```sh
64
+ bundle exec rake cargo:build
65
+ ```
66
+
67
+ * Commit all of the updated files:
68
+
69
+ ```sh
70
+ git commit -am "Bump to v$PRISM_VERSION"
71
+ ```
42
72
 
43
73
  ## Publishing
44
74
 
45
75
  * Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.
46
- * Run `bundle exec rake release` to publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
47
- * Either download the `wasm` artifact from GitHub actions or generate it yourself with `make wasm`.
48
- * Run `npm publish` to publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `ruby-prism` package to do this.
76
+ * Publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
77
+
78
+ ```sh
79
+ bundle exec rake release
80
+ ```
81
+
82
+ * Generate the `wasm` artifact (or download it from GitHub actions and put it in `javascript/src/prism.wasm`).
83
+
84
+ ```sh
85
+ make wasm
86
+ ```
87
+
88
+ * Publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `@ruby/prism` package to do this.
89
+
90
+ ```sh
91
+ npm publish
92
+ ```
93
+
94
+ * Publish the rust crate to [crates.io](crates.io). Note that you must have access to the `ruby-prism-sys` and `ruby-prism` crates to do this.
95
+
96
+ ```sh
97
+ bundle exec rake cargo:publish:real
98
+ ```
@@ -0,0 +1,19 @@
1
+ # ruby_parser translation
2
+
3
+ Prism ships with the ability to translate its syntax tree into the syntax tree used by the [seattlerb/ruby_parser](https://github.com/seattlerb/ruby_parser) gem. This allows you to use tools built on top of the `ruby_parser` gem with the `prism` parser.
4
+
5
+ ## Usage
6
+
7
+ You can call the `parse` and `parse_file` methods on the `Prism::Translation::RubyParser` module:
8
+
9
+ ```ruby
10
+ filepath = "path/to/file.rb"
11
+ Prism::Translation::RubyParser.parse_file(filepath)
12
+ ```
13
+
14
+ This will return to you `Sexp` objects that mirror the result of calling `RubyParser` methods, as in:
15
+
16
+ ```ruby
17
+ filepath = "path/to/file.rb"
18
+ RubyParser.new.parse(File.read(filepath), filepath)
19
+ ```
@@ -85,6 +85,8 @@ The header is structured like the following table:
85
85
  | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
86
86
  | string | the encoding name |
87
87
  | varsint | the start line |
88
+ | varuint | number of newline offsets |
89
+ | varuint* | newline offsets |
88
90
  | varuint | number of comments |
89
91
  | comment* | comments |
90
92
  | varuint | number of magic comments |