prism 0.19.0 → 0.24.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (69) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +102 -1
  3. data/Makefile +5 -0
  4. data/README.md +9 -6
  5. data/config.yml +236 -38
  6. data/docs/build_system.md +19 -2
  7. data/docs/cruby_compilation.md +27 -0
  8. data/docs/parser_translation.md +34 -0
  9. data/docs/parsing_rules.md +19 -0
  10. data/docs/releasing.md +84 -16
  11. data/docs/ruby_api.md +1 -1
  12. data/docs/ruby_parser_translation.md +19 -0
  13. data/docs/serialization.md +19 -5
  14. data/ext/prism/api_node.c +1989 -1525
  15. data/ext/prism/extension.c +130 -30
  16. data/ext/prism/extension.h +2 -2
  17. data/include/prism/ast.h +1700 -505
  18. data/include/prism/defines.h +8 -0
  19. data/include/prism/diagnostic.h +49 -7
  20. data/include/prism/encoding.h +17 -0
  21. data/include/prism/options.h +40 -14
  22. data/include/prism/parser.h +34 -18
  23. data/include/prism/util/pm_buffer.h +9 -0
  24. data/include/prism/util/pm_constant_pool.h +18 -0
  25. data/include/prism/util/pm_newline_list.h +4 -14
  26. data/include/prism/util/pm_strpbrk.h +4 -1
  27. data/include/prism/version.h +2 -2
  28. data/include/prism.h +19 -2
  29. data/lib/prism/debug.rb +11 -5
  30. data/lib/prism/desugar_compiler.rb +225 -80
  31. data/lib/prism/dot_visitor.rb +36 -14
  32. data/lib/prism/dsl.rb +302 -299
  33. data/lib/prism/ffi.rb +107 -76
  34. data/lib/prism/lex_compat.rb +17 -1
  35. data/lib/prism/node.rb +4580 -2607
  36. data/lib/prism/node_ext.rb +27 -4
  37. data/lib/prism/parse_result.rb +75 -29
  38. data/lib/prism/serialize.rb +633 -305
  39. data/lib/prism/translation/parser/compiler.rb +1838 -0
  40. data/lib/prism/translation/parser/lexer.rb +335 -0
  41. data/lib/prism/translation/parser/rubocop.rb +45 -0
  42. data/lib/prism/translation/parser.rb +190 -0
  43. data/lib/prism/translation/parser33.rb +12 -0
  44. data/lib/prism/translation/parser34.rb +12 -0
  45. data/lib/prism/translation/ripper.rb +696 -0
  46. data/lib/prism/translation/ruby_parser.rb +1521 -0
  47. data/lib/prism/translation.rb +11 -0
  48. data/lib/prism.rb +1 -1
  49. data/prism.gemspec +18 -7
  50. data/rbi/prism.rbi +150 -88
  51. data/rbi/prism_static.rbi +15 -3
  52. data/sig/prism.rbs +996 -961
  53. data/sig/prism_static.rbs +123 -46
  54. data/src/diagnostic.c +264 -219
  55. data/src/encoding.c +21 -26
  56. data/src/node.c +2 -6
  57. data/src/options.c +29 -5
  58. data/src/prettyprint.c +176 -44
  59. data/src/prism.c +1499 -564
  60. data/src/serialize.c +35 -21
  61. data/src/token_type.c +353 -4
  62. data/src/util/pm_buffer.c +11 -0
  63. data/src/util/pm_constant_pool.c +37 -11
  64. data/src/util/pm_newline_list.c +6 -15
  65. data/src/util/pm_string.c +0 -7
  66. data/src/util/pm_strpbrk.c +122 -14
  67. metadata +16 -5
  68. data/docs/building.md +0 -29
  69. data/lib/prism/ripper_compat.rb +0 -207
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 72e48ffbc7a58cb622a709f860470c4cf2178113a2ba6a2c9cd4f838229ccff8
4
- data.tar.gz: 709ecc0c274a141a868661f94d28aad3fda3032e1b253b2f699a15ff71200c37
3
+ metadata.gz: 827194fd8b477f9f021677ca386e35fd8300daa86a09fc11fe66a85ee54da97d
4
+ data.tar.gz: 2a44a3cf5e31260a949965aaf2aa5fba0cd8b1ac910ae694d22fd8e42f4d6b9b
5
5
  SHA512:
6
- metadata.gz: 42a21574ff5ca1ce62c836839a8c54ed7d6a3f943cfbbd13d860aa30f3d1cf2b6fd5add3082d7622b30e3403ecbf98a404f3c9ef831744e60d40deabeae44ba2
7
- data.tar.gz: 8ef39e9818ba2f13808a0e1131514709e6c7cbbbf2a0c784d047637d56b0024d35951a852c91d8f29c72477b5e923a6cc2687335cf44ccc761ab2e8e5eb5e521
6
+ metadata.gz: 41dab6cd85f9f57bf2e2256e7b7fd7ac44c46883ad6d79c0056bff3b34500b2ecb28a1febeb1deed5a49b862c4f4382856ad2195639f86e770c006d1b6a41f8e
7
+ data.tar.gz: 0474d64c9bbd3385f65b00bf83e6b2d3c9543785c85b42d50bce59327e6af7a8fc13d54e5b485b6c930997dabe0fd81c43835243fc96fb28beb3523395ddde71
data/CHANGELOG.md CHANGED
@@ -6,6 +6,102 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.24.0] - 2024-02-15
10
+
11
+ ### Added
12
+
13
+ - More support for `Prism::Translation::Ripper` is added.
14
+ - Support multiple versions for `Prism::Translation::Parser`.
15
+ - Improved memory usage in the FFI backend.
16
+ - Very large speed and memory improvements for creating the Ruby AST.
17
+
18
+ ### Changed
19
+
20
+ - Fix location for empty symbol in hash key.
21
+ - Fix parsing a rescue modifier on the value of an assignment when the LHS is a method call with arguments and no parentheses.
22
+
23
+ ## [0.23.0] - 2024-02-14
24
+
25
+ ### Added
26
+
27
+ - More support for `Prism::RipperCompat` is added.
28
+ - A significantly faster offset cache for `Prism::Translation::Parser` is added for files with multibyte characters.
29
+ - `Prism::Translation::RubyParser` is added.
30
+ - `Prism::ConstantPathTarget#full_name` is added.
31
+ - `version: "3.4.0"` is added as an option that is an alias for `version: "latest"`.
32
+ - Four new APIs are added to `Prism::Location`:
33
+ - `Prism::Location#start_code_units_offset`
34
+ - `Prism::Location#end_code_units_offset`
35
+ - `Prism::Location#start_code_units_column`
36
+ - `Prism::Location#end_code_units_column`
37
+ - Invalid multibyte characters are now validated within strings, lists, and heredocs.
38
+
39
+ ### Changed
40
+
41
+ - When defining `def !@`, the `name_loc` was previously only pointing to `!`, but now includes the `@`. The `name` is the same.
42
+ - `Prism::RipperCompat` has been moved to `Prism::Translation::Ripper`.
43
+ - Many of the error messages that prism produces have been changed to match the error messages that CRuby produces.
44
+
45
+ ## [0.22.0] - 2024-02-07
46
+
47
+ ### Added
48
+
49
+ - More support for `Prism::RipperCompat` is added.
50
+ - Support for Ruby 2.7 has been added, and the minimum Ruby requirement has been lowered to 2.7.
51
+
52
+ ### Changed
53
+
54
+ - The error for an invalid source encoding has a new `:argument` level to indicate it raises an argument error.
55
+ - `BeginNode` nodes that are used when a class, singleton class, module, method definition, or block have an inline `rescue`/`ensure`/`else` now have their opening locations set to the beginning of the respective keyword.
56
+ - Improved error messages for invalid characters.
57
+ - `Prism.parse_file` and similar APIs will raise more appropriate errors when the file does not exist or cannot be mapped.
58
+ - Correctly handle the `recover` parameter for `Prism::Translation::Parser`.
59
+
60
+ ## [0.21.0] - 2024-02-05
61
+
62
+ ### Added
63
+
64
+ - Add the `pm_constant_pool_find` API for finding a constant.
65
+
66
+ ### Changed
67
+
68
+ - Fixes for `Prism::Translation::Parser`.
69
+ - Ensure all errors flow through `parser.diagnostics.process`.
70
+ - Fix the find pattern node.
71
+ - Fix block forwarding with `NumberedParametersNode`.
72
+ - Ensure we can parse strings with invalid bytes for the encoding.
73
+ - Fix hash pairs in pattern matching.
74
+ - Properly reject operator writes on operator calls, e.g., `a.+ -= b`.
75
+ - Fix multi-byte escapes.
76
+ - Handle missing body in `begin` within the receiver of a method call.
77
+
78
+ ## [0.20.0] - 2024-02-01
79
+
80
+ ### Added
81
+
82
+ - String literal hash keys are now marked as frozen as static literal.
83
+ - `IndexTargetNode` now always has the `ATTRIBUTE_WRITE` flag.
84
+ - `Call*Node` nodes now have an `IGNORE_VISIBILITY` flag.
85
+ - We now support the `it` default parameter.
86
+ - Errors and warnings now have levels associated with them.
87
+ - Symbols now have correct encoding flags.
88
+ - We have now merged `parser-prism` in, which provides translation to the `whitequark/parser` AST.
89
+ - We now emit errors for invalid method definition receivers.
90
+
91
+ ### Changed
92
+
93
+ - We now emit errors on invalid pinned local variables.
94
+ - When passed scopes, it is now assumed that the innermost scope is the current binding.
95
+ - We now provide better error recovery for non terminated heredocs.
96
+ - Fix for `RationalNode#value` for non-decimal integers.
97
+ - Unary symbols `!@` and `~@` now unescape to `!` and `~`, respectively.
98
+ - `frozen_string_literal: false` now works properly.
99
+
100
+ ### Removed
101
+
102
+ - We've removed the `locals_body_index` field.
103
+ - We've removed the `verbose` option on the various parse APIs. Warnings are now always emitted with their associated level so that consumers can decide how to handle them.
104
+
9
105
  ## [0.19.0] - 2023-12-14
10
106
 
11
107
  ### Added
@@ -296,7 +392,12 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
296
392
 
297
393
  - 🎉 Initial release! 🎉
298
394
 
299
- [unreleased]: https://github.com/ruby/prism/compare/v0.19.0...HEAD
395
+ [unreleased]: https://github.com/ruby/prism/compare/v0.24.0...HEAD
396
+ [0.24.0]: https://github.com/ruby/prism/compare/v0.23.0...v0.24.0
397
+ [0.23.0]: https://github.com/ruby/prism/compare/v0.22.0...v0.23.0
398
+ [0.22.0]: https://github.com/ruby/prism/compare/v0.21.0...v0.22.0
399
+ [0.21.0]: https://github.com/ruby/prism/compare/v0.20.0...v0.21.0
400
+ [0.20.0]: https://github.com/ruby/prism/compare/v0.19.0...v0.20.0
300
401
  [0.19.0]: https://github.com/ruby/prism/compare/v0.18.0...v0.19.0
301
402
  [0.18.0]: https://github.com/ruby/prism/compare/v0.17.1...v0.18.0
302
403
  [0.17.1]: https://github.com/ruby/prism/compare/v0.17.0...v0.17.1
data/Makefile CHANGED
@@ -25,6 +25,7 @@ all: shared static
25
25
  shared: build/libprism.$(SOEXT)
26
26
  static: build/libprism.a
27
27
  wasm: javascript/src/prism.wasm
28
+ java-wasm: java-wasm/src/test/resources/prism.wasm
28
29
 
29
30
  build/libprism.$(SOEXT): $(SHARED_OBJECTS)
30
31
  $(ECHO) "linking $@"
@@ -38,6 +39,10 @@ javascript/src/prism.wasm: Makefile $(SOURCES) $(HEADERS)
38
39
  $(ECHO) "building $@"
39
40
  $(Q) $(WASI_SDK_PATH)/bin/clang --sysroot=$(WASI_SDK_PATH)/share/wasi-sysroot/ $(DEBUG_FLAGS) -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(CFLAGS) -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -o $@ $(SOURCES)
40
41
 
42
+ java-wasm/src/test/resources/prism.wasm: Makefile $(SOURCES) $(HEADERS)
43
+ $(ECHO) "building $@"
44
+ $(Q) $(WASI_SDK_PATH)/bin/clang $(DEBUG_FLAGS) -DPRISM_EXPORT_SYMBOLS -D_WASI_EMULATED_MMAN -lwasi-emulated-mman $(CPPFLAGS) $(CFLAGS) -Wl,--export-all -Wl,--no-entry -mexec-model=reactor -lc++ -lc++abi -o $@ $(SOURCES)
45
+
41
46
  build/shared/%.o: src/%.c Makefile $(HEADERS)
42
47
  $(ECHO) "compiling $@"
43
48
  $(Q) mkdir -p $(@D)
data/README.md CHANGED
@@ -32,8 +32,8 @@ The repository contains the infrastructure for both a shared library (libprism)
32
32
  │   └── prism.rb main entrypoint for the Ruby library
33
33
  ├── rakelib various Rake tasks for the project
34
34
  ├── rust
35
- │   ├── prism Rustified crate for the shared library
36
- │   └── prism-sys FFI binding for Rust
35
+ │   ├── ruby-prism Rustified crate for the shared library
36
+ │   └── ruby-prism-sys FFI binding for Rust
37
37
  ├── src
38
38
  │   ├── enc various encoding files
39
39
  │   ├── util various utility files
@@ -50,10 +50,9 @@ The repository contains the infrastructure for both a shared library (libprism)
50
50
 
51
51
  To compile the shared library, you will need:
52
52
 
53
- * A C99 compiler
54
- * autotools (autoconf, automake, libtool)
53
+ * C99 compiler
55
54
  * make
56
- * Ruby 3.3.0-preview1 or later
55
+ * Ruby 2.7.0 or later
57
56
 
58
57
  Once you have these dependencies, run:
59
58
 
@@ -79,8 +78,9 @@ to see the syntax tree for the expression `1 + 2`.
79
78
 
80
79
  See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additionally have documentation about the overall design of the project as well as various subtopics.
81
80
 
82
- * [Building](docs/building.md)
81
+ * [Build system](docs/build_system.md)
83
82
  * [Configuration](docs/configuration.md)
83
+ * [CRuby compilation](docs/cruby_compilation.md)
84
84
  * [Design](docs/design.md)
85
85
  * [Encoding](docs/encoding.md)
86
86
  * [Fuzzing](docs/fuzzing.md)
@@ -88,8 +88,11 @@ See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additio
88
88
  * [JavaScript](docs/javascript.md)
89
89
  * [Local variable depth](docs/local_variable_depth.md)
90
90
  * [Mapping](docs/mapping.md)
91
+ * [Parser translation](docs/parser_translation.md)
92
+ * [Parsing rules](docs/parsing_rules.md)
91
93
  * [Releasing](docs/releasing.md)
92
94
  * [Ripper](docs/ripper.md)
93
95
  * [Ruby API](docs/ruby_api.md)
96
+ * [RubyParser translation](docs/ruby_parser_translation.md)
94
97
  * [Serialization](docs/serialization.md)
95
98
  * [Testing](docs/testing.md)