prism 0.28.0 → 0.30.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7715842445523172c9593bae81ff5c1a889c6735b59f1c5a879fe63bbeacd724
-  data.tar.gz: 755dd34f1e20f374aa2910482a8fac4e709e332d5313a1c80bcc2b3fc9ffbad4
+  metadata.gz: 46cabe6b76e675b905cb9b2941faf00d35c0c5043e480c094952a89404e87587
+  data.tar.gz: 50a764fb701657e5936b764a7438f49ab074f45d5bee3e435bf70c9b891067be
 SHA512:
-  metadata.gz: 37206e932b34156ad6b1679ea379e8665d987bf9673d37a1e5f74c1abb1a442adbdc0d2045d75624e60507794bd8dd56c5b7bbc8864d7ab9cec1f9a6c1c44b05
-  data.tar.gz: 15fc69115a30581010fb65d540f8d2a0c0cd1f76d59754d9b3c1d2f4c2f4354f2f9cbb5087a7989df6b85ff5285248875d6ed40b05d2fca89753f1235948bb0d
+  metadata.gz: 58535a5049cf8d6b8ceb97721135f58ca2fa4e7311cce8c336b6d4d5674de38861a07728651859ffe926bf84db8c14454b543d5d26732174de4c67a2be7cb46b
+  data.tar.gz: 16ae75556351bd4c96a54dbd930e466c40695419c29ba9cf5b048dc1173fc397094974bea313258e7615a27f67c9b55b00d72dcee33ddacc1613169005b0e184

data/CHANGELOG.md

@@ -6,6 +6,44 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [0.30.0] - 2024-06-07
+
+### Added
+
+- More correctly raise mixed encoding errors.
+- Implement ambiguous binary operator warning.
+- Fix up regexp escapes with control and meta characters.
+- Fix up support for the `it` implicit local variable.
+- Heredoc identifiers now properly disallow CLRF.
+- Errors added for void value expressions in begin clauses.
+- Many updates to more closely match the `parser` gem in parser translation.
+- Many errors added for invalid regular expressions.
+
+### Changed
+
+- Handle parser translation missing the `parser` gem.
+- Handle ruby_parser translation missing the `ruby_parser` gem.
+- Various error messages have been updated to more closely match CRuby.
+- `RationalNode` now has a `numerator` and `denominator` field instead of a `numeric` field. For the Ruby API we provide a `RationalNode#numeric` method for backwards-compatibility.
+
+## [0.29.0] - 2024-05-10
+
+### Added
+
+- Added `Prism::CallNode#full_message_loc`, which gives the location including the `=` if there is one.
+- A warning for when `# shareable_constant_value` is not used on its own line.
+- An error for invalid implicit local variable writes.
+- Implicit hash patterns in array patterns are disallowed.
+- We now validate that Unicode escape sequences are not surrogates.
+
+### Changed
+
+- All fields named `operator` have been renamed to `binary_operator` for `*OperatorWriteNode` nodes. This is to make it easier to provide C++ support. In the Ruby API, the old fields are aliased to the new fields with a deprecation warning.
+- Many updated error messages to more closely match CRuby.
+- We ensure keyword parameters do not end in `!` or `?`.
+- Fixed some escaping in string literals with control sequences and hex escapes.
+- Fix a bug with RBS types when used outside the `ruby/prism` codebase.
+
 ## [0.28.0] - 2024-05-03
 
 ### Added
@@ -500,7 +538,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby/prism/compare/v0.28.0...HEAD
+[unreleased]: https://github.com/ruby/prism/compare/v0.30.0...HEAD
+[0.30.0]: https://github.com/ruby/prism/compare/v0.29.0...v0.30.0
+[0.29.0]: https://github.com/ruby/prism/compare/v0.28.0...v0.29.0
 [0.28.0]: https://github.com/ruby/prism/compare/v0.27.0...v0.28.0
 [0.27.0]: https://github.com/ruby/prism/compare/v0.26.0...v0.27.0
 [0.26.0]: https://github.com/ruby/prism/compare/v0.25.0...v0.26.0

data/CONTRIBUTING.md

@@ -10,10 +10,6 @@ The discussions page on the GitHub repository are open. If you have a question o
 
 If you want to contribute code, please first open or contribute to a discussion. A lot of the project is in flux, and we want to make sure that you are contributing to the right place. Once you have a discussion going, you can open a pull request with your changes. We will review your code and get it merged in.
 
-### Ruby Features
-
-Pattern matching and endless method definitions should be avoided as long as the latest TruffleRuby release does not support it.
-
 ## Tests
 
 We could always use more tests! One of the biggest challenges of this project is building up a big test suite. If you want to contribute tests, feel free to open a pull request. These will get merged in as soon as possible.

data/README.md

@@ -40,6 +40,7 @@ The repository contains the infrastructure for both a shared library (libprism)
 ├── rust
 │   ├── ruby-prism        Rustified crate for the shared library
 │   └── ruby-prism-sys    FFI binding for Rust
+├── sample                Sample code that uses the Ruby API for documentation
 ├── sig                   RBS type signatures for the Ruby library
 ├── src
 │   ├── util              various utility files

data/config.yml

@@ -7,13 +7,16 @@ errors:
   - ARGUMENT_BARE_HASH
   - ARGUMENT_BLOCK_FORWARDING
   - ARGUMENT_BLOCK_MULTI
+  - ARGUMENT_CONFLICT_AMPERSAND
+  - ARGUMENT_CONFLICT_STAR
+  - ARGUMENT_CONFLICT_STAR_STAR
   - ARGUMENT_FORMAL_CLASS
   - ARGUMENT_FORMAL_CONSTANT
   - ARGUMENT_FORMAL_GLOBAL
   - ARGUMENT_FORMAL_IVAR
   - ARGUMENT_FORWARDING_UNBOUND
   - ARGUMENT_IN
-  - ARGUMENT_NO_FORWARDING_AMP
+  - ARGUMENT_NO_FORWARDING_AMPERSAND
   - ARGUMENT_NO_FORWARDING_ELLIPSES
   - ARGUMENT_NO_FORWARDING_STAR
   - ARGUMENT_NO_FORWARDING_STAR_STAR
@@ -95,6 +98,7 @@ errors:
   - EXPECT_EXPRESSION_AFTER_SPLAT_HASH
   - EXPECT_EXPRESSION_AFTER_STAR
   - EXPECT_IDENT_REQ_PARAMETER
+  - EXPECT_IN_DELIMITER
   - EXPECT_LPAREN_REQ_PARAMETER
   - EXPECT_MESSAGE
   - EXPECT_RBRACKET
@@ -110,6 +114,7 @@ errors:
   - EXPRESSION_NOT_WRITABLE_FILE
   - EXPRESSION_NOT_WRITABLE_LINE
   - EXPRESSION_NOT_WRITABLE_NIL
+  - EXPRESSION_NOT_WRITABLE_NUMBERED
   - EXPRESSION_NOT_WRITABLE_SELF
   - EXPRESSION_NOT_WRITABLE_TRUE
   - FLOAT_PARSE
@@ -134,6 +139,7 @@ errors:
   - INVALID_BLOCK_EXIT
   - INVALID_CHARACTER
   - INVALID_ENCODING_MAGIC_COMMENT
+  - INVALID_ESCAPE_CHARACTER
   - INVALID_FLOAT_EXPONENT
   - INVALID_LOCAL_VARIABLE_READ
   - INVALID_LOCAL_VARIABLE_WRITE
@@ -142,11 +148,13 @@ errors:
   - INVALID_MULTIBYTE_ESCAPE
   - INVALID_NUMBER_BINARY
   - INVALID_NUMBER_DECIMAL
+  - INVALID_NUMBER_FRACTION
   - INVALID_NUMBER_HEXADECIMAL
   - INVALID_NUMBER_OCTAL
   - INVALID_NUMBER_UNDERSCORE_INNER
   - INVALID_NUMBER_UNDERSCORE_TRAILING
   - INVALID_PERCENT
+  - INVALID_PERCENT_EOF
   - INVALID_PRINTABLE_CHARACTER
   - INVALID_RETRY_AFTER_ELSE
   - INVALID_RETRY_AFTER_ENSURE
@@ -178,15 +186,17 @@ errors:
   - NO_LOCAL_VARIABLE
   - NOT_EXPRESSION
   - NUMBER_LITERAL_UNDERSCORE
+  - NUMBERED_PARAMETER_INNER_BLOCK
   - NUMBERED_PARAMETER_IT
   - NUMBERED_PARAMETER_ORDINARY
-  - NUMBERED_PARAMETER_OUTER_SCOPE
+  - NUMBERED_PARAMETER_OUTER_BLOCK
   - OPERATOR_MULTI_ASSIGN
   - OPERATOR_WRITE_ARGUMENTS
   - OPERATOR_WRITE_BLOCK
   - PARAMETER_ASSOC_SPLAT_MULTI
   - PARAMETER_BLOCK_MULTI
   - PARAMETER_CIRCULAR
+  - PARAMETER_FORWARDING_AFTER_REST
   - PARAMETER_METHOD_NAME
   - PARAMETER_NAME_DUPLICATED
   - PARAMETER_NO_DEFAULT
@@ -196,8 +206,8 @@ errors:
   - PARAMETER_SPLAT_MULTI
   - PARAMETER_STAR
   - PARAMETER_UNEXPECTED_FWD
-  - PARAMETER_WILD_LOOSE_COMMA
   - PARAMETER_UNEXPECTED_NO_KW
+  - PARAMETER_WILD_LOOSE_COMMA
   - PATTERN_CAPTURE_DUPLICATE
   - PATTERN_EXPRESSION_AFTER_BRACKET
   - PATTERN_EXPRESSION_AFTER_COMMA
@@ -209,8 +219,10 @@ errors:
   - PATTERN_EXPRESSION_AFTER_PIPE
   - PATTERN_EXPRESSION_AFTER_RANGE
   - PATTERN_EXPRESSION_AFTER_REST
+  - PATTERN_HASH_IMPLICIT
   - PATTERN_HASH_KEY
   - PATTERN_HASH_KEY_DUPLICATE
+  - PATTERN_HASH_KEY_INTERPOLATED
   - PATTERN_HASH_KEY_LABEL
   - PATTERN_HASH_KEY_LOCALS
   - PATTERN_IDENT_AFTER_HROCKET
@@ -224,6 +236,7 @@ errors:
   - REGEXP_INCOMPAT_CHAR_ENCODING
   - REGEXP_INVALID_UNICODE_RANGE
   - REGEXP_NON_ESCAPED_MBC
+  - REGEXP_PARSE_ERROR
   - REGEXP_TERM
   - REGEXP_UNKNOWN_OPTIONS
   - REGEXP_UTF8_CHAR_NON_UTF8_REGEXP
@@ -264,6 +277,7 @@ errors:
   - WRITE_TARGET_UNEXPECTED
   - XSTRING_TERM
 warnings:
+  - AMBIGUOUS_BINARY_OPERATOR
   - AMBIGUOUS_FIRST_ARGUMENT_MINUS
   - AMBIGUOUS_FIRST_ARGUMENT_PLUS
   - AMBIGUOUS_PREFIX_AMPERSAND
@@ -286,6 +300,7 @@ warnings:
   - KEYWORD_EOL
   - LITERAL_IN_CONDITION_DEFAULT
   - LITERAL_IN_CONDITION_VERBOSE
+  - SHAREABLE_CONSTANT_VALUE_LINE
   - SHEBANG_CARRIAGE_RETURN
   - UNEXPECTED_CARRIAGE_RETURN
   - UNREACHABLE_STATEMENT
@@ -1203,9 +1218,9 @@ nodes:
         type: constant
       - name: write_name
         type: constant
-      - name: operator
+      - name: binary_operator
         type: constant
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
@@ -1361,11 +1376,11 @@ nodes:
         type: constant
       - name: name_loc
         type: location
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
-      - name: operator
+      - name: binary_operator
         type: constant
     comment: |
       Represents assigning to a class variable using an operator that isn't `=`.
@@ -1471,11 +1486,11 @@ nodes:
         type: constant
       - name: name_loc
         type: location
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
-      - name: operator
+      - name: binary_operator
         type: constant
     comment: |
       Represents assigning to a constant using an operator that isn't `=`.
@@ -1559,11 +1574,11 @@ nodes:
       - name: target
         type: node
         kind: ConstantPathNode
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
-      - name: operator
+      - name: binary_operator
         type: constant
     comment: |
       Represents assigning to a constant path using an operator that isn't `=`.
@@ -1866,19 +1881,56 @@ nodes:
     fields:
       - name: index
         type: node
+        comment: |
+          The index expression for `for` loops.
+
+              for i in a end
+                  ^
       - name: collection
         type: node
+        comment: |
+          The collection to iterate over.
+
+              for i in a end
+                       ^
       - name: statements
         type: node?
         kind: StatementsNode
+        comment: |
+          Represents the body of statements to execute for each iteration of the loop.
+
+              for i in a
+                foo(i)
+                ^^^^^^
+              end
       - name: for_keyword_loc
         type: location
+        comment: |
+          The location of the `for` keyword.
+
+              for i in a end
+              ^^^
       - name: in_keyword_loc
         type: location
+        comment: |
+          The location of the `in` keyword.
+
+              for i in a end
+                    ^^
       - name: do_keyword_loc
         type: location?
+        comment: |
+          The location of the `do` keyword, if present.
+
+              for i in a do end
+                         ^^
       - name: end_keyword_loc
         type: location
+        comment: |
+          The location of the `end` keyword.
+
+              for i in a end
+                         ^^^
     comment: |
       Represents the use of the `for` keyword.
 
@@ -1930,11 +1982,11 @@ nodes:
         type: constant
       - name: name_loc
         type: location
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
-      - name: operator
+      - name: binary_operator
         type: constant
     comment: |
       Represents assigning to a global variable using an operator that isn't `=`.
@@ -2266,9 +2318,9 @@ nodes:
         type: location
       - name: block
         type: node?
-      - name: operator
+      - name: binary_operator
         type: constant
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
@@ -2354,11 +2406,11 @@ nodes:
         type: constant
       - name: name_loc
         type: location
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
-      - name: operator
+      - name: binary_operator
         type: constant
     comment: |
       Represents assigning to an instance variable using an operator that isn't `=`.
@@ -2556,6 +2608,12 @@ nodes:
 
           `foo #{bar} baz`
           ^^^^^^^^^^^^^^^^
+  - name: ItLocalVariableReadNode
+    comment: |
+      Represents reading from the implicit `it` local variable.
+
+          -> { it }
+               ^^
   - name: ItParametersNode
     comment: |
       Represents an implicit set of parameters through the use of the `it` keyword within a block or lambda.
@@ -2634,13 +2692,13 @@ nodes:
     fields:
       - name: name_loc
         type: location
-      - name: operator_loc
+      - name: binary_operator_loc
         type: location
       - name: value
         type: node
       - name: name
         type: constant
-      - name: operator
+      - name: binary_operator
         type: constant
       - name: depth
         type: uint32
@@ -2681,10 +2739,6 @@ nodes:
 
               _1     # name `:_1`
 
-          Finally, for the default `it` block parameter, the name is `0it`. This is to distinguish it from an `it` local variable that is explicitly declared.
-
-              it     # name `:0it`
-
       - name: depth
         type: uint32
         comment: |
@@ -3075,6 +3129,8 @@ nodes:
           # On parsing error of `f(**kwargs, ...)` or `f(**nil, ...)`, the keyword_rest value is moved here:
           - KeywordRestParameterNode
           - NoKeywordsParameterNode
+          # On parsing error of `f(..., ...)`, the first forwarding parameter is moved here:
+          - ForwardingParameterNode
       - name: keywords
         type: node[]
         kind:
@@ -3215,8 +3271,21 @@ nodes:
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   - name: RationalNode
     fields:
-      - name: numeric
-        type: node
+      - name: flags
+        type: flags
+        kind: IntegerBaseFlags
+      - name: numerator
+        type: integer
+        comment: |
+          The numerator of the rational number.
+
+              1.5r # numerator 3
+      - name: denominator
+        type: integer
+        comment: |
+          The denominator of the rational number.
+
+              1.5r # denominator 2
     comment: |
       Represents a rational number literal.
 
@@ -3555,7 +3624,7 @@ nodes:
                          ^^^^
       - name: then_keyword_loc
         type: location?
-        comment:
+        comment: |
           The location of the `then` keyword, if present.
 
               unless cond then bar end

data/docs/fuzzing.md

@@ -26,7 +26,7 @@ fuzz
 There are currently three fuzzing targets
 
 - `pm_serialize_parse` (parse)
-- `pm_regexp_named_capture_group_names` (regexp)
+- `pm_regexp_parse` (regexp)
 
 Respectively, fuzzing can be performed with
 

data/docs/ripper_translation.md

@@ -48,3 +48,25 @@ ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
 ```
 
 The exact names of the `on_*` methods are listed in the `Ripper` source.
+
+## Background
+
+It is helpful to understand the differences between the `Ripper` library and the `Prism` library. Both libraries perform parsing and provide you with APIs to manipulate and understand the resulting syntax tree. However, there are a few key differences.
+
+### Design
+
+`Ripper` is a streaming parser. This means as it is parsing Ruby code, it dispatches events back to the consumer. This allows quite a bit of flexibility. You can use it to build your own syntax tree or to find specific patterns in the code. `Prism` on the other hand returns to your the completed syntax tree _before_ it allows you to manipulate it. This means the tree that you get back is the only representation that can be generated by the parser _at parse time_ (but of course can be manipulated later).
+
+### Fields
+
+We use the term "field" to mean a piece of information on a syntax tree node. `Ripper` provides the minimal number of fields to accurately represent the syntax tree for the purposes of compilation/interpretation. For example, in the callbacks for nodes that are based on keywords (`class`, `module`, `for`, `while`, etc.) you are not given the keyword itself, you need to attach it on your own. In other cases, tokens are not necessarily dispatched at all, meaning you need to find them yourself. `Prism` provides the opposite: the maximum number of fields on nodes is provided. As a tradeoff, this requires more memory, but this is chosen to make it easier on consumers.
+
+### Maintainability
+
+The `Ripper` interface is not guaranteed in any way, and tends to change between patch versions of CRuby. This is largely due to the fact that `Ripper` is a by-product of the generated parser, as opposed to its own parser. As an example, in the expression `foo::bar = baz`, there are three different represents possible for the call operator, including:
+
+* `:"::"` - Ruby 1.9 to Ruby 3.1.4
+* `73` - Ruby 3.1.5 to Ruby 3.1.6
+* `[:@op, "::", [lineno, column]]` - Ruby 3.2.0 and later
+
+The `Prism` interface is guaranteed going forward to be the consistent, and the official Ruby syntax tree interface. This means you can rely on this interface without having to worry about individual changes between Ruby versions. It also is a gem, which means it is versioned based on the gem version, as opposed to being versioned based on the Ruby version. Finally, you can use `Prism` to parse multiple versions of Ruby, whereas `Ripper` is tied to the Ruby version it is running on.