prism 0.18.0 → 0.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec5c0e5a9423ea40bad7c6ea608cb43dc0250c4d7119b4960271f547697540c1
-  data.tar.gz: d505901505b8c4eee195171721d68fe05dde76de171e5a80115a301ba030cc34
+  metadata.gz: 72e48ffbc7a58cb622a709f860470c4cf2178113a2ba6a2c9cd4f838229ccff8
+  data.tar.gz: 709ecc0c274a141a868661f94d28aad3fda3032e1b253b2f699a15ff71200c37
 SHA512:
-  metadata.gz: 3f3a4b92227885288d47e147f72b5cd2a27886222339ac08426a32ffe980b5b1397cf0356db93fcbdb2f6a3d16b1aa8db7185acbfe2a92473ef6adcf2aeda93e
-  data.tar.gz: 20528ed404ebdfb5e2093054cd7d1b28fed5f8d95c84df217eee9759ca25603cbd381943db985e17b8e4851756605aa1fb2fe9d41c0d050dfee9d770ffd915d7
+  metadata.gz: 42a21574ff5ca1ce62c836839a8c54ed7d6a3f943cfbbd13d860aa30f3d1cf2b6fd5add3082d7622b30e3403ecbf98a404f3c9ef831744e60d40deabeae44ba2
+  data.tar.gz: 8ef39e9818ba2f13808a0e1131514709e6c7cbbbf2a0c784d047637d56b0024d35951a852c91d8f29c72477b5e923a6cc2687335cf44ccc761ab2e8e5eb5e521

data/CHANGELOG.md

@@ -6,6 +6,35 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [0.19.0] - 2023-12-14
+
+### Added
+
+- `ArrayNode` now has a `contains_splat?` flag if it has a splatted element in it.
+- All of the remaining encodings have been implemented.
+- Allow forwarding `&` in a method that has a `...` parameter.
+- Many statements that are found in non-statement positions are being properly rejected now.
+- Void values are now properly checked.
+- Referencing a parameter in its own default value is now properly rejected.
+- `DATA`/`__END__` is now parsed as its own field on parse result (`data_loc`) as opposed to as a comment.
+- Blank `*` now properly forwards into arrays.
+- `ImplicitRestNode` is introduced to represent the implicit rest of a destructure.
+- We now support negative start lines.
+- `StringNode#heredoc?`, `InterpolatedStringNode#heredoc?`, `XStringNode#heredoc?`, and `InterpolatedXStringNode#heredoc?` are introduced.
+- `NumberedParametersNode` is introduced to represent the implicit set of parameters when numbered parameters are used.
+- `Prism::parse_success?` and `Prism::parse_failure?` are introduced to bypass reifying the AST.
+- We now emit a warning for constant assignments in method definitions.
+- We now provide flags on strings and xstrings to indicate the correct encoding.
+- The hash pattern `rest` field now more accurately parses `**` and `**nil`.
+- The equality operators are now properly parsed as non-associative.
+
+### Changed
+
+- **BREAKING**: Many fields have changed positions within their nodes. This impacts the C API and the Ruby API if you are manually creating nodes through the initializer.
+- **BREAKING**: Almost all of the error messages have been updated to begin with lowercase characters to match ruby/spec.
+- Unterminated strings with only plain content are now always `StringNode` as opposed to `InterpolatedStringNode`
+- **BREAKING**: Call node has been split up when it is in the target position into `CallTargetNode` and `IndexTargetNode`.
+
 ## [0.18.0] - 2023-11-21
 
 ### Added
@@ -267,7 +296,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby/prism/compare/v0.18.0...HEAD
+[unreleased]: https://github.com/ruby/prism/compare/v0.19.0...HEAD
+[0.19.0]: https://github.com/ruby/prism/compare/v0.18.0...v0.19.0
 [0.18.0]: https://github.com/ruby/prism/compare/v0.17.1...v0.18.0
 [0.17.1]: https://github.com/ruby/prism/compare/v0.17.0...v0.17.1
 [0.17.0]: https://github.com/ruby/prism/compare/v0.16.0...v0.17.0

data/README.md

@@ -64,7 +64,7 @@ bundle install
 to fetch the Ruby dependencies. Finally, run:
 
 ```
-rake compile
+bundle exec rake compile
 ```
 
 to compile the shared library. It will be built in the `build` directory. To test that everything is working, run:
@@ -86,6 +86,7 @@ See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additio
 * [Fuzzing](docs/fuzzing.md)
 * [Heredocs](docs/heredocs.md)
 * [JavaScript](docs/javascript.md)
+* [Local variable depth](docs/local_variable_depth.md)
 * [Mapping](docs/mapping.md)
 * [Releasing](docs/releasing.md)
 * [Ripper](docs/ripper.md)

data/config.yml

@@ -331,27 +331,46 @@ tokens:
 flags:
   - name: ArgumentsNodeFlags
     values:
-      - name: KEYWORD_SPLAT
+      - name: CONTAINS_KEYWORD_SPLAT
         comment: "if arguments contain keyword splat"
     comment: Flags for arguments nodes.
+  - name: ArrayNodeFlags
+    values:
+      - name: CONTAINS_SPLAT
+        comment: "if array contains splat nodes"
+    comment: Flags for array nodes.
   - name: CallNodeFlags
     values:
       - name: SAFE_NAVIGATION
         comment: "&. operator"
       - name: VARIABLE_CALL
         comment: "a call that could have been a local variable"
+      - name: ATTRIBUTE_WRITE
+        comment: "a call that is an attribute write, so the value being written should be returned"
     comment: Flags for call nodes.
+  - name: EncodingFlags
+    values:
+      - name: FORCED_UTF8_ENCODING
+        comment: "internal bytes forced the encoding to UTF-8"
+      - name: FORCED_BINARY_ENCODING
+        comment: "internal bytes forced the encoding to binary"
+    comment: Flags for nodes that have unescaped content.
   - name: IntegerBaseFlags
     values:
       - name: BINARY
         comment: "0b prefix"
-      - name: OCTAL
-        comment: "0o or 0 prefix"
       - name: DECIMAL
         comment: "0d or no prefix"
+      - name: OCTAL
+        comment: "0o or 0 prefix"
       - name: HEXADECIMAL
         comment: "0x prefix"
     comment: Flags for integer nodes that correspond to the base of the integer.
+  - name: KeywordHashNodeFlags
+    values:
+      - name: STATIC_KEYS
+        comment: "a keyword hash which only has `AssocNode` elements all with static literal keys, which means the elements can be treated as keyword arguments"
+    comment: Flags for keyword hash nodes.
   - name: LoopFlags
     values:
       - name: BEGIN_MODIFIER
@@ -380,12 +399,31 @@ flags:
         comment: "s - forces the Windows-31J encoding"
       - name: UTF_8
         comment: "u - forces the UTF-8 encoding"
+      - name: FORCED_UTF8_ENCODING
+        comment: "internal bytes forced the encoding to UTF-8"
+      - name: FORCED_BINARY_ENCODING
+        comment: "internal bytes forced the encoding to binary"
+      - name: FORCED_US_ASCII_ENCODING
+        comment: "internal bytes forced the encoding to US-ASCII"
     comment: Flags for regular expression and match last line nodes.
   - name: StringFlags
     values:
+      - name: FORCED_UTF8_ENCODING
+        comment: "internal bytes forced the encoding to UTF-8"
+      - name: FORCED_BINARY_ENCODING
+        comment: "internal bytes forced the encoding to binary"
       - name: FROZEN
         comment: "frozen by virtue of a `frozen_string_literal` comment"
     comment: Flags for string nodes.
+  - name: SymbolFlags
+    values:
+      - name: FORCED_UTF8_ENCODING
+        comment: "internal bytes forced the encoding to UTF-8"
+      - name: FORCED_BINARY_ENCODING
+        comment: "internal bytes forced the encoding to binary"
+      - name: FORCED_US_ASCII_ENCODING
+        comment: "internal bytes forced the encoding to US-ASCII"
+    comment: Flags for symbol nodes.
 nodes:
   - name: AliasGlobalVariableNode
     fields:
@@ -441,11 +479,11 @@ nodes:
           ^^^^^^^^^^^^^^
   - name: ArgumentsNode
     fields:
-      - name: arguments
-        type: node[]
       - name: flags
         type: flags
         kind: ArgumentsNodeFlags
+      - name: arguments
+        type: node[]
     comment: |
       Represents a set of arguments to a method or a keyword.
 
@@ -453,6 +491,9 @@ nodes:
                  ^^^^^^^^^^^^^
   - name: ArrayNode
     fields:
+      - name: flags
+        type: flags
+        kind: ArrayNodeFlags
       - name: elements
         type: node[]
       - name: opening_loc
@@ -579,9 +620,10 @@ nodes:
     fields:
       - name: locals
         type: constant[]
+      - name: locals_body_index
+        type: uint32
       - name: parameters
         type: node?
-        kind: BlockParametersNode
       - name: body
         type: node?
       - name: opening_loc
@@ -641,15 +683,15 @@ nodes:
           ^^^^^^^^^
   - name: CallAndWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
         type: location?
       - name: message_loc
         type: location?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: read_name
         type: constant
       - name: write_name
@@ -665,10 +707,15 @@ nodes:
           ^^^^^^^^^^^^^^^^^
   - name: CallNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
         type: location?
+      - name: name
+        type: constant
       - name: message_loc
         type: location?
       - name: opening_loc
@@ -680,11 +727,6 @@ nodes:
         type: location?
       - name: block
         type: node?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
-      - name: name
-        type: constant
     comment: |
       Represents a method call, in all of the various forms that can take.
 
@@ -707,15 +749,15 @@ nodes:
           ^^^^^^^^
   - name: CallOperatorWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
         type: location?
       - name: message_loc
         type: location?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: read_name
         type: constant
       - name: write_name
@@ -733,15 +775,15 @@ nodes:
           ^^^^^^^^^^^^^^
   - name: CallOrWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
         type: location?
       - name: message_loc
         type: location?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: read_name
         type: constant
       - name: write_name
@@ -755,6 +797,32 @@ nodes:
 
           foo.bar ||= value
           ^^^^^^^^^^^^^^^^^
+  - name: CallTargetNode
+    fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
+      - name: receiver
+        type: node
+      - name: call_operator_loc
+        type: location
+      - name: name
+        type: constant
+      - name: message_loc
+        type: location
+    comment: |
+      Represents assigning to a method call.
+
+          foo.bar, = 1
+          ^^^^^^^
+
+          begin
+          rescue => foo.bar
+                    ^^^^^^^
+          end
+
+          for foo.bar in baz do end
+              ^^^^^^^
   - name: CapturePatternNode
     fields:
       - name: value
@@ -1096,6 +1164,8 @@ nodes:
         type: node?
       - name: locals
         type: constant[]
+      - name: locals_body_index
+        type: uint32
       - name: def_keyword_loc
         type: location
       - name: operator_loc
@@ -1219,15 +1289,15 @@ nodes:
                  ^^^^^^^^^^^^^^^^^^^^
   - name: FlipFlopNode
     fields:
+      - name: flags
+        type: flags
+        kind: RangeFlags
       - name: left
         type: node?
       - name: right
         type: node?
       - name: operator_loc
         type: location
-      - name: flags
-        type: flags
-        kind: RangeFlags
     comment: |
       Represents the use of the `..` or `...` operators to create flip flops.
 
@@ -1445,6 +1515,21 @@ nodes:
 
           { Foo: }
             ^^^^
+  - name: ImplicitRestNode
+    comment: |
+      Represents using a trailing comma to indicate an implicit rest parameter.
+
+          foo { |bar,| }
+                    ^
+
+          foo in [bar,]
+                     ^
+
+          for foo, in bar do end
+                 ^
+
+          foo, = bar
+             ^
   - name: InNode
     fields:
       - name: pattern
@@ -1463,6 +1548,9 @@ nodes:
                   ^^^^^^^^^^^
   - name: IndexAndWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
@@ -1476,9 +1564,6 @@ nodes:
         type: location
       - name: block
         type: node?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: operator_loc
         type: location
       - name: value
@@ -1490,6 +1575,9 @@ nodes:
           ^^^^^^^^^^^^^^^^^^^^^^
   - name: IndexOperatorWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
@@ -1503,9 +1591,6 @@ nodes:
         type: location
       - name: block
         type: node?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: operator
         type: constant
       - name: operator_loc
@@ -1519,6 +1604,9 @@ nodes:
           ^^^^^^^^^^^^^^^^^^^^^
   - name: IndexOrWriteNode
     fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
       - name: receiver
         type: node?
       - name: call_operator_loc
@@ -1532,9 +1620,6 @@ nodes:
         type: location
       - name: block
         type: node?
-      - name: flags
-        type: flags
-        kind: CallNodeFlags
       - name: operator_loc
         type: location
       - name: value
@@ -1544,6 +1629,35 @@ nodes:
 
           foo.bar[baz] ||= value
           ^^^^^^^^^^^^^^^^^^^^^^
+  - name: IndexTargetNode
+    fields:
+      - name: flags
+        type: flags
+        kind: CallNodeFlags
+      - name: receiver
+        type: node
+      - name: opening_loc
+        type: location
+      - name: arguments
+        type: node?
+        kind: ArgumentsNode
+      - name: closing_loc
+        type: location
+      - name: block
+        type: node?
+    comment: |
+      Represents assigning to an index.
+
+          foo[bar], = 1
+          ^^^^^^^^
+
+          begin
+          rescue => foo[bar]
+                    ^^^^^^^^
+          end
+
+          for foo[bar] in baz do end
+              ^^^^^^^^
   - name: InstanceVariableAndWriteNode
     fields:
       - name: name
@@ -1636,15 +1750,15 @@ nodes:
           ^
   - name: InterpolatedMatchLastLineNode
     fields:
+      - name: flags
+        type: flags
+        kind: RegularExpressionFlags
       - name: opening_loc
         type: location
       - name: parts
         type: node[]
       - name: closing_loc
         type: location
-      - name: flags
-        type: flags
-        kind: RegularExpressionFlags
     newline: parts
     comment: |
       Represents a regular expression literal that contains interpolation that
@@ -1655,15 +1769,15 @@ nodes:
              ^^^^^^^^^^^^^^^^
   - name: InterpolatedRegularExpressionNode
     fields:
+      - name: flags
+        type: flags
+        kind: RegularExpressionFlags
       - name: opening_loc
         type: location
       - name: parts
         type: node[]
       - name: closing_loc
         type: location
-      - name: flags
-        type: flags
-        kind: RegularExpressionFlags
     newline: parts
     comment: |
       Represents a regular expression literal that contains interpolation.
@@ -1714,6 +1828,9 @@ nodes:
           ^^^^^^^^^^^^^^^^
   - name: KeywordHashNode
     fields:
+      - name: flags
+        type: flags
+        kind: KeywordHashNodeFlags
       - name: elements
         type: node[]
     comment: |
@@ -1739,6 +1856,8 @@ nodes:
     fields:
       - name: locals
         type: constant[]
+      - name: locals_body_index
+        type: uint32
       - name: operator_loc
         type: location
       - name: opening_loc
@@ -1747,7 +1866,6 @@ nodes:
         type: location
       - name: parameters
         type: node?
-        kind: BlockParametersNode
       - name: body
         type: node?
     comment: |
@@ -1851,6 +1969,9 @@ nodes:
           ^^^^^^^
   - name: MatchLastLineNode
     fields:
+      - name: flags
+        type: flags
+        kind: RegularExpressionFlags
       - name: opening_loc
         type: location
       - name: content_loc
@@ -1859,9 +1980,6 @@ nodes:
         type: location
       - name: unescaped
         type: string
-      - name: flags
-        type: flags
-        kind: RegularExpressionFlags
     comment: |
       Represents a regular expression literal used in the predicate of a
       conditional to implicitly match against the last line read by an IO
@@ -1999,6 +2117,16 @@ nodes:
           def a(**nil)
                 ^^^^^
           end
+  - name: NumberedParametersNode
+    fields:
+      - name: maximum
+        type: uint8
+    comment: |
+      Represents an implicit set of parameters through the use of numbered
+      parameters within a block or lambda.
+
+          -> { _1 + _2 }
+          ^^^^^^^^^^^^^^
   - name: NumberedReferenceReadNode
     fields:
       - name: number
@@ -2059,7 +2187,6 @@ nodes:
         type: node[]
       - name: rest
         type: node?
-        kind: RestParameterNode
       - name: posts
         type: node[]
       - name: keywords
@@ -2159,15 +2286,15 @@ nodes:
     comment: The top level node of any parse tree.
   - name: RangeNode
     fields:
+      - name: flags
+        type: flags
+        kind: RangeFlags
       - name: left
         type: node?
       - name: right
         type: node?
       - name: operator_loc
         type: location
-      - name: flags
-        type: flags
-        kind: RangeFlags
     comment: |
       Represents the use of the `..` or `...` operators.
 
@@ -2193,6 +2320,9 @@ nodes:
           ^^^^
   - name: RegularExpressionNode
     fields:
+      - name: flags
+        type: flags
+        kind: RegularExpressionFlags
       - name: opening_loc
         type: location
       - name: content_loc
@@ -2201,9 +2331,6 @@ nodes:
         type: location
       - name: unescaped
         type: string
-      - name: flags
-        type: flags
-        kind: RegularExpressionFlags
     comment: |
       Represents a regular expression literal with no interpolation.
 
@@ -2418,6 +2545,9 @@ nodes:
           ^^^^^^^^^^^^^^
   - name: SymbolNode
     fields:
+      - name: flags
+        type: flags
+        kind: SymbolFlags
       - name: opening_loc
         type: location?
       - name: value_loc
@@ -2478,6 +2608,9 @@ nodes:
           ^^^^^^^^^^^^^^^^^^^^^^^
   - name: UntilNode
     fields:
+      - name: flags
+        type: flags
+        kind: LoopFlags
       - name: keyword_loc
         type: location
       - name: closing_loc
@@ -2487,9 +2620,6 @@ nodes:
       - name: statements
         type: node?
         kind: StatementsNode
-      - name: flags
-        type: flags
-        kind: LoopFlags
     newline: predicate
     comment: |
       Represents the use of the `until` keyword, either in the block form or the modifier form.
@@ -2517,6 +2647,9 @@ nodes:
           end
   - name: WhileNode
     fields:
+      - name: flags
+        type: flags
+        kind: LoopFlags
       - name: keyword_loc
         type: location
       - name: closing_loc
@@ -2526,9 +2659,6 @@ nodes:
       - name: statements
         type: node?
         kind: StatementsNode
-      - name: flags
-        type: flags
-        kind: LoopFlags
     newline: predicate
     comment: |
       Represents the use of the `while` keyword, either in the block form or the modifier form.
@@ -2540,6 +2670,9 @@ nodes:
           ^^^^^^^^^^^^^^^^^^^^
   - name: XStringNode
     fields:
+      - name: flags
+        type: flags
+        kind: EncodingFlags
       - name: opening_loc
         type: location
       - name: content_loc

data/docs/building.md

@@ -1,7 +1,6 @@
 # Building
 
-The following describes how to build prism from source.
-This comes directly from the [Makefile](../Makefile).
+The following describes how to build prism from source. This comes directly from the [Makefile](../Makefile).
 
 ## Common
 
@@ -20,3 +19,11 @@ If you want to build prism as a shared library and link against it, you should c
 
 * `-fPIC -shared` - Compile as a shared library
 * `-DPRISM_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)
+
+## Flags
+
+`make` respects the `MAKEFLAGS` environment variable. As such, to speed up the build you can run:
+
+```
+MAKEFLAGS="-j10" bundle exec rake compile
+```

data/docs/configuration.md

@@ -50,14 +50,15 @@ Optionally, every node can define a `child_nodes` key that is an array. This arr
 
 The available values for `type` are:
 
-* `node` - A child node that is a node itself. This is a `pm_node_t *` in C.
-* `node?` - A child node that is optionally present. This is also a `pm_node_t *` in C, but can be `NULL`.
-* `node[]` - A child node that is an array of nodes. This is a `pm_node_list_t` in C.
-* `string` - A child node that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is a `pm_string_t` in C.
-* `constant` - A variable-length integer that represents an index in the constant pool. This is a `pm_constant_id_t` in C.
-* `constant[]` - A child node that is an array of constants. This is a `pm_constant_id_list_t` in C.
-* `location` - A child node that is a location. This is a `pm_location_t` in C.
-* `location?` - A child node that is a location that is optionally present. This is a `pm_location_t` in C, but if the value is not present then the `start` and `end` fields will be `NULL`.
-* `uint32` - A child node that is a 32-bit unsigned integer. This is a `uint32_t` in C.
+* `node` - A field that is a node. This is a `pm_node_t *` in C.
+* `node?` - A field that is a node that is optionally present. This is also a `pm_node_t *` in C, but can be `NULL`.
+* `node[]` - A field that is an array of nodes. This is a `pm_node_list_t` in C.
+* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is a `pm_string_t` in C.
+* `constant` - A field that is an integer that represents an index in the constant pool. This is a `pm_constant_id_t` in C.
+* `constant[]` - A field that is an array of constants. This is a `pm_constant_id_list_t` in C.
+* `location` - A field that is a location. This is a `pm_location_t` in C.
+* `location?` - A field that is a location that is optionally present. This is a `pm_location_t` in C, but if the value is not present then the `start` and `end` fields will be `NULL`.
+* `uint8` - A field that is an 8-bit unsigned integer. This is a `uint8_t` in C.
+* `uint32` - A field that is a 32-bit unsigned integer. This is a `uint32_t` in C.
 
 If the type is `node` or `node?` then the value also accepts an optional `kind` key (a string). This key is expected to match to the name of another node type within `config.yml`. This changes a couple of places where code is templated out to use the more specific struct name instead of the generic `pm_node_t`. For example, with `kind: StatementsNode` the `pm_node_t *` in C becomes a `pm_statements_node_t *`.