yarp 0.8.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ef0d177f2be8f0d5ecd0ba54901e3cdf17fd6ec6f44a1b04160082ae5a1cb519
-  data.tar.gz: 776a4c7b981d4d6f3848bef030a448ea7d402f6ea75c20fcd87d3cbf935ecfe5
+  metadata.gz: '028bafce371c205e4f9a845aa0328a8911519b064ec92be9a970c0077088ca5c'
+  data.tar.gz: ed653e7a3734884de2f42d1df304be7eb1532b29d7acf63214c747ee84426ec0
 SHA512:
-  metadata.gz: 2e462ba8f51d24c8164ca67804a372d1bea6f6706828801cb2404af02ee183c301e221fef21725d59e013a64652eab06eae84b3e10a427f58ee7e0eed35c3950
-  data.tar.gz: 9594d53c55fc434c92c0601db17c548f473ca62812a0ab57b229d5b9753315198742c83f57a44744b6572def60b4b9659cfa1f93e4d86d66fd6b2f73e0db5234
+  metadata.gz: e33c681424d19626c0ee1d5c21265bcdfce4ec56909a5707d4bf7025ea5356b5ac41b080a1e54a8490124065a1595bd5833af8e488558f0ff01c4f14a92d0430
+  data.tar.gz: d55c0483e944f9a33e76a2baeef11841da8f8bf8755512ff90497d07b0fedf2dc568f6ad8c384831b6b382b1482e34177c6dbd8314d6570542c59faca71ad23e

data/CHANGELOG.md

@@ -6,6 +6,38 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [0.9.0] - 2023-08-25
+
+### Added
+
+- Regular expressions can now be bound by `\n`, `\r`, and a combination of `\r\n`.
+- Strings delimited by `%`, `%q`, and `%Q` can now be bound by `\n`, `\r`, and a combination of `\r\n`.
+- `IntegerNode#value` now returns the value of the integer as a Ruby `Integer`.
+- `FloatNode#value` now returns the value of the float as a Ruby `Float`.
+- `RationalNode#value` now returns the value of the rational as a Ruby `Rational`.
+- `ImaginaryNode#value` now returns the value of the imaginary as a Ruby `Complex`.
+- `ClassNode#name` is now a string that returns the name of just the class, without the namespace.
+- `ModuleNode#name` is now a string that returns the name of just the module, without the namespace.
+- Regular expressions and strings found after a heredoc declaration but before the heredoc body are now parsed correctly.
+- The serialization API now supports shared strings, which should help reduce the size of the serialized AST.
+- `*Node#copy` is introduced, which returns a copy of the node with the given overrides.
+- `Location#copy` is introduced, which returns a copy of the location with the given overrides.
+- `DesugarVisitor` is introduced, which provides a simpler AST for use in tools that want to process fewer node types.
+- `{ClassVariable,Constant,ConstantPath,GlobalVariable,InstanceVariable,LocalVariable}TargetNode` are introduced. These nodes represent the target of writes in locations where a value cannot be provided, like a multi write or a rescue reference.
+- `UntilNode#closing_loc` and `WhileNode#closing_loc` are now provided.
+- `Location#join` is now provided, which joins two locations together.
+- `YARP::parse_lex` and `YARP::parse_lex_file` are introduced to parse and lex in one result.
+
+### Changed
+
+- When there is a magic encoding comment, the encoding of the first token's source string is now properly reencoded.
+- Constants followed by unary `&` are now properly parsed as a call with a passed block argument.
+- Escaping multi-byte characters in a string literal will now properly escape the entire character.
+- `YARP.lex_compat` now has more accurate behavior when a byte-order mark is present in the file.
+- **BREAKING**: `AndWriteNode`, `OrWriteNode`, and `OperatorWriteNode` have been split back up into their `0.7.0` versions.
+- We now properly support spaces between the `encoding` and `=`/`:` in a magic encoding comment.
+- We now properly parse `-> foo: bar do end`.
+
 ## [0.8.0] - 2023-08-18
 
 ### Added
@@ -49,7 +81,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby/yarp/compare/v0.8.0...HEAD
+[unreleased]: https://github.com/ruby/yarp/compare/v0.9.0...HEAD
+[0.9.0]: https://github.com/ruby/yarp/compare/v0.8.0...v0.9.0
 [0.8.0]: https://github.com/ruby/yarp/compare/v0.7.0...v0.8.0
 [0.7.0]: https://github.com/ruby/yarp/compare/v0.6.0...v0.7.0
 [0.6.0]: https://github.com/ruby/yarp/compare/d60531...v0.6.0

data/README.md

@@ -14,7 +14,6 @@ The repository contains the infrastructure for both a shared library (librubypar
 │   ├── lex               runs the lexer on a file or string, prints the tokens, and compares to ripper
 │   └── parse             runs the parser on a file or string and prints the syntax tree
 ├── config.yml            specification for tokens and nodes in the tree
-├── configure.ac          configuration to generate the Makefile
 ├── docs                  documentation about the project
 ├── ext
 │   └── yarp
@@ -30,6 +29,7 @@ The repository contains the infrastructure for both a shared library (librubypar
 │   └── yarp.rb           main entrypoint for the Ruby library
 ├── rakelib               various Rake tasks for the project
 ├── rust
+│   ├── yarp              Rustified crate for the shared library
 │   └── yarp-sys          FFI binding for Rust
 ├── src
 │   ├── enc               various encoding files
@@ -38,8 +38,9 @@ The repository contains the infrastructure for both a shared library (librubypar
 ├── templates             contains ERB templates generated by templates/template.rb
 │   └── template.rb       generates code from the nodes and tokens configured by config.yml
 └── test
-    ├── fixtures          Ruby code used for testing
-    └── snapshots         snapshots of generated syntax trees corresponding to fixtures
+    └── yarp
+        ├── fixtures      Ruby code used for testing
+        └── snapshots     snapshots of generated syntax trees corresponding to fixtures
 ```
 
 ## Getting started

data/config.yml

@@ -399,19 +399,6 @@ nodes:
 
           left and right
           ^^^^^^^^^^^^^^
-  - name: AndWriteNode
-    child_nodes:
-      - name: target
-        type: node
-      - name: value
-        type: node
-      - name: operator_loc
-        type: location
-    comment: |
-      Represents the use of the `&&=` operator.
-
-          target &&= value
-          ^^^^^^^^^^^^^^^^
   - name: ArgumentsNode
     child_nodes:
       - name: arguments
@@ -731,17 +718,66 @@ nodes:
         type: node?
       - name: end_keyword_loc
         type: location
+      - name: name
+        type: string
     comment: |
       Represents a class declaration involving the `class` keyword.
 
           class Foo end
           ^^^^^^^^^^^^^
+  - name: ClassVariableAndWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `&&=` operator for assignment to a class variable.
+
+          @@target &&= value
+          ^^^^^^^^^^^^^^^^
+  - name: ClassVariableOperatorWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: operator
+        type: constant
+    comment: |
+      Represents assigning to a class variable using an operator that isn't `=`.
+
+          @@target += value
+          ^^^^^^^^^^^^^^^^^
+  - name: ClassVariableOrWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `||=` operator for assignment to a class variable.
+
+          @@target ||= value
+          ^^^^^^^^^^^^^^^^^^
   - name: ClassVariableReadNode
     comment: |
       Represents referencing a class variable.
 
           @@foo
           ^^^^^
+  - name: ClassVariableTargetNode
+    comment: |
+      Represents writing to a class variable in a context that doesn't have an explicit value.
+
+          @@foo, @@bar = baz
+          ^^^^^  ^^^^^
   - name: ClassVariableWriteNode
     child_nodes:
       - name: name_loc
@@ -755,6 +791,61 @@ nodes:
 
           @@foo = 1
           ^^^^^^^^^
+  - name: ConstantAndWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `&&=` operator for assignment to a constant.
+
+          Target &&= value
+          ^^^^^^^^^^^^^^^^
+  - name: ConstantOperatorWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: operator
+        type: constant
+    comment: |
+      Represents assigning to a constant using an operator that isn't `=`.
+
+          Target += value
+          ^^^^^^^^^^^^^^^
+  - name: ConstantOrWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `||=` operator for assignment to a constant.
+
+          Target ||= value
+          ^^^^^^^^^^^^^^^^
+  - name: ConstantPathAndWriteNode
+    child_nodes:
+      - name: target
+        type: node
+        kind: ConstantPathNode
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `&&=` operator for assignment to a constant path.
+
+          Parent::Child &&= value
+          ^^^^^^^^^^^^^^^^^^^^^^^
   - name: ConstantPathNode
     child_nodes:
       - name: parent
@@ -768,15 +859,58 @@ nodes:
 
           Foo::Bar
           ^^^^^^^^
-  - name: ConstantPathWriteNode
+  - name: ConstantPathOperatorWriteNode
     child_nodes:
       - name: target
         type: node
         kind: ConstantPathNode
       - name: operator_loc
-        type: location?
+        type: location
+      - name: value
+        type: node
+      - name: operator
+        type: constant
+    comment: |
+      Represents assigning to a constant path using an operator that isn't `=`.
+
+          Parent::Child += value
+          ^^^^^^^^^^^^^^^^^^^^^^
+  - name: ConstantPathOrWriteNode
+    child_nodes:
+      - name: target
+        type: node
+        kind: ConstantPathNode
+      - name: operator_loc
+        type: location
       - name: value
+        type: node
+    comment: |
+      Represents the use of the `||=` operator for assignment to a constant path.
+
+          Parent::Child ||= value
+          ^^^^^^^^^^^^^^^^^^^^^^^
+  - name: ConstantPathTargetNode
+    child_nodes:
+      - name: parent
         type: node?
+      - name: child
+        type: node
+      - name: delimiter_loc
+        type: location
+    comment: |
+      Represents writing to a constant path in a context that doesn't have an explicit value.
+
+          Foo::Foo, Bar::Bar = baz
+          ^^^^^^^^  ^^^^^^^^
+  - name: ConstantPathWriteNode
+    child_nodes:
+      - name: target
+        type: node
+        kind: ConstantPathNode
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
     comment: |
       Represents writing to a constant path.
 
@@ -794,14 +928,20 @@ nodes:
 
           Foo
           ^^^
+  - name: ConstantTargetNode
+    comment: |
+      Represents writing to a constant in a context that doesn't have an explicit value.
+
+          Foo, Bar = baz
+          ^^^  ^^^
   - name: ConstantWriteNode
     child_nodes:
       - name: name_loc
         type: location
       - name: value
-        type: node?
+        type: node
       - name: operator_loc
-        type: location?
+        type: location
     comment: |
       Represents writing to a constant.
 
@@ -1010,20 +1150,67 @@ nodes:
 
           super
           ^^^^^
+  - name: GlobalVariableAndWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `&&=` operator for assignment to a global variable.
+
+          $target &&= value
+          ^^^^^^^^^^^^^^^^^
+  - name: GlobalVariableOperatorWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: operator
+        type: constant
+    comment: |
+      Represents assigning to a global variable using an operator that isn't `=`.
+
+          $target += value
+          ^^^^^^^^^^^^^^^^
+  - name: GlobalVariableOrWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `||=` operator for assignment to a global variable.
+
+          $target ||= value
+          ^^^^^^^^^^^^^^^^^
   - name: GlobalVariableReadNode
     comment: |
       Represents referencing a global variable.
 
           $foo
           ^^^^
+  - name: GlobalVariableTargetNode
+    comment: |
+      Represents writing to a global variable in a context that doesn't have an explicit value.
+
+          $foo, $bar = baz
+          ^^^^  ^^^^
   - name: GlobalVariableWriteNode
     child_nodes:
       - name: name_loc
         type: location
       - name: operator_loc
-        type: location?
+        type: location
       - name: value
-        type: node?
+        type: node
     comment: |
       Represents writing to a global variable.
 
@@ -1109,20 +1296,67 @@ nodes:
 
           case a; in b then c end
                   ^^^^^^^^^^^
+  - name: InstanceVariableAndWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `&&=` operator for assignment to an instance variable.
+
+          @target &&= value
+          ^^^^^^^^^^^^^^^^^
+  - name: InstanceVariableOperatorWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: operator
+        type: constant
+    comment: |
+      Represents assigning to an instance variable using an operator that isn't `=`.
+
+          @target += value
+          ^^^^^^^^^^^^^^^^
+  - name: InstanceVariableOrWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+    comment: |
+      Represents the use of the `||=` operator for assignment to an instance variable.
+
+          @target ||= value
+          ^^^^^^^^^^^^^^^^^
   - name: InstanceVariableReadNode
     comment: |
       Represents referencing an instance variable.
 
           @foo
           ^^^^
+  - name: InstanceVariableTargetNode
+    comment: |
+      Represents writing to an instance variable in a context that doesn't have an explicit value.
+
+          @foo, @bar = baz
+          ^^^^  ^^^^
   - name: InstanceVariableWriteNode
     child_nodes:
       - name: name_loc
         type: location
       - name: value
-        type: node?
+        type: node
       - name: operator_loc
-        type: location?
+        type: location
     comment: |
       Represents writing to an instance variable.
 
@@ -1234,8 +1468,12 @@ nodes:
     child_nodes:
       - name: locals
         type: constant[]
+      - name: operator_loc
+        type: location
       - name: opening_loc
         type: location
+      - name: closing_loc
+        type: location
       - name: parameters
         type: node?
         kind: BlockParametersNode
@@ -1246,6 +1484,59 @@ nodes:
 
           ->(value) { value * 2 }
           ^^^^^^^^^^^^^^^^^^^^^^^
+  - name: LocalVariableAndWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: constant_id
+        type: constant
+      - name: depth
+        type: uint32
+    comment: |
+      Represents the use of the `&&=` operator for assignment to a local variable.
+
+          target &&= value
+          ^^^^^^^^^^^^^^^^
+  - name: LocalVariableOperatorWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: constant_id
+        type: constant
+      - name: operator_id
+        type: constant
+      - name: depth
+        type: uint32
+    comment: |
+      Represents assigning to a local variable using an operator that isn't `=`.
+
+          target += value
+          ^^^^^^^^^^^^^^^
+  - name: LocalVariableOrWriteNode
+    child_nodes:
+      - name: name_loc
+        type: location
+      - name: operator_loc
+        type: location
+      - name: value
+        type: node
+      - name: constant_id
+        type: constant
+      - name: depth
+        type: uint32
+    comment: |
+      Represents the use of the `||=` operator for assignment to a local variable.
+
+          target ||= value
+          ^^^^^^^^^^^^^^^^
   - name: LocalVariableReadNode
     child_nodes:
       - name: constant_id
@@ -1259,6 +1550,17 @@ nodes:
 
           foo
           ^^^
+  - name: LocalVariableTargetNode
+    child_nodes:
+      - name: constant_id
+        type: constant
+      - name: depth
+        type: uint32
+    comment: |
+      Represents writing to a local variable in a context that doesn't have an explicit value.
+
+          foo, bar = baz
+          ^^^  ^^^
   - name: LocalVariableWriteNode
     child_nodes:
       - name: constant_id
@@ -1266,11 +1568,11 @@ nodes:
       - name: depth
         type: uint32
       - name: value
-        type: node?
+        type: node
       - name: name_loc
         type: location
       - name: operator_loc
-        type: location?
+        type: location
     comment: |
       Represents writing to a local variable.
 
@@ -1318,6 +1620,8 @@ nodes:
         type: node?
       - name: end_keyword_loc
         type: location
+      - name: name
+        type: string
     comment: |
       Represents a module declaration involving the `module` keyword.
 
@@ -1376,21 +1680,6 @@ nodes:
 
           $1
           ^^
-  - name: OperatorWriteNode
-    child_nodes:
-      - name: target
-        type: node
-      - name: operator_loc
-        type: location
-      - name: operator
-        type: constant
-      - name: value
-        type: node
-    comment: |
-      Represents the use of an operator on a write.
-
-          target += value
-          ^^^^^^^^^^^^^^^
   - name: OptionalParameterNode
     child_nodes:
       - name: constant_id
@@ -1420,19 +1709,6 @@ nodes:
 
           left or right
           ^^^^^^^^^^^^^
-  - name: OrWriteNode
-    child_nodes:
-      - name: target
-        type: node
-      - name: value
-        type: node
-      - name: operator_loc
-        type: location
-    comment: |
-      Represents the use of the `||=` operator.
-
-          target ||= value
-          ^^^^^^^^^^^^^^^^
   - name: ParametersNode
     child_nodes:
       - name: requireds
@@ -1812,7 +2088,7 @@ nodes:
       - name: opening_loc
         type: location?
       - name: value_loc
-        type: location
+        type: location?
       - name: closing_loc
         type: location?
       - name: unescaped
@@ -1869,6 +2145,8 @@ nodes:
     child_nodes:
       - name: keyword_loc
         type: location
+      - name: closing_loc
+        type: location?
       - name: predicate
         type: node
       - name: statements
@@ -1906,6 +2184,8 @@ nodes:
     child_nodes:
       - name: keyword_loc
         type: location
+      - name: closing_loc
+        type: location?
       - name: predicate
         type: node
       - name: statements

data/docs/configuration.md

@@ -7,6 +7,7 @@ A lot of code in YARP's repository is templated from a single configuration file
 * `java/org/yarp/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
 * `java/org/yarp/Loader.java` - for defining how to deserialize the nodes in Java
 * `java/org/yarp/Nodes.java` - for defining the nodes in Java
+* `lib/yarp/mutation_visitor.rb` - for defining the mutation visitor for the nodes in Ruby
 * `lib/yarp/node.rb` - for defining the nodes in Ruby
 * `lib/yarp/serialize.rb` - for defining how to deserialize the nodes in Ruby
 * `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C

data/docs/ruby_api.md

@@ -20,4 +20,6 @@ The full API is documented below.
 * `YARP.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
 * `YARP.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
 * `YARP.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `YARP.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
+* `YARP.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
 * `YARP.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree

data/docs/serialization.md

@@ -73,7 +73,7 @@ The header is structured like the following table:
 | varint | content pool size |
 
 After the header comes the body of the serialized string.
-The body consistents of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
+The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
 Each node is structured like the following table:
 
 | # bytes | field |

data/docs/testing.md

@@ -12,9 +12,9 @@ These test specific YARP implementation details like comments, errors, and regul
 
 ### Snapshot tests
 
-Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/snapshots/` directory. For example, `test/fixtures/strings.txt` has a corresponding `test/snapshots/strings.txt`.
+Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/yarp/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/yarp/snapshots/` directory. For example, `test/yarp/fixtures/strings.txt` has a corresponding `test/yarp/snapshots/strings.txt`.
 
-If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/snapshots/` directory, one will be created so that it exists for the next test run.
+If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/yarp/snapshots/` directory, one will be created so that it exists for the next test run.
 
 ### Testing against repositories