syntax_tree 5.3.0 → 6.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3261679bf963ba7263bb3ae21e826ed1fa7a28c6afbf2a2b65b42101be16b7ac
-  data.tar.gz: 653c6c79a34db7686ceeadf83f0d8adb47093d1ab1d946636008ee4c31144258
+  metadata.gz: eb083ad4f07db18f2d09a8fef263962bd3593b8e93be19d241c7f4cc254a02bb
+  data.tar.gz: 68c448add2745ffbb26f5200b888e241f8ac56d2f8cb7c6ee6ae675dffb0f3b6
 SHA512:
-  metadata.gz: 9cefe2a22c594efc74aa0fbe300c986a414c890efb7204633481345ceb9b0bd3578d836046ca64e289621ce31ea343f0da40db1f9b3d2d7dbd7f0a11b020c725
-  data.tar.gz: 628ed098064cb9e6dc720666d59b5e9d8653f7e06cb7e0774887dfeb4ed4a5bdb217ea644b93f52a3730e2dfdc14334c0690018d145728a89fa99cbd041e4301
+  metadata.gz: 1e930dfe285bc15786b4d4de401144cc4b5ae90a1724ea9d5d9065b556ec7c9a238e4e9dfa47d72aff563bbc5a07269b2027916ab5e74bdf48896ff65c811e68
+  data.tar.gz: e148cc8409cbe08e1fcbc72f9cf5018d018264f996b79c9093fc51e588ea8836dbccbaa320f423a013341031369ba2047a0014c47116a65ca10f9b7add3ea9ec

data/.rubocop.yml

@@ -8,7 +8,6 @@ AllCops:
   TargetRubyVersion: 2.7
   Exclude:
   - '{.git,.github,bin,coverage,pkg,spec,test/fixtures,vendor,tmp}/**/*'
-  - test/ruby-syntax-fixtures/**/*
   - test.rb
 
 Gemspec/DevelopmentDependencies:
@@ -29,6 +28,9 @@ Lint/AmbiguousRange:
 Lint/BooleanSymbol:
   Enabled: false
 
+Lint/Debugger:
+  Enabled: false
+
 Lint/DuplicateBranch:
   Enabled: false
 
@@ -80,6 +82,9 @@ Security/Eval:
 Style/AccessorGrouping:
   Enabled: false
 
+Style/Alias:
+  Enabled: false
+
 Style/CaseEquality:
   Enabled: false
 
@@ -89,6 +94,9 @@ Style/CaseLikeIf:
 Style/ClassVars:
   Enabled: false
 
+Style/CombinableLoops:
+  Enabled: false
+
 Style/DocumentDynamicEvalDefinition:
   Enabled: false
 
@@ -110,6 +118,9 @@ Style/FormatStringToken:
 Style/GuardClause:
   Enabled: false
 
+Style/HashLikeCase:
+  Enabled: false
+
 Style/IdenticalConditionalBranches:
   Enabled: false
 

data/CHANGELOG.md

@@ -6,6 +6,68 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [6.0.0] - 2023-02-10
+
+### Added
+
+- `SyntaxTree::BasicVisitor::visit_methods` has been added to allow you to check multiple visit methods inside of a block. There _was_ a method called `visit_methods` previously, but it was undocumented because it was meant as a private API. That method has been renamed to `valid_visit_methods`.
+- `rake sorbet:rbi` has been added as a task within the repository to generate an RBI file corresponding to the nodes in the tree. This can be used to help aid consumers of Syntax Tree that are using Sorbet.
+- `SyntaxTree::Reflection` has been added to allow you to get information about the nodes in the tree. It is not required by default, since it takes a small amount of time to parse `node.rb` and get all of the information.
+- `SyntaxTree::Node#to_mermaid` has been added to allow you to generate a Mermaid diagram of the node and its children. This is useful for debugging and understanding the structure of the tree.
+- `SyntaxTree::Translation` has been added as an experimental API to transform the Syntax Tree syntax tree into the syntax trees represented by the whitequark/parser and rubocop/rubocop-ast gems.
+  - `SyntaxTree::Translation.to_parser(node, buffer)` will return a `Parser::AST::Node` object.
+  - `SyntaxTree::Translation.to_rubocop_ast(node, buffer)` will return a `RuboCop::AST::Node` object.
+- `SyntaxTree::index` and `SyntaxTree::index_file` have been added to allow you to get a list of all of the classes, modules, and methods defined in a given source string or file.
+- Various convenience methods have been added:
+  - `SyntaxTree::format_file` - which calls format with the result of reading the file
+  - `SyntaxTree::format_node` - which formats the node directly
+  - `SyntaxTree::parse_file` - which calls parse with the result of reading the file
+  - `SyntaxTree::search_file` - which calls search with the result of reading the file
+  - `SyntaxTree::Node#start_char` - which is the same as calling `node.location.start_char`
+  - `SyntaxTree::Node#end_char` - which is the same as calling `node.location.end_char`
+- `SyntaxTree::Assoc` nodes can now be formatted on their own without a parent hash node.
+- `SyntaxTree::BlockVar#arg0?` has been added to check if a single required block parameter is present and would potentially be expanded.
+- More experimental APIs have been added to the `SyntaxTree::YARV` module, including:
+  - `SyntaxTree::YARV::ControlFlowGraph`
+  - `SyntaxTree::YARV::DataFlowGraph`
+  - `SyntaxTree::YARV::SeaOfNodes`
+
+### Changed
+
+#### Major changes
+
+- *BREAKING* Updates to `WithEnvironment`:
+  - The `WithEnvironment` module has been renamed to `WithScope`.
+  - The `current_environment` method has been renamed to `current_scope`.
+  - The `with_current_environment` method has been removed.
+  - Previously scopes were always able to look up the tree, as in: `a = 1; def foo; a = 2; end` would see only a single `a` variable. That has been corrected.
+  - Previously accessing variables from inside of blocks that were not shadowed would mark them as being local to the block only. This has been correct.
+- *BREAKING* Lots of constants moved out of `SyntaxTree::Visitor` to just `SyntaxTree`:
+  * `SyntaxTree::Visitor::FieldVisitor` is now `SyntaxTree::FieldVisitor`
+  * `SyntaxTree::Visitor::JSONVisitor` is now `SyntaxTree::JSONVisitor`
+  * `SyntaxTree::Visitor::MatchVisitor` is now `SyntaxTree::MatchVisitor`
+  * `SyntaxTree::Visitor::MutationVisitor` is now `SyntaxTree::MutationVisitor`
+  * `SyntaxTree::Visitor::PrettyPrintVisitor` is now `SyntaxTree::PrettyPrintVisitor`
+- *BREAKING* Lots of constants are now autoloaded instead of required by default. This is only particularly relevant if you are in a forking environment and want to preload constants before forking for better memory usage with copy-on-write.
+- *BREAKING* The `SyntaxTree::Statements#initialize` method no longer accepts a parser as the first argument. It now mirrors the other nodes in that it accepts its children and location. As a result, Syntax Tree nodes are now marshalable (and therefore can be sent over DRb). Previously the `Statements` node was not able to be marshaled because it held a reference to the parser.
+
+#### Minor changes
+
+- Many places where embedded documents (`=begin` to `=end`) were being treated as real comments have been fixed for formatting.
+- Dynamic symbols in keyword pattern matching now have better formatting.
+- Endless method definitions used to have a `SyntaxTree::BodyStmt` node that had any kind of node as its `statements` field. That has been corrected to be more consistent such that now going from `def_node.bodystmt.statements` always returns a `SyntaxTree::Statements` node, which is more consistent.
+- We no longer assume that `fiddle` is able to be required, and only require it when it is actually needed.
+
+#### Tiny changes
+
+- Empty parameter nodes within blocks now have more accurate location information.
+- Pinned variables have more correct location information now. (Previously the location was just around the variable itself, but it now includes the pin.)
+- Array patterns in pattern matching now have more accurate location information when they are using parentheses with a constant present.
+- Find patterns in pattern matching now have more correct location information for their `left` and `right` fields.
+- Lots of nodes have more correct types in the comments on their attributes.
+- The expressions `break foo.bar :baz do |qux| qux end` and `next fun foo do end` now correctly parses as a control-flow statement with a method call that has a block attached, as opposed to a control-flow statement with a block attached.
+- The expression `self::a, b = 1, 2` would previously yield a `SyntaxTree::ConstPathField` node for the first element of the left-hand-side of the multiple assignment. Semantically this is incorrect, and we have fixed this to now be a `SyntaxTree::Field` node instead.
+
 ## [5.3.0] - 2023-01-26
 
 ### Added
@@ -497,7 +559,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v5.3.0...HEAD
+[unreleased]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v6.0.0...HEAD
+[6.0.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v5.3.0...v6.0.0
 [5.3.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v5.2.0...v5.3.0
 [5.2.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v5.1.0...v5.2.0
 [5.1.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v5.0.1...v5.1.0

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    syntax_tree (5.3.0)
+    syntax_tree (6.0.0)
       prettier_print (>= 1.2.0)
 
 GEM
@@ -19,7 +19,7 @@ GEM
     rake (13.0.6)
     regexp_parser (2.6.2)
     rexml (3.2.5)
-    rubocop (1.44.1)
+    rubocop (1.45.1)
       json (~> 2.3)
       parallel (~> 1.10)
       parser (>= 3.2.0.0)

data/README.md

@@ -40,9 +40,10 @@ It is built with only standard library dependencies. It additionally ships with
   - [construct_keys](#construct_keys)
 - [Visitor](#visitor)
   - [visit_method](#visit_method)
+  - [visit_methods](#visit_methods)
   - [BasicVisitor](#basicvisitor)
   - [MutationVisitor](#mutationvisitor)
-  - [WithEnvironment](#withenvironment)
+  - [WithScope](#withscope)
 - [Language server](#language-server)
   - [textDocument/formatting](#textdocumentformatting)
   - [textDocument/inlayHint](#textdocumentinlayhint)
@@ -340,7 +341,7 @@ This function takes an input string containing Ruby code, parses it into its und
 
 ### SyntaxTree.mutation(&block)
 
-This function yields a new mutation visitor to the block, and then returns the initialized visitor. It's effectively a shortcut for creating a `SyntaxTree::Visitor::MutationVisitor` without having to remember the class name. For more information on that visitor, see the definition below.
+This function yields a new mutation visitor to the block, and then returns the initialized visitor. It's effectively a shortcut for creating a `SyntaxTree::MutationVisitor` without having to remember the class name. For more information on that visitor, see the definition below.
 
 ### SyntaxTree.search(source, query, &block)
 
@@ -517,6 +518,26 @@ Did you mean?  visit_binary
 	from bin/console:8:in `<main>'
 ```
 
+### visit_methods
+
+Similar to `visit_method`, `visit_methods` also checks that methods defined are valid visit methods. This variation however accepts a block and checks that all methods defined within that block are valid visit methods. It's meant to be used like:
+
+```ruby
+class ArithmeticVisitor < SyntaxTree::Visitor
+  visit_methods do
+    def visit_binary(node)
+      # ...
+    end
+
+    def visit_int(node)
+      # ...
+    end
+  end
+end
+```
+
+This is only checked when the methods are defined and does not impose any kind of runtime overhead after that. It is very useful for upgrading versions of Syntax Tree in case these methods names change.
+
 ### BasicVisitor
 
 When you're defining your own visitor, by default it will walk down the tree even if you don't define `visit_*` methods. This is to ensure you can define a subset of the necessary methods in order to only interact with the nodes you're interested in. If you'd like to change this default to instead raise an error if you visit a node you haven't explicitly handled, you can instead inherit from `BasicVisitor`.
@@ -537,7 +558,7 @@ The `MutationVisitor` is a visitor that can be used to mutate the tree. It works
 
 ```ruby
 # Create a new visitor
-visitor = SyntaxTree::Visitor::MutationVisitor.new
+visitor = SyntaxTree::MutationVisitor.new
 
 # Specify that it should mutate If nodes with assignments in their predicates
 visitor.mutate("IfNode[predicate: Assign | OpAssign]") do |node|
@@ -567,20 +588,18 @@ SyntaxTree::Formatter.format(source, program.accept(visitor))
 # => "if (a = 1)\nend\n"
 ```
 
-### WithEnvironment
+### WithScope
 
-The `WithEnvironment` module can be included in visitors to automatically keep track of local variables and arguments
-defined inside each environment. A `current_environment` accessor is made available to the request, allowing it to find
-all usages and definitions of a local.
+The `WithScope` module can be included in visitors to automatically keep track of local variables and arguments defined inside each scope. A `current_scope` accessor is made available to the request, allowing it to find all usages and definitions of a local.
 
 ```ruby
 class MyVisitor < Visitor
-  include WithEnvironment
+  prepend WithScope
 
   def visit_ident(node)
     # find_local will return a Local for any local variables or arguments
     # present in the current environment or nil if the identifier is not a local
-    local = current_environment.find_local(node)
+    local = current_scope.find_local(node)
 
     puts local.type # the type of the local (:variable or :argument)
     puts local.definitions # the array of locations where this local is defined

data/Rakefile

@@ -4,6 +4,8 @@ require "bundler/gem_tasks"
 require "rake/testtask"
 require "syntax_tree/rake_tasks"
 
+Rake.add_rakelib "tasks"
+
 Rake::TestTask.new(:test) do |t|
   t.libs << "test"
   t.libs << "lib"
@@ -14,7 +16,16 @@ task default: :test
 
 configure = ->(task) do
   task.source_files =
-    FileList[%w[Gemfile Rakefile syntax_tree.gemspec lib/**/*.rb test/*.rb]]
+    FileList[
+      %w[
+        Gemfile
+        Rakefile
+        syntax_tree.gemspec
+        lib/**/*.rb
+        tasks/*.rake
+        test/*.rb
+      ]
+    ]
 
   # Since Syntax Tree supports back to Ruby 2.7.0, we need to make sure that we
   # format our code such that it's compatible with that version. This actually
@@ -26,10 +37,3 @@ end
 
 SyntaxTree::Rake::CheckTask.new(&configure)
 SyntaxTree::Rake::WriteTask.new(&configure)
-
-desc "Run mspec tests using YARV emulation"
-task :spec do
-  Dir["./spec/ruby/language/**/*_spec.rb"].each do |filepath|
-    sh "exe/yarv ./spec/mspec/bin/mspec-tag #{filepath}"
-  end
-end

data/bin/console

@@ -3,6 +3,7 @@
 
 require "bundler/setup"
 require "syntax_tree"
+require "syntax_tree/reflection"
 
 require "irb"
 IRB.start(__FILE__)

data/bin/whitequark

@@ -0,0 +1,79 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "parser/current"
+
+$:.unshift(File.expand_path("../lib", __dir__))
+require "syntax_tree"
+
+# First, opt in to every AST feature.
+Parser::Builders::Default.modernize
+
+# Modify the source map == check so that it doesn't check against the node
+# itself so we don't get into a recursive loop.
+Parser::Source::Map.prepend(
+  Module.new {
+    def ==(other)
+      self.class == other.class &&
+        (instance_variables - %i[@node]).map do |ivar|
+          instance_variable_get(ivar) == other.instance_variable_get(ivar)
+        end.reduce(:&)
+    end
+  }
+)
+
+# Next, ensure that we're comparing the nodes and also comparing the source
+# ranges so that we're getting all of the necessary information.
+Parser::AST::Node.prepend(
+  Module.new {
+    def ==(other)
+      super && (location == other.location)
+    end
+  }
+)
+
+source = ARGF.read
+
+parser = Parser::CurrentRuby.new
+parser.diagnostics.all_errors_are_fatal = true
+
+buffer = Parser::Source::Buffer.new("(string)", 1)
+buffer.source = source.dup.force_encoding(parser.default_encoding)
+
+stree = SyntaxTree::Translation.to_parser(SyntaxTree.parse(source), buffer)
+ptree = parser.parse(buffer)
+
+if stree == ptree
+  puts "Syntax trees are equivalent."
+elsif stree.inspect == ptree.inspect
+  warn "Syntax tree locations are different."
+
+  queue = [[stree, ptree]]
+  while (left, right = queue.shift)
+    if left.location != right.location
+      warn "Different node:"
+      pp left
+
+      warn "Different location:"
+
+      warn "Syntax Tree:"
+      pp left.location
+
+      warn "whitequark/parser:"
+      pp right.location
+
+      exit
+    end
+
+    left.children.zip(right.children).each do |left_child, right_child|
+      queue << [left_child, right_child] if left_child.is_a?(Parser::AST::Node)
+    end
+  end
+else
+  warn "Syntax Tree:"
+  pp stree
+
+  warn "whitequark/parser:"
+  pp ptree
+end

data/doc/changing_structure.md

@@ -0,0 +1,16 @@
+# Changing structure
+
+First and foremost, changing the structure of the tree in any way is a major breaking change. It forces the consumers to update their visitors, pattern matches, and method calls. It should not be taking lightly, and can only happen on a major version change. So keep that in mind.
+
+That said, if you do want to change the structure of the tree, there are a few steps that you have to take. They are enumerated below.
+
+1. Change the structure in the required node classes. This could mean adding/removing classes or adding/removing fields. Be sure to also update the `copy` and `===` methods to be sure that they are correct.
+2. Update the parser to correctly create the new structure.
+3. Update any visitor methods that are affected by the change. For example, if adding a new node make sure to create the new visit method alias in the `Visitor` class.
+4. Update the `FieldVisitor` class to be sure that the various serializers, pretty printers, and matchers all get updated accordingly.
+5. Update the `DSL` module to be sure that folks can correctly create nodes with the new structure.
+6. Ensure the formatting of the code hasn't changed. This can mostly be done by running the tests, but if there's a corner case that we don't cover that is now exposed by your change be sure to add test cases.
+7. Update the translation visitors to ensure we're still translating into other ASTs correctly.
+8. Update the YARV compiler visitor to ensure we're still compiling correctly.
+9. Make sure we aren't referencing the previous structure in any documentation or tests.
+10. Be sure to update `CHANGELOG.md` with a description of the change that you made.

data/lib/syntax_tree/basic_visitor.rb

@@ -29,7 +29,7 @@ module SyntaxTree
       def corrections
         @corrections ||=
           DidYouMean::SpellChecker.new(
-            dictionary: Visitor.visit_methods
+            dictionary: BasicVisitor.valid_visit_methods
           ).correct(visit_method)
       end
 
@@ -40,7 +40,40 @@ module SyntaxTree
       end
     end
 
+    # This module is responsible for checking all of the methods defined within
+    # a given block to ensure that they are valid visit methods.
+    class VisitMethodsChecker < Module
+      Status = Struct.new(:checking)
+
+      # This is the status of the checker. It's used to determine whether or not
+      # we should be checking the methods that are defined. It is kept as an
+      # instance variable so that it can be disabled later.
+      attr_reader :status
+
+      def initialize
+        # We need the status to be an instance variable so that it can be
+        # accessed by the disable! method, but also a local variable so that it
+        # can be captured by the define_method block.
+        status = @status = Status.new(true)
+
+        define_method(:method_added) do |name|
+          BasicVisitor.visit_method(name) if status.checking
+          super(name)
+        end
+      end
+
+      def disable!
+        status.checking = false
+      end
+    end
+
     class << self
+      # This is the list of all of the valid visit methods.
+      def valid_visit_methods
+        @valid_visit_methods ||=
+          Visitor.instance_methods.grep(/^visit_(?!child_nodes)/)
+      end
+
       # This method is here to help folks write visitors.
       #
       # It's not always easy to ensure you're writing the correct method name in
@@ -51,15 +84,21 @@ module SyntaxTree
       # name. It will raise an error if the visit method you're defining isn't
       # actually a method on the parent visitor.
       def visit_method(method_name)
-        return if visit_methods.include?(method_name)
+        return if valid_visit_methods.include?(method_name)
 
         raise VisitMethodError, method_name
       end
 
-      # This is the list of all of the valid visit methods.
+      # This method is here to help folks write visitors.
+      #
+      # Within the given block, every method that is defined will be checked to
+      # ensure it's a valid visit method using the BasicVisitor::visit_method
+      # method defined above.
       def visit_methods
-        @visit_methods ||=
-          Visitor.instance_methods.grep(/^visit_(?!child_nodes)/)
+        checker = VisitMethodsChecker.new
+        extend(checker)
+        yield
+        checker.disable!
       end
     end
 

data/lib/syntax_tree/cli.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "etc"
 require "optparse"
 
 module SyntaxTree
@@ -238,7 +239,7 @@ module SyntaxTree
     # representation.
     class Json < Action
       def run(item)
-        object = Visitor::JSONVisitor.new.visit(item.handler.parse(item.source))
+        object = item.handler.parse(item.source).accept(JSONVisitor.new)
         puts JSON.pretty_generate(object)
       end
     end
@@ -501,7 +502,6 @@ module SyntaxTree
           when "j", "json"
             Json.new(options)
           when "lsp"
-            require "syntax_tree/language_server"
             LanguageServer.new(print_width: options.print_width).run
             return 0
           when "m", "match"

data/lib/syntax_tree/dsl.rb

@@ -210,12 +210,17 @@ module SyntaxTree
     end
 
     # Create a new ClassDeclaration node.
-    def ClassDeclaration(constant, superclass, bodystmt)
+    def ClassDeclaration(
+      constant,
+      superclass,
+      bodystmt,
+      location = Location.default
+    )
       ClassDeclaration.new(
         constant: constant,
         superclass: superclass,
         bodystmt: bodystmt,
-        location: Location.default
+        location: location
       )
     end
 
@@ -225,12 +230,12 @@ module SyntaxTree
     end
 
     # Create a new Command node.
-    def Command(message, arguments, block)
+    def Command(message, arguments, block, location = Location.default)
       Command.new(
         message: message,
         arguments: arguments,
         block: block,
-        location: Location.default
+        location: location
       )
     end
 
@@ -247,8 +252,8 @@ module SyntaxTree
     end
 
     # Create a new Comment node.
-    def Comment(value, inline)
-      Comment.new(value: value, inline: inline, location: Location.default)
+    def Comment(value, inline, location = Location.default)
+      Comment.new(value: value, inline: inline, location: location)
     end
 
     # Create a new Const node.
@@ -285,14 +290,21 @@ module SyntaxTree
     end
 
     # Create a new DefNode node.
-    def DefNode(target, operator, name, params, bodystmt)
+    def DefNode(
+      target,
+      operator,
+      name,
+      params,
+      bodystmt,
+      location = Location.default
+    )
       DefNode.new(
         target: target,
         operator: operator,
         name: name,
         params: params,
         bodystmt: bodystmt,
-        location: Location.default
+        location: location
       )
     end
 
@@ -565,8 +577,8 @@ module SyntaxTree
     end
 
     # Create a new MethodAddBlock node.
-    def MethodAddBlock(call, block)
-      MethodAddBlock.new(call: call, block: block, location: Location.default)
+    def MethodAddBlock(call, block, location = Location.default)
+      MethodAddBlock.new(call: call, block: block, location: location)
     end
 
     # Create a new MLHS node.
@@ -779,7 +791,7 @@ module SyntaxTree
 
     # Create a new Statements node.
     def Statements(body)
-      Statements.new(nil, body: body, location: Location.default)
+      Statements.new(body: body, location: Location.default)
     end
 
     # Create a new StringContent node.