machete 0.4.0 → 0.5.0

data/.gitignore

@@ -2,4 +2,6 @@ lib/machete/parser.rb
 doc/
 .yardoc
 *.rbc
+.rbx/
 Gemfile.lock
+.rbenv-version

data/CHANGELOG

@@ -1,5 +1,18 @@
+0.5.0 (2012-08-19)
+------------------
+
+* Support for matching symbols using "^=", "$=" and "*=" operators.
+* Support for regexp literals and matching regexps using "*=" operator.
+* Extended symbol grammar to cover symbols denoting instance and class variable
+  names (:@foo, :@@bar).
+* Matchete#matches? and Machete#find accept the pattern also in compiled
+  form (instance of Machete:Matchers::Matcher class).
+* Works in Rubinius 1.9 mode.
+* Internal code improvements and fixes.
+
 0.4.0 (2011-10-18)
 ------------------
+
 * Support for "true", "false" and "nil" literals.
 * New "*=" operator matching part of a string.
 * Bundler support.

data/Gemfile

@@ -1,3 +1,7 @@
 source "http://rubygems.org"
 
 gemspec
+
+group :test do
+  gem "rake"
+end

data/README.md

@@ -1,12 +1,16 @@
 Machete
 =======
 
-Machete is a simple tool for matching Rubinius AST nodes against patterns. You can use it if you are writing any kind of tool that processes Ruby code and needs to do some work on specific types of nodes, needs to find patterns in the code, etc.
+Machete is a simple tool for matching Rubinius AST nodes against patterns. You
+can use it if you are writing any kind of tool that processes Ruby code and
+needs to do some work on specific types of nodes, needs to find patterns in the
+code, etc.
 
 Installation
 ------------
 
-You need to install [Rubinius](http://rubini.us/) first. You can then install Machete:
+You need to install current development version of [Rubinius](http://rubini.us/)
+first. You can then install Machete:
 
     $ gem install machete
 
@@ -17,19 +21,21 @@ First, require the library:
 
     require "machete"
 
-You can now use one of two methods Machete offers: `Machete.matches?` and `Machete.find`.
+You can now use one of two methods Machete offers: `Machete.matches?` and
+`Machete.find`.
 
 The `Machete.matches?` method matches a Rubinus AST node against a pattern:
 
-    Machete.matches?('foo.bar'.to_ast, 'Send<receiver = Send<receiver = Self, name = :foo>, name = :bar>')
+    Machete.matches?('foo.bar'.to_ast, 'Send<name = :bar>')
     # => true
 
-    Machete.matches?('42'.to_ast, 'Send<receiver = Send<receiver = Self, name = :foo>, name = :bar>')
+    Machete.matches?('42'.to_ast, 'Send<name = :bar>')
     # => false
 
 (See below for pattern syntax description.)
 
-The `Machete.find` method finds all nodes in a Rubinius AST tree matching a pattern:
+The `Machete.find` method finds all nodes in a Rubinius AST tree matching a
+pattern:
 
     Machete.find('42 + 43 + 44'.to_ast, 'FixnumLiteral')
     # => [
@@ -38,12 +44,24 @@ The `Machete.find` method finds all nodes in a Rubinius AST tree matching a patt
     #      #<Rubinius::AST::FixnumLiteral:0x10c0 @value=42 @line=1>
     #    ]
 
+Both `Machete.matches?` and `Machete.find` also accept patterns in their
+compiled form (instance of `Machete::Matchers::Matcher`):
+
+    Machete.matches?(
+      'foo.bar'.to_ast,
+      Machete::Matchers::NodeMatcher.new("Send",
+        :name => Machete::Matchers::LiteralMatcher.new(:bar)
+      )
+    )
+    # => true
+
 Pattern Syntax
 --------------
 
 ### Basics
 
-Rubinius AST consists of instances of classes that represent various types of nodes:
+Rubinius AST consists of instances of classes that represent various types of
+nodes:
 
     '42'.to_ast     # => #<Rubinius::AST::FixnumLiteral:0xf28 @value=42 @line=1>
     '"abcd"'.to_ast # => #<Rubinius::AST::StringLiteral:0xf60 @line=1 @string="abcd">
@@ -58,25 +76,31 @@ To specify multiple alternatives, use the choice operator:
     Machete.matches?('42'.to_ast,     'FixnumLiteral | StringLiteral') # => true
     Machete.matches?('"abcd"'.to_ast, 'FixnumLiteral | StringLiteral') # => true
 
-If you don't care about the node type at all, use the `any` keyword (this is most useful when matching arrays — see below):
+If you don't care about the node type at all, use the `any` keyword (this is
+most useful when matching arrays — see below):
 
     Machete.matches?('42'.to_ast,     'any') # => true
     Machete.matches?('"abcd"'.to_ast, 'any') # => true
 
 ### Node Attributes
 
-If you want to match a specific attribute of a node, specify its value inside `<...>` right after the node name:
+If you want to match a specific attribute of a node, specify its value inside
+`<...>` right after the node name:
 
     Machete.matches?('42'.to_ast, 'FixnumLiteral<value = 42>') # => true
     Machete.matches?('45'.to_ast, 'FixnumLiteral<value = 42>') # => false
 
-The attribute value can be `true`, `false`, `nil`, integer, string, symbol, array or other pattern. The last option means you can easily match nested nodes recursively. You can also specify multiple attributes:
+The attribute value can be `nil`, `true`, `false`, integer, symbol, string,
+regexp, array or other pattern. The last option means you can easily match
+nested nodes recursively. You can also specify multiple attributes:
 
     Machete.matches?('foo.bar'.to_ast, 'Send<receiver = Send<receiver = Self, name = :foo>, name = :bar>') # => true
 
-#### String Attributes
+#### String And Symbol Attributes
 
-When matching string attributes values, you don't have to do a whole-string match using the `=` operator. You can also match the beginning, the end or a part of a string attribute value using the `^=`, `$=` and `*=` operators:
+When matching string attributes values, you don't have to do a whole-string
+match using the `=` operator. You can also match the beginning, the end or a
+part of a string attribute value using the `^=`, `$=` and `*=` operators:
 
     Machete.matches?('"abcd"'.to_ast, 'StringLiteral<string ^= "ab">') # => true
     Machete.matches?('"efgh"'.to_ast, 'StringLiteral<string ^= "ab">') # => false
@@ -85,9 +109,31 @@ When matching string attributes values, you don't have to do a whole-string matc
     Machete.matches?('"abcd"'.to_ast, 'StringLiteral<string *= "bc">') # => true
     Machete.matches?('"efgh"'.to_ast, 'StringLiteral<string *= "bc">') # => false
 
+Match symbol attributes works in the same way:
+
+    Machete.matches?(':abcd'.to_ast, 'SymbolLiteral<value ^= :ab>') # => true
+    Machete.matches?(':efgh'.to_ast, 'SymbolLiteral<value ^= :ab>') # => false
+    Machete.matches?(':abcd'.to_ast, 'SymbolLiteral<value $= :cd>') # => true
+    Machete.matches?(':efgh'.to_ast, 'SymbolLiteral<value $= :cd>') # => false
+    Machete.matches?(':abcd'.to_ast, 'SymbolLiteral<value *= :bc>') # => true
+    Machete.matches?(':efgh'.to_ast, 'SymbolLiteral<value *= :bc>') # => false
+
+In addition, you can match string and symbol attributes using regular
+expressions together with the `*=` operator:
+
+    Machete.matches?('"abcd"'.to_ast, 'StringLiteral<string *= /bc/>') # => true
+    Machete.matches?('"efgh"'.to_ast, 'StringLiteral<string *= /bc/>') # => false
+
+    Machete.matches?(':abcd'.to_ast, 'SymbolLiteral<value *= /bc/>') # => true
+    Machete.matches?(':efgh'.to_ast, 'SymbolLiteral<value *= /bc/>') # => false
+
+The regular expressions can take the `i`, `m` and `x` options with the same
+semantics as in Ruby.
+
 #### Array Attributes
 
-When matching array attribute values, the simplest way is to specify the array elements exactly. They will be matched one-by-one.
+When matching array attribute values, the simplest way is to specify the array
+elements exactly. They will be matched one-by-one.
 
     Machete.matches?('[1, 2]'.to_ast, 'ArrayLiteral<body = [FixnumLiteral<value = 1>, FixnumLiteral<value = 2>]>') # => true
 
@@ -96,7 +142,9 @@ If you don't care about the node type of some array elements, you can use `any`:
     Machete.matches?('[1, 2]'.to_ast,      'ArrayLiteral<body = [any, FixnumLiteral<value = 2>]>') # => true
     Machete.matches?('["abcd", 2]'.to_ast, 'ArrayLiteral<body = [any, FixnumLiteral<value = 2>]>') # => true
 
-The best thing about array matching is that you can use quantifiers for elements: `*`, `+`, `?`, `{n}`, `{n,}`, `{,n}`, `{m,n}`. Their meaning is the same as in Perl-like regular expressions:
+The best thing about array matching is that you can use quantifiers for
+elements: `*`, `+`, `?`, `{n}`, `{n,}`, `{,n}`, `{m,n}`. Their meaning is the
+same as in Perl-like regular expressions:
 
     Machete.matches?('[2]'.to_ast,          'ArrayLiteral<body = [any*, FixnumLiteral<value = 2>]>') # => true
     Machete.matches?('[1, 2]'.to_ast,       'ArrayLiteral<body = [any*, FixnumLiteral<value = 2>]>') # => true
@@ -126,7 +174,8 @@ The best thing about array matching is that you can use quantifiers for elements
     Machete.matches?('[1, 1, 2]'.to_ast,    'ArrayLiteral<body = [any{1,2}, FixnumLiteral<value = 2>]>') # => true
     Machete.matches?('[1, 1, 1, 2]'.to_ast, 'ArrayLiteral<body = [any{1,2}, FixnumLiteral<value = 2>]>') # => false
 
-There are also two unusual quantifiers: `{even}` and `{odd}`. They specify that the quantified expression must repeat even or odd number of times:
+There are also two unusual quantifiers: `{even}` and `{odd}`. They specify that
+the quantified expression must repeat even or odd number of times:
 
     Machete.matches?('[1, 2]'.to_ast,       'ArrayLiteral<body = [any{even}, FixnumLiteral<value = 2>]>') # => false
     Machete.matches?('[1, 1, 2]'.to_ast,    'ArrayLiteral<body = [any{even}, FixnumLiteral<value = 2>]>') # => true
@@ -134,26 +183,40 @@ There are also two unusual quantifiers: `{even}` and `{odd}`. They specify that
     Machete.matches?('[1, 2]'.to_ast,       'ArrayLiteral<body = [any{odd}, FixnumLiteral<value = 2>]>') # => true
     Machete.matches?('[1, 1, 2]'.to_ast,    'ArrayLiteral<body = [any{odd}, FixnumLiteral<value = 2>]>') # => false
 
-These quantifiers are best used when matching hashes containing a specific key or value. This is because in Rubinius AST both hash keys and values are flattened into one array and the only thing distinguishing them is even or odd position.
+These quantifiers are best used when matching hashes containing a specific key
+or value. This is because in Rubinius AST both hash keys and values are
+flattened into one array and the only thing distinguishing them is even or odd
+position.
 
 ### More Information
 
-For more details about the syntax see the `lib/machete/parser.y` file which contains the pattern parser.
+For more details about the syntax see the `lib/machete/parser.y` file which
+contains the pattern parser.
 
 FAQ
 ---
 
-**Why did you chose Rubinius AST as a base? Aren't there other tools for Ruby parsing which are not VM-specific?**
+**Why did you chose Rubinius AST as a base? Aren't there other tools for Ruby
+parsing which are not VM-specific?**
 
 There are three other tools which were considered but each has its issues:
 
-* [parse_tree](http://parsetree.rubyforge.org/) — unmaintained and unsupported for 1.9
-* [ruby_parser](http://parsetree.rubyforge.org/) — sometimes reports wrong line numbers for the nodes (this is a killer for some use cases)
-* [Ripper](http://rubyforge.org/projects/ripper/) — usable but the generated AST is too low level (the patterns would be too complex and low-level)
+* [parse_tree](http://parsetree.rubyforge.org/) — unmaintained and unsupported
+  for 1.9
+* [ruby_parser](http://parsetree.rubyforge.org/) — sometimes reports wrong line
+   numbers for the nodes (this is a killer for some use cases)
+* [Ripper](http://rubyforge.org/projects/ripper/) — usable but the generated AST
+  is too low level (the patterns would be too complex and low-level)
 
 Rubinius AST is also by far the easiest to work with.
 
+Compatibility
+-------------
+
+Machete is compatible with both the 1.8 and 1.9 mode of Rubinius.
+
 Acknowledgement
 ---------------
 
-The general idea and inspiration for the pattern syntax was taken form Python's [2to3](http://docs.python.org/library/2to3.html) tool.
+The general idea and inspiration for the pattern syntax was taken form Python's
+[2to3](http://docs.python.org/library/2to3.html) tool.

data/Rakefile

@@ -5,8 +5,8 @@ desc "Generate the expression parser"
 task :parser do
   source = "lib/machete/parser.y"
   target = "lib/machete/parser.rb"
-  unless uptodate?(target, source)
-    system "racc -o #{target} #{source}" or exit 1
+  unless uptodate?(target, [source])
+    sh "racc -o #{target} #{source}"
   end
 end
 

data/VERSION

@@ -1 +1 @@
-0.4.0
+0.5.0

data/lib/machete.rb

@@ -3,53 +3,97 @@ require File.expand_path(File.dirname(__FILE__) + "/machete/parser")
 require File.expand_path(File.dirname(__FILE__) + "/machete/version")
 
 module Machete
-  # Matches a Rubinius AST node against a pattern.
-  #
-  # @param [Rubinius::AST::Node] node node to match
-  # @param [String] pattern pattern to match the node against (see {file:README.md} for syntax description)
-  #
-  # @example Succesfull match
-  #   Machete.matches?('foo.bar'.to_ast, 'Send<receiver = Send<receiver = Self, name = :foo>, name = :bar>')
-  #   # => true
-  #
-  # @example Failed match
-  #   Machete.matches?('42'.to_ast, 'Send<receiver = Send<receiver = Self, name = :foo>, name = :bar>')
-  #   # => false
-  #
-  # @return [Boolean] +true+ if the node matches the pattern, +false+ otherwise
-  #
-  # @raise [Matchete::Parser::SyntaxError] if the pattern is invalid
-  def self.matches?(node, pattern)
-    Parser.new.parse(pattern).matches?(node)
-  end
+  class << self
+    # Matches a Rubinius AST node against a pattern.
+    #
+    # @param [Rubinius::AST::Node] node node to match
+    # @param [String, Machete::Matchers::Matcher] pattern pattern to match the
+    #   node against, either as a string (see {file:README.md} for syntax
+    #   description) or in compiled form
+    #
+    # @example Test using a string pattern
+    #   Machete.matches?('foo.bar'.to_ast, 'Send<name = :bar>')
+    #   # => true
+    #
+    #   Machete.matches?('42'.to_ast, 'Send<name = :bar>')
+    #   # => false
+    #
+    # @example Test using a compiled pattern
+    #   Machete.matches?(
+    #     'foo.bar'.to_ast,
+    #     Machete::Matchers::NodeMatcher.new("Send",
+    #       :name => Machete::Matchers::LiteralMatcher.new(:bar)
+    #     )
+    #   )
+    #   # => true
+    #
+    #   Machete.matches?(
+    #     '42'.to_ast,
+    #     Machete::Matchers::NodeMatcher.new("Send",
+    #       :name => Machete::Matchers::LiteralMatcher.new(:bar)
+    #     )
+    #   )
+    #   # => false
+    #
+    # @return [Boolean] +true+ if the node matches the pattern, +false+
+    #   otherwise
+    #
+    # @raise [Matchete::Parser::SyntaxError] if the pattern is invalid
+    def matches?(node, pattern)
+      compiled_pattern(pattern).matches?(node)
+    end
+
+    # Finds all nodes in a Rubinius AST matching a pattern.
+    #
+    # @param [Rubinius::AST::Node] ast tree to search
+    # @param [String, Machete::Matchers::Matcher] pattern pattern to match the
+    #   nodes against, either as a string (see {file:README.md} for syntax
+    #   description) or in compiled form
+    #
+    # @example Search using a string pattern
+    #   Machete.find('42 + 43 + 44'.to_ast, 'FixnumLiteral')
+    #   # => [
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10b0 @value=44 @line=1>,
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10b8 @value=43 @line=1>,
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10c0 @value=42 @line=1>
+    #   #    ]
+    #
+    # @example Search using a compiled pattern
+    #   Machete.find(
+    #     '42 + 43 + 44'.to_ast,
+    #     Machete::Matchers::NodeMatcher.new("FixnumLiteral")
+    #   )
+    #   # => [
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10b0 @value=44 @line=1>,
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10b8 @value=43 @line=1>,
+    #   #      #<Rubinius::AST::FixnumLiteral:0x10c0 @value=42 @line=1>
+    #   #    ]
+    #
+    # @return [Array] list of matching nodes (in unspecified order)
+    #
+    # @raise [Matchete::Parser::SyntaxError] if the pattern is invalid
+    def find(ast, pattern)
+      matcher = compiled_pattern(pattern)
 
-  # Finds all nodes in a Rubinius AST matching a pattern.
-  #
-  # @param [Rubinius::AST::Node] ast tree to search
-  # @param [String] pattern pattern to match the nodes against (see {file:README.md} for syntax description)
-  #
-  # @example
-  #   Machete.find('42 + 43 + 44'.to_ast, 'FixnumLiteral')
-  #   # => [
-  #   #      #<Rubinius::AST::FixnumLiteral:0x10b0 @value=44 @line=1>,
-  #   #      #<Rubinius::AST::FixnumLiteral:0x10b8 @value=43 @line=1>,
-  #   #      #<Rubinius::AST::FixnumLiteral:0x10c0 @value=42 @line=1>
-  #   #    ]
-  #
-  # @return [Array] list of matching nodes (in unspecified order)
-  #
-  # @raise [Matchete::Parser::SyntaxError] if the pattern is invalid
-  def self.find(ast, pattern)
-    matcher = Parser.new.parse(pattern)
+      result = []
+      result << ast if matcher.matches?(ast)
 
-    result = []
-    result << ast if matcher.matches?(ast)
+      ast.walk(true) do |dummy, node|
+        result << node if matcher.matches?(node)
+        true
+      end
 
-    ast.walk(true) do |dummy, node|
-      result << node if matcher.matches?(node)
-      true
+      result
     end
 
-    result
+    private
+
+    def compiled_pattern(pattern)
+      if pattern.is_a?(String)
+        Parser.new.parse(pattern)
+      else
+        pattern
+      end
+    end
   end
 end

data/lib/machete/matchers.rb

@@ -1,7 +1,5 @@
 module Machete
-  # @private
   module Matchers
-    # @private
     class Quantifier
       # :min should be always set, :max can be nil (meaning infinity)
       attr_reader :matcher, :min, :max, :step
@@ -19,8 +17,10 @@ module Machete
       end
     end
 
-    # @private
-    class ChoiceMatcher
+    class Matcher
+    end
+
+    class ChoiceMatcher < Matcher
       attr_reader :alternatives
 
       def initialize(alternatives)
@@ -36,8 +36,7 @@ module Machete
       end
     end
 
-    # @private
-    class NodeMatcher
+    class NodeMatcher < Matcher
       attr_reader :class_name, :attrs
 
       def initialize(class_name, attrs = {})
@@ -56,8 +55,7 @@ module Machete
       end
     end
 
-    # @private
-    class ArrayMatcher
+    class ArrayMatcher < Matcher
       attr_reader :items
 
       def initialize(items)
@@ -119,8 +117,7 @@ module Machete
       end
     end
 
-    # @private
-    class LiteralMatcher
+    class LiteralMatcher < Matcher
       attr_reader :literal
 
       def initialize(literal)
@@ -136,8 +133,7 @@ module Machete
       end
     end
 
-    # @private
-    class StringRegexpMatcher
+    class RegexpMatcher < Matcher
       attr_reader :regexp
 
       def initialize(regexp)
@@ -147,14 +143,28 @@ module Machete
       def ==(other)
         other.instance_of?(self.class) && @regexp == other.regexp
       end
+    end
 
+    class SymbolRegexpMatcher < RegexpMatcher
+      def matches?(node)
+        node.is_a?(Symbol) && node.to_s =~ @regexp
+      end
+    end
+
+    class StringRegexpMatcher < RegexpMatcher
       def matches?(node)
         node.is_a?(String) && node =~ @regexp
       end
     end
 
-    # @private
-    class AnyMatcher
+    class IndifferentRegexpMatcher < RegexpMatcher
+      def matches?(node)
+        (node.is_a?(Symbol) && node.to_s =~ @regexp) ||
+        (node.is_a?(String) && node =~ @regexp)
+      end
+    end
+
+    class AnyMatcher < Matcher
       def ==(other)
         other.instance_of?(self.class)
       end