sexp_path 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,167 @@
1
+ = SexpPath
2
+
3
+ Structural pattern matching against S-Expressions.
4
+
5
+ SexpPath allows you to define patterns that can be matched against S-Expressions.
6
+ SexpPath draws inspiration from Regular Expressions, XPath, and CSS Selectors.
7
+
8
+ I'm still figuring out how SexpPath should work so either fork this, or send me
9
+ some feedback.
10
+ http://github.com/adamsanderson/sexp_path
11
+ netghost@gmail.com
12
+
13
+ == Installation
14
+
15
+ SexpPath is distributed as a ruby gem:
16
+
17
+ gem install adamsanderson-sexp_path
18
+
19
+ == Notation
20
+
21
+ In ruby you're most likely to come across S-Expressions when dealing with
22
+ ParseTree's representation of the abstract syntax tree. An S-Expression is
23
+ just a set of nested lists. The SexpProcessor library displays them like this:
24
+
25
+ s(:a, :b,
26
+ s(:c)
27
+ )
28
+
29
+ Where that means that there is a list containing `:a`, `:b`, and then another list which
30
+ contains `:c`. We will refer to `:a`,`:b`, and `:c` as atoms, while
31
+ `s( something )` is an S-Expression or Sexp.
32
+
33
+ == General Syntax
34
+
35
+ SexpPath is an internal ruby DSL, which means a SexpPath query is valid ruby code.
36
+ SexpPath queries are built with the SexpQueryBuilder through the Q? convenience
37
+ method:
38
+
39
+ Q?{ s(:a, :b, :c)} # Matches s(:a, :b, :c)
40
+
41
+ This will match the S-Expression `s(:a, :b, :c)`. If you want to match something
42
+ more complicated, you will probably want to employ one of the many matchers built
43
+ into SexpPath.
44
+
45
+ [Wild Card] Matches anything.
46
+
47
+ _ => s(), :a, or s(:cat)
48
+
49
+ [Atom] Matches any atom (or symbol).
50
+
51
+ atom => :a, :b, or :cat
52
+
53
+ [Pattern] Matches any atom that matches the given string or regular expression.
54
+
55
+ m('cat') => :cat
56
+ m(/rat/) => :rat, :brat, or :rate
57
+ m(/^test_/) => :test_sexp_path
58
+
59
+ [Includes] Matches any S-Expression that includes the sub expression.
60
+
61
+ include(:a) => s(:a), s(:a, :b), or s(:cat, :a, :b)
62
+ include( s(:cat) ) => s(:pet, s(:cat))
63
+
64
+ [Child] Matches any S-Expression that has the sub expression as a child.
65
+
66
+ child( s(:a) ) => s(:b, s(:a)) or even s(s(s(s(s( s(:a))))))
67
+
68
+ [Sibling] Matches any S-Expression that has the second expression as a sibling.
69
+ s(:a) >> s(:c) => s( s(:a), s(:b), s(:c) )
70
+
71
+ [Type] The sexp type is considered to be the first atom. This matches any expression that has the given type.
72
+
73
+ type(:a) => s(:a), s(:a, :b), or s(:a, :b, s(:c))
74
+
75
+ [Any] Matches any sub expression
76
+
77
+ any( s(:a), s(:b) ) => s(:a) or s(:b)
78
+ any( s(:a), s(atom, :b) ) => s(:a), s(:a, :b), or s(:cat, :b)
79
+
80
+ [All] Matches anything that satisfies all the sub expressions
81
+
82
+ all( s(:a, atom), s(atom, :b) ) => s(:a,:b)
83
+
84
+ [Not] Negates a matcher
85
+
86
+ -s(:a) => s(:a,:b), s(:b), but not s(:a)
87
+ s(is_not :a) => s(:b), s(:c), but not s(:a) or s(:a, :b)
88
+
89
+ == Searching
90
+
91
+ You may use any SexpPath to search an S-Expression. SexpPath defines the `/` operator as search,
92
+ so to search `s( s(:a) )` for `s(:a)` you may just do:
93
+
94
+ s( s(:a) ) / Q?{ s(:a) }
95
+
96
+ This will return a collection with just one result which is `s(:a)`. You could also do something
97
+ more interesting:
98
+
99
+ s( s(:a), s(:b) ) / Q?{ s(atom) }
100
+
101
+ This will return two matches which are `s(:a)` and `s(:b)`. You can also chain searches, so this
102
+ works just fine as well:
103
+
104
+ sexp = s(:class, :Calculator,
105
+ s(:defn, :add),
106
+ s(:defn, :sub)
107
+ )
108
+
109
+ sexp / Q?{ s(:class, atom, _) } / Q?{ s(:defn, _) }
110
+
111
+ In this case you would get back `s(:defn, :add)` and `s(:defn, :sub)`.
112
+
113
+ == Capturing
114
+
115
+ It is useful to also capture results from your queries. So using
116
+ the same Sexp from above we could modify our query to actually capture some names.
117
+ Capturing is done by using `%` operator followed by the name you would like the value
118
+ to be captured as.
119
+
120
+ sexp / Q?{ s(:class, atom % 'class_name', _) } / Q?{ s(:defn, _ % 'method_name') }
121
+
122
+ The results will now capture `:Calculator` in `class_name`, and then `:add` and `:sub`
123
+ in `method_name`.
124
+
125
+ == Examples
126
+
127
+ Here is an example of using SexpPath to grab all the classes and their methods from
128
+ a file:
129
+
130
+ require 'rubygems'
131
+ require 'sexp_path'
132
+ require 'parse_tree'
133
+
134
+ path = ARGV.shift
135
+ code = File.read(path)
136
+ sexp = Sexp.from_array(ParseTree.new.parse_tree_for_string(code, path))
137
+
138
+ class_query = Q?{ s(:class, atom % 'class_name', _, _) }
139
+ method_query = Q?{ s(:defn, atom % 'method_name', _ ) }
140
+
141
+ results = sexp / class_query / method_query
142
+
143
+ puts path
144
+ puts "-" * 80
145
+
146
+ results.each do |sexp_result|
147
+ class_name = sexp_result['class_name']
148
+ method_name = sexp_result['method_name']
149
+ puts "#{class_name}##{method_name}"
150
+ end
151
+
152
+ Neat huh? Check the `examples` folder for some more little apps.
153
+
154
+ == Project Information
155
+
156
+ Hop in and fork it or add some issues over at GitHub:
157
+ http://github.com/adamsanderson/sexp_path
158
+
159
+ Ideas for Hacking on SexpPath:
160
+
161
+ * More examples
162
+ * Add new matchers
163
+ * Connivence matchers, for instance canned matchers for matching ruby classes, methods, etc
164
+
165
+ I'd love to see what people do with this library, let me know if you find it useful.
166
+
167
+ Adam Sanderson, netghost@gmail.com
@@ -0,0 +1,41 @@
1
+ require 'rake'
2
+ require 'rake/testtask'
3
+ require 'rake/rdoctask'
4
+
5
+ begin
6
+ require 'jeweler'
7
+
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "sexp_path"
10
+ s.summary = "Pattern matching for S-Expressions (sexp)."
11
+ s.description = <<-DESC
12
+ Allows you to do example based pattern matching and queries against S Expressions (sexp).
13
+ DESC
14
+ s.email = "netghost@gmail.com"
15
+ s.homepage = "http://github.com/adamsanderson/sexp_path"
16
+ s.authors = ["Adam Sanderson"]
17
+ s.files = FileList["[A-Z]*", "{bin,lib,test,examples}/**/*"]
18
+
19
+ s.add_dependency 'sexp_processor', '~> 3.0'
20
+
21
+ # Testing
22
+ s.test_files = FileList["test/**/*_test.rb"]
23
+ s.add_development_dependency 'ParseTree', '~> 2.1'
24
+ end
25
+
26
+ rescue LoadError
27
+ puts "Jeweler not available. Install it for jeweler-related tasks with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
28
+ end
29
+
30
+ Rake::RDocTask.new do |t|
31
+ t.main = "README.rdoc"
32
+ t.rdoc_files.include("README.rdoc", "lib/**/*.rb")
33
+ end
34
+
35
+ Rake::TestTask.new do |t|
36
+ t.libs << 'lib'
37
+ t.pattern = 'test/**/*_test.rb'
38
+ t.verbose = false
39
+ end
40
+
41
+ task :default => :test
data/TODO ADDED
@@ -0,0 +1,4 @@
1
+ * Clean up class organization
2
+ * Document extending SexpPath
3
+ * Add examples of extending matchers
4
+ * AssertSanity!
@@ -0,0 +1,4 @@
1
+ ---
2
+ :minor: 4
3
+ :patch: 0
4
+ :major: 0
@@ -0,0 +1,28 @@
1
+ require 'rubygems'
2
+ require File.dirname(__FILE__) + '/../lib/sexp_path'
3
+ require 'parse_tree'
4
+
5
+ path = ARGV.shift
6
+ if !path
7
+ puts "Prints classes and methods in a file"
8
+ puts "usage:"
9
+ puts " ruby print_methods.rb <path>"
10
+ exit
11
+ end
12
+
13
+ code = File.read(path)
14
+ sexp = Sexp.from_array(ParseTree.new.parse_tree_for_string(code, path))
15
+
16
+ class_query = Q?{ s(:class, atom % 'class_name', _, _) }
17
+ method_query = Q?{ s(:defn, atom % 'method_name', _ ) }
18
+
19
+ results = sexp / class_query / method_query
20
+
21
+ puts path
22
+ puts "-" * 80
23
+
24
+ results.each do |sexp_result|
25
+ class_name = sexp_result['class_name']
26
+ method_name = sexp_result['method_name']
27
+ puts "#{class_name}##{method_name}"
28
+ end
@@ -0,0 +1,46 @@
1
+ require 'rubygems'
2
+ require File.dirname(__FILE__) + '/../lib/sexp_path'
3
+ require 'parse_tree'
4
+
5
+ # Example program, this will scan a file for anything
6
+ # matching the Sexp passed in.
7
+
8
+
9
+ pattern = ARGV.shift
10
+ paths = ARGV
11
+
12
+ if paths.empty? || !pattern
13
+ puts "Prints classes and methods in a file"
14
+ puts "usage:"
15
+ puts " ruby sexp_grep.rb <pattern> <path>"
16
+ puts "example:"
17
+ puts " ruby sexp_grep.rb t(:defn) *.rb"
18
+ exit
19
+ end
20
+
21
+ begin
22
+ # Generate the pattern, we use a little instance_eval trickery here.
23
+ pattern = SexpPath::SexpQueryBuilder.instance_eval(pattern)
24
+ rescue Exception=>ex
25
+ puts "Invalid Pattern: '#{pattern}'"
26
+ puts "Trace:"
27
+ puts ex
28
+ puts ex.backtrace
29
+ exit 1
30
+ end
31
+
32
+ # For each path the user defined, search for the SexpPath pattern
33
+ paths.each do |path|
34
+ # Parse it with ParseTree, and append line numbers
35
+ sexp = sexp = LineNumberingProcessor.process_file(path)
36
+ found = false
37
+
38
+ # Search it with the given pattern, printing any results
39
+ sexp.search_each(pattern) do |match|
40
+ if !found
41
+ puts path
42
+ found = true
43
+ end
44
+ puts "%4i: %s" % [match.sexp.line, match.sexp.inspect]
45
+ end
46
+ end
@@ -0,0 +1,57 @@
1
+ require 'rubygems'
2
+ require 'sexp_processor'
3
+
4
+ module SexpPath
5
+
6
+ # SexpPath Matchers are used to build SexpPath queries.
7
+ #
8
+ # See also: SexpQueryBuilder
9
+ module Matcher
10
+ end
11
+ end
12
+
13
+ sexp_path_root = File.dirname(__FILE__)+'/sexp_path/'
14
+ %w[
15
+ traverse
16
+ sexp_query_builder
17
+ sexp_result
18
+ sexp_collection
19
+
20
+ line_numbering_processor
21
+
22
+ matcher/base
23
+ matcher/any
24
+ matcher/all
25
+ matcher/not
26
+ matcher/child
27
+ matcher/block
28
+ matcher/atom
29
+ matcher/pattern
30
+ matcher/type
31
+ matcher/wild
32
+ matcher/include
33
+ matcher/sibling
34
+
35
+ ].each do |path|
36
+ require sexp_path_root+path
37
+ end
38
+
39
+ # Pattern building helper, see SexpQueryBuilder
40
+ def Q?(&block)
41
+ SexpPath::SexpQueryBuilder.do(&block)
42
+ end
43
+
44
+ # SexpPath extends Sexp with Traverse.
45
+ # This adds support for searching S-Expressions
46
+ class Sexp
47
+ include SexpPath::Traverse
48
+
49
+ # Extends Sexp to allow any Sexp to be used as a SexpPath matcher
50
+ def satisfy?(o, data={})
51
+ return false unless o.is_a? Sexp
52
+ return false unless length == o.length
53
+ each_with_index{|c,i| return false unless c.is_a?(Sexp) ? c.satisfy?( o[i], data ) : c == o[i] }
54
+
55
+ capture_match(o, data)
56
+ end
57
+ end
@@ -0,0 +1,60 @@
1
+ require 'parse_tree' rescue nil
2
+
3
+ # Transforms a Sexp, keeping track of newlines. This uses the internal ruby newline nodes
4
+ # so they must be included in the Sexp to be transformed. If ParseTree is being used, it should
5
+ # be configured to include newlines:
6
+ #
7
+ # parser = ParseTree.new(true) # true => include_newlines
8
+ #
9
+ # LineNumberingProcessor.rewrite_file(path) should be used as a short cut if ParseTree is available.
10
+ #
11
+ class LineNumberingProcessor < SexpProcessor
12
+ # Helper method for generating a Sexp with line numbers from a file at +path+.
13
+ #
14
+ # Only available if ParseTree is loaded.
15
+ def self.rewrite_file(path)
16
+ raise 'ParseTree must be installed.' unless Object.const_defined? :ParseTree
17
+
18
+ code = File.read(path)
19
+ sexp = Sexp.from_array(ParseTree.new(true).parse_tree_for_string(code, path).first)
20
+ processor = LineNumberingProcessor.new
21
+
22
+ # Fill in the first lines with a value
23
+ sexp.line = 0
24
+ sexp.file = path
25
+
26
+ # Rewrite the sexp so that everything gets a line number if possible.
27
+ processor.rewrite sexp
28
+ end
29
+
30
+ # Creates a new LineNumberingProcessor.
31
+ def initialize()
32
+ super
33
+ @unsupported.delete :newline
34
+ end
35
+
36
+ # Rewrites a Sexp using :newline nodes to fill in line and file information.
37
+ def rewrite exp
38
+ unless exp.nil?
39
+ if exp.sexp_type == :newline
40
+ @line = exp[1]
41
+ @file = exp[2]
42
+ end
43
+
44
+ exp.file ||= @file
45
+ exp.line ||= @line
46
+ end
47
+
48
+ super exp
49
+ end
50
+
51
+ private
52
+ # Removes newlines from the expression, they are read inside of rewrite, and used to give
53
+ # the other nodes a line number and file.
54
+ def rewrite_newline(exp)
55
+ # New lines look like:
56
+ # s(:newline, 21, "test/sample.rb", s(:call, nil, :private, s(:arglist)) )
57
+ sexp = exp[3]
58
+ rewrite(sexp)
59
+ end
60
+ end
@@ -0,0 +1,20 @@
1
+ # See SexpQueryBuilder.all
2
+ class SexpPath::Matcher::All < SexpPath::Matcher::Base
3
+ attr_reader :options
4
+
5
+ # Create an All matcher which will match all of the +options+.
6
+ def initialize(*options)
7
+ @options = options
8
+ end
9
+
10
+ # Satisfied when all sub expressions match +o+
11
+ def satisfy?(o, data={})
12
+ return nil unless options.all?{|exp| exp.is_a?(Sexp) ? exp.satisfy?(o, data) : exp == o}
13
+
14
+ capture_match o, data
15
+ end
16
+
17
+ def inspect
18
+ options.map{|o| o.inspect}.join(' & ')
19
+ end
20
+ end
@@ -0,0 +1,20 @@
1
+ # See SexpQueryBuilder.any
2
+ class SexpPath::Matcher::Any < SexpPath::Matcher::Base
3
+ attr_reader :options
4
+
5
+ # Create an Any matcher which will match any of the +options+.
6
+ def initialize(*options)
7
+ @options = options
8
+ end
9
+
10
+ # Satisfied when any sub expressions match +o+
11
+ def satisfy?(o, data={})
12
+ return nil unless options.any?{|exp| exp.is_a?(Sexp) ? exp.satisfy?(o, data) : exp == o}
13
+
14
+ capture_match o, data
15
+ end
16
+
17
+ def inspect
18
+ options.map{|o| o.inspect}.join(' | ')
19
+ end
20
+ end