syntax_tree 4.0.2 → 4.2.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 959bd2a452303ffb7565ee0ce9641cc1167609c7c2231fc12d476d35db44ae34
4
- data.tar.gz: d1d15bf2076e7ee83caf30d1371d3d2e3c43415349e5e89e39efca68ccda0339
3
+ metadata.gz: 444c1c74319e55d9c3a656ff1796e6658237c0433a257eb6e2c66d8625d47243
4
+ data.tar.gz: 2662a60c92265c6b257f6545ee1947269e1990f96003c60caef790eeb871eb32
5
5
  SHA512:
6
- metadata.gz: 814586875ea83553dcf691b374056617b0491af9237c5fc4332da559cec7cf459593b515549cdf33c2a5b3bd45a79b01b8fa9c6bb027646799f3fb935b1b37b1
7
- data.tar.gz: a7a70c1d9b77b7c9f3c460973760cabf7446fa794e1254da25536657e7094c9bbcdb6522f08a00a26fe317d982b31b48e9f99758e2a9dde093b5c7998cdf047a
6
+ metadata.gz: 9dddc08b2d7c2c3c7cec089f301b88220cf18a625ef7b155ad71810eaefd4cfe54c063279a4f817bc823b1eb586a222b106aa5889f3b736da901ca0404feedb5
7
+ data.tar.gz: eaabea185c2f53aaf62ea76b1bb89987d7c472538025ae3f3e457f97eafe38b2533895f504100ed534785ffebaa18a013dd8d1f27432da974beedf0b7202861c
data/.gitignore CHANGED
@@ -9,3 +9,4 @@
9
9
  /vendor/
10
10
 
11
11
  test.rb
12
+ query.txt
data/.rubocop.yml CHANGED
@@ -28,6 +28,9 @@ Lint/InterpolationCheck:
28
28
  Lint/MissingSuper:
29
29
  Enabled: false
30
30
 
31
+ Lint/RedundantRequireStatement:
32
+ Enabled: false
33
+
31
34
  Lint/UnusedMethodArgument:
32
35
  AllowUnusedKeywordArguments: true
33
36
 
data/CHANGELOG.md CHANGED
@@ -6,6 +6,23 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [4.2.0] - 2022-10-25
10
+
11
+ ### Added
12
+
13
+ - [#182](https://github.com/ruby-syntax-tree/syntax_tree/pull/182) - The new `stree expr` CLI command will function similarly to the `stree match` CLI command except that it only outputs the first expression of the program.
14
+ - [#182](https://github.com/ruby-syntax-tree/syntax_tree/pull/182) - Added the `SyntaxTree::Pattern` class for compiling `in` expressions into procs.
15
+
16
+ ### Changed
17
+
18
+ - [#182](https://github.com/ruby-syntax-tree/syntax_tree/pull/182) - Much more syntax is now supported by the search command.
19
+
20
+ ## [4.1.0] - 2022-10-24
21
+
22
+ ### Added
23
+
24
+ - [#180](https://github.com/ruby-syntax-tree/syntax_tree/pull/180) - The new `stree search` CLI command and the corresponding `SyntaxTree::Search` class for searching for a pattern against a given syntax tree.
25
+
9
26
  ## [4.0.2] - 2022-10-19
10
27
 
11
28
  ### Changed
@@ -397,7 +414,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
397
414
 
398
415
  - 🎉 Initial release! 🎉
399
416
 
400
- [unreleased]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.0.2...HEAD
417
+ [unreleased]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.2.0...HEAD
418
+ [4.2.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.1.0...v4.2.0
419
+ [4.1.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.0.2...v4.1.0
401
420
  [4.0.2]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.0.1...v4.0.2
402
421
  [4.0.1]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v4.0.0...v4.0.1
403
422
  [4.0.0]: https://github.com/ruby-syntax-tree/syntax_tree/compare/v3.6.3...v4.0.0
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- syntax_tree (4.0.2)
4
+ syntax_tree (4.2.0)
5
5
  prettier_print (>= 1.0.2)
6
6
 
7
7
  GEM
@@ -19,17 +19,17 @@ GEM
19
19
  rake (13.0.6)
20
20
  regexp_parser (2.6.0)
21
21
  rexml (3.2.5)
22
- rubocop (1.36.0)
22
+ rubocop (1.37.1)
23
23
  json (~> 2.3)
24
24
  parallel (~> 1.10)
25
25
  parser (>= 3.1.2.1)
26
26
  rainbow (>= 2.2.2, < 4.0)
27
27
  regexp_parser (>= 1.8, < 3.0)
28
28
  rexml (>= 3.2.5, < 4.0)
29
- rubocop-ast (>= 1.20.1, < 2.0)
29
+ rubocop-ast (>= 1.23.0, < 2.0)
30
30
  ruby-progressbar (~> 1.7)
31
31
  unicode-display_width (>= 1.4.0, < 3.0)
32
- rubocop-ast (1.22.0)
32
+ rubocop-ast (1.23.0)
33
33
  parser (>= 3.1.1.0)
34
34
  ruby-progressbar (1.11.0)
35
35
  simplecov (0.21.2)
data/README.md CHANGED
@@ -15,9 +15,11 @@ It is built with only standard library dependencies. It additionally ships with
15
15
  - [CLI](#cli)
16
16
  - [ast](#ast)
17
17
  - [check](#check)
18
+ - [expr](#expr)
18
19
  - [format](#format)
19
20
  - [json](#json)
20
21
  - [match](#match)
22
+ - [search](#search)
21
23
  - [write](#write)
22
24
  - [Configuration](#configuration)
23
25
  - [Globbing](#globbing)
@@ -25,6 +27,7 @@ It is built with only standard library dependencies. It additionally ships with
25
27
  - [SyntaxTree.read(filepath)](#syntaxtreereadfilepath)
26
28
  - [SyntaxTree.parse(source)](#syntaxtreeparsesource)
27
29
  - [SyntaxTree.format(source)](#syntaxtreeformatsource)
30
+ - [SyntaxTree.search(source, query, &block)](#syntaxtreesearchsource-query-block)
28
31
  - [Nodes](#nodes)
29
32
  - [child_nodes](#child_nodes)
30
33
  - [Pattern matching](#pattern-matching)
@@ -128,6 +131,24 @@ To change the print width that you are checking against, specify the `--print-wi
128
131
  stree check --print-width=100 path/to/file.rb
129
132
  ```
130
133
 
134
+ ### expr
135
+
136
+ This command will output a Ruby case-match expression that would match correctly against the first expression of the input.
137
+
138
+ ```sh
139
+ stree expr path/to/file.rb
140
+ ```
141
+
142
+ For a file that contains `1 + 1`, you will receive:
143
+
144
+ ```ruby
145
+ SyntaxTree::Binary[
146
+ left: SyntaxTree::Int[value: "1"],
147
+ operator: :+,
148
+ right: SyntaxTree::Int[value: "1"]
149
+ ]
150
+ ```
151
+
131
152
  ### format
132
153
 
133
154
  This command will output the formatted version of each of the listed files. Importantly, it will not write that content back to the source files. It is meant to display the formatted version only.
@@ -215,6 +236,29 @@ SyntaxTree::Program[
215
236
  ]
216
237
  ```
217
238
 
239
+ ### search
240
+
241
+ This command will search the given filepaths against the specified pattern to find nodes that match. The pattern is a Ruby pattern-matching expression that is matched against each node in the tree. It can optionally be loaded from a file if you specify a filepath as the pattern argument.
242
+
243
+ ```sh
244
+ stree search VarRef path/to/file.rb
245
+ ```
246
+
247
+ For a file that contains `Foo + Bar` you will receive:
248
+
249
+ ```
250
+ path/to/file.rb:1:0: Foo + Bar
251
+ path/to/file.rb:1:6: Foo + Bar
252
+ ```
253
+
254
+ If you put `VarRef` into a file instead (for example, `query.txt`), you would instead run:
255
+
256
+ ```sh
257
+ stree search query.txt path/to/file.rb
258
+ ```
259
+
260
+ Note that the output of the `match` CLI command creates a valid pattern that can be used as the input for this command.
261
+
218
262
  ### write
219
263
 
220
264
  This command will format the listed files and write that formatted version back to the source files. Note that this overwrites the original content, to be sure to be using a version control system.
@@ -288,6 +332,10 @@ This function takes an input string containing Ruby code and returns the syntax
288
332
 
289
333
  This function takes an input string containing Ruby code, parses it into its underlying syntax tree, and formats it back out to a string. You can optionally pass a second argument to this method as well that is the maximum width to print. It defaults to `80`.
290
334
 
335
+ ### SyntaxTree.search(source, query, &block)
336
+
337
+ This function takes an input string containing Ruby code, an input string containing a valid Ruby `in` clause expression that can be used to match against nodes in the tree (can be generated using `stree expr`, `stree match`, or `Node#construct_keys`), and a block. Each node that matches the given query will be yielded to the block. The block will receive the node as its only argument.
338
+
291
339
  ## Nodes
292
340
 
293
341
  There are many different node types in the syntax tree. They are meant to be treated as immutable structs containing links to child nodes with minimal logic contained within their implementation. However, for the most part they all respond to a certain set of APIs, listed below.
@@ -188,6 +188,20 @@ module SyntaxTree
188
188
  end
189
189
  end
190
190
 
191
+ # An action of the CLI that outputs a pattern-matching Ruby expression that
192
+ # would match the first expression of the input given.
193
+ class Expr < Action
194
+ def run(item)
195
+ case item.handler.parse(item.source)
196
+ in Program[statements: Statements[body: [expression]]]
197
+ puts expression.construct_keys
198
+ else
199
+ warn("The input to `stree expr` must be a single expression.")
200
+ exit(1)
201
+ end
202
+ end
203
+ end
204
+
191
205
  # An action of the CLI that formats the input source and prints it out.
192
206
  class Format < Action
193
207
  def run(item)
@@ -212,6 +226,44 @@ module SyntaxTree
212
226
  end
213
227
  end
214
228
 
229
+ # An action of the CLI that searches for the given pattern matching pattern
230
+ # in the given files.
231
+ class Search < Action
232
+ attr_reader :search
233
+
234
+ def initialize(query)
235
+ query = File.read(query) if File.readable?(query)
236
+ pattern =
237
+ begin
238
+ Pattern.new(query).compile
239
+ rescue Pattern::CompilationError => error
240
+ warn(error.message)
241
+ exit(1)
242
+ end
243
+
244
+ @search = SyntaxTree::Search.new(pattern)
245
+ end
246
+
247
+ def run(item)
248
+ search.scan(item.handler.parse(item.source)) do |node|
249
+ location = node.location
250
+ line = location.start_line
251
+
252
+ bold_range =
253
+ if line == location.end_line
254
+ location.start_column...location.end_column
255
+ else
256
+ location.start_column..
257
+ end
258
+
259
+ source = item.source.lines[line - 1].chomp
260
+ source[bold_range] = Color.bold(source[bold_range]).to_s
261
+
262
+ puts("#{item.filepath}:#{line}:#{location.start_column}: #{source}")
263
+ end
264
+ end
265
+ end
266
+
215
267
  # An action of the CLI that formats the input source and writes the
216
268
  # formatted output back to the file.
217
269
  class Write < Action
@@ -248,6 +300,10 @@ module SyntaxTree
248
300
  #{Color.bold("stree doc [--plugins=...] [-e SCRIPT] FILE")}
249
301
  Print out the doc tree that would be used to format the given files
250
302
 
303
+ #{Color.bold("stree expr [-e SCRIPT] FILE")}
304
+ Print out a pattern-matching Ruby expression that would match the first
305
+ expression of the given files
306
+
251
307
  #{Color.bold("stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE")}
252
308
  Print out the formatted version of the given files
253
309
 
@@ -263,6 +319,9 @@ module SyntaxTree
263
319
  #{Color.bold("stree lsp [--plugins=...] [--print-width=NUMBER]")}
264
320
  Run syntax tree in language server mode
265
321
 
322
+ #{Color.bold("stree search PATTERN [-e SCRIPT] FILE")}
323
+ Search for the given pattern in the given files
324
+
266
325
  #{Color.bold("stree version")}
267
326
  Output the current version of syntax tree
268
327
 
@@ -400,6 +459,10 @@ module SyntaxTree
400
459
  Debug.new(options)
401
460
  when "doc"
402
461
  Doc.new(options)
462
+ when "e", "expr"
463
+ Expr.new(options)
464
+ when "f", "format"
465
+ Format.new(options)
403
466
  when "help"
404
467
  puts HELP
405
468
  return 0
@@ -411,8 +474,8 @@ module SyntaxTree
411
474
  return 0
412
475
  when "m", "match"
413
476
  Match.new(options)
414
- when "f", "format"
415
- Format.new(options)
477
+ when "s", "search"
478
+ Search.new(arguments.shift)
416
479
  when "version"
417
480
  puts SyntaxTree::VERSION
418
481
  return 0
@@ -434,7 +497,7 @@ module SyntaxTree
434
497
  .glob(pattern)
435
498
  .each do |filepath|
436
499
  if File.readable?(filepath) &&
437
- options.ignore_files.none? { File.fnmatch?(_1, filepath) }
500
+ options.ignore_files.none? { File.fnmatch?(_1, filepath) }
438
501
  queue << FileItem.new(filepath)
439
502
  end
440
503
  end
@@ -1657,12 +1657,12 @@ module SyntaxTree
1657
1657
  # for older Ruby versions.
1658
1658
  unless :+.respond_to?(:name)
1659
1659
  using Module.new {
1660
- refine Symbol do
1661
- def name
1662
- to_s.freeze
1663
- end
1664
- end
1665
- }
1660
+ refine Symbol do
1661
+ def name
1662
+ to_s.freeze
1663
+ end
1664
+ end
1665
+ }
1666
1666
  end
1667
1667
 
1668
1668
  # [untyped] the left-hand side of the expression
@@ -0,0 +1,172 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SyntaxTree
4
+ # A pattern is an object that wraps a Ruby pattern matching expression. The
5
+ # expression would normally be passed to an `in` clause within a `case`
6
+ # expression or a rightward assignment expression. For example, in the
7
+ # following snippet:
8
+ #
9
+ # case node
10
+ # in Const[value: "SyntaxTree"]
11
+ # end
12
+ #
13
+ # the pattern is the `Const[value: "SyntaxTree"]` expression. Within Syntax
14
+ # Tree, every node generates these kinds of expressions using the
15
+ # #construct_keys method.
16
+ #
17
+ # The pattern gets compiled into an object that responds to call by running
18
+ # the #compile method. This method itself will run back through Syntax Tree to
19
+ # parse the expression into a tree, then walk the tree to generate the
20
+ # necessary callable objects. For example, if you wanted to compile the
21
+ # expression above into a callable, you would:
22
+ #
23
+ # callable = SyntaxTree::Pattern.new("Const[value: 'SyntaxTree']").compile
24
+ # callable.call(node)
25
+ #
26
+ # The callable object returned by #compile is guaranteed to respond to #call
27
+ # with a single argument, which is the node to match against. It also is
28
+ # guaranteed to respond to #===, which means it itself can be used in a `case`
29
+ # expression, as in:
30
+ #
31
+ # case node
32
+ # when callable
33
+ # end
34
+ #
35
+ # If the query given to the initializer cannot be compiled into a valid
36
+ # matcher (either because of a syntax error or because it is using syntax we
37
+ # do not yet support) then a SyntaxTree::Pattern::CompilationError will be
38
+ # raised.
39
+ class Pattern
40
+ # Raised when the query given to a pattern is either invalid Ruby syntax or
41
+ # is using syntax that we don't yet support.
42
+ class CompilationError < StandardError
43
+ def initialize(repr)
44
+ super(<<~ERROR)
45
+ Syntax Tree was unable to compile the pattern you provided to search
46
+ into a usable expression. It failed on to understand the node
47
+ represented by:
48
+
49
+ #{repr}
50
+
51
+ Note that not all syntax supported by Ruby's pattern matching syntax
52
+ is also supported by Syntax Tree's code search. If you're using some
53
+ syntax that you believe should be supported, please open an issue on
54
+ GitHub at https://github.com/ruby-syntax-tree/syntax_tree/issues/new.
55
+ ERROR
56
+ end
57
+ end
58
+
59
+ attr_reader :query
60
+
61
+ def initialize(query)
62
+ @query = query
63
+ end
64
+
65
+ def compile
66
+ program =
67
+ begin
68
+ SyntaxTree.parse("case nil\nin #{query}\nend")
69
+ rescue Parser::ParseError
70
+ raise CompilationError, query
71
+ end
72
+
73
+ compile_node(program.statements.body.first.consequent.pattern)
74
+ end
75
+
76
+ private
77
+
78
+ def combine_and(left, right)
79
+ ->(node) { left.call(node) && right.call(node) }
80
+ end
81
+
82
+ def combine_or(left, right)
83
+ ->(node) { left.call(node) || right.call(node) }
84
+ end
85
+
86
+ def compile_node(root)
87
+ case root
88
+ in AryPtn[constant:, requireds:, rest: nil, posts: []]
89
+ compiled_constant = compile_node(constant) if constant
90
+
91
+ preprocessed = requireds.map { |required| compile_node(required) }
92
+
93
+ compiled_requireds = ->(node) do
94
+ deconstructed = node.deconstruct
95
+
96
+ deconstructed.length == preprocessed.length &&
97
+ preprocessed
98
+ .zip(deconstructed)
99
+ .all? { |(matcher, value)| matcher.call(value) }
100
+ end
101
+
102
+ if compiled_constant
103
+ combine_and(compiled_constant, compiled_requireds)
104
+ else
105
+ compiled_requireds
106
+ end
107
+ in Binary[left:, operator: :|, right:]
108
+ combine_or(compile_node(left), compile_node(right))
109
+ in Const[value:] if SyntaxTree.const_defined?(value)
110
+ clazz = SyntaxTree.const_get(value)
111
+
112
+ ->(node) { node.is_a?(clazz) }
113
+ in Const[value:] if Object.const_defined?(value)
114
+ clazz = Object.const_get(value)
115
+
116
+ ->(node) { node.is_a?(clazz) }
117
+ in ConstPathRef[
118
+ parent: VarRef[value: Const[value: "SyntaxTree"]], constant:
119
+ ]
120
+ compile_node(constant)
121
+ in DynaSymbol[parts: []]
122
+ symbol = :""
123
+
124
+ ->(node) { node == symbol }
125
+ in DynaSymbol[parts: [TStringContent[value:]]]
126
+ symbol = value.to_sym
127
+
128
+ ->(attribute) { attribute == value }
129
+ in HshPtn[constant:, keywords:, keyword_rest: nil]
130
+ compiled_constant = compile_node(constant)
131
+
132
+ preprocessed =
133
+ keywords.to_h do |keyword, value|
134
+ raise NoMatchingPatternError unless keyword.is_a?(Label)
135
+ [keyword.value.chomp(":").to_sym, compile_node(value)]
136
+ end
137
+
138
+ compiled_keywords = ->(node) do
139
+ deconstructed = node.deconstruct_keys(preprocessed.keys)
140
+
141
+ preprocessed.all? do |keyword, matcher|
142
+ matcher.call(deconstructed[keyword])
143
+ end
144
+ end
145
+
146
+ if compiled_constant
147
+ combine_and(compiled_constant, compiled_keywords)
148
+ else
149
+ compiled_keywords
150
+ end
151
+ in RegexpLiteral[parts: [TStringContent[value:]]]
152
+ regexp = /#{value}/
153
+
154
+ ->(attribute) { regexp.match?(attribute) }
155
+ in StringLiteral[parts: []]
156
+ ->(attribute) { attribute == "" }
157
+ in StringLiteral[parts: [TStringContent[value:]]]
158
+ ->(attribute) { attribute == value }
159
+ in SymbolLiteral[value:]
160
+ symbol = value.value.to_sym
161
+
162
+ ->(attribute) { attribute == symbol }
163
+ in VarRef[value: Const => value]
164
+ compile_node(value)
165
+ in VarRef[value: Kw[value: "nil"]]
166
+ ->(attribute) { attribute.nil? }
167
+ end
168
+ rescue NoMatchingPatternError
169
+ raise CompilationError, PP.pp(root, +"").chomp
170
+ end
171
+ end
172
+ end
@@ -0,0 +1,26 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SyntaxTree
4
+ # Provides an interface for searching for a pattern of nodes against a
5
+ # subtree of an AST.
6
+ class Search
7
+ attr_reader :pattern
8
+
9
+ def initialize(pattern)
10
+ @pattern = pattern
11
+ end
12
+
13
+ def scan(root)
14
+ return to_enum(__method__, root) unless block_given?
15
+ queue = [root]
16
+
17
+ until queue.empty?
18
+ node = queue.shift
19
+ next unless node
20
+
21
+ yield node if pattern.call(node)
22
+ queue += node.child_nodes
23
+ end
24
+ end
25
+ end
26
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SyntaxTree
4
- VERSION = "4.0.2"
4
+ VERSION = "4.2.0"
5
5
  end
data/lib/syntax_tree.rb CHANGED
@@ -21,6 +21,8 @@ require_relative "syntax_tree/visitor/environment"
21
21
  require_relative "syntax_tree/visitor/with_environment"
22
22
 
23
23
  require_relative "syntax_tree/parser"
24
+ require_relative "syntax_tree/pattern"
25
+ require_relative "syntax_tree/search"
24
26
 
25
27
  # Syntax Tree is a suite of tools built on top of the internal CRuby parser. It
26
28
  # provides the ability to generate a syntax tree from source, as well as the
@@ -73,4 +75,10 @@ module SyntaxTree
73
75
 
74
76
  File.read(filepath, encoding: encoding)
75
77
  end
78
+
79
+ # Searches through the given source using the given pattern and yields each
80
+ # node in the tree that matches the pattern to the given block.
81
+ def self.search(source, query, &block)
82
+ Search.new(Pattern.new(query).compile).scan(parse(source), &block)
83
+ end
76
84
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: syntax_tree
3
3
  version: !ruby/object:Gem::Version
4
- version: 4.0.2
4
+ version: 4.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Kevin Newton
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-10-19 00:00:00.000000000 Z
11
+ date: 2022-10-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: prettier_print
@@ -129,12 +129,14 @@ files:
129
129
  - lib/syntax_tree/language_server/inlay_hints.rb
130
130
  - lib/syntax_tree/node.rb
131
131
  - lib/syntax_tree/parser.rb
132
+ - lib/syntax_tree/pattern.rb
132
133
  - lib/syntax_tree/plugin/single_quotes.rb
133
134
  - lib/syntax_tree/plugin/trailing_comma.rb
134
135
  - lib/syntax_tree/rake/check_task.rb
135
136
  - lib/syntax_tree/rake/task.rb
136
137
  - lib/syntax_tree/rake/write_task.rb
137
138
  - lib/syntax_tree/rake_tasks.rb
139
+ - lib/syntax_tree/search.rb
138
140
  - lib/syntax_tree/version.rb
139
141
  - lib/syntax_tree/visitor.rb
140
142
  - lib/syntax_tree/visitor/environment.rb