ruby_parser 3.0.0.a8 → 3.0.0.a9

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of ruby_parser might be problematic. Click here for more details.

data.tar.gz.sig CHANGED
Binary file
data/.autotest CHANGED
@@ -13,6 +13,12 @@ Autotest.add_hook :initialize do |at|
13
13
  at.add_exception '.diff'
14
14
  at.add_exception 'rubycorpus'
15
15
 
16
+ dirs = Dir["corpus*"] + Dir["gauntlet*"]
17
+
18
+ dirs.each do |f|
19
+ at.add_exception f
20
+ end
21
+
16
22
  at.libs << ':../../minitest/dev/lib'
17
23
  at.testlib = "minitest/autorun"
18
24
 
data/History.txt CHANGED
@@ -1,3 +1,43 @@
1
+ === 3.0.0.a9 / 2012-10-22
2
+
3
+ Holy crap! I've hit a 99.92% success rate against 558k files! 492
4
+ failed parses to go. NOW is the time to start testing ruby_parser
5
+ against your code!
6
+
7
+ * 16 minor enhancements:
8
+
9
+ * 1.9 parser: Finished magic encoding and utf-8 bom support.
10
+ * 1.9: Added leading dot support eg "a\n.b".
11
+ * 1.9: Added partial handling for *arg and opt=val args in block args.
12
+ * 1.9: OMFG Encoding is the worst thing ever. Overhauled encoding support once again. It is much better at doing last ditch guessing of the encoding of the source file.
13
+ * 1.9: added String#grep for legacy support.
14
+ * Added Sexp#block_pass?
15
+ * Added ability to delete slow files when they timeout.
16
+ * Added block_dup_check and refactored grammar code to use it for block arg handling.
17
+ * Added in_lex_state?(*states) to clean up lexer code.
18
+ * Added tentative support for optional block args. THIS IS SUBJECT TO CHANGE!!
19
+ * Added toggleable debugging to StackState to make my life easier
20
+ * All untested uses have not moved but have been laced with poison.
21
+ * Finally got good tests for most forms of double-block arg errors.
22
+ * Moved all _known_ uses of SyntaxError to RubyParser::SyntaxError
23
+ * f_block_optarg now always returns a block node.
24
+ * ruby_parse_extract_error uses Find.find instead of globbing so dot files aren't missed.
25
+
26
+ * 12 bug fixes:
27
+
28
+ * 1.9: Completely ignore IndexError in unread_many because... you know... it sucks.
29
+ * 1.9: Fixed lex state after lexing ? in trinary.
30
+ * 1.9: Fixed lex state in some ternarys.
31
+ * 1.9: Fixed parsing of "1 ? b('') : 2\na d: 3"... ARE YOU NOT GLAD?!?!
32
+ * Fix Timeout differences between 1.8 and 1.9 :(
33
+ * Fixed emacs-style encodings to deal with no whitespace.
34
+ * Fixed error message for bad % codes. (whitequark)
35
+ * Fixed lexing of :a==>b vs :a===b vs :a==>b. P.S. Your space bar is broken. chump.
36
+ * Fixed lexing of rare numeric formats.
37
+ * Fixed magic comment encodings on DOS files ... :/
38
+ * Fixed ruby_parse_extract_error to exit non-zero on errors.
39
+ * Removed Symbol#is_argument and switch usage to is_arg?
40
+
1
41
  === 3.0.0.a8 / 2012-09-26
2
42
 
3
43
  * 1 bug fix:
data/README.txt CHANGED
@@ -13,24 +13,19 @@ base types.
13
13
 
14
14
  As an example:
15
15
 
16
- def conditional1(arg1)
17
- if arg1 == 0 then
18
- return 1
19
- end
16
+ def conditional1 arg1
17
+ return 1 if arg1 == 0
20
18
  return 0
21
19
  end
22
20
 
23
21
  becomes:
24
22
 
25
- s(:defn, :conditional1,
26
- s(:args, :arg1),
27
- s(:scope,
28
- s(:block,
29
- s(:if,
30
- s(:call, s(:lvar, :arg1), :==, s(:arglist, s(:lit, 0))),
23
+ s(:defn, :conditional1, s(:args, :arg1),
24
+ s(:if,
25
+ s(:call, s(:lvar, :arg1), :==, s(:lit, 0)),
31
26
  s(:return, s(:lit, 1)),
32
27
  nil),
33
- s(:return, s(:lit, 0)))))
28
+ s(:return, s(:lit, 0)))
34
29
 
35
30
  == FEATURES/PROBLEMS:
36
31
 
data/Rakefile CHANGED
@@ -171,6 +171,11 @@ task :debug => :isolate do
171
171
  end
172
172
  end
173
173
 
174
+ task :debug_ruby do
175
+ file = ENV["F"] || ENV["FILE"]
176
+ sh "ruby19 -cwy #{file} 2>&1 | ./yuck.rb"
177
+ end
178
+
174
179
  task :extract => :isolate do
175
180
  ENV["V"] ||= "19"
176
181
  Rake.application[:parser].invoke # this way we can have DEBUG set
@@ -2,6 +2,8 @@
2
2
 
3
3
  $d ||= false
4
4
  $d ||= ENV["DELETE"]
5
+ $t ||= false
6
+ $t ||= ENV["DELETE_TIMEOUT"]
5
7
  $q ||= false
6
8
  $q ||= ENV["QUIET"]
7
9
 
@@ -13,6 +15,9 @@ ARGV.push "-" if ARGV.empty?
13
15
  class Racc::Parser
14
16
  def extract_defs
15
17
  ss = lexer.src
18
+
19
+ raise "can't access source. possible encoding issue" unless ss
20
+
16
21
  src = ss.string
17
22
  pre_error = src[0...ss.pos]
18
23
 
@@ -43,7 +48,15 @@ end
43
48
 
44
49
  def expand path
45
50
  if File.directory? path then
46
- Dir.glob File.join path, "**/{*.rb,*.rake,Rakefile}"
51
+ require 'find'
52
+
53
+ files = []
54
+
55
+ Find.find(*Dir[path]) do |f|
56
+ files << f if File.file? f
57
+ end
58
+
59
+ files
47
60
  else
48
61
  Dir.glob path
49
62
  end
@@ -52,8 +65,6 @@ end
52
65
  def process_error parser
53
66
  defs = parser.extract_defs
54
67
 
55
- orig_size = defs.size
56
-
57
68
  if parser.retest_for_errors defs then
58
69
  warn "Can't reproduce error with just methods, punting..."
59
70
  return
@@ -70,7 +81,7 @@ def process_error parser
70
81
  end
71
82
  end
72
83
  rescue RuntimeError, Racc::ParseError => e
73
- warn "# error: #{e.message.strip}"
84
+ warn "# process error: #{e.message.strip}"
74
85
  end
75
86
 
76
87
  def process file
@@ -81,17 +92,21 @@ def process file
81
92
  parser.process(ruby, file)
82
93
  warn "good"
83
94
  File.unlink file if $d
95
+ rescue Timeout::Error
96
+ $exit = 1
97
+ warn "TIMEOUT parsing #{file}. Skipping."
98
+ File.unlink file if $t
84
99
  rescue StandardError, SyntaxError, Racc::ParseError => e
100
+ $exit = 1
85
101
  warn ""
86
102
  warn "# error: #{e.message.strip}" unless $q
87
103
  warn ""
88
104
  return if $q
89
105
 
90
106
  process_error parser
91
- rescue Timeout::Error
92
- warn "TIMEOUT parsing #{file}. Skipping."
93
107
  end
94
108
 
109
+ $exit = 0
95
110
  $stdout.sync = true
96
111
 
97
112
  ARGV.each do |path|
@@ -100,3 +115,5 @@ ARGV.each do |path|
100
115
  process file
101
116
  end
102
117
  end
118
+
119
+ exit $exit
data/lib/ruby18_parser.rb CHANGED
@@ -3345,10 +3345,10 @@ end
3345
3345
 
3346
3346
  def _reduce_54(val, _values, result)
3347
3347
  result = new_call nil, val[0].to_sym, val[1]
3348
+
3348
3349
  if val[2] then
3349
- if result[0] == :block_pass then
3350
- raise "both block arg and actual block given"
3351
- end
3350
+ block_dup_check result, val[2]
3351
+
3352
3352
  result, operation = val[2], result
3353
3353
  result.insert 1, operation
3354
3354
  end
@@ -3364,6 +3364,14 @@ end
3364
3364
 
3365
3365
  def _reduce_56(val, _values, result)
3366
3366
  result = new_call val[0], val[2].to_sym, val[3]
3367
+ raise "no2"
3368
+
3369
+ if val[4] then
3370
+ block_dup_check result, val[4]
3371
+
3372
+ val[2] << result
3373
+ result = val[2]
3374
+ end
3367
3375
 
3368
3376
  result
3369
3377
  end
@@ -3376,10 +3384,11 @@ end
3376
3384
 
3377
3385
  def _reduce_58(val, _values, result)
3378
3386
  result = new_call val[0], val[2].to_sym, val[3]
3387
+ raise "no3"
3388
+
3379
3389
  if val[4] then
3380
- if result[0] == :block_pass then # REFACTOR
3381
- raise "both block arg and actual block given"
3382
- end
3390
+ block_dup_check result, val[4]
3391
+
3383
3392
  val[2] << result
3384
3393
  result = val[2]
3385
3394
  end
@@ -4497,6 +4506,8 @@ end
4497
4506
 
4498
4507
  def _reduce_293(val, _values, result)
4499
4508
  call, iter = val[0], val[1]
4509
+ block_dup_check call, iter
4510
+
4500
4511
  iter.insert 1, call
4501
4512
  result = iter
4502
4513
 
@@ -4902,8 +4913,7 @@ def _reduce_361(val, _values, result)
4902
4913
  end
4903
4914
 
4904
4915
  def _reduce_362(val, _values, result)
4905
- raise SyntaxError, "Both block arg and actual block given." if
4906
- val[0] && val[0][0] == :blockpass
4916
+ block_dup_check val[0], val[1]
4907
4917
 
4908
4918
  result = val[1]
4909
4919
  result.insert 1, val[0]
data/lib/ruby18_parser.y CHANGED
@@ -258,10 +258,10 @@ rule
258
258
  | operation command_args cmd_brace_block
259
259
  {
260
260
  result = new_call nil, val[0].to_sym, val[1]
261
+
261
262
  if val[2] then
262
- if result[0] == :block_pass then
263
- raise "both block arg and actual block given"
264
- end
263
+ block_dup_check result, val[2]
264
+
265
265
  result, operation = val[2], result
266
266
  result.insert 1, operation
267
267
  end
@@ -273,6 +273,14 @@ rule
273
273
  | primary_value tDOT operation2 command_args cmd_brace_block
274
274
  {
275
275
  result = new_call val[0], val[2].to_sym, val[3]
276
+ raise "no2"
277
+
278
+ if val[4] then
279
+ block_dup_check result, val[4]
280
+
281
+ val[2] << result
282
+ result = val[2]
283
+ end
276
284
  }
277
285
  | primary_value tCOLON2 operation2 command_args =tLOWEST
278
286
  {
@@ -281,10 +289,11 @@ rule
281
289
  | primary_value tCOLON2 operation2 command_args cmd_brace_block
282
290
  {
283
291
  result = new_call val[0], val[2].to_sym, val[3]
292
+ raise "no3"
293
+
284
294
  if val[4] then
285
- if result[0] == :block_pass then # REFACTOR
286
- raise "both block arg and actual block given"
287
- end
295
+ block_dup_check result, val[4]
296
+
288
297
  val[2] << result
289
298
  result = val[2]
290
299
  end
@@ -997,6 +1006,8 @@ rule
997
1006
  | method_call brace_block
998
1007
  {
999
1008
  call, iter = val[0], val[1]
1009
+ block_dup_check call, iter
1010
+
1000
1011
  iter.insert 1, call
1001
1012
  result = iter
1002
1013
  }
@@ -1287,8 +1298,7 @@ rule
1287
1298
 
1288
1299
  block_call: command do_block
1289
1300
  {
1290
- raise SyntaxError, "Both block arg and actual block given." if
1291
- val[0] && val[0][0] == :blockpass
1301
+ block_dup_check val[0], val[1]
1292
1302
 
1293
1303
  result = val[1]
1294
1304
  result.insert 1, val[0]
data/lib/ruby19_parser.rb CHANGED
@@ -2452,7 +2452,7 @@ racc_reduce_table = [
2452
2452
  4, 271, :_reduce_386,
2453
2453
  1, 272, :_reduce_none,
2454
2454
  2, 272, :_reduce_388,
2455
- 1, 273, :_reduce_none,
2455
+ 1, 273, :_reduce_389,
2456
2456
  3, 273, :_reduce_390,
2457
2457
  1, 274, :_reduce_none,
2458
2458
  1, 274, :_reduce_none,
@@ -2593,7 +2593,7 @@ racc_reduce_table = [
2593
2593
  3, 266, :_reduce_527,
2594
2594
  3, 309, :_reduce_528,
2595
2595
  3, 310, :_reduce_529,
2596
- 1, 267, :_reduce_none,
2596
+ 1, 267, :_reduce_530,
2597
2597
  3, 267, :_reduce_531,
2598
2598
  1, 307, :_reduce_532,
2599
2599
  3, 307, :_reduce_533,
@@ -3443,9 +3443,8 @@ end
3443
3443
  def _reduce_60(val, _values, result)
3444
3444
  result = new_call nil, val[0].to_sym, val[1]
3445
3445
  if val[2] then
3446
- if result[0] == :block_pass then
3447
- raise "both block arg and actual block given"
3448
- end
3446
+ block_dup_check result, val[2]
3447
+
3449
3448
  result, operation = val[2], result
3450
3449
  result.insert 1, operation
3451
3450
  end
@@ -3460,7 +3459,13 @@ def _reduce_61(val, _values, result)
3460
3459
  end
3461
3460
 
3462
3461
  def _reduce_62(val, _values, result)
3463
- result = new_call val[0], val[2].to_sym, val[3]
3462
+ recv, _, msg, args, block = val
3463
+ call = new_call recv, msg.to_sym, args
3464
+
3465
+ block_dup_check call, block
3466
+
3467
+ block.insert 1, call
3468
+ result = block
3464
3469
 
3465
3470
  result
3466
3471
  end
@@ -3472,14 +3477,13 @@ def _reduce_63(val, _values, result)
3472
3477
  end
3473
3478
 
3474
3479
  def _reduce_64(val, _values, result)
3475
- result = new_call val[0], val[2].to_sym, val[3]
3476
- if val[4] then
3477
- if result[0] == :block_pass then # REFACTOR
3478
- raise "both block arg and actual block given"
3479
- end
3480
- val[2] << result
3481
- result = val[2]
3482
- end
3480
+ recv, _, msg, args, block = val
3481
+ call = new_call recv, msg.to_sym, args
3482
+
3483
+ block_dup_check call, block
3484
+
3485
+ block.insert 1, call
3486
+ result = block
3483
3487
 
3484
3488
  result
3485
3489
  end
@@ -4356,7 +4360,7 @@ end
4356
4360
  # reduce 257 omitted
4357
4361
 
4358
4362
  def _reduce_258(val, _values, result)
4359
- result = lexer.cmdarg.stack.dup
4363
+ result = lexer.cmdarg.stack.dup # TODO: smell?
4360
4364
  lexer.cmdarg.push true
4361
4365
 
4362
4366
  result
@@ -4546,7 +4550,7 @@ def _reduce_293(val, _values, result)
4546
4550
  end
4547
4551
 
4548
4552
  def _reduce_294(val, _values, result)
4549
- raise "no3: #{val.inspect}"
4553
+ raise "no3\non#{val.inspect}"
4550
4554
 
4551
4555
  result
4552
4556
  end
@@ -4565,7 +4569,8 @@ end
4565
4569
 
4566
4570
  def _reduce_297(val, _values, result)
4567
4571
  call, iter = val[0], val[1]
4568
- iter.insert 1, call
4572
+ block_dup_check call, iter
4573
+ iter.insert 1, call # FIX
4569
4574
  result = iter
4570
4575
 
4571
4576
  result
@@ -4887,49 +4892,49 @@ def _reduce_358(val, _values, result)
4887
4892
  end
4888
4893
 
4889
4894
  def _reduce_359(val, _values, result)
4890
- raise "no9: #{val.inspect}"
4895
+ result = block_var val[0], val[3], nil
4891
4896
 
4892
4897
  result
4893
4898
  end
4894
4899
 
4895
4900
  def _reduce_360(val, _values, result)
4896
- raise "no10: #{val.inspect}"
4901
+ raise "no10\non: #{val.inspect}"
4897
4902
 
4898
4903
  result
4899
4904
  end
4900
4905
 
4901
4906
  def _reduce_361(val, _values, result)
4902
- raise "no11: #{val.inspect}"
4907
+ raise "no11\non: #{val.inspect}"
4903
4908
 
4904
4909
  result
4905
4910
  end
4906
4911
 
4907
4912
  def _reduce_362(val, _values, result)
4908
- raise "no12: #{val.inspect}"
4913
+ raise "no12\non: #{val.inspect}"
4909
4914
 
4910
4915
  result
4911
4916
  end
4912
4917
 
4913
4918
  def _reduce_363(val, _values, result)
4914
- raise "no13: #{val.inspect}"
4919
+ raise "no13\non: #{val.inspect}"
4915
4920
 
4916
4921
  result
4917
4922
  end
4918
4923
 
4919
4924
  def _reduce_364(val, _values, result)
4920
- raise "no14: #{val.inspect}"
4925
+ raise "no14\non: #{val.inspect}"
4921
4926
 
4922
4927
  result
4923
4928
  end
4924
4929
 
4925
4930
  def _reduce_365(val, _values, result)
4926
- raise "no15: #{val.inspect}"
4931
+ raise "no15\non: #{val.inspect}"
4927
4932
 
4928
4933
  result
4929
4934
  end
4930
4935
 
4931
4936
  def _reduce_366(val, _values, result)
4932
- raise "no16: #{val.inspect}"
4937
+ raise "no16\non: #{val.inspect}"
4933
4938
 
4934
4939
  result
4935
4940
  end
@@ -4947,7 +4952,12 @@ def _reduce_368(val, _values, result)
4947
4952
  end
4948
4953
 
4949
4954
  def _reduce_369(val, _values, result)
4950
- result = block_args19 val, "3"
4955
+ arg, _, opt, block = val
4956
+
4957
+ result = arg
4958
+ result.concat opt[1..-1].map { |s| s[1] }
4959
+ result << "&#{block.last}".to_sym if block
4960
+ result << opt
4951
4961
 
4952
4962
  result
4953
4963
  end
@@ -4995,7 +5005,12 @@ def _reduce_376(val, _values, result)
4995
5005
  end
4996
5006
 
4997
5007
  def _reduce_377(val, _values, result)
4998
- result = block_args19 val, "11"
5008
+ opt, block = val
5009
+
5010
+ result = s(:args)
5011
+ result.concat opt[1..-1].map { |s| s[1] }
5012
+ result << "&#{block.last}".to_sym if block
5013
+ result << opt
4999
5014
 
5000
5015
  result
5001
5016
  end
@@ -5013,7 +5028,11 @@ def _reduce_379(val, _values, result)
5013
5028
  end
5014
5029
 
5015
5030
  def _reduce_380(val, _values, result)
5016
- result = block_args19 val, "14"
5031
+ rest, _, args, block = val
5032
+
5033
+ result = args
5034
+ result[1,0] = rest
5035
+ result << "&#{block.last}".to_sym if block
5017
5036
 
5018
5037
  result
5019
5038
  end
@@ -5056,11 +5075,15 @@ def _reduce_388(val, _values, result)
5056
5075
  result
5057
5076
  end
5058
5077
 
5059
- # reduce 389 omitted
5078
+ def _reduce_389(val, _values, result)
5079
+ result = [val[0]]
5080
+
5081
+ result
5082
+ end
5060
5083
 
5061
5084
  def _reduce_390(val, _values, result)
5062
- result = val[0] << val[2]
5063
- raise "no18: #{val.inspect}"
5085
+ result = val[0].concat val[2]
5086
+ raise "no18\non: #{val.inspect}"
5064
5087
 
5065
5088
  result
5066
5089
  end
@@ -5135,8 +5158,12 @@ def _reduce_400(val, _values, result)
5135
5158
  end
5136
5159
 
5137
5160
  def _reduce_401(val, _values, result)
5138
- raise SyntaxError, "Both block arg and actual block given." if
5139
- val[0] && val[0][0] == :blockpass
5161
+ # TODO:
5162
+ # if (nd_type($1) == NODE_YIELD) {
5163
+ # compile_error(PARSER_ARG "block given to yield");
5164
+
5165
+ syntax_error "Both block arg and actual block given." if
5166
+ val[0].block_pass?
5140
5167
 
5141
5168
  result = val[1]
5142
5169
  result.insert 1, val[0]
@@ -5886,10 +5913,15 @@ def _reduce_529(val, _values, result)
5886
5913
  result
5887
5914
  end
5888
5915
 
5889
- # reduce 530 omitted
5916
+ def _reduce_530(val, _values, result)
5917
+ result = s(:block, val[0])
5918
+
5919
+ result
5920
+ end
5890
5921
 
5891
5922
  def _reduce_531(val, _values, result)
5892
- raise "no22: #{val.inspect}"
5923
+ result = val[0]
5924
+ result << val[2]
5893
5925
 
5894
5926
  result
5895
5927
  end