ruby_parser 3.0.0 → 3.19.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (58) hide show
  1. checksums.yaml +7 -0
  2. checksums.yaml.gz.sig +0 -0
  3. data/.autotest +36 -19
  4. data/History.rdoc +1297 -0
  5. data/Manifest.txt +35 -7
  6. data/{README.txt → README.rdoc} +44 -14
  7. data/Rakefile +308 -110
  8. data/bin/ruby_parse +3 -1
  9. data/bin/ruby_parse_extract_error +36 -16
  10. data/compare/normalize.rb +218 -0
  11. data/debugging.md +190 -0
  12. data/gauntlet.md +107 -0
  13. data/lib/.document +1 -0
  14. data/lib/rp_extensions.rb +53 -0
  15. data/lib/rp_stringscanner.rb +33 -0
  16. data/lib/ruby20_parser.rb +10973 -0
  17. data/lib/ruby20_parser.y +2683 -0
  18. data/lib/ruby21_parser.rb +10980 -0
  19. data/lib/ruby21_parser.y +2700 -0
  20. data/lib/ruby22_parser.rb +11123 -0
  21. data/lib/ruby22_parser.y +2711 -0
  22. data/lib/ruby23_parser.rb +11132 -0
  23. data/lib/ruby23_parser.y +2713 -0
  24. data/lib/ruby24_parser.rb +11231 -0
  25. data/lib/ruby24_parser.y +2721 -0
  26. data/lib/ruby25_parser.rb +11231 -0
  27. data/lib/ruby25_parser.y +2721 -0
  28. data/lib/ruby26_parser.rb +11253 -0
  29. data/lib/ruby26_parser.y +2736 -0
  30. data/lib/ruby27_parser.rb +12980 -0
  31. data/lib/ruby27_parser.y +3324 -0
  32. data/lib/ruby30_parser.rb +13242 -0
  33. data/lib/ruby30_parser.y +3447 -0
  34. data/lib/ruby31_parser.rb +13622 -0
  35. data/lib/ruby31_parser.y +3481 -0
  36. data/lib/ruby3_parser.yy +3536 -0
  37. data/lib/ruby_lexer.rb +933 -1232
  38. data/lib/ruby_lexer.rex +185 -0
  39. data/lib/ruby_lexer.rex.rb +399 -0
  40. data/lib/ruby_lexer_strings.rb +638 -0
  41. data/lib/ruby_parser.rb +97 -3
  42. data/lib/ruby_parser.yy +3465 -0
  43. data/lib/ruby_parser_extras.rb +1216 -687
  44. data/test/test_ruby_lexer.rb +2249 -1092
  45. data/test/test_ruby_parser.rb +5156 -975
  46. data/test/test_ruby_parser_extras.rb +47 -77
  47. data/tools/munge.rb +250 -0
  48. data/tools/ripper.rb +44 -0
  49. data.tar.gz.sig +1 -1
  50. metadata +200 -155
  51. metadata.gz.sig +0 -0
  52. data/.gemtest +0 -0
  53. data/History.txt +0 -482
  54. data/lib/gauntlet_rubyparser.rb +0 -120
  55. data/lib/ruby18_parser.rb +0 -5747
  56. data/lib/ruby18_parser.y +0 -1873
  57. data/lib/ruby19_parser.rb +0 -6110
  58. data/lib/ruby19_parser.y +0 -2078
data/History.txt DELETED
@@ -1,482 +0,0 @@
1
- === 3.0.0 / 2012-11-02
2
-
3
- I've hit 99.967% success rate against 558k files! 3.6σ!! 182 files
4
- failed to parse and spot checking them shows that they're pretty much
5
- lost causes. I'm sure I missed some stuff, but it is more important at
6
- this point to release a version to get more people using it in more
7
- diverse ways than I can come up with.
8
-
9
- * 3 minor enhancements:
10
-
11
- * Added RBStringScanner#charpos using #string_to_pos.
12
- * Added RBStringScanner#string_to_pos using String#byteslice.
13
- * Optimized regexp used for escape chars in strings. (presidentbeef)
14
-
15
- * 3 bug fixes:
16
-
17
- * Fixed current_line and unread_many depending on StringScanner#pos
18
- * Fixed parsing of 'a[] = b'
19
- * Fixed the worst abuse of heredocs EVER. Just because it is valid doesn't mean you should.
20
-
21
- === 3.0.0.a10 / 2012-10-26
22
-
23
- * 1 major enhancement:
24
-
25
- * iter nodes are now structurally the same as defs. Block args are COMPLETELY different.
26
-
27
- * 4 minor enhancements:
28
-
29
- * 1.8 and 1.9 now treat f { |(...)| } differently, per MRI.
30
- * 1.9: Added __ENCODING__ support. (witlessbird)
31
- * 1.9: Finished coverage for masgn variants in block args
32
- * 1.9: covered all forms of block args: optional, splat, block, and destructuring
33
-
34
- === 3.0.0.a9 / 2012-10-22
35
-
36
- Holy crap! I've hit a 99.92% success rate against 558k files! 492
37
- failed parses to go. NOW is the time to start testing ruby_parser
38
- against your code!
39
-
40
- * 16 minor enhancements:
41
-
42
- * 1.9 parser: Finished magic encoding and utf-8 bom support.
43
- * 1.9: Added leading dot support eg "a\n.b".
44
- * 1.9: Added partial handling for *arg and opt=val args in block args.
45
- * 1.9: OMFG Encoding is the worst thing ever. Overhauled encoding support once again. It is much better at doing last ditch guessing of the encoding of the source file.
46
- * 1.9: added String#grep for legacy support.
47
- * Added Sexp#block_pass?
48
- * Added ability to delete slow files when they timeout.
49
- * Added block_dup_check and refactored grammar code to use it for block arg handling.
50
- * Added in_lex_state?(*states) to clean up lexer code.
51
- * Added tentative support for optional block args. THIS IS SUBJECT TO CHANGE!!
52
- * Added toggleable debugging to StackState to make my life easier
53
- * All untested uses have not moved but have been laced with poison.
54
- * Finally got good tests for most forms of double-block arg errors.
55
- * Moved all _known_ uses of SyntaxError to RubyParser::SyntaxError
56
- * f_block_optarg now always returns a block node.
57
- * ruby_parse_extract_error uses Find.find instead of globbing so dot files aren't missed.
58
-
59
- * 12 bug fixes:
60
-
61
- * 1.9: Completely ignore IndexError in unread_many because... you know... it sucks.
62
- * 1.9: Fixed lex state after lexing ? in trinary.
63
- * 1.9: Fixed lex state in some ternarys.
64
- * 1.9: Fixed parsing of "1 ? b('') : 2\na d: 3"... ARE YOU NOT GLAD?!?!
65
- * Fix Timeout differences between 1.8 and 1.9 :(
66
- * Fixed emacs-style encodings to deal with no whitespace.
67
- * Fixed error message for bad % codes. (whitequark)
68
- * Fixed lexing of :a==>b vs :a===b vs :a==>b. P.S. Your space bar is broken. chump.
69
- * Fixed lexing of rare numeric formats.
70
- * Fixed magic comment encodings on DOS files ... :/
71
- * Fixed ruby_parse_extract_error to exit non-zero on errors.
72
- * Removed Symbol#is_argument and switch usage to is_arg?
73
-
74
- === 3.0.0.a8 / 2012-09-26
75
-
76
- * 1 bug fix:
77
-
78
- * Remove method #d from global namespace. (alexch)
79
-
80
- === 3.0.0.a7 / 2012-09-21
81
-
82
- * 3 minor enhancements:
83
-
84
- * Reorganized ruby_parse_extract_error so it will start much faster with a bunch of glob directories
85
- * RubyParserStuff#process takes optional time arg and raises Timeout::Error if it goes too long. You should rescue that, ya know...
86
- * ruby_parse_extract_error now checks *.rake and Rakefile on dir scan.
87
-
88
- * 3 bug fixes:
89
-
90
- * 1.9: Fixed ternary state tracking so {a:f{f()},b:nil} won't trip up the lexer.
91
- * Fixed optional values in block args (no20/no21)
92
- * ruby_parse_extract_error skips non-files. Some asshats put .rb on their dirs. :(
93
-
94
- === 3.0.0.a6 / 2012-08-20
95
-
96
- * 2 minor enhancements:
97
-
98
- * 1.8: Added basic encoding support to 1.8 parser. Assumes -Ku.
99
- * 1.9: Added encoding magic comment support to 1.9 parser.
100
-
101
- * 8 bug fixes:
102
-
103
- * 1.9: Fixed lexing of -1 w/in conditionals. yeah... I dunno.
104
- * 1.9: Fixed parsing of a do | | end.
105
- * 1.9: Fixed parsing of not(x).
106
- * 1.9: Fixed parsing of op_asgn + rescue: 'a ||= b rescue nil'
107
- * 1.9: added \r to the EOL backslash handler. dos files blow
108
- * 1.9: hacked in a workaround for 1.9 specific regexps running in 1.8.
109
- * Added #reset to RubyParser proxy class
110
- * Fixed lexing of conditional w/ %() tokens
111
-
112
- === 3.0.0.a5 / 2012-07-31
113
-
114
- * 5 bug fixes:
115
-
116
- * 1.9: Fix construction of 'f(:x, y: nil,)' w/ trailing comma.
117
- * 1.9: cleaned up lexing exit lex_state handling. Fixes bug parsing 'if f :x; end'
118
- * 1.9: fixed building of right-leaning masgns: 'f { |a, (b, c)| }'
119
- * 1.9: fixed lexing 'when *splat'
120
- * 1.9: fixed lexing of regexps in whens
121
-
122
- === 3.0.0.a4 / 2012-07-26
123
-
124
- * 10 minor enhancements:
125
-
126
- * 'rake debug' defaults to 1.9 parser since that's all I'm doing these days
127
- * 1.9: Fixed f { |(a, b, ...), ...| ... } handling.
128
- * Added 'rake extract F=path' task to quickly extract errors from large files
129
- * Added on_error handler to provide more readable error message.
130
- * Aliased #process to #parse.
131
- * Renamed #parse to #process (legacy name), added default path of '(string)'
132
- * cleaned ruby_parse_extract_error output and fixed to 1.9 parser
133
- * ruby_parse_extract_error expands shell globs from ARGV
134
- * ruby_parse_extract_error should also capture RuntimeError
135
- * yyerror(msg) now warns with the message instead of ignoring it.
136
-
137
- * 3 bug fixes:
138
-
139
- * 1.9: Fixed bug lexing/parsing [ in rhs.
140
- * 1.9: Fixed f { |((a, b), c)| ... } handling
141
- * 1.9: fixed newline handling during expr_value
142
-
143
- === 3.0.0.a3 / 2012-07-03
144
-
145
- * 1 major enhancement:
146
-
147
- * OMG! Implemented all the 1.9 arg crap and I went from 500 lines of
148
- structural diff to 32!
149
-
150
- * 17 minor enhancements:
151
-
152
- * 1.9: !(...) is now a method call. ugh
153
- * 1.9: Added __ENCODING__ keyword (returns Unsupported atm).
154
- * 1.9: Added support for "a.()" thingy. whatever you call it. Have I ever mentioned that ruby has too much syntax? I mean really...
155
- * 1.9: Moved kRETURN/kBREAK/kNEXT from command_call to command
156
- * 1.9: Removed when_args and refactored into args production.
157
- * 1.9: Support for not/! being a call to #!. why matz? why?!?
158
- * 1.9: Support for trailing comma support in opt call args.
159
- * 1.9: Switched keyword support to deal with 1.8 vs 1.9 differences.
160
- * 1.9: refactored and created command_asgn
161
- * 1.9: removed case/else from primary
162
- * 1.9: reworked mlhs and family. Now horrible things like 'a, *, c = f' work
163
- * 1.9: reworked top level rules from program to expr. Lots of lame refactorings in mri.
164
- * Moved 'a[]' from primary down to method_call
165
- * Removed 'args trailer' production from assoc_list (eg {1, 2}).
166
- * Removed 1.9 support for : as a terminator in do/then
167
- * RubyParser no longer subclasses the 18 parser, so I had to change some consts around.
168
- * Undeprecated RubyParser - but now it is a compound parser that tries 1.9 first.
169
-
170
- * 3 bug fixes:
171
-
172
- * 1.9: fixed bang method name.
173
- * Fixed case/when/splat handling on both 1.8 and 1.9.
174
- * Removed lambda_body production from lambda
175
-
176
- === 3.0.0.a2 / 2012-06-19
177
-
178
- * 1 minor enhancement:
179
-
180
- * Updated to sexp_processor ~> 4.0
181
-
182
- * 1 bug fix:
183
-
184
- * Fixed new_module to remove scope/block. Not enough tests on module variance.
185
-
186
- * 1 known issue:
187
-
188
- * No parsing of "a.()" thingy. (reported by julian7) (patches welcome!)
189
-
190
- === 3.0.0.a1 / 2012-05-22
191
-
192
- This is the first alpha release of the 3.0.0 series. It is probably
193
- woefully incomplete, bug ridden, and hasn't showered in several days.
194
- Please please please beat the crap out of it and send
195
- bugs/patches/complaints/suggestions.
196
-
197
- * 5 major enhancements:
198
-
199
- * 1.9 parsing! Thanks to EVERYONE for submitting patches for this!
200
- * Removed :arglist from everything but :op_asgn1
201
- * Removed :block from resbody
202
- * Removed :block from when
203
- * Removed :block nodes inside of scope nodes (defn/defs/class/sclass).
204
- * Removed :scope nodes in defn/defs/class/sclass nodes.
205
- * (probably more sexp cleanup to come before 3.0.0 final)
206
-
207
- * 25 minor enhancements:
208
-
209
- * 1.9: Fix \!a. (wanabe)
210
- * 1.9: Method calling with postargs. (wanabe)
211
- * 1.9: Method definition with postargs. (wanabe)
212
- * 1.9: Support lambda args without parentheses. (wanabe)
213
- * Added R arg to `rake debug` to debug ruby straight up
214
- * Added RubyParser, subclassing Ruby18Parser but warning on instantiation.
215
- * Added backref_assign_error (needs tests)
216
- * Added bin/ruby_parse_extract_error to help with error submissions
217
- * Added debug task to help quickly get into a bug
218
- * Added more 18 vs 19 lexing tests for ?c.
219
- * Added ruby_parser.rb that pulls everything together in proper order.
220
- * Added tLABEL. (brynary)
221
- * Branched ruby_parser.y to ruby18_parser.y
222
- * Fix to pass test_lambda_args_block__19 test. (mrmargolis)
223
- * Got rid of one instance of unread_many. (Confusion)
224
- * Moved everything from RubyParser to RubyParserStuff and included module in both.
225
- * Refactored 1.9 args handling
226
- * Refactored and added new_resbody to ruby_parser_extras.
227
- * Refactored and added new_when
228
- * Refactored tests infrastructure and added both 1.8 and 1.9 test branches.
229
- * Removed unused methods: unread, begin_of_line? was_begin_of_line. (YAY!) (Confusion)
230
- * Renamed ruby_parser.y to ruby19_parser.y
231
- * RubyLexer now takes a version specifier.
232
- * Started doing comparative refactoring between MRI's 1.9 parser and RP's. Shouldn't differ functionally except where we missed stuff in RP.
233
- * `rake debug` prints the output if it succeeds.
234
-
235
- * 21 bug fixes:
236
-
237
- * Added missing gvar arg error. (1.8)
238
- * Attach parser files to isolate to ensure they can build
239
- * Conditionalize handling of tLABEL to ruby19+. Fixes issue #33.
240
- * DOH. I deactivated some tests and never reactivated them. (Confusion ftw)
241
- * Duplicate the input so that heredoc processing doesn't morph original. (banister)
242
- * Entirely reworked block arg handling. (1.8)
243
- * Fix ?x char literal. (nobu)
244
- * Fixed 4/5 of literal lambda tests (jamie)
245
- * Fixed deps for parser
246
- * Fixed lexing of ?c for ruby 1.8 and 1.9.
247
- * Fixed more Ruby 1.9 args tests (brynary)
248
- * Fixed reswords to match MRI (1.8, 1.9)
249
- * Fixed symbols with no spaces in method calls (e.g. foo:bar) (YAY! brynary)
250
- * Fixed ternary_nil_no_space and other ternary edge cases for 1.9. (lastobelus)
251
- * Fixed test_call_not_equal__19. First bug bounty! (albus522)
252
- * Made lambda w/o arg list zero out the arg slot.
253
- * Renamed awords to qwords to match stupid MRI naming. (1.8, 1.9) :(
254
- * Rolled out brynary's symbols-no-spaces (foo:bar) changes when parsing 1.8 code
255
- * Split 1.8 from 1.9 open paren lexer. Gawd that's ugly code.
256
- * Split block_var from for_var. (1.8, 1.9)
257
- * Use binread (and emulate in ruby 1.8) to avoid encoding issues
258
-
259
- === 2.3.1 / 2011-09-21
260
-
261
- * 2 bug fixes:
262
-
263
- * Fixed line numbers at end of special var+whitespace (larsch)
264
- * Holy crap I was smokin' something good... Fixed 1.9.3 warning
265
-
266
- === 2.3.0 / 2011-09-06
267
-
268
- * 2 minor enhancements:
269
-
270
- * Add -g flag to parser compile if DEBUG
271
- * Lexer now embeds line number in yacc_value for keywords, helping fix up line numbers
272
-
273
- * 3 bug fixes:
274
-
275
- * Fix method line numbers when no args and no parens (quix)
276
- * Fixed line numbers on return/break/next w/ result expr. (pjnz)
277
- * Fixed some lexing state in order to parse: 'f (1), 2' as 'f(1, 2)'. (invernizzi)
278
-
279
- === 2.2.0 / 2011-08-23
280
-
281
- * 2 minor enhancements:
282
-
283
- * Moved Keyword, Environment, and StackState inside of RubyParser
284
- * Added proper dsym and dsym->sym support.
285
-
286
- * 3 bug fixes:
287
-
288
- * Added extra (failing) tests for call/iter line number checking (quix)
289
- * Fixed line numbers for certain call/iter edge cases
290
- * Fixed parsing of: alias :"<<" :">>".
291
-
292
- === 2.1.0 / 2011-08-15
293
-
294
- * 2 minor enhancements:
295
-
296
- * Added new accessor canonicalize_conditions to toggle conditional canonicalization (on by default). (confused)
297
- * Awesome cleanup: Replaced call to append_block by block_append. (Confusion)
298
-
299
- * 2 bug fixes:
300
-
301
- * Fixed handling last line of =begin/=end. (raybaxter)
302
- * Fixed source line numbers after heredocs. (jbarreneche)
303
-
304
- === 2.0.6 / 2011-02-18
305
-
306
- * 1 minor enhancement:
307
-
308
- * Switched to hoe's racc plugin to clean up rakefile and builds
309
-
310
- * 1 bug fix:
311
-
312
- * Fixed empty =begin/end.
313
-
314
- === 2.0.5 / 2010-09-01
315
-
316
- * 1 minor enhancement:
317
-
318
- * Started merging like lexical cases to try to squeeze some optimization out
319
-
320
- === 2.0.4 / 2009-08-18
321
-
322
- * 1 minor enhancement:
323
-
324
- * Changed requires around to be more accurate.
325
-
326
- * 4 bug fixes:
327
-
328
- * Fixed .autotest for minitest
329
- * Fixed emacs escape lexing bug: "\C-\\" (maglev/gemstone)
330
- * Fixed octal lexing edgecases. (maglev/gemstone)
331
- * Fixed regexp lexing edgecases. (maglev/gemstone)
332
-
333
- === 2.0.3 / 2009-06-23
334
-
335
- * 4 minor enhancements:
336
-
337
- * Removed dead code handling d/regex match node cruft.
338
- * Switched to minitest
339
- * Updated .autotest and rakefile wrt rcov for new hoe capabilities
340
- * Updated hoe for new capabilities
341
-
342
- * 4 bug fixes:
343
-
344
- * Environment#all now deals with strange edge cases in RAD.
345
- * Fixed packaging/compilation issue.
346
- * Minor 1.9 fixes
347
- * hoe -> flay -> rubyparser rakefile circularity fixed
348
-
349
- === 2.0.2 / 2009-01-20
350
-
351
- * 2 minor enhancements:
352
-
353
- * Added gauntlet_rubyparser plugin. YAY for easy massive bug-hunting.
354
- * Promoted Sexp's file/line/comments to sexp_processor.
355
-
356
- * 4 bug fixes:
357
-
358
- * Fixed and improved the readme
359
- * Fixed lexing heredoc newlines.
360
- * Fixed line numbers on defns.
361
- * Fixed rdoc generation bug pointed out by hugh sasse (who rocks)
362
-
363
- === 2.0.1 / 2008-11-04
364
-
365
- * 2 minor enhancements:
366
-
367
- * Updated for changes to splat node in many contexts.
368
- * Made PT a developer dep
369
-
370
- === 2.0.0 / 2008-10-22
371
-
372
- * 1 major enhancement
373
-
374
- * Brought on the AWESOME! 4x faster! no known lexing/parsing bugs!
375
-
376
- * 71 minor enhancements
377
-
378
- * 1.9: Added Fixnum#ord.
379
- * 1.9: Added missing Regexp constants and did it so it'd work on 1.9.
380
- * Added #store_comment and #comments
381
- * Added StringScanner #begin_of_line?
382
- * Added a bunch of tests for regexp escape chars, #parse_string, #read_escape, ? numbers, ? whitespace.
383
- * Added a hack for rubinius' r2l eval bug.
384
- * Added a new token type tSTRING that bypasses tSTRING_BEG/END entirely. Only does non-interpolated strings and then falls back to the old way. MUCH cleaner tho.
385
- * Added bin/ruby_parse
386
- * Added compare rule to Rakefile.
387
- * Added coverage files/dirs to clean rule.
388
- * Added file and line numbers to all sexp nodes. Column/ranges to come.
389
- * Added lex_state change for lvars at the end of yylex.
390
- * Added lexed comments to defn/defs/class/module nodes.
391
- * Added stats gathering for yylex. Reordered yylex for avg data
392
- * Added tSYMBOL token type and parser rule to speed up symbol lexing.
393
- * Added tally output for getch, unread, and unread_many.
394
- * Added tests for ambigous uminus/uplus, backtick in cmdarg, square and curly brackets, numeric gvars, eos edge cases, string quoting %<> and %%%.
395
- * All cases throughout yylex now return directly if they match, no passthroughs.
396
- * All lexer cases now slurp entire token in one swoop.
397
- * All zarrays are now just empty arrays.
398
- * Changed s(:block_arg, :blah) to :"&blah" in args sexp.
399
- * Cleaned up lexer error handling. Now just raises all over.
400
- * Cleaned up read_escape and regx_options
401
- * Cleaned up tokadd_string (for some definition of cleaned).
402
- * Converted single quoted strings to new tSTRING token type.
403
- * Coverage is currently 94.4% on lexer.
404
- * Done what I can to clean up heredoc lexing... still sucks.
405
- * Flattened resbodies in rescue node. Fixed .autotest file.
406
- * Folded lex_keywords back in now that it screams.
407
- * Found very last instanceof ILiteralNode in the code. haha!
408
- * Got the tests subclassing PTTC and cleaned up a lot. YAY
409
- * Handle yield(*ary) properly
410
- * MASSIVELY cleaned out =begin/=end comment processor.
411
- * Massive overhaul on Keyword class. All hail the mighty Hash!
412
- * Massively cleaned up ident= edge cases and fixed a stupid bug from jruby.
413
- * Merged @/@@ scanner together, going to try to do the same everywhere.
414
- * Refactored fix_arg_lex_state, common across the lexer.
415
- * Refactored new_fcall into new_call.
416
- * Refactored some code to get better profile numbers.
417
- * Refactored some more #fix_arg_lex_state.
418
- * Refactored tail of yylex into its own method.
419
- * Removed Module#kill
420
- * Removed Token, replaced with Sexp.
421
- * Removed all parse_number and parse_quote tests.
422
- * Removed argspush, argscat. YAY!
423
- * Removed as many token_buffer.split(//)'s as possible. 1 to go.
424
- * Removed begins from compstmts
425
- * Removed buffer arg for tokadd_string.
426
- * Removed crufty (?) solo '@' token... wtf was that anyhow?
427
- * Removed most jruby/stringio cruft from StringScanner.
428
- * Removed one unread_many... 2 to go. They're harder.
429
- * Removed store_comment, now done directly.
430
- * Removed token_buffer. Now I just use token ivar.
431
- * Removed use of s() from lexer. Changed the way line numbers are gathered.
432
- * Renamed *qwords to *awords.
433
- * Renamed StringScanner to RPStringScanner (a subclass) to fix namespace trashing.
434
- * Renamed parse to process and aliased to parse.
435
- * Renamed token_buffer to string_buffer since that arcane shit still needs it.
436
- * Resolved the rest of the lexing issues I brought up w/ ruby-core.
437
- * Revamped tokadd_escape.
438
- * Rewrote Keyword and KWtable.
439
- * Rewrote RubyLexer using StringScanner.
440
- * Rewrote tokadd_escape. 79 lines down to 21.
441
- * Split out lib/ruby_parser_extras.rb so lexer is standalone.
442
- * Started to clean up the parser and make it as skinny as possible
443
- * Stripped out as much code as possible.
444
- * Stripped yylex of some dead code.
445
- * Switched from StringIO to StringScanner.
446
- * Updated rakefile for new hoe.
447
- * Uses pure ruby racc if ENV['PURE_RUBY'], otherwise use c.
448
- * Wrote a ton of lexer tests. Coverage is as close to 100% as possible.
449
- * Wrote args to clean up the big nasty args processing grammar section.
450
- * lex_strterm is now a plain array, removed RubyLexer#s(...).
451
- * yield and super now flatten args.
452
-
453
- * 21+ bug fixes:
454
-
455
- * I'm sure this list is missing a lot:
456
- * Fixed 2 bugs both involving attrasgn (and ilk) esp when lhs is an array.
457
- * Fixed a bug in the lexer for strings with single digit hex escapes.
458
- * Fixed a bug parsing: a (args) { expr }... the space caused a different route to be followed and all hell broke loose.
459
- * Fixed a bug with x\n=beginvar not putting begin back.
460
- * Fixed attrasgn to have arglists, not arrays.
461
- * Fixed bug in defn/defs with block fixing.
462
- * Fixed class/module's name slot if colon2/3.
463
- * Fixed dstr with empty interpolation body.
464
- * Fixed for 1.9 string/char changes.
465
- * Fixed lexer BS wrt determining token type of words.
466
- * Fixed lexer BS wrt pass through values and lexing words. SO STUPID.
467
- * Fixed lexing of floats.
468
- * Fixed lexing of identifiers followed by equals. I hope.
469
- * Fixed masgn with splat on lhs
470
- * Fixed new_super to deal with block_pass correctly.
471
- * Fixed parser's treatment of :colon2 and :colon3.
472
- * Fixed regexp scanning of escaped numbers, ANY number is valid, not just octs.
473
- * Fixed string scanning of escaped octs, allowing 1-3 chars.
474
- * Fixed unescape for \n
475
- * Fixed: omg this is stupid. '()' was returning bare nil
476
- * Fixed: remove_begin now goes to the end, not sure why it didn't before.
477
-
478
- === 1.0.0 / 2007-12-20
479
-
480
- * 1 major enhancement
481
- * Birthday!
482
-
@@ -1,120 +0,0 @@
1
- #!/usr/bin/ruby -ws
2
-
3
- $f ||= false
4
-
5
- $:.unshift "../../ruby_parser/dev/lib"
6
- $:.unshift "../../ruby2ruby/dev/lib"
7
-
8
- require 'rubygems'
9
- require 'ruby2ruby'
10
- require 'ruby_parser'
11
-
12
- require 'gauntlet'
13
-
14
- class RubyParserGauntlet < Gauntlet
15
- def initialize
16
- super
17
-
18
- self.data = Hash.new { |h,k| h[k] = {} }
19
- old_data = load_yaml data_file
20
- self.data.merge! old_data
21
- end
22
-
23
- def should_skip? name
24
- if $f then
25
- if Hash === data[name] then
26
- ! data[name].empty?
27
- else
28
- data[name]
29
- end
30
- else
31
- data[name] == true # yes, == true on purpose
32
- end
33
- end
34
-
35
- def diff_pp o1, o2
36
- require 'pp'
37
-
38
- File.open("/tmp/a.#{$$}", "w") do |f|
39
- PP.pp o1, f
40
- end
41
-
42
- File.open("/tmp/b.#{$$}", "w") do |f|
43
- PP.pp o2, f
44
- end
45
-
46
- `diff -u /tmp/a.#{$$} /tmp/b.#{$$}`
47
- ensure
48
- File.unlink "/tmp/a.#{$$}" rescue nil
49
- File.unlink "/tmp/b.#{$$}" rescue nil
50
- end
51
-
52
- def broke name, file, msg
53
- warn "bad"
54
- self.data[name][file] = msg
55
- self.dirty = true
56
- end
57
-
58
- def process path, name
59
- begin
60
- $stderr.print " #{path}: "
61
- rp = RubyParser.new
62
- r2r = Ruby2Ruby.new
63
-
64
- old_ruby = File.read(path)
65
-
66
- begin
67
- old_sexp = rp.process old_ruby
68
- rescue Racc::ParseError => e
69
- self.data[name][path] = :unparsable
70
- self.dirty = true
71
- return
72
- end
73
-
74
- new_ruby = r2r.process old_sexp.deep_clone
75
-
76
- begin
77
- new_sexp = rp.process new_ruby
78
- rescue Racc::ParseError => e
79
- broke name, path, "couldn't parse new_ruby: #{e.message.strip}"
80
- return
81
- end
82
-
83
- if old_sexp != new_sexp then
84
- broke name, path, diff_pp(old_sexp, new_sexp)
85
- return
86
- end
87
-
88
- self.data[name][path] = true
89
- self.dirty = true
90
-
91
- warn "good"
92
- rescue Interrupt
93
- puts "User cancelled"
94
- exit 1
95
- rescue Exception => e
96
- broke name, path, " UNKNOWN ERROR: #{e}: #{e.message.strip}"
97
- end
98
- end
99
-
100
- def run name
101
- warn name
102
- Dir["**/*.rb"].sort.each do |path|
103
- next if path =~ /gemspec.rb/ # HACK
104
- next if data[name][path] == true
105
- process path, name
106
- end
107
-
108
- if self.data[name].values.all? { |v| v == true } then
109
- warn " ALL GOOD!"
110
- self.data[name] = true
111
- self.dirty = true
112
- end
113
- end
114
- end
115
-
116
- filter = ARGV.shift
117
- filter = Regexp.new filter if filter
118
-
119
- gauntlet = RubyParserGauntlet.new
120
- gauntlet.run_the_gauntlet filter