ruby_parser 3.0.0 → 3.19.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. checksums.yaml +7 -0
  2. checksums.yaml.gz.sig +0 -0
  3. data/.autotest +36 -19
  4. data/History.rdoc +1297 -0
  5. data/Manifest.txt +35 -7
  6. data/{README.txt → README.rdoc} +44 -14
  7. data/Rakefile +308 -110
  8. data/bin/ruby_parse +3 -1
  9. data/bin/ruby_parse_extract_error +36 -16
  10. data/compare/normalize.rb +218 -0
  11. data/debugging.md +190 -0
  12. data/gauntlet.md +107 -0
  13. data/lib/.document +1 -0
  14. data/lib/rp_extensions.rb +53 -0
  15. data/lib/rp_stringscanner.rb +33 -0
  16. data/lib/ruby20_parser.rb +10973 -0
  17. data/lib/ruby20_parser.y +2683 -0
  18. data/lib/ruby21_parser.rb +10980 -0
  19. data/lib/ruby21_parser.y +2700 -0
  20. data/lib/ruby22_parser.rb +11123 -0
  21. data/lib/ruby22_parser.y +2711 -0
  22. data/lib/ruby23_parser.rb +11132 -0
  23. data/lib/ruby23_parser.y +2713 -0
  24. data/lib/ruby24_parser.rb +11231 -0
  25. data/lib/ruby24_parser.y +2721 -0
  26. data/lib/ruby25_parser.rb +11231 -0
  27. data/lib/ruby25_parser.y +2721 -0
  28. data/lib/ruby26_parser.rb +11253 -0
  29. data/lib/ruby26_parser.y +2736 -0
  30. data/lib/ruby27_parser.rb +12980 -0
  31. data/lib/ruby27_parser.y +3324 -0
  32. data/lib/ruby30_parser.rb +13242 -0
  33. data/lib/ruby30_parser.y +3447 -0
  34. data/lib/ruby31_parser.rb +13622 -0
  35. data/lib/ruby31_parser.y +3481 -0
  36. data/lib/ruby3_parser.yy +3536 -0
  37. data/lib/ruby_lexer.rb +933 -1232
  38. data/lib/ruby_lexer.rex +185 -0
  39. data/lib/ruby_lexer.rex.rb +399 -0
  40. data/lib/ruby_lexer_strings.rb +638 -0
  41. data/lib/ruby_parser.rb +97 -3
  42. data/lib/ruby_parser.yy +3465 -0
  43. data/lib/ruby_parser_extras.rb +1216 -687
  44. data/test/test_ruby_lexer.rb +2249 -1092
  45. data/test/test_ruby_parser.rb +5156 -975
  46. data/test/test_ruby_parser_extras.rb +47 -77
  47. data/tools/munge.rb +250 -0
  48. data/tools/ripper.rb +44 -0
  49. data.tar.gz.sig +1 -1
  50. metadata +200 -155
  51. metadata.gz.sig +0 -0
  52. data/.gemtest +0 -0
  53. data/History.txt +0 -482
  54. data/lib/gauntlet_rubyparser.rb +0 -120
  55. data/lib/ruby18_parser.rb +0 -5747
  56. data/lib/ruby18_parser.y +0 -1873
  57. data/lib/ruby19_parser.rb +0 -6110
  58. data/lib/ruby19_parser.y +0 -2078
data/History.txt DELETED
@@ -1,482 +0,0 @@
1
- === 3.0.0 / 2012-11-02
2
-
3
- I've hit 99.967% success rate against 558k files! 3.6σ!! 182 files
4
- failed to parse and spot checking them shows that they're pretty much
5
- lost causes. I'm sure I missed some stuff, but it is more important at
6
- this point to release a version to get more people using it in more
7
- diverse ways than I can come up with.
8
-
9
- * 3 minor enhancements:
10
-
11
- * Added RBStringScanner#charpos using #string_to_pos.
12
- * Added RBStringScanner#string_to_pos using String#byteslice.
13
- * Optimized regexp used for escape chars in strings. (presidentbeef)
14
-
15
- * 3 bug fixes:
16
-
17
- * Fixed current_line and unread_many depending on StringScanner#pos
18
- * Fixed parsing of 'a[] = b'
19
- * Fixed the worst abuse of heredocs EVER. Just because it is valid doesn't mean you should.
20
-
21
- === 3.0.0.a10 / 2012-10-26
22
-
23
- * 1 major enhancement:
24
-
25
- * iter nodes are now structurally the same as defs. Block args are COMPLETELY different.
26
-
27
- * 4 minor enhancements:
28
-
29
- * 1.8 and 1.9 now treat f { |(...)| } differently, per MRI.
30
- * 1.9: Added __ENCODING__ support. (witlessbird)
31
- * 1.9: Finished coverage for masgn variants in block args
32
- * 1.9: covered all forms of block args: optional, splat, block, and destructuring
33
-
34
- === 3.0.0.a9 / 2012-10-22
35
-
36
- Holy crap! I've hit a 99.92% success rate against 558k files! 492
37
- failed parses to go. NOW is the time to start testing ruby_parser
38
- against your code!
39
-
40
- * 16 minor enhancements:
41
-
42
- * 1.9 parser: Finished magic encoding and utf-8 bom support.
43
- * 1.9: Added leading dot support eg "a\n.b".
44
- * 1.9: Added partial handling for *arg and opt=val args in block args.
45
- * 1.9: OMFG Encoding is the worst thing ever. Overhauled encoding support once again. It is much better at doing last ditch guessing of the encoding of the source file.
46
- * 1.9: added String#grep for legacy support.
47
- * Added Sexp#block_pass?
48
- * Added ability to delete slow files when they timeout.
49
- * Added block_dup_check and refactored grammar code to use it for block arg handling.
50
- * Added in_lex_state?(*states) to clean up lexer code.
51
- * Added tentative support for optional block args. THIS IS SUBJECT TO CHANGE!!
52
- * Added toggleable debugging to StackState to make my life easier
53
- * All untested uses have not moved but have been laced with poison.
54
- * Finally got good tests for most forms of double-block arg errors.
55
- * Moved all _known_ uses of SyntaxError to RubyParser::SyntaxError
56
- * f_block_optarg now always returns a block node.
57
- * ruby_parse_extract_error uses Find.find instead of globbing so dot files aren't missed.
58
-
59
- * 12 bug fixes:
60
-
61
- * 1.9: Completely ignore IndexError in unread_many because... you know... it sucks.
62
- * 1.9: Fixed lex state after lexing ? in trinary.
63
- * 1.9: Fixed lex state in some ternarys.
64
- * 1.9: Fixed parsing of "1 ? b('') : 2\na d: 3"... ARE YOU NOT GLAD?!?!
65
- * Fix Timeout differences between 1.8 and 1.9 :(
66
- * Fixed emacs-style encodings to deal with no whitespace.
67
- * Fixed error message for bad % codes. (whitequark)
68
- * Fixed lexing of :a==>b vs :a===b vs :a==>b. P.S. Your space bar is broken. chump.
69
- * Fixed lexing of rare numeric formats.
70
- * Fixed magic comment encodings on DOS files ... :/
71
- * Fixed ruby_parse_extract_error to exit non-zero on errors.
72
- * Removed Symbol#is_argument and switch usage to is_arg?
73
-
74
- === 3.0.0.a8 / 2012-09-26
75
-
76
- * 1 bug fix:
77
-
78
- * Remove method #d from global namespace. (alexch)
79
-
80
- === 3.0.0.a7 / 2012-09-21
81
-
82
- * 3 minor enhancements:
83
-
84
- * Reorganized ruby_parse_extract_error so it will start much faster with a bunch of glob directories
85
- * RubyParserStuff#process takes optional time arg and raises Timeout::Error if it goes too long. You should rescue that, ya know...
86
- * ruby_parse_extract_error now checks *.rake and Rakefile on dir scan.
87
-
88
- * 3 bug fixes:
89
-
90
- * 1.9: Fixed ternary state tracking so {a:f{f()},b:nil} won't trip up the lexer.
91
- * Fixed optional values in block args (no20/no21)
92
- * ruby_parse_extract_error skips non-files. Some asshats put .rb on their dirs. :(
93
-
94
- === 3.0.0.a6 / 2012-08-20
95
-
96
- * 2 minor enhancements:
97
-
98
- * 1.8: Added basic encoding support to 1.8 parser. Assumes -Ku.
99
- * 1.9: Added encoding magic comment support to 1.9 parser.
100
-
101
- * 8 bug fixes:
102
-
103
- * 1.9: Fixed lexing of -1 w/in conditionals. yeah... I dunno.
104
- * 1.9: Fixed parsing of a do | | end.
105
- * 1.9: Fixed parsing of not(x).
106
- * 1.9: Fixed parsing of op_asgn + rescue: 'a ||= b rescue nil'
107
- * 1.9: added \r to the EOL backslash handler. dos files blow
108
- * 1.9: hacked in a workaround for 1.9 specific regexps running in 1.8.
109
- * Added #reset to RubyParser proxy class
110
- * Fixed lexing of conditional w/ %() tokens
111
-
112
- === 3.0.0.a5 / 2012-07-31
113
-
114
- * 5 bug fixes:
115
-
116
- * 1.9: Fix construction of 'f(:x, y: nil,)' w/ trailing comma.
117
- * 1.9: cleaned up lexing exit lex_state handling. Fixes bug parsing 'if f :x; end'
118
- * 1.9: fixed building of right-leaning masgns: 'f { |a, (b, c)| }'
119
- * 1.9: fixed lexing 'when *splat'
120
- * 1.9: fixed lexing of regexps in whens
121
-
122
- === 3.0.0.a4 / 2012-07-26
123
-
124
- * 10 minor enhancements:
125
-
126
- * 'rake debug' defaults to 1.9 parser since that's all I'm doing these days
127
- * 1.9: Fixed f { |(a, b, ...), ...| ... } handling.
128
- * Added 'rake extract F=path' task to quickly extract errors from large files
129
- * Added on_error handler to provide more readable error message.
130
- * Aliased #process to #parse.
131
- * Renamed #parse to #process (legacy name), added default path of '(string)'
132
- * cleaned ruby_parse_extract_error output and fixed to 1.9 parser
133
- * ruby_parse_extract_error expands shell globs from ARGV
134
- * ruby_parse_extract_error should also capture RuntimeError
135
- * yyerror(msg) now warns with the message instead of ignoring it.
136
-
137
- * 3 bug fixes:
138
-
139
- * 1.9: Fixed bug lexing/parsing [ in rhs.
140
- * 1.9: Fixed f { |((a, b), c)| ... } handling
141
- * 1.9: fixed newline handling during expr_value
142
-
143
- === 3.0.0.a3 / 2012-07-03
144
-
145
- * 1 major enhancement:
146
-
147
- * OMG! Implemented all the 1.9 arg crap and I went from 500 lines of
148
- structural diff to 32!
149
-
150
- * 17 minor enhancements:
151
-
152
- * 1.9: !(...) is now a method call. ugh
153
- * 1.9: Added __ENCODING__ keyword (returns Unsupported atm).
154
- * 1.9: Added support for "a.()" thingy. whatever you call it. Have I ever mentioned that ruby has too much syntax? I mean really...
155
- * 1.9: Moved kRETURN/kBREAK/kNEXT from command_call to command
156
- * 1.9: Removed when_args and refactored into args production.
157
- * 1.9: Support for not/! being a call to #!. why matz? why?!?
158
- * 1.9: Support for trailing comma support in opt call args.
159
- * 1.9: Switched keyword support to deal with 1.8 vs 1.9 differences.
160
- * 1.9: refactored and created command_asgn
161
- * 1.9: removed case/else from primary
162
- * 1.9: reworked mlhs and family. Now horrible things like 'a, *, c = f' work
163
- * 1.9: reworked top level rules from program to expr. Lots of lame refactorings in mri.
164
- * Moved 'a[]' from primary down to method_call
165
- * Removed 'args trailer' production from assoc_list (eg {1, 2}).
166
- * Removed 1.9 support for : as a terminator in do/then
167
- * RubyParser no longer subclasses the 18 parser, so I had to change some consts around.
168
- * Undeprecated RubyParser - but now it is a compound parser that tries 1.9 first.
169
-
170
- * 3 bug fixes:
171
-
172
- * 1.9: fixed bang method name.
173
- * Fixed case/when/splat handling on both 1.8 and 1.9.
174
- * Removed lambda_body production from lambda
175
-
176
- === 3.0.0.a2 / 2012-06-19
177
-
178
- * 1 minor enhancement:
179
-
180
- * Updated to sexp_processor ~> 4.0
181
-
182
- * 1 bug fix:
183
-
184
- * Fixed new_module to remove scope/block. Not enough tests on module variance.
185
-
186
- * 1 known issue:
187
-
188
- * No parsing of "a.()" thingy. (reported by julian7) (patches welcome!)
189
-
190
- === 3.0.0.a1 / 2012-05-22
191
-
192
- This is the first alpha release of the 3.0.0 series. It is probably
193
- woefully incomplete, bug ridden, and hasn't showered in several days.
194
- Please please please beat the crap out of it and send
195
- bugs/patches/complaints/suggestions.
196
-
197
- * 5 major enhancements:
198
-
199
- * 1.9 parsing! Thanks to EVERYONE for submitting patches for this!
200
- * Removed :arglist from everything but :op_asgn1
201
- * Removed :block from resbody
202
- * Removed :block from when
203
- * Removed :block nodes inside of scope nodes (defn/defs/class/sclass).
204
- * Removed :scope nodes in defn/defs/class/sclass nodes.
205
- * (probably more sexp cleanup to come before 3.0.0 final)
206
-
207
- * 25 minor enhancements:
208
-
209
- * 1.9: Fix \!a. (wanabe)
210
- * 1.9: Method calling with postargs. (wanabe)
211
- * 1.9: Method definition with postargs. (wanabe)
212
- * 1.9: Support lambda args without parentheses. (wanabe)
213
- * Added R arg to `rake debug` to debug ruby straight up
214
- * Added RubyParser, subclassing Ruby18Parser but warning on instantiation.
215
- * Added backref_assign_error (needs tests)
216
- * Added bin/ruby_parse_extract_error to help with error submissions
217
- * Added debug task to help quickly get into a bug
218
- * Added more 18 vs 19 lexing tests for ?c.
219
- * Added ruby_parser.rb that pulls everything together in proper order.
220
- * Added tLABEL. (brynary)
221
- * Branched ruby_parser.y to ruby18_parser.y
222
- * Fix to pass test_lambda_args_block__19 test. (mrmargolis)
223
- * Got rid of one instance of unread_many. (Confusion)
224
- * Moved everything from RubyParser to RubyParserStuff and included module in both.
225
- * Refactored 1.9 args handling
226
- * Refactored and added new_resbody to ruby_parser_extras.
227
- * Refactored and added new_when
228
- * Refactored tests infrastructure and added both 1.8 and 1.9 test branches.
229
- * Removed unused methods: unread, begin_of_line? was_begin_of_line. (YAY!) (Confusion)
230
- * Renamed ruby_parser.y to ruby19_parser.y
231
- * RubyLexer now takes a version specifier.
232
- * Started doing comparative refactoring between MRI's 1.9 parser and RP's. Shouldn't differ functionally except where we missed stuff in RP.
233
- * `rake debug` prints the output if it succeeds.
234
-
235
- * 21 bug fixes:
236
-
237
- * Added missing gvar arg error. (1.8)
238
- * Attach parser files to isolate to ensure they can build
239
- * Conditionalize handling of tLABEL to ruby19+. Fixes issue #33.
240
- * DOH. I deactivated some tests and never reactivated them. (Confusion ftw)
241
- * Duplicate the input so that heredoc processing doesn't morph original. (banister)
242
- * Entirely reworked block arg handling. (1.8)
243
- * Fix ?x char literal. (nobu)
244
- * Fixed 4/5 of literal lambda tests (jamie)
245
- * Fixed deps for parser
246
- * Fixed lexing of ?c for ruby 1.8 and 1.9.
247
- * Fixed more Ruby 1.9 args tests (brynary)
248
- * Fixed reswords to match MRI (1.8, 1.9)
249
- * Fixed symbols with no spaces in method calls (e.g. foo:bar) (YAY! brynary)
250
- * Fixed ternary_nil_no_space and other ternary edge cases for 1.9. (lastobelus)
251
- * Fixed test_call_not_equal__19. First bug bounty! (albus522)
252
- * Made lambda w/o arg list zero out the arg slot.
253
- * Renamed awords to qwords to match stupid MRI naming. (1.8, 1.9) :(
254
- * Rolled out brynary's symbols-no-spaces (foo:bar) changes when parsing 1.8 code
255
- * Split 1.8 from 1.9 open paren lexer. Gawd that's ugly code.
256
- * Split block_var from for_var. (1.8, 1.9)
257
- * Use binread (and emulate in ruby 1.8) to avoid encoding issues
258
-
259
- === 2.3.1 / 2011-09-21
260
-
261
- * 2 bug fixes:
262
-
263
- * Fixed line numbers at end of special var+whitespace (larsch)
264
- * Holy crap I was smokin' something good... Fixed 1.9.3 warning
265
-
266
- === 2.3.0 / 2011-09-06
267
-
268
- * 2 minor enhancements:
269
-
270
- * Add -g flag to parser compile if DEBUG
271
- * Lexer now embeds line number in yacc_value for keywords, helping fix up line numbers
272
-
273
- * 3 bug fixes:
274
-
275
- * Fix method line numbers when no args and no parens (quix)
276
- * Fixed line numbers on return/break/next w/ result expr. (pjnz)
277
- * Fixed some lexing state in order to parse: 'f (1), 2' as 'f(1, 2)'. (invernizzi)
278
-
279
- === 2.2.0 / 2011-08-23
280
-
281
- * 2 minor enhancements:
282
-
283
- * Moved Keyword, Environment, and StackState inside of RubyParser
284
- * Added proper dsym and dsym->sym support.
285
-
286
- * 3 bug fixes:
287
-
288
- * Added extra (failing) tests for call/iter line number checking (quix)
289
- * Fixed line numbers for certain call/iter edge cases
290
- * Fixed parsing of: alias :"<<" :">>".
291
-
292
- === 2.1.0 / 2011-08-15
293
-
294
- * 2 minor enhancements:
295
-
296
- * Added new accessor canonicalize_conditions to toggle conditional canonicalization (on by default). (confused)
297
- * Awesome cleanup: Replaced call to append_block by block_append. (Confusion)
298
-
299
- * 2 bug fixes:
300
-
301
- * Fixed handling last line of =begin/=end. (raybaxter)
302
- * Fixed source line numbers after heredocs. (jbarreneche)
303
-
304
- === 2.0.6 / 2011-02-18
305
-
306
- * 1 minor enhancement:
307
-
308
- * Switched to hoe's racc plugin to clean up rakefile and builds
309
-
310
- * 1 bug fix:
311
-
312
- * Fixed empty =begin/end.
313
-
314
- === 2.0.5 / 2010-09-01
315
-
316
- * 1 minor enhancement:
317
-
318
- * Started merging like lexical cases to try to squeeze some optimization out
319
-
320
- === 2.0.4 / 2009-08-18
321
-
322
- * 1 minor enhancement:
323
-
324
- * Changed requires around to be more accurate.
325
-
326
- * 4 bug fixes:
327
-
328
- * Fixed .autotest for minitest
329
- * Fixed emacs escape lexing bug: "\C-\\" (maglev/gemstone)
330
- * Fixed octal lexing edgecases. (maglev/gemstone)
331
- * Fixed regexp lexing edgecases. (maglev/gemstone)
332
-
333
- === 2.0.3 / 2009-06-23
334
-
335
- * 4 minor enhancements:
336
-
337
- * Removed dead code handling d/regex match node cruft.
338
- * Switched to minitest
339
- * Updated .autotest and rakefile wrt rcov for new hoe capabilities
340
- * Updated hoe for new capabilities
341
-
342
- * 4 bug fixes:
343
-
344
- * Environment#all now deals with strange edge cases in RAD.
345
- * Fixed packaging/compilation issue.
346
- * Minor 1.9 fixes
347
- * hoe -> flay -> rubyparser rakefile circularity fixed
348
-
349
- === 2.0.2 / 2009-01-20
350
-
351
- * 2 minor enhancements:
352
-
353
- * Added gauntlet_rubyparser plugin. YAY for easy massive bug-hunting.
354
- * Promoted Sexp's file/line/comments to sexp_processor.
355
-
356
- * 4 bug fixes:
357
-
358
- * Fixed and improved the readme
359
- * Fixed lexing heredoc newlines.
360
- * Fixed line numbers on defns.
361
- * Fixed rdoc generation bug pointed out by hugh sasse (who rocks)
362
-
363
- === 2.0.1 / 2008-11-04
364
-
365
- * 2 minor enhancements:
366
-
367
- * Updated for changes to splat node in many contexts.
368
- * Made PT a developer dep
369
-
370
- === 2.0.0 / 2008-10-22
371
-
372
- * 1 major enhancement
373
-
374
- * Brought on the AWESOME! 4x faster! no known lexing/parsing bugs!
375
-
376
- * 71 minor enhancements
377
-
378
- * 1.9: Added Fixnum#ord.
379
- * 1.9: Added missing Regexp constants and did it so it'd work on 1.9.
380
- * Added #store_comment and #comments
381
- * Added StringScanner #begin_of_line?
382
- * Added a bunch of tests for regexp escape chars, #parse_string, #read_escape, ? numbers, ? whitespace.
383
- * Added a hack for rubinius' r2l eval bug.
384
- * Added a new token type tSTRING that bypasses tSTRING_BEG/END entirely. Only does non-interpolated strings and then falls back to the old way. MUCH cleaner tho.
385
- * Added bin/ruby_parse
386
- * Added compare rule to Rakefile.
387
- * Added coverage files/dirs to clean rule.
388
- * Added file and line numbers to all sexp nodes. Column/ranges to come.
389
- * Added lex_state change for lvars at the end of yylex.
390
- * Added lexed comments to defn/defs/class/module nodes.
391
- * Added stats gathering for yylex. Reordered yylex for avg data
392
- * Added tSYMBOL token type and parser rule to speed up symbol lexing.
393
- * Added tally output for getch, unread, and unread_many.
394
- * Added tests for ambigous uminus/uplus, backtick in cmdarg, square and curly brackets, numeric gvars, eos edge cases, string quoting %<> and %%%.
395
- * All cases throughout yylex now return directly if they match, no passthroughs.
396
- * All lexer cases now slurp entire token in one swoop.
397
- * All zarrays are now just empty arrays.
398
- * Changed s(:block_arg, :blah) to :"&blah" in args sexp.
399
- * Cleaned up lexer error handling. Now just raises all over.
400
- * Cleaned up read_escape and regx_options
401
- * Cleaned up tokadd_string (for some definition of cleaned).
402
- * Converted single quoted strings to new tSTRING token type.
403
- * Coverage is currently 94.4% on lexer.
404
- * Done what I can to clean up heredoc lexing... still sucks.
405
- * Flattened resbodies in rescue node. Fixed .autotest file.
406
- * Folded lex_keywords back in now that it screams.
407
- * Found very last instanceof ILiteralNode in the code. haha!
408
- * Got the tests subclassing PTTC and cleaned up a lot. YAY
409
- * Handle yield(*ary) properly
410
- * MASSIVELY cleaned out =begin/=end comment processor.
411
- * Massive overhaul on Keyword class. All hail the mighty Hash!
412
- * Massively cleaned up ident= edge cases and fixed a stupid bug from jruby.
413
- * Merged @/@@ scanner together, going to try to do the same everywhere.
414
- * Refactored fix_arg_lex_state, common across the lexer.
415
- * Refactored new_fcall into new_call.
416
- * Refactored some code to get better profile numbers.
417
- * Refactored some more #fix_arg_lex_state.
418
- * Refactored tail of yylex into its own method.
419
- * Removed Module#kill
420
- * Removed Token, replaced with Sexp.
421
- * Removed all parse_number and parse_quote tests.
422
- * Removed argspush, argscat. YAY!
423
- * Removed as many token_buffer.split(//)'s as possible. 1 to go.
424
- * Removed begins from compstmts
425
- * Removed buffer arg for tokadd_string.
426
- * Removed crufty (?) solo '@' token... wtf was that anyhow?
427
- * Removed most jruby/stringio cruft from StringScanner.
428
- * Removed one unread_many... 2 to go. They're harder.
429
- * Removed store_comment, now done directly.
430
- * Removed token_buffer. Now I just use token ivar.
431
- * Removed use of s() from lexer. Changed the way line numbers are gathered.
432
- * Renamed *qwords to *awords.
433
- * Renamed StringScanner to RPStringScanner (a subclass) to fix namespace trashing.
434
- * Renamed parse to process and aliased to parse.
435
- * Renamed token_buffer to string_buffer since that arcane shit still needs it.
436
- * Resolved the rest of the lexing issues I brought up w/ ruby-core.
437
- * Revamped tokadd_escape.
438
- * Rewrote Keyword and KWtable.
439
- * Rewrote RubyLexer using StringScanner.
440
- * Rewrote tokadd_escape. 79 lines down to 21.
441
- * Split out lib/ruby_parser_extras.rb so lexer is standalone.
442
- * Started to clean up the parser and make it as skinny as possible
443
- * Stripped out as much code as possible.
444
- * Stripped yylex of some dead code.
445
- * Switched from StringIO to StringScanner.
446
- * Updated rakefile for new hoe.
447
- * Uses pure ruby racc if ENV['PURE_RUBY'], otherwise use c.
448
- * Wrote a ton of lexer tests. Coverage is as close to 100% as possible.
449
- * Wrote args to clean up the big nasty args processing grammar section.
450
- * lex_strterm is now a plain array, removed RubyLexer#s(...).
451
- * yield and super now flatten args.
452
-
453
- * 21+ bug fixes:
454
-
455
- * I'm sure this list is missing a lot:
456
- * Fixed 2 bugs both involving attrasgn (and ilk) esp when lhs is an array.
457
- * Fixed a bug in the lexer for strings with single digit hex escapes.
458
- * Fixed a bug parsing: a (args) { expr }... the space caused a different route to be followed and all hell broke loose.
459
- * Fixed a bug with x\n=beginvar not putting begin back.
460
- * Fixed attrasgn to have arglists, not arrays.
461
- * Fixed bug in defn/defs with block fixing.
462
- * Fixed class/module's name slot if colon2/3.
463
- * Fixed dstr with empty interpolation body.
464
- * Fixed for 1.9 string/char changes.
465
- * Fixed lexer BS wrt determining token type of words.
466
- * Fixed lexer BS wrt pass through values and lexing words. SO STUPID.
467
- * Fixed lexing of floats.
468
- * Fixed lexing of identifiers followed by equals. I hope.
469
- * Fixed masgn with splat on lhs
470
- * Fixed new_super to deal with block_pass correctly.
471
- * Fixed parser's treatment of :colon2 and :colon3.
472
- * Fixed regexp scanning of escaped numbers, ANY number is valid, not just octs.
473
- * Fixed string scanning of escaped octs, allowing 1-3 chars.
474
- * Fixed unescape for \n
475
- * Fixed: omg this is stupid. '()' was returning bare nil
476
- * Fixed: remove_begin now goes to the end, not sure why it didn't before.
477
-
478
- === 1.0.0 / 2007-12-20
479
-
480
- * 1 major enhancement
481
- * Birthday!
482
-
@@ -1,120 +0,0 @@
1
- #!/usr/bin/ruby -ws
2
-
3
- $f ||= false
4
-
5
- $:.unshift "../../ruby_parser/dev/lib"
6
- $:.unshift "../../ruby2ruby/dev/lib"
7
-
8
- require 'rubygems'
9
- require 'ruby2ruby'
10
- require 'ruby_parser'
11
-
12
- require 'gauntlet'
13
-
14
- class RubyParserGauntlet < Gauntlet
15
- def initialize
16
- super
17
-
18
- self.data = Hash.new { |h,k| h[k] = {} }
19
- old_data = load_yaml data_file
20
- self.data.merge! old_data
21
- end
22
-
23
- def should_skip? name
24
- if $f then
25
- if Hash === data[name] then
26
- ! data[name].empty?
27
- else
28
- data[name]
29
- end
30
- else
31
- data[name] == true # yes, == true on purpose
32
- end
33
- end
34
-
35
- def diff_pp o1, o2
36
- require 'pp'
37
-
38
- File.open("/tmp/a.#{$$}", "w") do |f|
39
- PP.pp o1, f
40
- end
41
-
42
- File.open("/tmp/b.#{$$}", "w") do |f|
43
- PP.pp o2, f
44
- end
45
-
46
- `diff -u /tmp/a.#{$$} /tmp/b.#{$$}`
47
- ensure
48
- File.unlink "/tmp/a.#{$$}" rescue nil
49
- File.unlink "/tmp/b.#{$$}" rescue nil
50
- end
51
-
52
- def broke name, file, msg
53
- warn "bad"
54
- self.data[name][file] = msg
55
- self.dirty = true
56
- end
57
-
58
- def process path, name
59
- begin
60
- $stderr.print " #{path}: "
61
- rp = RubyParser.new
62
- r2r = Ruby2Ruby.new
63
-
64
- old_ruby = File.read(path)
65
-
66
- begin
67
- old_sexp = rp.process old_ruby
68
- rescue Racc::ParseError => e
69
- self.data[name][path] = :unparsable
70
- self.dirty = true
71
- return
72
- end
73
-
74
- new_ruby = r2r.process old_sexp.deep_clone
75
-
76
- begin
77
- new_sexp = rp.process new_ruby
78
- rescue Racc::ParseError => e
79
- broke name, path, "couldn't parse new_ruby: #{e.message.strip}"
80
- return
81
- end
82
-
83
- if old_sexp != new_sexp then
84
- broke name, path, diff_pp(old_sexp, new_sexp)
85
- return
86
- end
87
-
88
- self.data[name][path] = true
89
- self.dirty = true
90
-
91
- warn "good"
92
- rescue Interrupt
93
- puts "User cancelled"
94
- exit 1
95
- rescue Exception => e
96
- broke name, path, " UNKNOWN ERROR: #{e}: #{e.message.strip}"
97
- end
98
- end
99
-
100
- def run name
101
- warn name
102
- Dir["**/*.rb"].sort.each do |path|
103
- next if path =~ /gemspec.rb/ # HACK
104
- next if data[name][path] == true
105
- process path, name
106
- end
107
-
108
- if self.data[name].values.all? { |v| v == true } then
109
- warn " ALL GOOD!"
110
- self.data[name] = true
111
- self.dirty = true
112
- end
113
- end
114
- end
115
-
116
- filter = ARGV.shift
117
- filter = Regexp.new filter if filter
118
-
119
- gauntlet = RubyParserGauntlet.new
120
- gauntlet.run_the_gauntlet filter