ruby_parser 3.13.1 → 3.21.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 92087c0c745fee6dd8a060c3180c8254e0f23fd0d252b23f3988a3d92890ce64
-  data.tar.gz: 4ff9742257ffdf393892e293b1548587f552b6bb92fc2484bda6805c27cf1cff
+  metadata.gz: 6130eb28e9d379051c5b2f5c83b2c77ca217e82e0ca0aa53736128810fd1ff35
+  data.tar.gz: 7359cfb0974a66a0d07bb75899d92c2082a23e326bd6aed56ae5c7790e2819be
 SHA512:
-  metadata.gz: 8585bd8ef2918e9a31907b71dfada37f2f4d2755685ff976702bcfa90ec0137afd1b6e71bced5c631f8549188dc62169c8137eb3cf8475b393ec65174a0ac7bc
-  data.tar.gz: ead3c27d85cae963568e98c53ded6d7199da7ea781b29895eecdd54756af714b154528dc20920b3f8982519d1fd6a2f80c933489844c1d81052cc991b88172d6
+  metadata.gz: 0d9e8d398b4f7f412406c23ec13557d4c80f4c10fffbf9563992df2cbbda7c0c9ca46e933025c20930d7434336b63d6485d8cb5463e484e5a04c676d7207a8eb
+  data.tar.gz: a34f3b845c73d919b5e502d07f2683e823dd126372cac137eb7c4b504ad4b90f6a25f46f8eb30ef62cba55ef50b7eb3b148d426dcdeabeca87e3465b0622b548

data/.autotest

@@ -7,20 +7,20 @@ require 'autotest/rcov' if ENV['RCOV']
 Autotest.add_hook :initialize do |at|
   at.extra_files << "../../sexp_processor/dev/lib/pt_testcase.rb"
   at.libs << ":../../sexp_processor/dev/lib"
-  at.add_exception "unit"
-  at.add_exception "coverage"
-  at.add_exception "coverage.info"
   at.add_exception ".diff"
+  at.add_exception ".rdoc"
+  at.add_exception ".yml"
+  at.add_exception ".txt"
   at.add_exception ".output"
-  at.add_exception "lib/ruby18_parser.rb"
-  at.add_exception "lib/ruby19_parser.rb"
-  at.add_exception "lib/ruby20_parser.rb"
-  at.add_exception "lib/ruby21_parser.rb"
-  at.add_exception "lib/ruby22_parser.rb"
-  at.add_exception "lib/ruby20_parser.y"
-  at.add_exception "lib/ruby21_parser.y"
-  at.add_exception "lib/ruby22_parser.y"
-  at.add_exception "lib/ruby_lexer.rex.rb"
+  at.add_exception "bin"
+  at.add_exception "compare"
+  at.add_exception "coverage"
+  at.add_exception "coverage.info"
+  at.add_exception "misc"
+
+  Dir["lib/ruby??_parser.{rb,y}"].each do |f|
+    at.add_exception f
+  end
 
   Dir["gauntlet*"].each do |f|
     at.add_exception f
@@ -30,12 +30,12 @@ Autotest.add_hook :initialize do |at|
   at.testlib = "minitest/autorun"
 
   at.add_mapping(/^lib\/.*\.y$/) do |f, _|
-    g = File.basename(f, ".y").gsub("_", "_?").gsub(/1[89]|2[012]/, '')
+    g = File.basename(f, ".y").gsub("_", "_?").gsub(/2\d/, '')
     at.files_matching %r%^test/.*#{g}.rb$%
   end
 
   at.add_mapping(/^lib\/.*\.yy$/) do |f, _|
-    g = File.basename(f, ".yy").gsub("_", "_?").gsub(/1[89]|2[012]/, '')
+    g = File.basename(f, ".yy").gsub("_", "_?")
     at.files_matching %r%^test/.*#{g}.rb$%
   end
 
@@ -52,27 +52,16 @@ Autotest.add_hook :initialize do |at|
     at.files_matching(/test_.*rb$/)
   end
 
-  %w(TestEnvironment TestStackState).each do |klass|
+  %w(TestEnvironment TestStackState TestValueExpr).each do |klass|
     at.extra_class_map[klass] = "test/test_ruby_parser_extras.rb"
   end
 
-  %w(TestRuby18Parser TestRuby19Parser TestRuby20Parser TestRuby21Parser TestRuby22Parser TestRuby23Parser TestParseTree).each do |klass| # HACK
-    at.extra_class_map[klass] = "test/test_ruby_parser.rb"
+  Dir["lib/ruby??_parser.rb"].each do |s|
+    n = s[/\d+/]
+    at.extra_class_map["TestRubyParserV#{n}"] = "test/test_ruby_parser.rb"
   end
 end
 
 Autotest.add_hook :run_command do |at, _|
   system "rake parser lexer DEBUG=1"
 end
-
-class Autotest
-  def ruby
-    File.expand_path "~/.multiruby/install/1.9.3-p194/bin/ruby"
-  end
-end if ENV["R19"]
-
-class Autotest
-  def ruby
-    File.expand_path "~/.multiruby/install/2.0.0-p195/bin/ruby"
-  end
-end if ENV["R20"]

data/History.rdoc

@@ -1,3 +1,315 @@
+=== 3.21.0 / 2024-01-15
+
+* 18 minor enhancements:
+
+  * (internal) Added simple RubyLexer#comment.
+  * (internal) Removed RubyLexer#comments and #ignore_body_comments.
+  * (internal) Removed RubyParser#comments.
+  * (internal) Renamed files from lib/ruby##_parser.* to lib/ruby_parser##.*
+  * (internal) Renamed lib/ruby3_parser.yy to lib/ruby_parser3.yy
+  * (internal) Renamed lib/ruby_parser.yy to lib/ruby_parser2.yy
+  * Added tentative 3.2 support. (torrocus)
+  * Made `rake compare` a multitask (28s -> 10s!!).
+  * Made `rake compare` depend on `generate` (from clean: 28s -> 18s).
+  * No longer package versioned .y files.
+  * Overhaul of parser/lexer comment processing.
+  * Parser productions for k_class/k_module/k_def carry their comment with them.
+  * Restructured build tasks to utilize Hoe's .yy support! 8.5s -> 1.2s!!!
+  * RubyParser new_(class|module|defn|defs) handle comment processing via sexp.
+  * RubyParser new_(class|module|defn|defs) handle normalizing production sexps.
+  * RubyParser#next_token clears out lexer's comment when applicable.
+  * Simplified parser productions for defn_head, defs_head. No more normalizing.
+  * Updated `rake compare` steps for lrama in ruby 3.3+.
+
+* 6 bug fixes:
+
+  * Cleaned out cruft from `rake compare` code.
+  * Fixed compare/normalize.rb for lrama differences.
+  * Made `parser` task depend on `isolate` task to pick up racc generator.
+  * Made racc a runtime dependency (was dev only) now that ruby 3.3 doesn't ship with the racc runtime. (wildmaples)
+  * Updated README.rdoc updating instructions to match current filenames.
+  * Updated compare versions and added 3.3.
+
+=== 3.20.3 / 2023-07-11
+
+* 2 minor enhancements:
+
+  * Added Parser#in_argdef and integrated into 3.x parsers.
+  * Improved tools/munge.rb to handler MRI 3.2 output
+
+* 2 bug fixes:
+
+  * Fixed process_dots to properly deal with paren-less forward_args. (eric1234)
+  * Fixed tools/ripper.rb to properly print ripper sexp at the end
+
+=== 3.20.2 / 2023-06-06
+
+* 1 bug fix:
+
+  * 3.2: fixed parsing of f(*) and f(**). (agrobbin)
+
+=== 3.20.1 / 2023-05-16
+
+* 1 minor enhancement:
+
+  * Fixes Sexp#line_max in parser for many constructs: paren_args, arrays of various sorts, calls, classes, modules, etc.
+
+=== 3.20.0 / 2023-03-04
+
+* 1 major enhancement:
+
+  * Added tentative 3.2 support.
+
+* 1 minor enhancement:
+
+  * Change minimum ruby version to 2.6. (want higher)
+
+* 1 bug fix:
+
+  * Fix up compare tasks for ruby 3.2 differences.
+
+=== 3.19.2 / 2022-12-03
+
+* 5 bug fixes:
+
+  * Fixed bug parsing array patterns using %w, %i, etc.
+  * Fixed dedenting when heredoc has only empty line and interpolation. (mvz)
+  * Fixed dsym unescaping. (mvz)
+  * Fixed vars in pattern matching that should be lasgn, not lvar.
+  * find_pat mid-patterns are now splatted inline, not wrapped in an array_pat.
+
+=== 3.19.1 / 2022-04-05
+
+* 2 bug fixes:
+
+  * Added comments to endless defn and defs. (mvz)
+  * Fixed endless method bug handling attrset names.
+
+=== 3.19.0 / 2022-03-29
+
+* 1 major enhancement:
+
+  * Added tentative 3.1 support.
+
+* 7 minor enhancements:
+
+  * 3.1: bare RHS assoc: { y: } => s(:hash, s(:lit, :y), nil)
+  * 3.1: calls w/ unnamed block args (bare &)
+  * 3.1: endless defn/defs w/ paren-less calls (aka commands)
+  * 3.1: pattern capture to nonlocal vars, eg: ^@a, ^$b, ^@@c
+  * 3.1: pattern: ^(expr) => expr
+  * Improved steps for adding new versions.
+  * Improved steps for running gauntlets.
+
+* 2 bug fixes:
+
+  * Bumped 2.6+ cached versions for rake compare.
+  * Skip test_regexp_esc_C_slash on ruby 3.1.0 because of MRI bug.
+
+=== 3.18.1 / 2021-11-10
+
+* 1 minor enhancement:
+
+  * All parser tests are now explicitly testing line numbers at every level.
+
+* 3 bug fixes:
+
+  * Fixed endless method with noargs. (mitsuru)
+  * Fixed line numbers on some yield forms.
+  * Handle and clearly report if unifdef is missing.
+
+=== 3.18.0 / 2021-10-27
+
+Holy crap... 58 commits! 2.7 and 3.0 are feature complete. Strings
+& heredocs have been rewritten.
+
+* 9 major enhancements:
+
+  * !!! Rewrote lexer (and friends) for strings, heredocs, and %*[] constructs.
+  * Massive overhaul on line numbers.
+  * Freeze input! Finally!!! No more modifying the input string for heredocs.
+  * Overhauled RPStringScanner. Removed OLD compatibility methods!
+  * Removed Sexp methods: value, to_sym, add, add_all, node_type, values.
+    * value moved to sexp_processor.
+  * Removed String#grep monkey-patch.
+  * Removed String#lineno monkey-patch.
+  * Removed string_to_pos, charpos, etc hacks for ancient ruby versions.
+  * Removed unread_many... NO! NO EDITING THE INPUT STRING!
+
+* 31 minor enhancements:
+
+  * 2.7/3.0: many more pattern edge cases
+  * 2.7: Added `mlhs = rhs rescue expr`
+  * 2.7: refactored destructured args (`|(k,v)|`) and unfactored(?!) case_body/args.
+  * 3.0: excessed_comma
+  * 3.0: finished most everything: endless methods, patterns, etc.
+  * 3.0: refactored / added new pattern changes
+  * Added RubyLexer#in_heredoc? (ie, is there old_ss ?)
+  * Added RubyLexer#old_ss and old_lineno and removed much of SSStack(ish).
+  * Added Symbol#end_with? when necessary
+  * Added TALLY and DEBUG options for ss.getch and ss.scan
+  * Added ignore_body_comments to make parser productions more clear.
+  * Added support for no_kwarg (eg `def f(**nil)`).
+  * Added support for no_kwarg in blocks (eg `f { |**nil| }`).
+  * Augmented generated parser files to have frozen_string_literal comments and fixed tests.
+  * Broke out 3.0 parser into its own to ease development.
+  * Bumped dependencies on sexp_processor and oedipus_lex.
+  * Clean generated 3.x files.
+  * Extracted all string scanner methods to their own module.
+  * Fixed some precedence decls.
+  * Implemented most of pattern matching for 2.7+.
+  * Improve lex_state= to report location in verbose debug mode.
+  * Made it easier to debug with a particular version of ruby via rake.
+  * Make sure ripper uses the same version of ruby we specified.
+  * Moved all string/heredoc/etc code to ruby_lexer_strings.rb
+  * Remove warning from newer bisons.
+  * Sprinkled in some frozen_string_literal, but mostly helped by oedipus bump.
+  * Switch to comparing against ruby binary since ripper is buggy.
+  * bugs task should try both bug*.rb and bad*.rb.
+  * endless methods
+  * f_any_kwrest refactoring.
+  * refactored defn/defs
+
+* 15 bug fixes:
+
+  * Cleaned a bunch of old hacks. Initializing RubyLexer w/ Parser is cleaner now.
+  * Corrected some lex_state errors in process_token_keyword.
+  * Fixed ancient ruby2 change (use #lines) in ruby_parse_extract_error.
+  * Fixed bug where else without rescue only raises on 2.6+
+  * Fixed caller for getch and scan when DEBUG=1
+  * Fixed comments in the middle of message cascades.
+  * Fixed differences w/ symbol productions against ruby 2.7.
+  * Fixed dsym to use string_contents production.
+  * Fixed error in bdot2/3 in some edge cases. Fixed p_alt line.
+  * Fixed heredoc dedenting in the presence of empty lines. (mvz)
+  * Fixed some leading whitespace / comment processing
+  * Fixed up how class/module/defn/defs comments were collected.
+  * Overhauled ripper.rb to deal with buggy ripper w/ yydebug.
+  * Removed dsym from literal.
+  * Removed tUBANG lexeme but kept it distinct as a method name (eg: `def !@`).
+
+=== 3.17.0 / 2021-08-03
+
+* 1 minor enhancement:
+
+  * Added support for arg forwarding (eg `def f(...); m(...); end`) (presidentbeef)
+
+=== 3.16.0 / 2021-05-15
+
+* 1 major enhancement:
+
+  * Added tentative 3.0 support.
+
+* 3 minor enhancements:
+
+  * Added lexing for "beginless range" (bdots).
+  * Added parsing for bdots.
+  * Updated rake compare task to download xz files, bumped versions, etc
+
+* 4 bug fixes:
+
+  * Bump rake dependency to >= 10, < 15. (presidentbeef)
+  * Bump sexp_processor dependency to 4.15.1+. (pravi)
+  * Fixed minor state mismatch at the end of parsing to make diffing a little cleaner.
+  * Fixed normalizer to deal with new bison token syntax
+
+=== 3.15.1 / 2021-01-10
+
+* 1 bug fix:
+
+  * Bumped ruby version to include < 4 (trunk).
+
+=== 3.15.0 / 2020-08-31
+
+* 1 major enhancement:
+
+  * Added tentative 2.7 support.
+
+* 1 minor enhancement:
+
+  * Improved ruby_parse_extract_error's handling of moving slow files out.
+
+* 22 bug fixes:
+
+  * Bumped ruby version to include 3.0 (trunk).
+  * Fix an error related to empty ensure bodies. (presidentbeef)
+  * Fix handling of bad magic encoding comment.
+  * Fixed SystemStackError when parsing a huoooge hash, caused by a splat arg.
+  * Fixed a number of errors parsing do blocks in strange edge cases.
+  * Fixed a string backslash lexing bug when the string is an invalid encoding. (nijikon, gmcgibbon)
+  * Fixed bug assigning line number to some arg nodes.
+  * Fixed bug concatinating string literals with differing encodings.
+  * Fixed bug lexing heredoc w/ nasty mix of \r\n and \n.
+  * Fixed bug lexing multiple codepoints in \u{0000 1111 2222} forms.
+  * Fixed bug setting line numbers in empty xstrings in some contexts.
+  * Fixed edge case on call w/ begin + do block as an arg.
+  * Fixed handling of UTF BOM.
+  * Fixed handling of lexer state across string interpolation braces.
+  * Fixed infinite loop when lexing backslash+cr+newline (aka dos-files)
+  * Fixed lambda + do block edge case.
+  * Fixed lexing of some ?\M... and ?\C... edge cases.
+  * Fixed more do/brace block edge case failures.
+  * Fixed parsing bug where splat was used in the middle of a list.
+  * Fixed parsing of interpolation in heredoc-like strings. (presidentbeef)
+  * Fixed parsing some esoteric edge cases in op_asgn.
+  * Fixed unicode processing in ident chars so now they better mix.
+
+=== 3.14.2 / 2020-02-06
+
+* 1 minor enhancement:
+
+  * Cleaned up call_args and removed arg_blk_pass from ruby_parser_extras.rb! Yay!
+
+=== 3.14.1 / 2019-10-29
+
+* 1 minor enhancement:
+
+  * Declared that ruby_parser supports ruby 2.2 and up.
+
+* 3 bug fixes:
+
+  * Fixed a problem with %W with a null-byte terminator. (wtf?) (spohlenz)
+  * Fixed line numbering for command (eg methods without parentheses) arguments. (mvz)
+  * Fixed lineno on new dxstrs. (presidentbeef)
+
+=== 3.14.0 / 2019-09-24
+
+* 8 minor enhancements:
+
+  * Added Sexp#line_min & new line number debugging feature.
+  * Allow bin/ruby_parse to use RP_TIMEOUT to tweak default timeout for large files & slow debugging modes.
+  * Did you know that custom inspect can't be > 65 chars? 65! Me neither. Fixed.
+  * For now, value_expr == remove_begin.
+  * Improved error messages for unterminated regexps.
+  * Moved all STR_* consts into State::Values and made them State instances.
+  * Overhauled RubyLexer::State + Values to allow for completely separate groups of States.
+  * RubyParserExtras#remove_begin removes nested begins and patches line numbers.
+
+* 22 bug fixes:
+
+  * Changed EOF signaling to return [false, false] per doco.
+  * Changed RubyParserStuff#remove_begin to keep inner line number, if any. (mvz)
+  * Differentiated between lambda call and stabby proc. Added :lambda sexp. (whitequark)
+  * Extended State#== to check equal? first as an optimization.
+  * Fixed a bug in tab indent width.
+  * Fixed a line numbering bug for assignables.
+  * Fixed a line numbering bug for defns w/ nil bodies.
+  * Fixed another missing line number, driven out by ruby_to_ruby bootstrap.
+  * Fixed dedenting squiggle heredocs if interpolation is in the mix. (mvz)
+  * Fixed differentiation of `{ |a| ... }` vs `{ |a,| ... }`. Fixes #236 (whitequark).
+  * Fixed lex_state for symbols.
+  * Fixed lex_state for tSTRING_END and tREGEXP_END.
+  * Fixed line numbers for BEGIN block. (mvz)
+  * Fixed op_asgn1 w/ command_call having array instead of arglist. (mvz)
+  * Fixed parsing of () in a command-call (eg p).
+  * Fixed remaining missing line numbers by forcing extra checks during test and blowing up. They may still be wrong (esp after heredocs, ugh), but they're there! Special thank you to mvz for pushing on this and providing tests and PRs.
+  * Fixed some lex_state versioning issues with closing braces/bracket on ruby<25.
+  * Keep carriage return escapes in heredocs. (mvz)
+  * Massive overhaul of line number code.
+  * More line number fixes and extra tests from mvz. Thank you!
+  * Parsing BEGIN block when not at top level should raise an error. (mvz)
+  * Removed irb task from Rakefile as it has been pushed up to Hoe and doubles up here.
+
 === 3.13.1 / 2019-03-25
 
 * 1 major enhancement:

data/Manifest.txt

@@ -7,28 +7,29 @@ bin/ruby_parse
 bin/ruby_parse_extract_error
 compare/normalize.rb
 debugging.md
+gauntlet.md
 lib/.document
 lib/rp_extensions.rb
 lib/rp_stringscanner.rb
-lib/ruby20_parser.rb
-lib/ruby20_parser.y
-lib/ruby21_parser.rb
-lib/ruby21_parser.y
-lib/ruby22_parser.rb
-lib/ruby22_parser.y
-lib/ruby23_parser.rb
-lib/ruby23_parser.y
-lib/ruby24_parser.rb
-lib/ruby24_parser.y
-lib/ruby25_parser.rb
-lib/ruby25_parser.y
-lib/ruby26_parser.rb
-lib/ruby26_parser.y
 lib/ruby_lexer.rb
 lib/ruby_lexer.rex
 lib/ruby_lexer.rex.rb
+lib/ruby_lexer_strings.rb
 lib/ruby_parser.rb
-lib/ruby_parser.yy
+lib/ruby_parser2.yy
+lib/ruby_parser20.rb
+lib/ruby_parser21.rb
+lib/ruby_parser22.rb
+lib/ruby_parser23.rb
+lib/ruby_parser24.rb
+lib/ruby_parser25.rb
+lib/ruby_parser26.rb
+lib/ruby_parser27.rb
+lib/ruby_parser3.yy
+lib/ruby_parser30.rb
+lib/ruby_parser31.rb
+lib/ruby_parser32.rb
+lib/ruby_parser33.rb
 lib/ruby_parser_extras.rb
 test/test_ruby_lexer.rb
 test/test_ruby_parser.rb

data/README.rdoc

@@ -7,9 +7,9 @@ rdoc :: http://docs.seattlerb.org/ruby_parser
 == DESCRIPTION:
 
 ruby_parser (RP) is a ruby parser written in pure ruby (utilizing
-racc--which does by default use a C extension). RP's output is
-the same as ParseTree's output: s-expressions using ruby's arrays and
-base types.
+racc--which does by default use a C extension). It outputs
+s-expressions which can be manipulated and converted back to ruby via
+the ruby2ruby gem.
 
 As an example:
 
@@ -32,6 +32,10 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
 * 1.8 parser is at 99.9739% accuracy, 3.651 sigma
 * 1.9 parser is at 99.9940% accuracy, 4.013 sigma
 * 2.0 parser is at 99.9939% accuracy, 4.008 sigma
+* 2.6 parser is at 99.9972% accuracy, 4.191 sigma
+* 3.0 parser has a 100% parse rate.
+  * Tested against 2,672,412 unique ruby files across 167k gems.
+  * As do all the others now, basically.
 
 == FEATURES/PROBLEMS:
 
@@ -61,15 +65,15 @@ You can also use Ruby19Parser, Ruby18Parser, or RubyParser.for_current_ruby:
 
 To add a new version:
 
-* New parser should be generated from lib/ruby_parser.yy.
-* Extend lib/ruby_parser.yy with new class name.
-* Add new version number to V2 in Rakefile for rule creation.
+* New parser should be generated from lib/ruby_parser[23].yy.
+* Extend lib/ruby_parser[23].yy with new class name.
+* Add new version number to V2/V3 in Rakefile for rule creation.
+* Add new `ruby_parse "x.y.z"` line to Rakefile for rake compare (line ~300).
 * Require generated parser in lib/ruby_parser.rb.
+* Add new V## = ::Ruby##Parser; end to ruby_parser.rb (bottom of file).
 * Add empty TestRubyParserShared##Plus module and TestRubyParserV## to test/test_ruby_parser.rb.
 * Extend Manifest.txt with generated file names.
-* Extend sexp_processor's pt_testcase.rb to match version
-  * add_19tests needs to have the version added
-  * VER_RE needs to have the regexp expanded
+* Add new version number to sexp_processor's pt_testcase.rb in all_versions.
 
 Until all of these are done, you won't have a clean test run.