ruby_parser 3.15.1 → 3.18.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0adcb0c06b7ee6d29360bd8dd9641dfcce867cdd7167f59bef5134c3092cb091
-  data.tar.gz: ef51691d5a0e2f09a38a0c6d8bd8d3fccdebf19bd1c1ce41a99f761112a43c4a
+  metadata.gz: dad59e40690dfb746fea4399e13d05f97b43cf562c2559c67d1f9b5868da86c5
+  data.tar.gz: 0c14605ecf34929f6d4865c20f085b6ef2a35533a70e530f261f9d04f44998cb
 SHA512:
-  metadata.gz: 85e6684d7a8a6207f1040ca97768d4792ba52e2f039eee5fa83047e8e5c83a7a913f98fc5384ac8b3387829078e294e22fb009163ac4eee272557cbed715849e
-  data.tar.gz: 7e996bef3411d3f63dfd8e2af388bddb16dc8a4206bd309611aeda038feb4c66041fb2b7ba5136015d4b0453e34fdebf8c1f8d408d68dccb9031514ebff6f728
+  metadata.gz: 4fa69e55ac577e55fa2945ef9c135b47359a646e654bc8cf9f2ead973852adb97350cf67ac5da549592ea583f4824cd5f6930a167fa19cf3f914cb6d6888093d
+  data.tar.gz: 0c13eb02ffaf3b3c60ef358be2fa8f70b951e947170d3eeb8a94e5c37bec5dc8282376e4ab3f03de99e49b6e2aac2aa13169b31c30de1067e1a290c1fd65803a

data/History.rdoc

@@ -1,3 +1,110 @@
+=== 3.18.1 / 2021-11-10
+
+* 1 minor enhancement:
+
+  * All parser tests are now explicitly testing line numbers at every level.
+
+* 3 bug fixes:
+
+  * Fixed endless method with noargs. (mitsuru)
+  * Fixed line numbers on some yield forms.
+  * Handle and clearly report if unifdef is missing.
+
+=== 3.18.0 / 2021-10-27
+
+Holy crap... 58 commits! 2.7 and 3.0 are feature complete. Strings
+& heredocs have been rewritten.
+
+* 9 major enhancements:
+
+  * !!! Rewrote lexer (and friends) for strings, heredocs, and %*[] constructs.
+  * Massive overhaul on line numbers.
+  * Freeze input! Finally!!! No more modifying the input string for heredocs.
+  * Overhauled RPStringScanner. Removed OLD compatibility methods!
+  * Removed Sexp methods: value, to_sym, add, add_all, node_type, values.
+    * value moved to sexp_processor.
+  * Removed String#grep monkey-patch.
+  * Removed String#lineno monkey-patch.
+  * Removed string_to_pos, charpos, etc hacks for ancient ruby versions.
+  * Removed unread_many... NO! NO EDITING THE INPUT STRING!
+
+* 31 minor enhancements:
+
+  * 2.7/3.0: many more pattern edge cases
+  * 2.7: Added `mlhs = rhs rescue expr`
+  * 2.7: refactored destructured args (`|(k,v)|`) and unfactored(?!) case_body/args.
+  * 3.0: excessed_comma
+  * 3.0: finished most everything: endless methods, patterns, etc.
+  * 3.0: refactored / added new pattern changes
+  * Added RubyLexer#in_heredoc? (ie, is there old_ss ?)
+  * Added RubyLexer#old_ss and old_lineno and removed much of SSStack(ish).
+  * Added Symbol#end_with? when necessary
+  * Added TALLY and DEBUG options for ss.getch and ss.scan
+  * Added ignore_body_comments to make parser productions more clear.
+  * Added support for no_kwarg (eg `def f(**nil)`).
+  * Added support for no_kwarg in blocks (eg `f { |**nil| }`).
+  * Augmented generated parser files to have frozen_string_literal comments and fixed tests.
+  * Broke out 3.0 parser into its own to ease development.
+  * Bumped dependencies on sexp_processor and oedipus_lex.
+  * Clean generated 3.x files.
+  * Extracted all string scanner methods to their own module.
+  * Fixed some precedence decls.
+  * Implemented most of pattern matching for 2.7+.
+  * Improve lex_state= to report location in verbose debug mode.
+  * Made it easier to debug with a particular version of ruby via rake.
+  * Make sure ripper uses the same version of ruby we specified.
+  * Moved all string/heredoc/etc code to ruby_lexer_strings.rb
+  * Remove warning from newer bisons.
+  * Sprinkled in some frozen_string_literal, but mostly helped by oedipus bump.
+  * Switch to comparing against ruby binary since ripper is buggy.
+  * bugs task should try both bug*.rb and bad*.rb.
+  * endless methods
+  * f_any_kwrest refactoring.
+  * refactored defn/defs
+
+* 15 bug fixes:
+
+  * Cleaned a bunch of old hacks. Initializing RubyLexer w/ Parser is cleaner now.
+  * Corrected some lex_state errors in process_token_keyword.
+  * Fixed ancient ruby2 change (use #lines) in ruby_parse_extract_error.
+  * Fixed bug where else without rescue only raises on 2.6+
+  * Fixed caller for getch and scan when DEBUG=1
+  * Fixed comments in the middle of message cascades.
+  * Fixed differences w/ symbol productions against ruby 2.7.
+  * Fixed dsym to use string_contents production.
+  * Fixed error in bdot2/3 in some edge cases. Fixed p_alt line.
+  * Fixed heredoc dedenting in the presence of empty lines. (mvz)
+  * Fixed some leading whitespace / comment processing
+  * Fixed up how class/module/defn/defs comments were collected.
+  * Overhauled ripper.rb to deal with buggy ripper w/ yydebug.
+  * Removed dsym from literal.
+  * Removed tUBANG lexeme but kept it distinct as a method name (eg: `def !@`).
+
+=== 3.17.0 / 2021-08-03
+
+* 1 minor enhancement:
+
+  * Added support for arg forwarding (eg `def f(...); m(...); end`) (presidentbeef)
+
+=== 3.16.0 / 2021-05-15
+
+* 1 major enhancement:
+
+  * Added tentative 3.0 support.
+
+* 3 minor enhancements:
+
+  * Added lexing for "beginless range" (bdots).
+  * Added parsing for bdots.
+  * Updated rake compare task to download xz files, bumped versions, etc
+
+* 4 bug fixes:
+
+  * Bump rake dependency to >= 10, < 15. (presidentbeef)
+  * Bump sexp_processor dependency to 4.15.1+. (pravi)
+  * Fixed minor state mismatch at the end of parsing to make diffing a little cleaner.
+  * Fixed normalizer to deal with new bison token syntax
+
 === 3.15.1 / 2021-01-10
 
 * 1 bug fix:

data/Manifest.txt

@@ -7,6 +7,7 @@ bin/ruby_parse
 bin/ruby_parse_extract_error
 compare/normalize.rb
 debugging.md
+gauntlet.md
 lib/.document
 lib/rp_extensions.rb
 lib/rp_stringscanner.rb
@@ -26,9 +27,13 @@ lib/ruby26_parser.rb
 lib/ruby26_parser.y
 lib/ruby27_parser.rb
 lib/ruby27_parser.y
+lib/ruby30_parser.rb
+lib/ruby30_parser.y
+lib/ruby3_parser.yy
 lib/ruby_lexer.rb
 lib/ruby_lexer.rex
 lib/ruby_lexer.rex.rb
+lib/ruby_lexer_strings.rb
 lib/ruby_parser.rb
 lib/ruby_parser.yy
 lib/ruby_parser_extras.rb

data/README.rdoc

@@ -32,6 +32,7 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
 * 1.8 parser is at 99.9739% accuracy, 3.651 sigma
 * 1.9 parser is at 99.9940% accuracy, 4.013 sigma
 * 2.0 parser is at 99.9939% accuracy, 4.008 sigma
+* 2.6 parser is at 99.9972% accuracy, 4.191 sigma
 
 == FEATURES/PROBLEMS:
 

data/Rakefile

@@ -14,25 +14,37 @@ Hoe.add_include_dirs "../../minitest/dev/lib"
 Hoe.add_include_dirs "../../oedipus_lex/dev/lib"
 
 V2   = %w[20 21 22 23 24 25 26 27]
-V2.replace [V2.last] if ENV["FAST"] # HACK
+V3   = %w[30]
+
+VERS = V2 + V3
+
+ENV["FAST"] = VERS.last if ENV["FAST"] && !VERS.include?(ENV["FAST"])
+VERS.replace [ENV["FAST"]] if ENV["FAST"]
 
 Hoe.spec "ruby_parser" do
   developer "Ryan Davis", "ryand-ruby@zenspider.com"
 
   license "MIT"
 
-  dependency "sexp_processor", "~> 4.9"
-  dependency "rake", "< 11", :developer
-  dependency "oedipus_lex", "~> 2.5", :developer
+  dependency "sexp_processor", "~> 4.16"
+  dependency "rake", [">= 10", "< 15"], :developer
+  dependency "oedipus_lex", "~> 2.6", :developer
+
+  # NOTE: Ryan!!! Stop trying to fix this dependency! Isolate just
+  # can't handle having a faux-gem half-installed! Stop! Just `gem
+  # install racc` and move on. Revisit this ONLY once racc-compiler
+  # gets split out.
+
+  dependency "racc", "~> 1.5", :developer
 
   require_ruby_version [">= 2.1", "< 4"]
 
   if plugin? :perforce then     # generated files
-    V2.each do |n|
+    VERS.each do |n|
       self.perforce_ignore << "lib/ruby#{n}_parser.rb"
     end
 
-    V2.each do |n|
+    VERS.each do |n|
       self.perforce_ignore << "lib/ruby#{n}_parser.y"
     end
 
@@ -46,8 +58,44 @@ Hoe.spec "ruby_parser" do
   end
 end
 
+def maybe_add_to_top path, string
+  file = File.read path
+
+  return if file.start_with? string
+
+  warn "Altering top of #{path}"
+  tmp_path = "#{path}.tmp"
+  File.open(tmp_path, "w") do |f|
+    f.puts string
+    f.puts
+
+    f.write file
+    # TODO: make this deal with encoding comments properly?
+  end
+  File.rename tmp_path, path
+end
+
+def unifdef?
+  @unifdef ||= system("which unifdef") or abort <<~EOM
+    unifdef not found!
+
+    Please install 'unifdef' package on your system or `rake generate` on a mac.
+  EOM
+end
+
 V2.each do |n|
   file "lib/ruby#{n}_parser.y" => "lib/ruby_parser.yy" do |t|
+    unifdef?
+    cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
+    sh cmd
+  end
+
+  file "lib/ruby#{n}_parser.rb" => "lib/ruby#{n}_parser.y"
+end
+
+V3.each do |n|
+  file "lib/ruby#{n}_parser.y" => "lib/ruby3_parser.yy" do |t|
+    unifdef?
     cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
     sh cmd
   end
@@ -57,6 +105,12 @@ end
 
 file "lib/ruby_lexer.rex.rb" => "lib/ruby_lexer.rex"
 
+task :parser do |t|
+  t.prerequisite_tasks.grep(Rake::FileTask).select(&:already_invoked).each do |f|
+    maybe_add_to_top f.name, "# frozen_string_literal: true"
+  end
+end
+
 task :generate => [:lexer, :parser]
 
 task :clean do
@@ -65,6 +119,7 @@ task :clean do
         Dir["coverage.info"] +
         Dir["coverage"] +
         Dir["lib/ruby2*_parser.y"] +
+        Dir["lib/ruby3*_parser.y"] +
         Dir["lib/*.output"])
 end
 
@@ -92,7 +147,7 @@ end
 
 def dl v
   dir = v[/^\d+\.\d+/]
-  url = "https://cache.ruby-lang.org/pub/ruby/#{dir}/ruby-#{v}.tar.bz2"
+  url = "https://cache.ruby-lang.org/pub/ruby/#{dir}/ruby-#{v}.tar.xz"
   path = File.basename url
   unless File.exist? path then
     system "curl -O #{url}"
@@ -104,7 +159,7 @@ def ruby_parse version
   rp_txt    = "rp#{v}.txt"
   mri_txt   = "mri#{v}.txt"
   parse_y   = "parse#{v}.y"
-  tarball   = "ruby-#{version}.tar.bz2"
+  tarball   = "ruby-#{version}.tar.xz"
   ruby_dir  = "ruby-#{version}"
   diff      = "diff#{v}.diff"
   rp_out    = "lib/ruby#{v}_parser.output"
@@ -124,15 +179,18 @@ def ruby_parse version
     end
   end
 
+  desc "fetch all tarballs"
+  task :fetch => c_tarball
+
   file c_parse_y => c_tarball do
     in_compare do
       extract_glob = case version
-                     when /2\.7/
+                     when /2\.7|3\.0/
                        "{id.h,parse.y,tool/{id2token.rb,lib/vpath.rb}}"
                      else
                        "{id.h,parse.y,tool/{id2token.rb,vpath.rb}}"
                      end
-      system "tar yxf #{tarball} #{ruby_dir}/#{extract_glob}"
+      system "tar Jxf #{tarball} #{ruby_dir}/#{extract_glob}"
 
       Dir.chdir ruby_dir do
         if File.exist? "tool/id2token.rb" then
@@ -141,15 +199,20 @@ def ruby_parse version
           sh "expand parse.y > ../#{parse_y}"
         end
 
-        ruby "-pi", "-e", 'gsub(/^%define\s+api\.pure/, "%pure-parser")', "../#{parse_y}"
+        ruby "-pi", "-e", 'gsub(/^%pure-parser/, "%define api.pure")', "../#{parse_y}"
       end
       sh "rm -rf #{ruby_dir}"
     end
   end
 
+  bison = Dir["/opt/homebrew/opt/bison/bin/bison",
+              "/usr/local/opt/bison/bin/bison",
+              `which bison`.chomp,
+             ].first
+
   file c_mri_txt => [c_parse_y, normalize] do
     in_compare do
-      sh "bison -r all #{parse_y}"
+      sh "#{bison} -r all #{parse_y}"
       sh "./normalize.rb parse#{v}.output > #{mri_txt}"
       rm ["parse#{v}.output", "parse#{v}.tab.c"]
     end
@@ -190,17 +253,50 @@ def ruby_parse version
   end
 end
 
+task :versions do
+  require "open-uri"
+  require "net/http" # avoid require issues in threads
+  require "net/https"
+
+  versions = %w[ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.0 ]
+
+  base_url = "https://cache.ruby-lang.org/pub/ruby"
+
+  class Array
+    def human_sort
+      sort_by { |item| item.to_s.split(/(\d+)/).map { |e| [e.to_i, e] } }
+    end
+  end
+
+  versions = versions.map { |ver|
+    Thread.new {
+      URI
+        .parse("#{base_url}/#{ver}/")
+        .read
+        .scan(/ruby-\d+\.\d+\.\d+[-\w.]*?.tar.gz/)
+        .reject { |s| s =~ /-(?:rc|preview)\d/ }
+        .human_sort
+        .last
+        .delete_prefix("ruby-")
+        .delete_suffix ".tar.gz"
+    }
+  }.map(&:value).sort
+
+  puts versions.map { |v| "ruby_parse %p" % [v] }
+end
+
 ruby_parse "2.0.0-p648"
-ruby_parse "2.1.9"
-ruby_parse "2.2.9"
+ruby_parse "2.1.10"
+ruby_parse "2.2.10"
 ruby_parse "2.3.8"
-ruby_parse "2.4.9"
-ruby_parse "2.5.8"
-ruby_parse "2.6.6"
-ruby_parse "2.7.1"
+ruby_parse "2.4.10"
+ruby_parse "2.5.9"
+ruby_parse "2.6.8"
+ruby_parse "2.7.4"
+ruby_parse "3.0.2"
 
 task :debug => :isolate do
-  ENV["V"] ||= V2.last
+  ENV["V"] ||= VERS.last
   Rake.application[:parser].invoke # this way we can have DEBUG set
   Rake.application[:lexer].invoke # this way we can have DEBUG set
 
@@ -215,7 +311,7 @@ task :debug => :isolate do
   time = (ENV["RP_TIMEOUT"] || 10).to_i
 
   n = ENV["BUG"]
-  file = (n && "bug#{n}.rb") || ENV["F"] || ENV["FILE"] || "bug.rb"
+  file = (n && "bug#{n}.rb") || ENV["F"] || ENV["FILE"] || "debug.rb"
   ruby = ENV["R"] || ENV["RUBY"]
 
   if ruby then
@@ -238,19 +334,22 @@ task :debug => :isolate do
 end
 
 task :debug3 do
-  file    = ENV["F"] || "bug.rb"
-  verbose = ENV["V"] ? "-v" : ""
+  file    = ENV["F"] || "debug.rb"
+  version = ENV["V"] || ""
+  verbose = ENV["VERBOSE"] ? "-v" : ""
   munge    = "./tools/munge.rb #{verbose}"
 
   abort "Need a file to parse, via: F=path.rb" unless file
 
   ENV.delete "V"
 
-  sh "ruby -v"
-  sh "ruby -y #{file} 2>&1 | #{munge} > tmp/ruby"
-  sh "./tools/ripper.rb -d #{file} | #{munge} > tmp/rip"
+  ruby = "ruby#{version}"
+
+  sh "#{ruby} -v"
+  sh "#{ruby} -y #{file} 2>&1 | #{munge} > tmp/ruby"
+  sh "#{ruby} ./tools/ripper.rb -d #{file} | #{munge} > tmp/rip"
   sh "rake debug F=#{file} DEBUG=1 2>&1 | #{munge} > tmp/rp"
-  sh "diff -U 999 -d tmp/{rip,rp}"
+  sh "diff -U 999 -d tmp/{ruby,rp}"
 end
 
 task :cmp do
@@ -262,16 +361,25 @@ task :cmp3 do
 end
 
 task :extract => :isolate do
-  ENV["V"] ||= V2.last
+  ENV["V"] ||= VERS.last
   Rake.application[:parser].invoke # this way we can have DEBUG set
 
-  file = ENV["F"] || ENV["FILE"]
+  file = ENV["F"] || ENV["FILE"] || abort("Need to provide F=<path>")
 
   ruby "-Ilib", "bin/ruby_parse_extract_error", file
 end
 
+task :parse => :isolate do
+  ENV["V"] ||= VERS.last
+  Rake.application[:parser].invoke # this way we can have DEBUG set
+
+  file = ENV["F"] || ENV["FILE"] || abort("Need to provide F=<path>")
+
+  ruby "-Ilib", "bin/ruby_parse", file
+end
+
 task :bugs do
-  sh "for f in bug*.rb ; do #{Gem.ruby} -S rake debug F=$f && rm $f ; done"
+  sh "for f in bug*.rb bad*.rb ; do #{Gem.ruby} -S rake debug F=$f && rm $f ; done"
 end
 
 # vim: syntax=Ruby

data/bin/ruby_parse_extract_error

@@ -21,7 +21,7 @@ class RubyParser
     src = ss.string
     pre_error = src[0...ss.pos]
 
-    defs = pre_error.grep(/^ *(?:def|it)/)
+    defs = pre_error.lines.grep(/^ *(?:def|it)/)
 
     raise "can't figure out where the bad code starts" unless defs.last
 

data/compare/normalize.rb

@@ -84,6 +84,7 @@ def munge s
 
              "' '",             "tSPACE", # needs to be later to avoid bad hits
 
+             "%empty",          "none", # newer bison
              "/* empty */",     "none",
              /^\s*$/,           "none",
 
@@ -140,6 +141,7 @@ def munge s
              '"do for block"',     "kDO_BLOCK",
              '"do for condition"', "kDO_COND",
              '"do for lambda"',    "kDO_LAMBDA",
+             "tLABEL",             "kLABEL",
 
              # UGH
              "k_LINE__",       "k__LINE__",
@@ -155,7 +157,10 @@ def munge s
              /\"(\w+) \(?modifier\)?\"/, proc { |x| "k#{$1.upcase}_MOD" },
              /\"(\w+)\"/,                proc { |x| "k#{$1.upcase}" },
 
-             /@(\d+)(\s+|$)/,       "",
+             /\$?@(\d+)(\s+|$)/,    "", # newer bison
+
+             # TODO: remove for 3.0 work:
+             "lex_ctxt ", "" # 3.0 production that's mostly noise right now
             ]
 
   renames.each_slice(2) do |(a, b)|
@@ -174,7 +179,7 @@ ARGF.each_line do |line|
 
   case line.strip
   when /^$/ then
-  when /^(\d+) (\$?\w+): (.*)/ then    # yacc
+  when /^(\d+) (\$?[@\w]+): (.*)/ then    # yacc
     rule = $2
     order << rule unless rules.has_key? rule
     rules[rule] << munge($3)
@@ -199,7 +204,7 @@ ARGF.each_line do |line|
   when /^\cL/ then                     # byacc
     break
   else
-    warn "unparsed: #{$.}: #{line.chomp}"
+    warn "unparsed: #{$.}: #{line.strip.inspect}"
   end
 end
 

data/debugging.md

@@ -55,3 +55,136 @@ From there? Good luck. I'm currently trying to backtrack from rule
 reductions to state change differences. I'd like to figure out a way
 to go from this sort of diff to a reasonable test that checks state
 changes but I don't have that set up at this point.
+
+## Adding New Grammar Productions
+
+Ruby adds stuff to the parser ALL THE TIME. It's actually hard to keep
+up with, but I've added some tools and shown what a typical workflow
+looks like. Let's say you want to add ruby 2.7's "beginless range" (eg
+`..42`).
+
+Whenever there's a language feature missing, I start with comparing
+the parse trees between MRI and RP:
+
+### Structural Comparing
+
+There's a bunch of rake tasks `compare27`, `compare26`, etc that try
+to normalize and diff MRI's parse.y parse tree (just the structure of
+the tree in yacc) to ruby\_parser's parse tree (racc). It's the first
+thing I do when I'm adding a new version. Stub out all the version
+differences, and then start to diff the structure and move
+ruby\_parser towards the new changes.
+
+Some differences are just gonna be there... but here's an example of a
+real diff between MRI 2.7 and ruby_parser as of today:
+
+```diff
+     arg tDOT3 arg
+     arg tDOT2
+     arg tDOT3
+-    tBDOT2 arg
+-    tBDOT3 arg
+     arg tPLUS arg
+     arg tMINUS arg
+     arg tSTAR2 arg
+```
+
+This is a new language feature that ruby_parser doesn't handle yet.
+It's in MRI (the left hand side of the diff) but not ruby\_parser (the
+right hand side) so it is a `-` or missing line.
+
+Some other diffs will have both `+` and `-` lines. That usually
+happens when MRI has been refactoring the grammar. Sometimes I choose
+to adapt those refactorings and sometimes it starts to get too
+difficult to maintain multiple versions of ruby parsing in a single
+file.
+
+But! This structural comparing is always a place you should look when
+ruby_parser is failing to parse something. Maybe it just hasn't been
+implemented yet and the easiest place to look is the diff.
+
+### Starting Test First
+
+The next thing I do is to add a parser test to cover that feature. I
+usually start with the parser and work backwards towards the lexer as
+needed, as I find it structures things properly and keeps things goal
+oriented.
+
+So, make a new parser test, usually in the versioned section of the
+parser tests.
+
+```
+  def test_beginless2
+    rb = "..10\n; ..a\n; c"
+    pt = s(:block,
+           s(:dot2, nil, s(:lit, 0).line(1)).line(1),
+           s(:dot2, nil, s(:call, nil, :a).line(2)).line(2),
+           s(:call, nil, :c).line(3)).line(1)
+
+    assert_parse_line rb, pt, 1
+
+    flunk "not done yet"
+  end
+```
+
+(In this case copied and modified the tests for open ranges from 2.6)
+and run it to get my first error:
+
+```
+% rake N=/beginless/
+
+...
+
+E
+
+Finished in 0.021814s, 45.8421 runs/s, 0.0000 assertions/s.
+
+  1) Error:
+TestRubyParserV27#test_whatevs:
+Racc::ParseError: (string):1 :: parse error on value ".." (tDOT2)
+    GEMS/2.7.0/gems/racc-1.5.0/lib/racc/parser.rb:538:in `on_error'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1304:in `on_error'
+    (eval):3:in `_racc_do_parse_c'
+    (eval):3:in `do_parse'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1329:in `block in process'
+    RUBY/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `block in catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:110:in `timeout'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1317:in `process'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4198:in `assert_parse'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4221:in `assert_parse_line'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4451:in `test_whatevs'
+```
+
+For starters, we know the missing production is for `tBDOT2 arg`. It
+is currently blowing up because it is getting `tDOT2` and simply
+doesn't know what to do with it, so it raises the error. As the diff
+suggests, that's the wrong token to begin with, so it is probably time
+to also create a lexer test:
+
+```
+def test_yylex_bdot2
+  assert_lex3("..42",
+              s(:dot2, nil, s(:lit, 42)),
+
+              :tBDOT2,   "..", EXPR_BEG,
+              :tINTEGER, "42", EXPR_NUM)
+
+  flunk "not done yet"
+end
+```
+
+This one is mostly speculative at this point. It says "if we're lexing
+this string, we should get this sexp if we fully parse it, and the
+lexical stream should look like this"... That last bit is mostly made
+up at this point. Sometimes I don't know exactly what expression state
+things should be in until I start really digging in.
+
+At this point, I have 2 failing tests that are directing me in the
+right direction. It's now a matter of digging through
+`compare/parse26.y` to see how the lexer differs and implementing
+it...
+
+But this is a good start to the doco for now. I'll add more later.