ruby_parser 3.17.0 → 3.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ff6be95e278654e341f5279fed2fd7f0c9a96d93b2fd23ba1ff4b181d593be18
-  data.tar.gz: ab91b782eb2e77cdd855fa68f4699614b6160ebcca623dd8be25719b410b4206
+  metadata.gz: 36780d9d3244dd62d13430987076d5e81ae2e536d6d2bfd259f8a612da3d94cc
+  data.tar.gz: bec4b32e7f7a8d9ae8e3202f30230f351a2fedc6e2ac4e984260486dbb7529c6
 SHA512:
-  metadata.gz: a469da9dadd1eeb35a48dbb34548e70feed8ca83b2d27e41c6bf940cf9dd779622fbddcc4b3c50534f46c6de42f1f085754739d76051413866ee6557fe84050d
-  data.tar.gz: d182a507b167a6c9af4a7a48e748f55066462888b646a38a08ab12c4365a57645c5216baa2d64c99c8d7b6ebcb4e4b22219a9da9a7d0bd945c00fc21104d7343
+  metadata.gz: f28d02d2b14687e365bab3a353348b93a9df993be2d1afd3f2783b5b97ca016a6ca2f834ef61ebb4a4eae3decc38e1351349679f951f901bef09c25f23d44322
+  data.tar.gz: 276ecce4db1f72ed2ce0d276679e65419225a46b885d0050aa7ba6382b45033ccd24b5006a0d382f0aecdbb6c5a5fd93e3e826adeafccc3c47ee051b76772eee

data/History.rdoc

@@ -1,3 +1,79 @@
+=== 3.18.0 / 2021-10-27
+
+Holy crap... 58 commits! 2.7 and 3.0 are feature complete. Strings
+& heredocs have been rewritten.
+
+* 9 major enhancements:
+
+  * !!! Rewrote lexer (and friends) for strings, heredocs, and %*[] constructs.
+  * Massive overhaul on line numbers.
+  * Freeze input! Finally!!! No more modifying the input string for heredocs.
+  * Overhauled RPStringScanner. Removed OLD compatibility methods!
+  * Removed Sexp methods: value, to_sym, add, add_all, node_type, values.
+    * value moved to sexp_processor.
+  * Removed String#grep monkey-patch.
+  * Removed String#lineno monkey-patch.
+  * Removed string_to_pos, charpos, etc hacks for ancient ruby versions.
+  * Removed unread_many... NO! NO EDITING THE INPUT STRING!
+
+* 31 minor enhancements:
+
+  * 2.7/3.0: many more pattern edge cases
+  * 2.7: Added `mlhs = rhs rescue expr`
+  * 2.7: refactored destructured args (`|(k,v)|`) and unfactored(?!) case_body/args.
+  * 3.0: excessed_comma
+  * 3.0: finished most everything: endless methods, patterns, etc.
+  * 3.0: refactored / added new pattern changes
+  * Added RubyLexer#in_heredoc? (ie, is there old_ss ?)
+  * Added RubyLexer#old_ss and old_lineno and removed much of SSStack(ish).
+  * Added Symbol#end_with? when necessary
+  * Added TALLY and DEBUG options for ss.getch and ss.scan
+  * Added ignore_body_comments to make parser productions more clear.
+  * Added support for no_kwarg (eg `def f(**nil)`).
+  * Added support for no_kwarg in blocks (eg `f { |**nil| }`).
+  * Augmented generated parser files to have frozen_string_literal comments and fixed tests.
+  * Broke out 3.0 parser into its own to ease development.
+  * Bumped dependencies on sexp_processor and oedipus_lex.
+  * Clean generated 3.x files.
+  * Extracted all string scanner methods to their own module.
+  * Fixed some precedence decls.
+  * Implemented most of pattern matching for 2.7+.
+  * Improve lex_state= to report location in verbose debug mode.
+  * Made it easier to debug with a particular version of ruby via rake.
+  * Make sure ripper uses the same version of ruby we specified.
+  * Moved all string/heredoc/etc code to ruby_lexer_strings.rb
+  * Remove warning from newer bisons.
+  * Sprinkled in some frozen_string_literal, but mostly helped by oedipus bump.
+  * Switch to comparing against ruby binary since ripper is buggy.
+  * bugs task should try both bug*.rb and bad*.rb.
+  * endless methods
+  * f_any_kwrest refactoring.
+  * refactored defn/defs
+
+* 15 bug fixes:
+
+  * Cleaned a bunch of old hacks. Initializing RubyLexer w/ Parser is cleaner now.
+  * Corrected some lex_state errors in process_token_keyword.
+  * Fixed ancient ruby2 change (use #lines) in ruby_parse_extract_error.
+  * Fixed bug where else without rescue only raises on 2.6+
+  * Fixed caller for getch and scan when DEBUG=1
+  * Fixed comments in the middle of message cascades.
+  * Fixed differences w/ symbol productions against ruby 2.7.
+  * Fixed dsym to use string_contents production.
+  * Fixed error in bdot2/3 in some edge cases. Fixed p_alt line.
+  * Fixed heredoc dedenting in the presence of empty lines. (mvz)
+  * Fixed some leading whitespace / comment processing
+  * Fixed up how class/module/defn/defs comments were collected.
+  * Overhauled ripper.rb to deal with buggy ripper w/ yydebug.
+  * Removed dsym from literal.
+  * Removed tUBANG lexeme but kept it distinct as a method name (eg: `def !@`).
+
+=== 3.17.0 / 2021-08-03
+
+* 1 minor enhancement:
+
+  * Added support for arg forwarding (eg `def f(...); m(...); end`) (presidentbeef)
+
 === 3.16.0 / 2021-05-15
 
 * 1 major enhancement:

data/Manifest.txt

@@ -7,6 +7,7 @@ bin/ruby_parse
 bin/ruby_parse_extract_error
 compare/normalize.rb
 debugging.md
+gauntlet.md
 lib/.document
 lib/rp_extensions.rb
 lib/rp_stringscanner.rb
@@ -28,9 +29,11 @@ lib/ruby27_parser.rb
 lib/ruby27_parser.y
 lib/ruby30_parser.rb
 lib/ruby30_parser.y
+lib/ruby3_parser.yy
 lib/ruby_lexer.rb
 lib/ruby_lexer.rex
 lib/ruby_lexer.rex.rb
+lib/ruby_lexer_strings.rb
 lib/ruby_parser.rb
 lib/ruby_parser.yy
 lib/ruby_parser_extras.rb

data/README.rdoc

@@ -32,6 +32,7 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
 * 1.8 parser is at 99.9739% accuracy, 3.651 sigma
 * 1.9 parser is at 99.9940% accuracy, 4.013 sigma
 * 2.0 parser is at 99.9939% accuracy, 4.008 sigma
+* 2.6 parser is at 99.9972% accuracy, 4.191 sigma
 
 == FEATURES/PROBLEMS:
 

data/Rakefile

@@ -13,17 +13,22 @@ Hoe.add_include_dirs "../../sexp_processor/dev/lib"
 Hoe.add_include_dirs "../../minitest/dev/lib"
 Hoe.add_include_dirs "../../oedipus_lex/dev/lib"
 
-V2   = %w[20 21 22 23 24 25 26 27 30]
-V2.replace [V2.last] if ENV["FAST"] # HACK
+V2   = %w[20 21 22 23 24 25 26 27]
+V3   = %w[30]
+
+VERS = V2 + V3
+
+ENV["FAST"] = VERS.last if ENV["FAST"] && !VERS.include?(ENV["FAST"])
+VERS.replace [ENV["FAST"]] if ENV["FAST"]
 
 Hoe.spec "ruby_parser" do
   developer "Ryan Davis", "ryand-ruby@zenspider.com"
 
   license "MIT"
 
-  dependency "sexp_processor", ["~> 4.15", ">= 4.15.1"]
+  dependency "sexp_processor", "~> 4.16"
   dependency "rake", [">= 10", "< 15"], :developer
-  dependency "oedipus_lex", "~> 2.5", :developer
+  dependency "oedipus_lex", "~> 2.6", :developer
 
   # NOTE: Ryan!!! Stop trying to fix this dependency! Isolate just
   # can't handle having a faux-gem half-installed! Stop! Just `gem
@@ -35,11 +40,11 @@ Hoe.spec "ruby_parser" do
   require_ruby_version [">= 2.1", "< 4"]
 
   if plugin? :perforce then     # generated files
-    V2.each do |n|
+    VERS.each do |n|
       self.perforce_ignore << "lib/ruby#{n}_parser.rb"
     end
 
-    V2.each do |n|
+    VERS.each do |n|
       self.perforce_ignore << "lib/ruby#{n}_parser.y"
     end
 
@@ -53,6 +58,23 @@ Hoe.spec "ruby_parser" do
   end
 end
 
+def maybe_add_to_top path, string
+  file = File.read path
+
+  return if file.start_with? string
+
+  warn "Altering top of #{path}"
+  tmp_path = "#{path}.tmp"
+  File.open(tmp_path, "w") do |f|
+    f.puts string
+    f.puts
+
+    f.write file
+    # TODO: make this deal with encoding comments properly?
+  end
+  File.rename tmp_path, path
+end
+
 V2.each do |n|
   file "lib/ruby#{n}_parser.y" => "lib/ruby_parser.yy" do |t|
     cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
@@ -62,8 +84,23 @@ V2.each do |n|
   file "lib/ruby#{n}_parser.rb" => "lib/ruby#{n}_parser.y"
 end
 
+V3.each do |n|
+  file "lib/ruby#{n}_parser.y" => "lib/ruby3_parser.yy" do |t|
+    cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
+    sh cmd
+  end
+
+  file "lib/ruby#{n}_parser.rb" => "lib/ruby#{n}_parser.y"
+end
+
 file "lib/ruby_lexer.rex.rb" => "lib/ruby_lexer.rex"
 
+task :parser do |t|
+  t.prerequisite_tasks.grep(Rake::FileTask).select(&:already_invoked).each do |f|
+    maybe_add_to_top f.name, "# frozen_string_literal: true"
+  end
+end
+
 task :generate => [:lexer, :parser]
 
 task :clean do
@@ -72,6 +109,7 @@ task :clean do
         Dir["coverage.info"] +
         Dir["coverage"] +
         Dir["lib/ruby2*_parser.y"] +
+        Dir["lib/ruby3*_parser.y"] +
         Dir["lib/*.output"])
 end
 
@@ -151,7 +189,7 @@ def ruby_parse version
           sh "expand parse.y > ../#{parse_y}"
         end
 
-        ruby "-pi", "-e", 'gsub(/^%define\s+api\.pure/, "%pure-parser")', "../#{parse_y}"
+        ruby "-pi", "-e", 'gsub(/^%pure-parser/, "%define api.pure")', "../#{parse_y}"
       end
       sh "rm -rf #{ruby_dir}"
     end
@@ -248,7 +286,7 @@ ruby_parse "2.7.4"
 ruby_parse "3.0.2"
 
 task :debug => :isolate do
-  ENV["V"] ||= V2.last
+  ENV["V"] ||= VERS.last
   Rake.application[:parser].invoke # this way we can have DEBUG set
   Rake.application[:lexer].invoke # this way we can have DEBUG set
 
@@ -263,7 +301,7 @@ task :debug => :isolate do
   time = (ENV["RP_TIMEOUT"] || 10).to_i
 
   n = ENV["BUG"]
-  file = (n && "bug#{n}.rb") || ENV["F"] || ENV["FILE"] || "bug.rb"
+  file = (n && "bug#{n}.rb") || ENV["F"] || ENV["FILE"] || "debug.rb"
   ruby = ENV["R"] || ENV["RUBY"]
 
   if ruby then
@@ -286,19 +324,22 @@ task :debug => :isolate do
 end
 
 task :debug3 do
-  file    = ENV["F"] || "bug.rb"
-  verbose = ENV["V"] ? "-v" : ""
+  file    = ENV["F"] || "debug.rb"
+  version = ENV["V"] || ""
+  verbose = ENV["VERBOSE"] ? "-v" : ""
   munge    = "./tools/munge.rb #{verbose}"
 
   abort "Need a file to parse, via: F=path.rb" unless file
 
   ENV.delete "V"
 
-  sh "ruby -v"
-  sh "ruby -y #{file} 2>&1 | #{munge} > tmp/ruby"
-  sh "./tools/ripper.rb -d #{file} | #{munge} > tmp/rip"
+  ruby = "ruby#{version}"
+
+  sh "#{ruby} -v"
+  sh "#{ruby} -y #{file} 2>&1 | #{munge} > tmp/ruby"
+  sh "#{ruby} ./tools/ripper.rb -d #{file} | #{munge} > tmp/rip"
   sh "rake debug F=#{file} DEBUG=1 2>&1 | #{munge} > tmp/rp"
-  sh "diff -U 999 -d tmp/{rip,rp}"
+  sh "diff -U 999 -d tmp/{ruby,rp}"
 end
 
 task :cmp do
@@ -310,16 +351,25 @@ task :cmp3 do
 end
 
 task :extract => :isolate do
-  ENV["V"] ||= V2.last
+  ENV["V"] ||= VERS.last
   Rake.application[:parser].invoke # this way we can have DEBUG set
 
-  file = ENV["F"] || ENV["FILE"]
+  file = ENV["F"] || ENV["FILE"] || abort("Need to provide F=<path>")
 
   ruby "-Ilib", "bin/ruby_parse_extract_error", file
 end
 
+task :parse => :isolate do
+  ENV["V"] ||= VERS.last
+  Rake.application[:parser].invoke # this way we can have DEBUG set
+
+  file = ENV["F"] || ENV["FILE"] || abort("Need to provide F=<path>")
+
+  ruby "-Ilib", "bin/ruby_parse", file
+end
+
 task :bugs do
-  sh "for f in bug*.rb ; do #{Gem.ruby} -S rake debug F=$f && rm $f ; done"
+  sh "for f in bug*.rb bad*.rb ; do #{Gem.ruby} -S rake debug F=$f && rm $f ; done"
 end
 
 # vim: syntax=Ruby

data/bin/ruby_parse_extract_error

@@ -21,7 +21,7 @@ class RubyParser
     src = ss.string
     pre_error = src[0...ss.pos]
 
-    defs = pre_error.grep(/^ *(?:def|it)/)
+    defs = pre_error.lines.grep(/^ *(?:def|it)/)
 
     raise "can't figure out where the bad code starts" unless defs.last
 

data/compare/normalize.rb

@@ -84,6 +84,7 @@ def munge s
 
              "' '",             "tSPACE", # needs to be later to avoid bad hits
 
+             "%empty",          "none", # newer bison
              "/* empty */",     "none",
              /^\s*$/,           "none",
 
@@ -140,6 +141,7 @@ def munge s
              '"do for block"',     "kDO_BLOCK",
              '"do for condition"', "kDO_COND",
              '"do for lambda"',    "kDO_LAMBDA",
+             "tLABEL",             "kLABEL",
 
              # UGH
              "k_LINE__",       "k__LINE__",
@@ -155,7 +157,10 @@ def munge s
              /\"(\w+) \(?modifier\)?\"/, proc { |x| "k#{$1.upcase}_MOD" },
              /\"(\w+)\"/,                proc { |x| "k#{$1.upcase}" },
 
-             /@(\d+)(\s+|$)/,       "",
+             /\$?@(\d+)(\s+|$)/,    "", # newer bison
+
+             # TODO: remove for 3.0 work:
+             "lex_ctxt ", "" # 3.0 production that's mostly noise right now
             ]
 
   renames.each_slice(2) do |(a, b)|

data/gauntlet.md

@@ -0,0 +1,106 @@
+# Running the Gauntlet
+
+## Maintaining a Gem Mirror
+
+I use rubygems-mirror to keep an archive of all the latest rubygems on
+an external disk. Here is the config:
+
+```
+---
+- from: https://rubygems.org
+  to: /Volumes/StuffA/gauntlet/mirror
+  parallelism: 10
+  retries: 3
+  delete: true
+  skiperror: true
+  hashdir: true
+```
+
+And I update using rake:
+
+```
+% cd ~/Work/git/rubygems/rubygems-mirror
+% git down
+% rake mirror:latest
+% /Volumes/StuffA/gauntlet/bin/cleanup.rb
+```
+
+This rather quickly updates my mirror to the latest versions of
+everything and then deletes all old versions. I then run a cleanup
+script that fixes the file dates to their publication date and deletes
+any gems that have invalid specs. This can argue with the mirror a
+bit, but it is pretty minimal (currently ~20 bad gems).
+
+## Curating an Archive of Ruby Files
+
+Next, I process the gem mirror into a much more digestable structure
+using `hash.rb` (TODO: needs a better name):
+
+```
+% cd RP
+% /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
+... waaaait ...
+% mv hashed.noindex gauntlet.$(today).noindex
+% lrztar gauntlet.$(today).noindex
+% mv gauntlet.$(today).noindex.lrz /Volumes/StuffA/gauntlet/
+```
+
+This script filters all the newer gems (TODO: WHY?), unpacks them,
+finds all the files that look like they're valid ruby, ensures they're
+valid ruby (using the current version of ruby to compile them), and
+then moves them into a SHA dir structure that looks something like
+this:
+
+```
+hashed.noindex/a/b/c/<full_file_sha>.rb
+```
+
+This removes all duplicates and puts everything in a fairly even,
+wide, flat directory layout.
+
+This process takes a very long time, even with a lot of
+parallelization. There are currently about 160k gems in the mirror.
+Unpacking, validating, SHA'ing everything is disk and CPU intensive.
+The `.noindex` extension stops spotlight from indexing the continous
+churn of files being unpacked and moved and saves time.
+
+Finally, I rename and archive it all up (currently using lrztar, but
+I'm not in love with it).
+
+### Stats
+
+```
+9696 % find gauntlet.$(today).noindex -type f | lc
+  561270
+3.5G gauntlet.2021-08-06.noindex
+239M gauntlet.2021-08-06.noindex.tar.lrz
+```
+
+So I wind up with a little over half a million unique ruby files to
+parse. It's about 3.5g but compresses very nicely down to 240m
+
+## Running the Gauntlet
+
+Assuming you're starting from scratch, unpack the archive once:
+
+```
+% lrzuntar gauntlet.$(today).noindex.lrz
+```
+
+Then, either run a single process (easier to read):
+
+```
+% ./gauntlet/bin/gauntlet.rb gauntlet/*.noindex/?
+```
+
+Or max out your machine using xargs (note the `-P 16` and choose accordingly):
+
+```
+% ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
+```
+
+In another terminal I usually monitor the progress like so:
+
+```
+% while true ; do clear; fd . -t d -t e gauntlet/*.noindex -X rmdir -p 2> /dev/null ; for D in gauntlet/*.noindex/? ; do echo -n "$D: "; fd .rb $D | wc -l ; done ; echo ; sleep 30 ; done
+```

data/lib/rp_extensions.rb

@@ -12,26 +12,24 @@ class Regexp
 end
 # :startdoc:
 
-############################################################
-# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
-
-unless "".respond_to?(:grep) then
-  class String
-    def grep re
-      lines.grep re
-    end
+class Array
+  def prepend *vals
+    self[0,0] = vals
   end
-end
+end unless [].respond_to?(:prepend)
 
-class String
-  ##
-  # This is a hack used by the lexer to sneak in line numbers at the
-  # identifier level. This should be MUCH smaller than making
-  # process_token return [value, lineno] and modifying EVERYTHING that
-  # reduces tIDENTIFIER.
+# :stopdoc:
+class Symbol
+  def end_with? o
+    self.to_s.end_with? o
+  end
+end unless :woot.respond_to?(:end_with?)
+# :startdoc:
 
-  attr_accessor :lineno
+############################################################
+# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
 
+class String
   def clean_caller
     self.sub(File.dirname(__FILE__), "./lib").sub(/:in.*/, "")
   end if $DEBUG
@@ -40,34 +38,15 @@ end
 require "sexp"
 
 class Sexp
-  attr_writer :paren
+  attr_writer :paren # TODO: retire
 
   def paren
     @paren ||= false
   end
 
-  def value
-    raise "multi item sexp" if size > 2
-    last
-  end
-
-  def to_sym
-    raise "no: #{self.inspect}.to_sym is a bug"
-    self.value.to_sym
-  end
-
-  alias :add :<<
-
-  def add_all x
-    self.concat x.sexp_body
-  end
-
   def block_pass?
     any? { |s| Sexp === s && s.sexp_type == :block_pass }
   end
-
-  alias :node_type :sexp_type
-  alias :values :sexp_body # TODO: retire
 end
 
 # END HACK

data/lib/rp_stringscanner.rb

@@ -1,64 +1,33 @@
 require "strscan"
 
 class RPStringScanner < StringScanner
-#   if ENV['TALLY'] then
-#     alias :old_getch :getch
-#     def getch
-#       warn({:getch => caller[0]}.inspect)
-#       old_getch
-#     end
-#   end
-
-  if "".respond_to? :encoding then
-    if "".respond_to? :byteslice then
-      def string_to_pos
-        string.byteslice(0, pos)
-      end
-    else
-      def string_to_pos
-        string.bytes.first(pos).pack("c*").force_encoding(string.encoding)
-      end
-    end
-
-    def charpos
-      string_to_pos.length
-    end
-  else
-    alias :charpos :pos
-
-    def string_to_pos
-      string[0..pos]
-    end
-  end
-
-  def unread_many str # TODO: remove this entirely - we should not need it
-    warn({:unread_many => caller[0]}.inspect) if ENV['TALLY']
-    begin
-      string[charpos, 0] = str
-    rescue IndexError
-      # HACK -- this is a bandaid on a dirty rag on an open festering wound
-    end
-  end
-
-  if ENV['DEBUG'] then
-    alias :old_getch :getch
+  if ENV["DEBUG"] || ENV["TALLY"] then
     def getch
-      c = self.old_getch
-      p :getch => [c, caller.first]
+      c = super
+      where = caller.drop_while { |s| s =~ /(getch|nextc).$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d getch:where
+      else
+        d getch:[c, where]
+      end
       c
     end
 
-    alias :old_scan :scan
     def scan re
-      s = old_scan re
-      where = caller[1].split(/:/).first(2).join(":")
-      d :scan => [s, where] if s
+      s = super
+      where = caller.drop_while { |x| x =~ /scan.$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d scan:[where]
+      else
+        d scan:[s, where] if s
+      end
       s
     end
-  end
 
-  def d o
-    $stderr.puts o.inspect
+    def d o
+      STDERR.puts o.inspect
+    end
   end
 end
-