ruby_parser 3.15.0 → 3.19.2

data/debugging.md

@@ -55,3 +55,136 @@ From there? Good luck. I'm currently trying to backtrack from rule
 reductions to state change differences. I'd like to figure out a way
 to go from this sort of diff to a reasonable test that checks state
 changes but I don't have that set up at this point.
+
+## Adding New Grammar Productions
+
+Ruby adds stuff to the parser ALL THE TIME. It's actually hard to keep
+up with, but I've added some tools and shown what a typical workflow
+looks like. Let's say you want to add ruby 2.7's "beginless range" (eg
+`..42`).
+
+Whenever there's a language feature missing, I start with comparing
+the parse trees between MRI and RP:
+
+### Structural Comparing
+
+There's a bunch of rake tasks `compare27`, `compare26`, etc that try
+to normalize and diff MRI's parse.y parse tree (just the structure of
+the tree in yacc) to ruby\_parser's parse tree (racc). It's the first
+thing I do when I'm adding a new version. Stub out all the version
+differences, and then start to diff the structure and move
+ruby\_parser towards the new changes.
+
+Some differences are just gonna be there... but here's an example of a
+real diff between MRI 2.7 and ruby_parser as of today:
+
+```diff
+     arg tDOT3 arg
+     arg tDOT2
+     arg tDOT3
+-    tBDOT2 arg
+-    tBDOT3 arg
+     arg tPLUS arg
+     arg tMINUS arg
+     arg tSTAR2 arg
+```
+
+This is a new language feature that ruby_parser doesn't handle yet.
+It's in MRI (the left hand side of the diff) but not ruby\_parser (the
+right hand side) so it is a `-` or missing line.
+
+Some other diffs will have both `+` and `-` lines. That usually
+happens when MRI has been refactoring the grammar. Sometimes I choose
+to adapt those refactorings and sometimes it starts to get too
+difficult to maintain multiple versions of ruby parsing in a single
+file.
+
+But! This structural comparing is always a place you should look when
+ruby_parser is failing to parse something. Maybe it just hasn't been
+implemented yet and the easiest place to look is the diff.
+
+### Starting Test First
+
+The next thing I do is to add a parser test to cover that feature. I
+usually start with the parser and work backwards towards the lexer as
+needed, as I find it structures things properly and keeps things goal
+oriented.
+
+So, make a new parser test, usually in the versioned section of the
+parser tests.
+
+```
+  def test_beginless2
+    rb = "..10\n; ..a\n; c"
+    pt = s(:block,
+           s(:dot2, nil, s(:lit, 0).line(1)).line(1),
+           s(:dot2, nil, s(:call, nil, :a).line(2)).line(2),
+           s(:call, nil, :c).line(3)).line(1)
+
+    assert_parse_line rb, pt, 1
+
+    flunk "not done yet"
+  end
+```
+
+(In this case copied and modified the tests for open ranges from 2.6)
+and run it to get my first error:
+
+```
+% rake N=/beginless/
+
+...
+
+E
+
+Finished in 0.021814s, 45.8421 runs/s, 0.0000 assertions/s.
+
+  1) Error:
+TestRubyParserV27#test_whatevs:
+Racc::ParseError: (string):1 :: parse error on value ".." (tDOT2)
+    GEMS/2.7.0/gems/racc-1.5.0/lib/racc/parser.rb:538:in `on_error'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1304:in `on_error'
+    (eval):3:in `_racc_do_parse_c'
+    (eval):3:in `do_parse'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1329:in `block in process'
+    RUBY/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `block in catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:33:in `catch'
+    RUBY/lib/ruby/2.7.0/timeout.rb:110:in `timeout'
+    WORK/ruby_parser/dev/lib/ruby_parser_extras.rb:1317:in `process'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4198:in `assert_parse'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4221:in `assert_parse_line'
+    WORK/ruby_parser/dev/test/test_ruby_parser.rb:4451:in `test_whatevs'
+```
+
+For starters, we know the missing production is for `tBDOT2 arg`. It
+is currently blowing up because it is getting `tDOT2` and simply
+doesn't know what to do with it, so it raises the error. As the diff
+suggests, that's the wrong token to begin with, so it is probably time
+to also create a lexer test:
+
+```
+def test_yylex_bdot2
+  assert_lex3("..42",
+              s(:dot2, nil, s(:lit, 42)),
+
+              :tBDOT2,   "..", EXPR_BEG,
+              :tINTEGER, "42", EXPR_NUM)
+
+  flunk "not done yet"
+end
+```
+
+This one is mostly speculative at this point. It says "if we're lexing
+this string, we should get this sexp if we fully parse it, and the
+lexical stream should look like this"... That last bit is mostly made
+up at this point. Sometimes I don't know exactly what expression state
+things should be in until I start really digging in.
+
+At this point, I have 2 failing tests that are directing me in the
+right direction. It's now a matter of digging through
+`compare/parse26.y` to see how the lexer differs and implementing
+it...
+
+But this is a good start to the doco for now. I'll add more later.

data/gauntlet.md

@@ -0,0 +1,107 @@
+# Running the Gauntlet
+
+## Maintaining a Gem Mirror
+
+I use rubygems-mirror to keep an archive of all the latest rubygems on
+an external disk. Here is the config:
+
+```
+---
+- from: https://rubygems.org
+  to: /Volumes/StuffA/gauntlet/mirror
+  parallelism: 10
+  retries: 3
+  delete: true
+  skiperror: true
+  hashdir: true
+```
+
+And I update using rake:
+
+```
+% cd GIT/rubygems/rubygems-mirror
+% git down
+% rake mirror:latest
+% /Volumes/StuffA/gauntlet/bin/cleanup.rb -y -v
+```
+
+This rather quickly updates my mirror to the latest versions of
+everything and then deletes all old versions. I then run a cleanup
+script that fixes the file dates to their publication date and deletes
+any gems that have invalid specs. This can argue with the mirror a
+bit, but it is pretty minimal (currently ~20 bad gems).
+
+## Curating an Archive of Ruby Files
+
+Next, I process the gem mirror into a much more digestable structure
+using `unpack_gems.rb`.
+
+```
+% cd RP/gauntlet
+% time caffeinate /Volumes/StuffA/gauntlet/bin/unpack_gems.rb -v [-a] ; say done
+... waaaait ...
+% DIR=gauntlet.$(today).(all|new).noindex
+% mv hashed.noindex $DIR
+% tar vc -T <(fd -tf . $DIR | sort) | zstd -5 -T0 --long > archives/$DIR.tar.zst ; say done
+% ./bin/sync.sh
+```
+
+This script filters all the newer (< 1 year old) gems (unless `-a` is
+used), unpacks them, finds all the files that look like they're valid
+ruby, ensures they're valid ruby (using the current version of ruby to
+compile them), and then moves them into a SHA dir structure that looks
+something like this:
+
+```
+hashed.noindex/a/b/c/<full_file_sha>.rb
+```
+
+This removes all duplicates and puts everything in a fairly even,
+wide, flat directory layout.
+
+This process takes a very long time, even with a lot of
+parallelization. There are currently about 160k gems in the mirror.
+Unpacking, validating, SHA'ing everything is disk and CPU intensive.
+The `.noindex` extension stops spotlight from indexing the continous
+churn of files being unpacked and moved and saves time.
+
+Finally, I rename and archive it all up (currently using zstd to
+compress).
+
+### Stats
+
+```
+9696 % find gauntlet.$(today).noindex -type f | lc
+  561270
+3.5G gauntlet.2021-08-06.noindex
+239M gauntlet.2021-08-06.noindex.tar.zst
+```
+
+So I wind up with a little over half a million unique ruby files to
+parse. It's about 3.5g but compresses very nicely down to 240m
+
+## Running the Gauntlet
+
+Assuming you're starting from scratch, unpack the archive once:
+
+```
+% zstdcat gauntlet.$(today).noindex.tar.zst | tar x
+```
+
+Then, either run a single process (easier to read):
+
+```
+% ./gauntlet/bin/gauntlet.rb gauntlet/*.noindex/?
+```
+
+Or max out your machine using xargs (note the `-P 16` and choose accordingly):
+
+```
+% ls -d gauntlet/*.noindex/?/? | time xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
+```
+
+In another terminal I usually monitor the progress like so:
+
+```
+% while true ; do clear; fd . -t d -t e gauntlet/*.noindex -X rmdir -p 2> /dev/null ; for D in gauntlet/*.noindex/? ; do echo -n "$D: "; fd .rb $D | wc -l ; done ; echo ; sleep 30 ; done
+```

data/lib/rp_extensions.rb

@@ -12,26 +12,24 @@ class Regexp
 end
 # :startdoc:
 
-############################################################
-# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
-
-unless "".respond_to?(:grep) then
-  class String
-    def grep re
-      lines.grep re
-    end
+class Array
+  def prepend *vals
+    self[0,0] = vals
   end
-end
+end unless [].respond_to?(:prepend)
 
-class String
-  ##
-  # This is a hack used by the lexer to sneak in line numbers at the
-  # identifier level. This should be MUCH smaller than making
-  # process_token return [value, lineno] and modifying EVERYTHING that
-  # reduces tIDENTIFIER.
+# :stopdoc:
+class Symbol
+  def end_with? o
+    self.to_s.end_with? o
+  end
+end unless :woot.respond_to?(:end_with?)
+# :startdoc:
 
-  attr_accessor :lineno
+############################################################
+# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
 
+class String
   def clean_caller
     self.sub(File.dirname(__FILE__), "./lib").sub(/:in.*/, "")
   end if $DEBUG
@@ -40,34 +38,15 @@ end
 require "sexp"
 
 class Sexp
-  attr_writer :paren
+  attr_writer :paren # TODO: retire
 
   def paren
     @paren ||= false
   end
 
-  def value
-    raise "multi item sexp" if size > 2
-    last
-  end
-
-  def to_sym
-    raise "no: #{self.inspect}.to_sym is a bug"
-    self.value.to_sym
-  end
-
-  alias :add :<<
-
-  def add_all x
-    self.concat x.sexp_body
-  end
-
   def block_pass?
     any? { |s| Sexp === s && s.sexp_type == :block_pass }
   end
-
-  alias :node_type :sexp_type
-  alias :values :sexp_body # TODO: retire
 end
 
 # END HACK

data/lib/rp_stringscanner.rb

@@ -1,64 +1,33 @@
 require "strscan"
 
 class RPStringScanner < StringScanner
-#   if ENV['TALLY'] then
-#     alias :old_getch :getch
-#     def getch
-#       warn({:getch => caller[0]}.inspect)
-#       old_getch
-#     end
-#   end
-
-  if "".respond_to? :encoding then
-    if "".respond_to? :byteslice then
-      def string_to_pos
-        string.byteslice(0, pos)
-      end
-    else
-      def string_to_pos
-        string.bytes.first(pos).pack("c*").force_encoding(string.encoding)
-      end
-    end
-
-    def charpos
-      string_to_pos.length
-    end
-  else
-    alias :charpos :pos
-
-    def string_to_pos
-      string[0..pos]
-    end
-  end
-
-  def unread_many str # TODO: remove this entirely - we should not need it
-    warn({:unread_many => caller[0]}.inspect) if ENV['TALLY']
-    begin
-      string[charpos, 0] = str
-    rescue IndexError
-      # HACK -- this is a bandaid on a dirty rag on an open festering wound
-    end
-  end
-
-  if ENV['DEBUG'] then
-    alias :old_getch :getch
+  if ENV["DEBUG"] || ENV["TALLY"] then
     def getch
-      c = self.old_getch
-      p :getch => [c, caller.first]
+      c = super
+      where = caller.drop_while { |s| s =~ /(getch|nextc).$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d getch:where
+      else
+        d getch:[c, where]
+      end
       c
     end
 
-    alias :old_scan :scan
     def scan re
-      s = old_scan re
-      where = caller[1].split(/:/).first(2).join(":")
-      d :scan => [s, where] if s
+      s = super
+      where = caller.drop_while { |x| x =~ /scan.$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d scan:[where]
+      else
+        d scan:[s, where] if s
+      end
       s
     end
-  end
 
-  def d o
-    $stderr.puts o.inspect
+    def d o
+      STDERR.puts o.inspect
+    end
   end
 end
-