ruby_parser 3.15.0 → 3.18.0

data/gauntlet.md

@@ -0,0 +1,106 @@
+# Running the Gauntlet
+
+## Maintaining a Gem Mirror
+
+I use rubygems-mirror to keep an archive of all the latest rubygems on
+an external disk. Here is the config:
+
+```
+---
+- from: https://rubygems.org
+  to: /Volumes/StuffA/gauntlet/mirror
+  parallelism: 10
+  retries: 3
+  delete: true
+  skiperror: true
+  hashdir: true
+```
+
+And I update using rake:
+
+```
+% cd ~/Work/git/rubygems/rubygems-mirror
+% git down
+% rake mirror:latest
+% /Volumes/StuffA/gauntlet/bin/cleanup.rb
+```
+
+This rather quickly updates my mirror to the latest versions of
+everything and then deletes all old versions. I then run a cleanup
+script that fixes the file dates to their publication date and deletes
+any gems that have invalid specs. This can argue with the mirror a
+bit, but it is pretty minimal (currently ~20 bad gems).
+
+## Curating an Archive of Ruby Files
+
+Next, I process the gem mirror into a much more digestable structure
+using `hash.rb` (TODO: needs a better name):
+
+```
+% cd RP
+% /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
+... waaaait ...
+% mv hashed.noindex gauntlet.$(today).noindex
+% lrztar gauntlet.$(today).noindex
+% mv gauntlet.$(today).noindex.lrz /Volumes/StuffA/gauntlet/
+```
+
+This script filters all the newer gems (TODO: WHY?), unpacks them,
+finds all the files that look like they're valid ruby, ensures they're
+valid ruby (using the current version of ruby to compile them), and
+then moves them into a SHA dir structure that looks something like
+this:
+
+```
+hashed.noindex/a/b/c/<full_file_sha>.rb
+```
+
+This removes all duplicates and puts everything in a fairly even,
+wide, flat directory layout.
+
+This process takes a very long time, even with a lot of
+parallelization. There are currently about 160k gems in the mirror.
+Unpacking, validating, SHA'ing everything is disk and CPU intensive.
+The `.noindex` extension stops spotlight from indexing the continous
+churn of files being unpacked and moved and saves time.
+
+Finally, I rename and archive it all up (currently using lrztar, but
+I'm not in love with it).
+
+### Stats
+
+```
+9696 % find gauntlet.$(today).noindex -type f | lc
+  561270
+3.5G gauntlet.2021-08-06.noindex
+239M gauntlet.2021-08-06.noindex.tar.lrz
+```
+
+So I wind up with a little over half a million unique ruby files to
+parse. It's about 3.5g but compresses very nicely down to 240m
+
+## Running the Gauntlet
+
+Assuming you're starting from scratch, unpack the archive once:
+
+```
+% lrzuntar gauntlet.$(today).noindex.lrz
+```
+
+Then, either run a single process (easier to read):
+
+```
+% ./gauntlet/bin/gauntlet.rb gauntlet/*.noindex/?
+```
+
+Or max out your machine using xargs (note the `-P 16` and choose accordingly):
+
+```
+% ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
+```
+
+In another terminal I usually monitor the progress like so:
+
+```
+% while true ; do clear; fd . -t d -t e gauntlet/*.noindex -X rmdir -p 2> /dev/null ; for D in gauntlet/*.noindex/? ; do echo -n "$D: "; fd .rb $D | wc -l ; done ; echo ; sleep 30 ; done
+```

data/lib/rp_extensions.rb

@@ -12,26 +12,24 @@ class Regexp
 end
 # :startdoc:
 
-############################################################
-# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
-
-unless "".respond_to?(:grep) then
-  class String
-    def grep re
-      lines.grep re
-    end
+class Array
+  def prepend *vals
+    self[0,0] = vals
   end
-end
+end unless [].respond_to?(:prepend)
 
-class String
-  ##
-  # This is a hack used by the lexer to sneak in line numbers at the
-  # identifier level. This should be MUCH smaller than making
-  # process_token return [value, lineno] and modifying EVERYTHING that
-  # reduces tIDENTIFIER.
+# :stopdoc:
+class Symbol
+  def end_with? o
+    self.to_s.end_with? o
+  end
+end unless :woot.respond_to?(:end_with?)
+# :startdoc:
 
-  attr_accessor :lineno
+############################################################
+# HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK HACK
 
+class String
   def clean_caller
     self.sub(File.dirname(__FILE__), "./lib").sub(/:in.*/, "")
   end if $DEBUG
@@ -40,34 +38,15 @@ end
 require "sexp"
 
 class Sexp
-  attr_writer :paren
+  attr_writer :paren # TODO: retire
 
   def paren
     @paren ||= false
   end
 
-  def value
-    raise "multi item sexp" if size > 2
-    last
-  end
-
-  def to_sym
-    raise "no: #{self.inspect}.to_sym is a bug"
-    self.value.to_sym
-  end
-
-  alias :add :<<
-
-  def add_all x
-    self.concat x.sexp_body
-  end
-
   def block_pass?
     any? { |s| Sexp === s && s.sexp_type == :block_pass }
   end
-
-  alias :node_type :sexp_type
-  alias :values :sexp_body # TODO: retire
 end
 
 # END HACK

data/lib/rp_stringscanner.rb

@@ -1,64 +1,33 @@
 require "strscan"
 
 class RPStringScanner < StringScanner
-#   if ENV['TALLY'] then
-#     alias :old_getch :getch
-#     def getch
-#       warn({:getch => caller[0]}.inspect)
-#       old_getch
-#     end
-#   end
-
-  if "".respond_to? :encoding then
-    if "".respond_to? :byteslice then
-      def string_to_pos
-        string.byteslice(0, pos)
-      end
-    else
-      def string_to_pos
-        string.bytes.first(pos).pack("c*").force_encoding(string.encoding)
-      end
-    end
-
-    def charpos
-      string_to_pos.length
-    end
-  else
-    alias :charpos :pos
-
-    def string_to_pos
-      string[0..pos]
-    end
-  end
-
-  def unread_many str # TODO: remove this entirely - we should not need it
-    warn({:unread_many => caller[0]}.inspect) if ENV['TALLY']
-    begin
-      string[charpos, 0] = str
-    rescue IndexError
-      # HACK -- this is a bandaid on a dirty rag on an open festering wound
-    end
-  end
-
-  if ENV['DEBUG'] then
-    alias :old_getch :getch
+  if ENV["DEBUG"] || ENV["TALLY"] then
     def getch
-      c = self.old_getch
-      p :getch => [c, caller.first]
+      c = super
+      where = caller.drop_while { |s| s =~ /(getch|nextc).$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d getch:where
+      else
+        d getch:[c, where]
+      end
       c
     end
 
-    alias :old_scan :scan
     def scan re
-      s = old_scan re
-      where = caller[1].split(/:/).first(2).join(":")
-      d :scan => [s, where] if s
+      s = super
+      where = caller.drop_while { |x| x =~ /scan.$/ }.first
+      where = where.split(/:/).first(2).join(":")
+      if ENV["TALLY"] then
+        d scan:[where]
+      else
+        d scan:[s, where] if s
+      end
       s
     end
-  end
 
-  def d o
-    $stderr.puts o.inspect
+    def d o
+      STDERR.puts o.inspect
+    end
   end
 end
-