ruby_parser 3.18.0 → 3.19.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 36780d9d3244dd62d13430987076d5e81ae2e536d6d2bfd259f8a612da3d94cc
-  data.tar.gz: bec4b32e7f7a8d9ae8e3202f30230f351a2fedc6e2ac4e984260486dbb7529c6
+  metadata.gz: 2901404db3548ad6415060c14df20ae11431fff4a97420659187601a534eb256
+  data.tar.gz: 69705481ccbd6a8ffafbb81c25c40890a4683a19440196a19bbcc119f093612a
 SHA512:
-  metadata.gz: f28d02d2b14687e365bab3a353348b93a9df993be2d1afd3f2783b5b97ca016a6ca2f834ef61ebb4a4eae3decc38e1351349679f951f901bef09c25f23d44322
-  data.tar.gz: 276ecce4db1f72ed2ce0d276679e65419225a46b885d0050aa7ba6382b45033ccd24b5006a0d382f0aecdbb6c5a5fd93e3e826adeafccc3c47ee051b76772eee
+  metadata.gz: 1a5d89ba1fed1a0a49b5eb71eeb8cc6f80a396d6eaf49bec5742d62cd503916b90136ae689de4fa3a5de180544ba3ef6e8fca03e52bd8349b32c65dfa7962a28
+  data.tar.gz: c68ae86de3cbd9a1b86b6c5acbd0ecc8d809a30c42f7b50217d5038f262a6f61a3baa739332285d59158af9d3c9c3c8d8742e2eca499cf41536f14332e9f7eb9

data/History.rdoc

@@ -1,3 +1,43 @@
+=== 3.19.1 / 2022-04-05
+
+* 2 bug fixes:
+
+  * Added comments to endless defn and defs. (mvz)
+  * Fixed endless method bug handling attrset names.
+
+=== 3.19.0 / 2022-03-29
+
+* 1 major enhancement:
+
+  * Added tentative 3.1 support.
+
+* 7 minor enhancements:
+
+  * 3.1: bare RHS assoc: { y: } => s(:hash, s(:lit, :y), nil)
+  * 3.1: calls w/ unnamed block args (bare &)
+  * 3.1: endless defn/defs w/ paren-less calls (aka commands)
+  * 3.1: pattern capture to nonlocal vars, eg: ^@a, ^$b, ^@@c
+  * 3.1: pattern: ^(expr) => expr
+  * Improved steps for adding new versions.
+  * Improved steps for running gauntlets.
+
+* 2 bug fixes:
+
+  * Bumped 2.6+ cached versions for rake compare.
+  * Skip test_regexp_esc_C_slash on ruby 3.1.0 because of MRI bug.
+
+=== 3.18.1 / 2021-11-10
+
+* 1 minor enhancement:
+
+  * All parser tests are now explicitly testing line numbers at every level.
+
+* 3 bug fixes:
+
+  * Fixed endless method with noargs. (mitsuru)
+  * Fixed line numbers on some yield forms.
+  * Handle and clearly report if unifdef is missing.
+
 === 3.18.0 / 2021-10-27
 
 Holy crap... 58 commits! 2.7 and 3.0 are feature complete. Strings

data/Manifest.txt

@@ -29,6 +29,8 @@ lib/ruby27_parser.rb
 lib/ruby27_parser.y
 lib/ruby30_parser.rb
 lib/ruby30_parser.y
+lib/ruby31_parser.rb
+lib/ruby31_parser.y
 lib/ruby3_parser.yy
 lib/ruby_lexer.rb
 lib/ruby_lexer.rex

data/README.rdoc

@@ -33,6 +33,9 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
 * 1.9 parser is at 99.9940% accuracy, 4.013 sigma
 * 2.0 parser is at 99.9939% accuracy, 4.008 sigma
 * 2.6 parser is at 99.9972% accuracy, 4.191 sigma
+* 3.0 parser has a 100% parse rate.
+  * Tested against 2,672,412 unique ruby files across 167k gems.
+  * As do all the others now, basically.
 
 == FEATURES/PROBLEMS:
 
@@ -62,15 +65,14 @@ You can also use Ruby19Parser, Ruby18Parser, or RubyParser.for_current_ruby:
 
 To add a new version:
 
-* New parser should be generated from lib/ruby_parser.yy.
-* Extend lib/ruby_parser.yy with new class name.
-* Add new version number to V2 in Rakefile for rule creation.
+* New parser should be generated from lib/ruby[3]_parser.yy.
+* Extend lib/ruby[3]_parser.yy with new class name.
+* Add new version number to V2/V3 in Rakefile for rule creation.
+* Add new (full) version to `ruby_parse` section of Rakefile for rake compare
 * Require generated parser in lib/ruby_parser.rb.
 * Add empty TestRubyParserShared##Plus module and TestRubyParserV## to test/test_ruby_parser.rb.
 * Extend Manifest.txt with generated file names.
-* Extend sexp_processor's pt_testcase.rb to match version
-  * add_19tests needs to have the version added
-  * VER_RE needs to have the regexp expanded
+* Add new version number to sexp_processor's pt_testcase.rb in all_versions
 
 Until all of these are done, you won't have a clean test run.
 

data/Rakefile

@@ -14,7 +14,7 @@ Hoe.add_include_dirs "../../minitest/dev/lib"
 Hoe.add_include_dirs "../../oedipus_lex/dev/lib"
 
 V2   = %w[20 21 22 23 24 25 26 27]
-V3   = %w[30]
+V3   = %w[30 31]
 
 VERS = V2 + V3
 
@@ -75,8 +75,17 @@ def maybe_add_to_top path, string
   File.rename tmp_path, path
 end
 
+def unifdef?
+  @unifdef ||= system("which unifdef") or abort <<~EOM
+    unifdef not found!
+
+    Please install 'unifdef' package on your system or `rake generate` on a mac.
+  EOM
+end
+
 V2.each do |n|
   file "lib/ruby#{n}_parser.y" => "lib/ruby_parser.yy" do |t|
+    unifdef?
     cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
     sh cmd
   end
@@ -86,6 +95,7 @@ end
 
 V3.each do |n|
   file "lib/ruby#{n}_parser.y" => "lib/ruby3_parser.yy" do |t|
+    unifdef?
     cmd = 'unifdef -tk -DV=%s -UDEAD %s > %s || true' % [n, t.source, t.name]
     sh cmd
   end
@@ -174,8 +184,8 @@ def ruby_parse version
 
   file c_parse_y => c_tarball do
     in_compare do
-      extract_glob = case version
-                     when /2\.7|3\.0/
+      extract_glob = case
+                     when version > "2.7" then
                        "{id.h,parse.y,tool/{id2token.rb,lib/vpath.rb}}"
                      else
                        "{id.h,parse.y,tool/{id2token.rb,vpath.rb}}"
@@ -248,7 +258,7 @@ task :versions do
   require "net/http" # avoid require issues in threads
   require "net/https"
 
-  versions = %w[ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.0 ]
+  versions = VERS.map { |s| s.split(//).join "." }
 
   base_url = "https://cache.ruby-lang.org/pub/ruby"
 
@@ -281,9 +291,10 @@ ruby_parse "2.2.10"
 ruby_parse "2.3.8"
 ruby_parse "2.4.10"
 ruby_parse "2.5.9"
-ruby_parse "2.6.8"
-ruby_parse "2.7.4"
-ruby_parse "3.0.2"
+ruby_parse "2.6.9"
+ruby_parse "2.7.5"
+ruby_parse "3.0.3"
+ruby_parse "3.1.1"
 
 task :debug => :isolate do
   ENV["V"] ||= VERS.last

data/gauntlet.md

@@ -19,10 +19,10 @@ an external disk. Here is the config:
 And I update using rake:
 
 ```
-% cd ~/Work/git/rubygems/rubygems-mirror
+% cd GIT/rubygems/rubygems-mirror
 % git down
 % rake mirror:latest
-% /Volumes/StuffA/gauntlet/bin/cleanup.rb
+% /Volumes/StuffA/gauntlet/bin/cleanup.rb -y -v
 ```
 
 This rather quickly updates my mirror to the latest versions of
@@ -34,22 +34,23 @@ bit, but it is pretty minimal (currently ~20 bad gems).
 ## Curating an Archive of Ruby Files
 
 Next, I process the gem mirror into a much more digestable structure
-using `hash.rb` (TODO: needs a better name):
+using `unpack_gems.rb`.
 
 ```
-% cd RP
-% /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
+% cd RP/gauntlet
+% time caffeinate /Volumes/StuffA/gauntlet/bin/unpack_gems.rb -v [-a] ; say done
 ... waaaait ...
-% mv hashed.noindex gauntlet.$(today).noindex
-% lrztar gauntlet.$(today).noindex
-% mv gauntlet.$(today).noindex.lrz /Volumes/StuffA/gauntlet/
+% DIR=gauntlet.$(today).(all|new).noindex
+% mv hashed.noindex $DIR
+% tar vc -T <(fd -tf . $DIR | sort) | zstd -5 -T0 --long > archives/$DIR.tar.zst ; say done
+% ./bin/sync.sh
 ```
 
-This script filters all the newer gems (TODO: WHY?), unpacks them,
-finds all the files that look like they're valid ruby, ensures they're
-valid ruby (using the current version of ruby to compile them), and
-then moves them into a SHA dir structure that looks something like
-this:
+This script filters all the newer (< 1 year old) gems (unless `-a` is
+used), unpacks them, finds all the files that look like they're valid
+ruby, ensures they're valid ruby (using the current version of ruby to
+compile them), and then moves them into a SHA dir structure that looks
+something like this:
 
 ```
 hashed.noindex/a/b/c/<full_file_sha>.rb
@@ -64,8 +65,8 @@ Unpacking, validating, SHA'ing everything is disk and CPU intensive.
 The `.noindex` extension stops spotlight from indexing the continous
 churn of files being unpacked and moved and saves time.
 
-Finally, I rename and archive it all up (currently using lrztar, but
-I'm not in love with it).
+Finally, I rename and archive it all up (currently using zstd to
+compress).
 
 ### Stats
 
@@ -73,7 +74,7 @@ I'm not in love with it).
 9696 % find gauntlet.$(today).noindex -type f | lc
   561270
 3.5G gauntlet.2021-08-06.noindex
-239M gauntlet.2021-08-06.noindex.tar.lrz
+239M gauntlet.2021-08-06.noindex.tar.zst
 ```
 
 So I wind up with a little over half a million unique ruby files to
@@ -84,7 +85,7 @@ parse. It's about 3.5g but compresses very nicely down to 240m
 Assuming you're starting from scratch, unpack the archive once:
 
 ```
-% lrzuntar gauntlet.$(today).noindex.lrz
+% zstdcat gauntlet.$(today).noindex.tar.zst | tar x
 ```
 
 Then, either run a single process (easier to read):
@@ -96,7 +97,7 @@ Then, either run a single process (easier to read):
 Or max out your machine using xargs (note the `-P 16` and choose accordingly):
 
 ```
-% ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
+% ls -d gauntlet/*.noindex/?/? | time xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
 ```
 
 In another terminal I usually monitor the progress like so: