brakeman 5.2.0 → 5.2.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (72) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGES.md +17 -0
  3. data/bundle/load.rb +3 -3
  4. data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/MIT-LICENSE.txt +0 -0
  5. data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel/processor_count.rb +2 -3
  6. data/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib/parallel/version.rb +4 -0
  7. data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel.rb +84 -4
  8. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/History.rdoc +28 -0
  9. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/Manifest.txt +2 -0
  10. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/README.rdoc +8 -6
  11. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/compare/normalize.rb +0 -0
  12. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/debugging.md +0 -0
  13. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/gauntlet.md +19 -18
  14. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_extensions.rb +0 -0
  15. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_stringscanner.rb +0 -0
  16. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby20_parser.rb +10973 -0
  17. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby20_parser.y +14 -27
  18. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby21_parser.rb +10980 -0
  19. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby21_parser.y +14 -27
  20. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby22_parser.rb +11123 -0
  21. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby22_parser.y +14 -27
  22. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby23_parser.rb +11132 -0
  23. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby23_parser.y +14 -27
  24. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby24_parser.rb +11231 -0
  25. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby24_parser.y +14 -27
  26. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby25_parser.rb +11231 -0
  27. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby25_parser.y +14 -27
  28. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby26_parser.rb +11253 -0
  29. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby26_parser.y +14 -27
  30. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby27_parser.rb +12980 -0
  31. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby27_parser.y +19 -41
  32. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby30_parser.rb +13242 -0
  33. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby30_parser.y +65 -90
  34. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby31_parser.rb +13622 -0
  35. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1/lib/ruby3_parser.yy → ruby_parser-3.19.1/lib/ruby31_parser.y} +110 -105
  36. data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby3_parser.yy +3536 -0
  37. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rb +0 -0
  38. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex +0 -0
  39. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex.rb +0 -0
  40. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer_strings.rb +0 -0
  41. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.rb +2 -0
  42. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.yy +19 -41
  43. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser_extras.rb +55 -2
  44. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/munge.rb +0 -0
  45. data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/ripper.rb +0 -0
  46. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/History.rdoc +6 -0
  47. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/Manifest.txt +0 -0
  48. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/README.rdoc +0 -0
  49. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/composite_sexp_processor.rb +0 -0
  50. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/pt_testcase.rb +7 -3
  51. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp.rb +0 -0
  52. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp_matcher.rb +0 -0
  53. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp_processor.rb +1 -1
  54. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/strict_sexp.rb +0 -0
  55. data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/unique.rb +0 -0
  56. data/lib/brakeman/checks/check_sql.rb +3 -2
  57. data/lib/brakeman/checks/check_unsafe_reflection.rb +7 -2
  58. data/lib/brakeman/processors/alias_processor.rb +41 -2
  59. data/lib/brakeman/report/ignore/interactive.rb +2 -2
  60. data/lib/brakeman/version.rb +1 -1
  61. data/lib/brakeman/warning_codes.rb +2 -0
  62. metadata +54 -52
  63. data/bundle/ruby/2.7.0/gems/parallel-1.21.0/lib/parallel/version.rb +0 -4
  64. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby20_parser.rb +0 -7128
  65. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby21_parser.rb +0 -7182
  66. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby22_parser.rb +0 -7228
  67. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby23_parser.rb +0 -7237
  68. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby24_parser.rb +0 -7268
  69. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby25_parser.rb +0 -7268
  70. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby26_parser.rb +0 -7287
  71. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby27_parser.rb +0 -8517
  72. data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby30_parser.rb +0 -8751
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6cb338d80c6615c14b65edf49dd428ad57bb033e71366a2f1cc599253d28fb11
4
- data.tar.gz: f310c08560f4e5dd9d55983ba41e8ad64ac5cce07a805415ddd51658069c8fb9
3
+ metadata.gz: 2ae08a71e19d6c694a9e567fda5793a56ab174d106f44b178f7e9f69c3057814
4
+ data.tar.gz: fd24750e512d528b3fd9cecb344f4788a58e1aa8ffd6b28ff7c88bc7f034a3e8
5
5
  SHA512:
6
- metadata.gz: d79a9b1253b5bce89082ea81ed8ece812299c7df06a68248cb2e03866e32a57d685615e0b9f5f5ed36250f70d1f64705eced85dab9c4497dd529aa6416055755
7
- data.tar.gz: 89279ff60b5a728a10017c905cbc43da143eb78064d4531b913bcb545d566ffdcfad6744e718eb1ac082cec4333ea7a0b2bdf29c39b77cae236fabf9cdeb8cca
6
+ metadata.gz: e0e2d7fde5907d8158b21803876b0dd77e659ce8cae42c25e23021b46bd2c9d8c5d0dd13edff64f7fd721a8d1bca92af4a9fbbf2505e47c791557a3316c0f3d2
7
+ data.tar.gz: 38f894b42f893a6ce45db047f2d21c1529b63de0dd19e00a69475a72cbb4c2d9738f0f25edc60460d61607182d5a5c01ccbb77a6ae6eeae69ec4e88f9345a2e1
data/CHANGES.md CHANGED
@@ -1,3 +1,20 @@
1
+ # 5.2.3 - 2022-05-01
2
+
3
+ * Fix error with hash shorthand syntax
4
+ * Match order of interactive options with help message (Rory O'Kane)
5
+
6
+ # 5.2.2 - 2022-04-06
7
+
8
+ * Update `ruby_parser` for Ruby 3.1 support (Merek Skubela)
9
+ * Handle `nil` when joining values (Dan Buettner)
10
+ * Update message for unsafe reflection (Pedro Baracho)
11
+ * Add additional String methods for SQL injection check
12
+ * Respect equality in `if` conditions
13
+
14
+ # 5.2.1 - 2022-01-30
15
+
16
+ * Add warning codes for EOL software warnings
17
+
1
18
  # 5.2.0 - 2021-12-15
2
19
 
3
20
  * Initial Rails 7 support
data/bundle/load.rb CHANGED
@@ -1,7 +1,6 @@
1
1
  path = File.expand_path('../..', __FILE__)
2
- $:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib"
2
+ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib"
3
3
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/temple-0.8.2/lib"
4
- $:.unshift "#{path}/bundle/ruby/2.7.0/gems/sexp_processor-4.16.0/lib"
5
4
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/tilt-2.0.10/lib"
6
5
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/unicode-display_width-1.8.0/lib"
7
6
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/slim-4.1.0/lib"
@@ -9,7 +8,8 @@ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/highline-2.0.3/lib"
9
8
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby2ruby-2.4.4/lib"
10
9
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/terminal-table-1.8.0/lib"
11
10
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/haml-5.2.2/lib"
12
- $:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.21.0/lib"
11
+ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib"
12
+ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/sexp_processor-4.16.1/lib"
13
13
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-legacy-1.0.0/lib"
14
14
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/erubis-2.7.0/lib"
15
15
  $:.unshift "#{path}/bundle/ruby/2.7.0/gems/rexml-3.2.5/lib"
@@ -1,11 +1,10 @@
1
1
  # frozen_string_literal: true
2
- require 'etc'
3
-
4
2
  module Parallel
5
3
  # TODO: inline this method into parallel.rb and kill physical_processor_count in next major release
6
4
  module ProcessorCount
7
5
  # Number of processors seen by the OS, used for process scheduling
8
6
  def processor_count
7
+ require 'etc'
9
8
  @processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Etc.nprocessors)
10
9
  end
11
10
 
@@ -19,7 +18,7 @@ module Parallel
19
18
  when /linux/
20
19
  cores = {} # unique physical ID / core ID combinations
21
20
  phy = 0
22
- IO.read("/proc/cpuinfo").scan(/^physical id.*|^core id.*/) do |ln|
21
+ File.read("/proc/cpuinfo").scan(/^physical id.*|^core id.*/) do |ln|
23
22
  if ln.start_with?("physical")
24
23
  phy = ln[/\d+/]
25
24
  elsif ln.start_with?("core")
@@ -0,0 +1,4 @@
1
+ # frozen_string_literal: true
2
+ module Parallel
3
+ VERSION = Version = '1.22.1' # rubocop:disable Naming/ConstantName
4
+ end
@@ -264,6 +264,9 @@ module Parallel
264
264
  elsif options[:in_threads]
265
265
  method = :in_threads
266
266
  size = options[method]
267
+ elsif options[:in_ractors]
268
+ method = :in_ractors
269
+ size = options[method]
267
270
  else
268
271
  method = :in_processes
269
272
  if Process.respond_to?(:fork)
@@ -285,6 +288,8 @@ module Parallel
285
288
  work_direct(job_factory, options, &block)
286
289
  elsif method == :in_threads
287
290
  work_in_threads(job_factory, options.merge(count: size), &block)
291
+ elsif method == :in_ractors
292
+ work_in_ractors(job_factory, options.merge(count: size), &block)
288
293
  else
289
294
  work_in_processes(job_factory, options.merge(count: size), &block)
290
295
  end
@@ -382,6 +387,72 @@ module Parallel
382
387
  exception || results
383
388
  end
384
389
 
390
+ def work_in_ractors(job_factory, options)
391
+ exception = nil
392
+ results = []
393
+ results_mutex = Mutex.new # arrays are not thread-safe on jRuby
394
+
395
+ callback = options[:ractor]
396
+ if block_given? || !callback
397
+ raise ArgumentError, "pass the code you want to execute as `ractor: [ClassName, :method_name]`"
398
+ end
399
+
400
+ # build
401
+ ractors = Array.new(options.fetch(:count)) do
402
+ Ractor.new do
403
+ loop do
404
+ got = receive
405
+ (klass, method_name), item, index = got
406
+ break if index == :break
407
+ begin
408
+ Ractor.yield [nil, klass.send(method_name, item), item, index]
409
+ rescue StandardError => e
410
+ Ractor.yield [e, nil, item, index]
411
+ end
412
+ end
413
+ end
414
+ end
415
+
416
+ # start
417
+ ractors.dup.each do |ractor|
418
+ if set = job_factory.next
419
+ item, index = set
420
+ instrument_start item, index, options
421
+ ractor.send [callback, item, index]
422
+ else
423
+ ractor.send([[nil, nil], nil, :break]) # stop the ractor
424
+ ractors.delete ractor
425
+ end
426
+ end
427
+
428
+ # replace with new items
429
+ while set = job_factory.next
430
+ item_next, index_next = set
431
+ done, (exception, result, item, index) = Ractor.select(*ractors)
432
+ if exception
433
+ ractors.delete done
434
+ break
435
+ end
436
+ instrument_finish item, index, result, options
437
+ results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
438
+
439
+ instrument_start item_next, index_next, options
440
+ done.send([callback, item_next, index_next])
441
+ end
442
+
443
+ # finish
444
+ ractors.each do |ractor|
445
+ (new_exception, result, item, index) = ractor.take
446
+ exception ||= new_exception
447
+ next if new_exception
448
+ instrument_finish item, index, result, options
449
+ results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
450
+ ractor.send([[nil, nil], nil, :break]) # stop the ractor
451
+ end
452
+
453
+ exception || results
454
+ end
455
+
385
456
  def work_in_processes(job_factory, options, &blk)
386
457
  workers = create_workers(job_factory, options, &blk)
387
458
  results = []
@@ -426,6 +497,7 @@ module Parallel
426
497
  end
427
498
  end
428
499
  end
500
+
429
501
  exception || results
430
502
  end
431
503
 
@@ -521,12 +593,20 @@ module Parallel
521
593
  end
522
594
 
523
595
  def with_instrumentation(item, index, options)
524
- on_start = options[:start]
525
- on_finish = options[:finish]
526
- options[:mutex].synchronize { on_start.call(item, index) } if on_start
596
+ instrument_start(item, index, options)
527
597
  result = yield
528
- options[:mutex].synchronize { on_finish.call(item, index, result) } if on_finish
598
+ instrument_finish(item, index, result, options)
529
599
  result unless options[:preserve_results] == false
530
600
  end
601
+
602
+ def instrument_finish(item, index, result, options)
603
+ return unless on_finish = options[:finish]
604
+ options[:mutex].synchronize { on_finish.call(item, index, result) }
605
+ end
606
+
607
+ def instrument_start(item, index, options)
608
+ return unless on_start = options[:start]
609
+ options[:mutex].synchronize { on_start.call(item, index) }
610
+ end
531
611
  end
532
612
  end
@@ -1,3 +1,31 @@
1
+ === 3.19.1 / 2022-04-05
2
+
3
+ * 2 bug fixes:
4
+
5
+ * Added comments to endless defn and defs. (mvz)
6
+ * Fixed endless method bug handling attrset names.
7
+
8
+ === 3.19.0 / 2022-03-29
9
+
10
+ * 1 major enhancement:
11
+
12
+ * Added tentative 3.1 support.
13
+
14
+ * 7 minor enhancements:
15
+
16
+ * 3.1: bare RHS assoc: { y: } => s(:hash, s(:lit, :y), nil)
17
+ * 3.1: calls w/ unnamed block args (bare &)
18
+ * 3.1: endless defn/defs w/ paren-less calls (aka commands)
19
+ * 3.1: pattern capture to nonlocal vars, eg: ^@a, ^$b, ^@@c
20
+ * 3.1: pattern: ^(expr) => expr
21
+ * Improved steps for adding new versions.
22
+ * Improved steps for running gauntlets.
23
+
24
+ * 2 bug fixes:
25
+
26
+ * Bumped 2.6+ cached versions for rake compare.
27
+ * Skip test_regexp_esc_C_slash on ruby 3.1.0 because of MRI bug.
28
+
1
29
  === 3.18.1 / 2021-11-10
2
30
 
3
31
  * 1 minor enhancement:
@@ -29,6 +29,8 @@ lib/ruby27_parser.rb
29
29
  lib/ruby27_parser.y
30
30
  lib/ruby30_parser.rb
31
31
  lib/ruby30_parser.y
32
+ lib/ruby31_parser.rb
33
+ lib/ruby31_parser.y
32
34
  lib/ruby3_parser.yy
33
35
  lib/ruby_lexer.rb
34
36
  lib/ruby_lexer.rex
@@ -33,6 +33,9 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
33
33
  * 1.9 parser is at 99.9940% accuracy, 4.013 sigma
34
34
  * 2.0 parser is at 99.9939% accuracy, 4.008 sigma
35
35
  * 2.6 parser is at 99.9972% accuracy, 4.191 sigma
36
+ * 3.0 parser has a 100% parse rate.
37
+ * Tested against 2,672,412 unique ruby files across 167k gems.
38
+ * As do all the others now, basically.
36
39
 
37
40
  == FEATURES/PROBLEMS:
38
41
 
@@ -62,15 +65,14 @@ You can also use Ruby19Parser, Ruby18Parser, or RubyParser.for_current_ruby:
62
65
 
63
66
  To add a new version:
64
67
 
65
- * New parser should be generated from lib/ruby_parser.yy.
66
- * Extend lib/ruby_parser.yy with new class name.
67
- * Add new version number to V2 in Rakefile for rule creation.
68
+ * New parser should be generated from lib/ruby[3]_parser.yy.
69
+ * Extend lib/ruby[3]_parser.yy with new class name.
70
+ * Add new version number to V2/V3 in Rakefile for rule creation.
71
+ * Add new (full) version to `ruby_parse` section of Rakefile for rake compare
68
72
  * Require generated parser in lib/ruby_parser.rb.
69
73
  * Add empty TestRubyParserShared##Plus module and TestRubyParserV## to test/test_ruby_parser.rb.
70
74
  * Extend Manifest.txt with generated file names.
71
- * Extend sexp_processor's pt_testcase.rb to match version
72
- * add_19tests needs to have the version added
73
- * VER_RE needs to have the regexp expanded
75
+ * Add new version number to sexp_processor's pt_testcase.rb in all_versions
74
76
 
75
77
  Until all of these are done, you won't have a clean test run.
76
78
 
@@ -19,10 +19,10 @@ an external disk. Here is the config:
19
19
  And I update using rake:
20
20
 
21
21
  ```
22
- % cd ~/Work/git/rubygems/rubygems-mirror
22
+ % cd GIT/rubygems/rubygems-mirror
23
23
  % git down
24
24
  % rake mirror:latest
25
- % /Volumes/StuffA/gauntlet/bin/cleanup.rb
25
+ % /Volumes/StuffA/gauntlet/bin/cleanup.rb -y -v
26
26
  ```
27
27
 
28
28
  This rather quickly updates my mirror to the latest versions of
@@ -34,22 +34,23 @@ bit, but it is pretty minimal (currently ~20 bad gems).
34
34
  ## Curating an Archive of Ruby Files
35
35
 
36
36
  Next, I process the gem mirror into a much more digestable structure
37
- using `hash.rb` (TODO: needs a better name):
37
+ using `unpack_gems.rb`.
38
38
 
39
39
  ```
40
- % cd RP
41
- % /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
40
+ % cd RP/gauntlet
41
+ % time caffeinate /Volumes/StuffA/gauntlet/bin/unpack_gems.rb -v [-a] ; say done
42
42
  ... waaaait ...
43
- % mv hashed.noindex gauntlet.$(today).noindex
44
- % lrztar gauntlet.$(today).noindex
45
- % mv gauntlet.$(today).noindex.lrz /Volumes/StuffA/gauntlet/
43
+ % DIR=gauntlet.$(today).(all|new).noindex
44
+ % mv hashed.noindex $DIR
45
+ % tar vc -T <(fd -tf . $DIR | sort) | zstd -5 -T0 --long > archives/$DIR.tar.zst ; say done
46
+ % ./bin/sync.sh
46
47
  ```
47
48
 
48
- This script filters all the newer gems (TODO: WHY?), unpacks them,
49
- finds all the files that look like they're valid ruby, ensures they're
50
- valid ruby (using the current version of ruby to compile them), and
51
- then moves them into a SHA dir structure that looks something like
52
- this:
49
+ This script filters all the newer (< 1 year old) gems (unless `-a` is
50
+ used), unpacks them, finds all the files that look like they're valid
51
+ ruby, ensures they're valid ruby (using the current version of ruby to
52
+ compile them), and then moves them into a SHA dir structure that looks
53
+ something like this:
53
54
 
54
55
  ```
55
56
  hashed.noindex/a/b/c/<full_file_sha>.rb
@@ -64,8 +65,8 @@ Unpacking, validating, SHA'ing everything is disk and CPU intensive.
64
65
  The `.noindex` extension stops spotlight from indexing the continous
65
66
  churn of files being unpacked and moved and saves time.
66
67
 
67
- Finally, I rename and archive it all up (currently using lrztar, but
68
- I'm not in love with it).
68
+ Finally, I rename and archive it all up (currently using zstd to
69
+ compress).
69
70
 
70
71
  ### Stats
71
72
 
@@ -73,7 +74,7 @@ I'm not in love with it).
73
74
  9696 % find gauntlet.$(today).noindex -type f | lc
74
75
  561270
75
76
  3.5G gauntlet.2021-08-06.noindex
76
- 239M gauntlet.2021-08-06.noindex.tar.lrz
77
+ 239M gauntlet.2021-08-06.noindex.tar.zst
77
78
  ```
78
79
 
79
80
  So I wind up with a little over half a million unique ruby files to
@@ -84,7 +85,7 @@ parse. It's about 3.5g but compresses very nicely down to 240m
84
85
  Assuming you're starting from scratch, unpack the archive once:
85
86
 
86
87
  ```
87
- % lrzuntar gauntlet.$(today).noindex.lrz
88
+ % zstdcat gauntlet.$(today).noindex.tar.zst | tar x
88
89
  ```
89
90
 
90
91
  Then, either run a single process (easier to read):
@@ -96,7 +97,7 @@ Then, either run a single process (easier to read):
96
97
  Or max out your machine using xargs (note the `-P 16` and choose accordingly):
97
98
 
98
99
  ```
99
- % ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
100
+ % ls -d gauntlet/*.noindex/?/? | time xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
100
101
  ```
101
102
 
102
103
  In another terminal I usually monitor the progress like so: