brakeman 5.2.0 → 5.2.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGES.md +17 -0
- data/bundle/load.rb +3 -3
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/MIT-LICENSE.txt +0 -0
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel/processor_count.rb +2 -3
- data/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib/parallel/version.rb +4 -0
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel.rb +84 -4
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/History.rdoc +28 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/Manifest.txt +2 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/README.rdoc +8 -6
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/compare/normalize.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/debugging.md +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/gauntlet.md +19 -18
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_extensions.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_stringscanner.rb +0 -0
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby20_parser.rb +10973 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby20_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby21_parser.rb +10980 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby21_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby22_parser.rb +11123 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby22_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby23_parser.rb +11132 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby23_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby24_parser.rb +11231 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby24_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby25_parser.rb +11231 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby25_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby26_parser.rb +11253 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby26_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby27_parser.rb +12980 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby27_parser.y +19 -41
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby30_parser.rb +13242 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby30_parser.y +65 -90
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby31_parser.rb +13622 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1/lib/ruby3_parser.yy → ruby_parser-3.19.1/lib/ruby31_parser.y} +110 -105
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby3_parser.yy +3536 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer_strings.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.rb +2 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.yy +19 -41
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser_extras.rb +55 -2
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/munge.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/ripper.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/History.rdoc +6 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/Manifest.txt +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/README.rdoc +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/composite_sexp_processor.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/pt_testcase.rb +7 -3
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp_matcher.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/sexp_processor.rb +1 -1
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/strict_sexp.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{sexp_processor-4.16.0 → sexp_processor-4.16.1}/lib/unique.rb +0 -0
- data/lib/brakeman/checks/check_sql.rb +3 -2
- data/lib/brakeman/checks/check_unsafe_reflection.rb +7 -2
- data/lib/brakeman/processors/alias_processor.rb +41 -2
- data/lib/brakeman/report/ignore/interactive.rb +2 -2
- data/lib/brakeman/version.rb +1 -1
- data/lib/brakeman/warning_codes.rb +2 -0
- metadata +54 -52
- data/bundle/ruby/2.7.0/gems/parallel-1.21.0/lib/parallel/version.rb +0 -4
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby20_parser.rb +0 -7128
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby21_parser.rb +0 -7182
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby22_parser.rb +0 -7228
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby23_parser.rb +0 -7237
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby24_parser.rb +0 -7268
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby25_parser.rb +0 -7268
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby26_parser.rb +0 -7287
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby27_parser.rb +0 -8517
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby30_parser.rb +0 -8751
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2ae08a71e19d6c694a9e567fda5793a56ab174d106f44b178f7e9f69c3057814
|
4
|
+
data.tar.gz: fd24750e512d528b3fd9cecb344f4788a58e1aa8ffd6b28ff7c88bc7f034a3e8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e0e2d7fde5907d8158b21803876b0dd77e659ce8cae42c25e23021b46bd2c9d8c5d0dd13edff64f7fd721a8d1bca92af4a9fbbf2505e47c791557a3316c0f3d2
|
7
|
+
data.tar.gz: 38f894b42f893a6ce45db047f2d21c1529b63de0dd19e00a69475a72cbb4c2d9738f0f25edc60460d61607182d5a5c01ccbb77a6ae6eeae69ec4e88f9345a2e1
|
data/CHANGES.md
CHANGED
@@ -1,3 +1,20 @@
|
|
1
|
+
# 5.2.3 - 2022-05-01
|
2
|
+
|
3
|
+
* Fix error with hash shorthand syntax
|
4
|
+
* Match order of interactive options with help message (Rory O'Kane)
|
5
|
+
|
6
|
+
# 5.2.2 - 2022-04-06
|
7
|
+
|
8
|
+
* Update `ruby_parser` for Ruby 3.1 support (Merek Skubela)
|
9
|
+
* Handle `nil` when joining values (Dan Buettner)
|
10
|
+
* Update message for unsafe reflection (Pedro Baracho)
|
11
|
+
* Add additional String methods for SQL injection check
|
12
|
+
* Respect equality in `if` conditions
|
13
|
+
|
14
|
+
# 5.2.1 - 2022-01-30
|
15
|
+
|
16
|
+
* Add warning codes for EOL software warnings
|
17
|
+
|
1
18
|
# 5.2.0 - 2021-12-15
|
2
19
|
|
3
20
|
* Initial Rails 7 support
|
data/bundle/load.rb
CHANGED
@@ -1,7 +1,6 @@
|
|
1
1
|
path = File.expand_path('../..', __FILE__)
|
2
|
-
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.
|
2
|
+
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib"
|
3
3
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/temple-0.8.2/lib"
|
4
|
-
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/sexp_processor-4.16.0/lib"
|
5
4
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/tilt-2.0.10/lib"
|
6
5
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/unicode-display_width-1.8.0/lib"
|
7
6
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/slim-4.1.0/lib"
|
@@ -9,7 +8,8 @@ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/highline-2.0.3/lib"
|
|
9
8
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby2ruby-2.4.4/lib"
|
10
9
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/terminal-table-1.8.0/lib"
|
11
10
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/haml-5.2.2/lib"
|
12
|
-
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.
|
11
|
+
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib"
|
12
|
+
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/sexp_processor-4.16.1/lib"
|
13
13
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-legacy-1.0.0/lib"
|
14
14
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/erubis-2.7.0/lib"
|
15
15
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/rexml-3.2.5/lib"
|
File without changes
|
data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel/processor_count.rb
RENAMED
@@ -1,11 +1,10 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
-
require 'etc'
|
3
|
-
|
4
2
|
module Parallel
|
5
3
|
# TODO: inline this method into parallel.rb and kill physical_processor_count in next major release
|
6
4
|
module ProcessorCount
|
7
5
|
# Number of processors seen by the OS, used for process scheduling
|
8
6
|
def processor_count
|
7
|
+
require 'etc'
|
9
8
|
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Etc.nprocessors)
|
10
9
|
end
|
11
10
|
|
@@ -19,7 +18,7 @@ module Parallel
|
|
19
18
|
when /linux/
|
20
19
|
cores = {} # unique physical ID / core ID combinations
|
21
20
|
phy = 0
|
22
|
-
|
21
|
+
File.read("/proc/cpuinfo").scan(/^physical id.*|^core id.*/) do |ln|
|
23
22
|
if ln.start_with?("physical")
|
24
23
|
phy = ln[/\d+/]
|
25
24
|
elsif ln.start_with?("core")
|
@@ -264,6 +264,9 @@ module Parallel
|
|
264
264
|
elsif options[:in_threads]
|
265
265
|
method = :in_threads
|
266
266
|
size = options[method]
|
267
|
+
elsif options[:in_ractors]
|
268
|
+
method = :in_ractors
|
269
|
+
size = options[method]
|
267
270
|
else
|
268
271
|
method = :in_processes
|
269
272
|
if Process.respond_to?(:fork)
|
@@ -285,6 +288,8 @@ module Parallel
|
|
285
288
|
work_direct(job_factory, options, &block)
|
286
289
|
elsif method == :in_threads
|
287
290
|
work_in_threads(job_factory, options.merge(count: size), &block)
|
291
|
+
elsif method == :in_ractors
|
292
|
+
work_in_ractors(job_factory, options.merge(count: size), &block)
|
288
293
|
else
|
289
294
|
work_in_processes(job_factory, options.merge(count: size), &block)
|
290
295
|
end
|
@@ -382,6 +387,72 @@ module Parallel
|
|
382
387
|
exception || results
|
383
388
|
end
|
384
389
|
|
390
|
+
def work_in_ractors(job_factory, options)
|
391
|
+
exception = nil
|
392
|
+
results = []
|
393
|
+
results_mutex = Mutex.new # arrays are not thread-safe on jRuby
|
394
|
+
|
395
|
+
callback = options[:ractor]
|
396
|
+
if block_given? || !callback
|
397
|
+
raise ArgumentError, "pass the code you want to execute as `ractor: [ClassName, :method_name]`"
|
398
|
+
end
|
399
|
+
|
400
|
+
# build
|
401
|
+
ractors = Array.new(options.fetch(:count)) do
|
402
|
+
Ractor.new do
|
403
|
+
loop do
|
404
|
+
got = receive
|
405
|
+
(klass, method_name), item, index = got
|
406
|
+
break if index == :break
|
407
|
+
begin
|
408
|
+
Ractor.yield [nil, klass.send(method_name, item), item, index]
|
409
|
+
rescue StandardError => e
|
410
|
+
Ractor.yield [e, nil, item, index]
|
411
|
+
end
|
412
|
+
end
|
413
|
+
end
|
414
|
+
end
|
415
|
+
|
416
|
+
# start
|
417
|
+
ractors.dup.each do |ractor|
|
418
|
+
if set = job_factory.next
|
419
|
+
item, index = set
|
420
|
+
instrument_start item, index, options
|
421
|
+
ractor.send [callback, item, index]
|
422
|
+
else
|
423
|
+
ractor.send([[nil, nil], nil, :break]) # stop the ractor
|
424
|
+
ractors.delete ractor
|
425
|
+
end
|
426
|
+
end
|
427
|
+
|
428
|
+
# replace with new items
|
429
|
+
while set = job_factory.next
|
430
|
+
item_next, index_next = set
|
431
|
+
done, (exception, result, item, index) = Ractor.select(*ractors)
|
432
|
+
if exception
|
433
|
+
ractors.delete done
|
434
|
+
break
|
435
|
+
end
|
436
|
+
instrument_finish item, index, result, options
|
437
|
+
results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
|
438
|
+
|
439
|
+
instrument_start item_next, index_next, options
|
440
|
+
done.send([callback, item_next, index_next])
|
441
|
+
end
|
442
|
+
|
443
|
+
# finish
|
444
|
+
ractors.each do |ractor|
|
445
|
+
(new_exception, result, item, index) = ractor.take
|
446
|
+
exception ||= new_exception
|
447
|
+
next if new_exception
|
448
|
+
instrument_finish item, index, result, options
|
449
|
+
results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
|
450
|
+
ractor.send([[nil, nil], nil, :break]) # stop the ractor
|
451
|
+
end
|
452
|
+
|
453
|
+
exception || results
|
454
|
+
end
|
455
|
+
|
385
456
|
def work_in_processes(job_factory, options, &blk)
|
386
457
|
workers = create_workers(job_factory, options, &blk)
|
387
458
|
results = []
|
@@ -426,6 +497,7 @@ module Parallel
|
|
426
497
|
end
|
427
498
|
end
|
428
499
|
end
|
500
|
+
|
429
501
|
exception || results
|
430
502
|
end
|
431
503
|
|
@@ -521,12 +593,20 @@ module Parallel
|
|
521
593
|
end
|
522
594
|
|
523
595
|
def with_instrumentation(item, index, options)
|
524
|
-
|
525
|
-
on_finish = options[:finish]
|
526
|
-
options[:mutex].synchronize { on_start.call(item, index) } if on_start
|
596
|
+
instrument_start(item, index, options)
|
527
597
|
result = yield
|
528
|
-
|
598
|
+
instrument_finish(item, index, result, options)
|
529
599
|
result unless options[:preserve_results] == false
|
530
600
|
end
|
601
|
+
|
602
|
+
def instrument_finish(item, index, result, options)
|
603
|
+
return unless on_finish = options[:finish]
|
604
|
+
options[:mutex].synchronize { on_finish.call(item, index, result) }
|
605
|
+
end
|
606
|
+
|
607
|
+
def instrument_start(item, index, options)
|
608
|
+
return unless on_start = options[:start]
|
609
|
+
options[:mutex].synchronize { on_start.call(item, index) }
|
610
|
+
end
|
531
611
|
end
|
532
612
|
end
|
@@ -1,3 +1,31 @@
|
|
1
|
+
=== 3.19.1 / 2022-04-05
|
2
|
+
|
3
|
+
* 2 bug fixes:
|
4
|
+
|
5
|
+
* Added comments to endless defn and defs. (mvz)
|
6
|
+
* Fixed endless method bug handling attrset names.
|
7
|
+
|
8
|
+
=== 3.19.0 / 2022-03-29
|
9
|
+
|
10
|
+
* 1 major enhancement:
|
11
|
+
|
12
|
+
* Added tentative 3.1 support.
|
13
|
+
|
14
|
+
* 7 minor enhancements:
|
15
|
+
|
16
|
+
* 3.1: bare RHS assoc: { y: } => s(:hash, s(:lit, :y), nil)
|
17
|
+
* 3.1: calls w/ unnamed block args (bare &)
|
18
|
+
* 3.1: endless defn/defs w/ paren-less calls (aka commands)
|
19
|
+
* 3.1: pattern capture to nonlocal vars, eg: ^@a, ^$b, ^@@c
|
20
|
+
* 3.1: pattern: ^(expr) => expr
|
21
|
+
* Improved steps for adding new versions.
|
22
|
+
* Improved steps for running gauntlets.
|
23
|
+
|
24
|
+
* 2 bug fixes:
|
25
|
+
|
26
|
+
* Bumped 2.6+ cached versions for rake compare.
|
27
|
+
* Skip test_regexp_esc_C_slash on ruby 3.1.0 because of MRI bug.
|
28
|
+
|
1
29
|
=== 3.18.1 / 2021-11-10
|
2
30
|
|
3
31
|
* 1 minor enhancement:
|
@@ -33,6 +33,9 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
|
|
33
33
|
* 1.9 parser is at 99.9940% accuracy, 4.013 sigma
|
34
34
|
* 2.0 parser is at 99.9939% accuracy, 4.008 sigma
|
35
35
|
* 2.6 parser is at 99.9972% accuracy, 4.191 sigma
|
36
|
+
* 3.0 parser has a 100% parse rate.
|
37
|
+
* Tested against 2,672,412 unique ruby files across 167k gems.
|
38
|
+
* As do all the others now, basically.
|
36
39
|
|
37
40
|
== FEATURES/PROBLEMS:
|
38
41
|
|
@@ -62,15 +65,14 @@ You can also use Ruby19Parser, Ruby18Parser, or RubyParser.for_current_ruby:
|
|
62
65
|
|
63
66
|
To add a new version:
|
64
67
|
|
65
|
-
* New parser should be generated from lib/
|
66
|
-
* Extend lib/
|
67
|
-
* Add new version number to V2 in Rakefile for rule creation.
|
68
|
+
* New parser should be generated from lib/ruby[3]_parser.yy.
|
69
|
+
* Extend lib/ruby[3]_parser.yy with new class name.
|
70
|
+
* Add new version number to V2/V3 in Rakefile for rule creation.
|
71
|
+
* Add new (full) version to `ruby_parse` section of Rakefile for rake compare
|
68
72
|
* Require generated parser in lib/ruby_parser.rb.
|
69
73
|
* Add empty TestRubyParserShared##Plus module and TestRubyParserV## to test/test_ruby_parser.rb.
|
70
74
|
* Extend Manifest.txt with generated file names.
|
71
|
-
*
|
72
|
-
* add_19tests needs to have the version added
|
73
|
-
* VER_RE needs to have the regexp expanded
|
75
|
+
* Add new version number to sexp_processor's pt_testcase.rb in all_versions
|
74
76
|
|
75
77
|
Until all of these are done, you won't have a clean test run.
|
76
78
|
|
File without changes
|
File without changes
|
@@ -19,10 +19,10 @@ an external disk. Here is the config:
|
|
19
19
|
And I update using rake:
|
20
20
|
|
21
21
|
```
|
22
|
-
% cd
|
22
|
+
% cd GIT/rubygems/rubygems-mirror
|
23
23
|
% git down
|
24
24
|
% rake mirror:latest
|
25
|
-
% /Volumes/StuffA/gauntlet/bin/cleanup.rb
|
25
|
+
% /Volumes/StuffA/gauntlet/bin/cleanup.rb -y -v
|
26
26
|
```
|
27
27
|
|
28
28
|
This rather quickly updates my mirror to the latest versions of
|
@@ -34,22 +34,23 @@ bit, but it is pretty minimal (currently ~20 bad gems).
|
|
34
34
|
## Curating an Archive of Ruby Files
|
35
35
|
|
36
36
|
Next, I process the gem mirror into a much more digestable structure
|
37
|
-
using `
|
37
|
+
using `unpack_gems.rb`.
|
38
38
|
|
39
39
|
```
|
40
|
-
% cd RP
|
41
|
-
% /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
|
40
|
+
% cd RP/gauntlet
|
41
|
+
% time caffeinate /Volumes/StuffA/gauntlet/bin/unpack_gems.rb -v [-a] ; say done
|
42
42
|
... waaaait ...
|
43
|
-
%
|
44
|
-
%
|
45
|
-
%
|
43
|
+
% DIR=gauntlet.$(today).(all|new).noindex
|
44
|
+
% mv hashed.noindex $DIR
|
45
|
+
% tar vc -T <(fd -tf . $DIR | sort) | zstd -5 -T0 --long > archives/$DIR.tar.zst ; say done
|
46
|
+
% ./bin/sync.sh
|
46
47
|
```
|
47
48
|
|
48
|
-
This script filters all the newer gems (
|
49
|
-
finds all the files that look like they're valid
|
50
|
-
valid ruby (using the current version of ruby to
|
51
|
-
then moves them into a SHA dir structure that looks
|
52
|
-
this:
|
49
|
+
This script filters all the newer (< 1 year old) gems (unless `-a` is
|
50
|
+
used), unpacks them, finds all the files that look like they're valid
|
51
|
+
ruby, ensures they're valid ruby (using the current version of ruby to
|
52
|
+
compile them), and then moves them into a SHA dir structure that looks
|
53
|
+
something like this:
|
53
54
|
|
54
55
|
```
|
55
56
|
hashed.noindex/a/b/c/<full_file_sha>.rb
|
@@ -64,8 +65,8 @@ Unpacking, validating, SHA'ing everything is disk and CPU intensive.
|
|
64
65
|
The `.noindex` extension stops spotlight from indexing the continous
|
65
66
|
churn of files being unpacked and moved and saves time.
|
66
67
|
|
67
|
-
Finally, I rename and archive it all up (currently using
|
68
|
-
|
68
|
+
Finally, I rename and archive it all up (currently using zstd to
|
69
|
+
compress).
|
69
70
|
|
70
71
|
### Stats
|
71
72
|
|
@@ -73,7 +74,7 @@ I'm not in love with it).
|
|
73
74
|
9696 % find gauntlet.$(today).noindex -type f | lc
|
74
75
|
561270
|
75
76
|
3.5G gauntlet.2021-08-06.noindex
|
76
|
-
239M gauntlet.2021-08-06.noindex.tar.
|
77
|
+
239M gauntlet.2021-08-06.noindex.tar.zst
|
77
78
|
```
|
78
79
|
|
79
80
|
So I wind up with a little over half a million unique ruby files to
|
@@ -84,7 +85,7 @@ parse. It's about 3.5g but compresses very nicely down to 240m
|
|
84
85
|
Assuming you're starting from scratch, unpack the archive once:
|
85
86
|
|
86
87
|
```
|
87
|
-
%
|
88
|
+
% zstdcat gauntlet.$(today).noindex.tar.zst | tar x
|
88
89
|
```
|
89
90
|
|
90
91
|
Then, either run a single process (easier to read):
|
@@ -96,7 +97,7 @@ Then, either run a single process (easier to read):
|
|
96
97
|
Or max out your machine using xargs (note the `-P 16` and choose accordingly):
|
97
98
|
|
98
99
|
```
|
99
|
-
% ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
|
100
|
+
% ls -d gauntlet/*.noindex/?/? | time xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
|
100
101
|
```
|
101
102
|
|
102
103
|
In another terminal I usually monitor the progress like so:
|
File without changes
|
data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_stringscanner.rb
RENAMED
File without changes
|