brakeman 5.2.1 → 5.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGES.md +8 -0
- data/bundle/load.rb +2 -2
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/MIT-LICENSE.txt +0 -0
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel/processor_count.rb +2 -3
- data/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib/parallel/version.rb +4 -0
- data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel.rb +84 -4
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/History.rdoc +28 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/Manifest.txt +2 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/README.rdoc +8 -6
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/compare/normalize.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/debugging.md +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/gauntlet.md +19 -18
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_extensions.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_stringscanner.rb +0 -0
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby20_parser.rb +10973 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby20_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby21_parser.rb +10980 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby21_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby22_parser.rb +11123 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby22_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby23_parser.rb +11132 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby23_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby24_parser.rb +11231 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby24_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby25_parser.rb +11231 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby25_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby26_parser.rb +11253 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby26_parser.y +14 -27
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby27_parser.rb +12980 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby27_parser.y +19 -41
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby30_parser.rb +13242 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby30_parser.y +65 -90
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby31_parser.rb +13622 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1/lib/ruby3_parser.yy → ruby_parser-3.19.1/lib/ruby31_parser.y} +110 -105
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib/ruby3_parser.yy +3536 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer.rex.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_lexer_strings.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.rb +2 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser.yy +19 -41
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/ruby_parser_extras.rb +55 -2
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/munge.rb +0 -0
- data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/tools/ripper.rb +0 -0
- data/lib/brakeman/checks/check_sql.rb +3 -2
- data/lib/brakeman/checks/check_unsafe_reflection.rb +7 -2
- data/lib/brakeman/processors/alias_processor.rb +17 -1
- data/lib/brakeman/version.rb +1 -1
- metadata +44 -42
- data/bundle/ruby/2.7.0/gems/parallel-1.21.0/lib/parallel/version.rb +0 -4
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby20_parser.rb +0 -7128
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby21_parser.rb +0 -7182
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby22_parser.rb +0 -7228
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby23_parser.rb +0 -7237
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby24_parser.rb +0 -7268
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby25_parser.rb +0 -7268
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby26_parser.rb +0 -7287
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby27_parser.rb +0 -8517
- data/bundle/ruby/2.7.0/gems/ruby_parser-3.18.1/lib/ruby30_parser.rb +0 -8751
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7673373d923b1d6b2e4c2158a94681d01de5e5c8eb29561795c9fbe8bd879f6b
|
|
4
|
+
data.tar.gz: 44e5c623eb5fd6fe62ec7d0956a32cee82082a2c5983012ff7c172b10c6dc79a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2e3132eaffeb28f50ab40afda87c7a1f2b209044c71614bd6d68f55632479a148e08ac4dc995f370b75de352c896711ed09698056fa763b3449d2776e2ba2fb3
|
|
7
|
+
data.tar.gz: 31e069689731927b3f096fd39cb0f1f4bf9c7269e7af4487eed12b34e3a7711de77b353423975b3625f5883c4a3b4c5f9785ddf44ca4d4367007b9d44ee51205
|
data/CHANGES.md
CHANGED
|
@@ -1,3 +1,11 @@
|
|
|
1
|
+
# 5.2.2 - 2022-04-06
|
|
2
|
+
|
|
3
|
+
* Update `ruby_parser` for Ruby 3.1 support (Merek Skubela)
|
|
4
|
+
* Handle `nil` when joining values (Dan Buettner)
|
|
5
|
+
* Update message for unsafe reflection (Pedro Baracho)
|
|
6
|
+
* Add additional String methods for SQL injection check
|
|
7
|
+
* Respect equality in `if` conditions
|
|
8
|
+
|
|
1
9
|
# 5.2.1 - 2022-01-30
|
|
2
10
|
|
|
3
11
|
* Add warning codes for EOL software warnings
|
data/bundle/load.rb
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
path = File.expand_path('../..', __FILE__)
|
|
2
|
-
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.
|
|
2
|
+
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-3.19.1/lib"
|
|
3
3
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/temple-0.8.2/lib"
|
|
4
4
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/sexp_processor-4.16.0/lib"
|
|
5
5
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/tilt-2.0.10/lib"
|
|
@@ -9,7 +9,7 @@ $:.unshift "#{path}/bundle/ruby/2.7.0/gems/highline-2.0.3/lib"
|
|
|
9
9
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby2ruby-2.4.4/lib"
|
|
10
10
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/terminal-table-1.8.0/lib"
|
|
11
11
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/haml-5.2.2/lib"
|
|
12
|
-
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.
|
|
12
|
+
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/parallel-1.22.1/lib"
|
|
13
13
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/ruby_parser-legacy-1.0.0/lib"
|
|
14
14
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/erubis-2.7.0/lib"
|
|
15
15
|
$:.unshift "#{path}/bundle/ruby/2.7.0/gems/rexml-3.2.5/lib"
|
|
File without changes
|
data/bundle/ruby/2.7.0/gems/{parallel-1.21.0 → parallel-1.22.1}/lib/parallel/processor_count.rb
RENAMED
|
@@ -1,11 +1,10 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
|
-
require 'etc'
|
|
3
|
-
|
|
4
2
|
module Parallel
|
|
5
3
|
# TODO: inline this method into parallel.rb and kill physical_processor_count in next major release
|
|
6
4
|
module ProcessorCount
|
|
7
5
|
# Number of processors seen by the OS, used for process scheduling
|
|
8
6
|
def processor_count
|
|
7
|
+
require 'etc'
|
|
9
8
|
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Etc.nprocessors)
|
|
10
9
|
end
|
|
11
10
|
|
|
@@ -19,7 +18,7 @@ module Parallel
|
|
|
19
18
|
when /linux/
|
|
20
19
|
cores = {} # unique physical ID / core ID combinations
|
|
21
20
|
phy = 0
|
|
22
|
-
|
|
21
|
+
File.read("/proc/cpuinfo").scan(/^physical id.*|^core id.*/) do |ln|
|
|
23
22
|
if ln.start_with?("physical")
|
|
24
23
|
phy = ln[/\d+/]
|
|
25
24
|
elsif ln.start_with?("core")
|
|
@@ -264,6 +264,9 @@ module Parallel
|
|
|
264
264
|
elsif options[:in_threads]
|
|
265
265
|
method = :in_threads
|
|
266
266
|
size = options[method]
|
|
267
|
+
elsif options[:in_ractors]
|
|
268
|
+
method = :in_ractors
|
|
269
|
+
size = options[method]
|
|
267
270
|
else
|
|
268
271
|
method = :in_processes
|
|
269
272
|
if Process.respond_to?(:fork)
|
|
@@ -285,6 +288,8 @@ module Parallel
|
|
|
285
288
|
work_direct(job_factory, options, &block)
|
|
286
289
|
elsif method == :in_threads
|
|
287
290
|
work_in_threads(job_factory, options.merge(count: size), &block)
|
|
291
|
+
elsif method == :in_ractors
|
|
292
|
+
work_in_ractors(job_factory, options.merge(count: size), &block)
|
|
288
293
|
else
|
|
289
294
|
work_in_processes(job_factory, options.merge(count: size), &block)
|
|
290
295
|
end
|
|
@@ -382,6 +387,72 @@ module Parallel
|
|
|
382
387
|
exception || results
|
|
383
388
|
end
|
|
384
389
|
|
|
390
|
+
def work_in_ractors(job_factory, options)
|
|
391
|
+
exception = nil
|
|
392
|
+
results = []
|
|
393
|
+
results_mutex = Mutex.new # arrays are not thread-safe on jRuby
|
|
394
|
+
|
|
395
|
+
callback = options[:ractor]
|
|
396
|
+
if block_given? || !callback
|
|
397
|
+
raise ArgumentError, "pass the code you want to execute as `ractor: [ClassName, :method_name]`"
|
|
398
|
+
end
|
|
399
|
+
|
|
400
|
+
# build
|
|
401
|
+
ractors = Array.new(options.fetch(:count)) do
|
|
402
|
+
Ractor.new do
|
|
403
|
+
loop do
|
|
404
|
+
got = receive
|
|
405
|
+
(klass, method_name), item, index = got
|
|
406
|
+
break if index == :break
|
|
407
|
+
begin
|
|
408
|
+
Ractor.yield [nil, klass.send(method_name, item), item, index]
|
|
409
|
+
rescue StandardError => e
|
|
410
|
+
Ractor.yield [e, nil, item, index]
|
|
411
|
+
end
|
|
412
|
+
end
|
|
413
|
+
end
|
|
414
|
+
end
|
|
415
|
+
|
|
416
|
+
# start
|
|
417
|
+
ractors.dup.each do |ractor|
|
|
418
|
+
if set = job_factory.next
|
|
419
|
+
item, index = set
|
|
420
|
+
instrument_start item, index, options
|
|
421
|
+
ractor.send [callback, item, index]
|
|
422
|
+
else
|
|
423
|
+
ractor.send([[nil, nil], nil, :break]) # stop the ractor
|
|
424
|
+
ractors.delete ractor
|
|
425
|
+
end
|
|
426
|
+
end
|
|
427
|
+
|
|
428
|
+
# replace with new items
|
|
429
|
+
while set = job_factory.next
|
|
430
|
+
item_next, index_next = set
|
|
431
|
+
done, (exception, result, item, index) = Ractor.select(*ractors)
|
|
432
|
+
if exception
|
|
433
|
+
ractors.delete done
|
|
434
|
+
break
|
|
435
|
+
end
|
|
436
|
+
instrument_finish item, index, result, options
|
|
437
|
+
results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
|
|
438
|
+
|
|
439
|
+
instrument_start item_next, index_next, options
|
|
440
|
+
done.send([callback, item_next, index_next])
|
|
441
|
+
end
|
|
442
|
+
|
|
443
|
+
# finish
|
|
444
|
+
ractors.each do |ractor|
|
|
445
|
+
(new_exception, result, item, index) = ractor.take
|
|
446
|
+
exception ||= new_exception
|
|
447
|
+
next if new_exception
|
|
448
|
+
instrument_finish item, index, result, options
|
|
449
|
+
results_mutex.synchronize { results[index] = (options[:preserve_results] == false ? nil : result) }
|
|
450
|
+
ractor.send([[nil, nil], nil, :break]) # stop the ractor
|
|
451
|
+
end
|
|
452
|
+
|
|
453
|
+
exception || results
|
|
454
|
+
end
|
|
455
|
+
|
|
385
456
|
def work_in_processes(job_factory, options, &blk)
|
|
386
457
|
workers = create_workers(job_factory, options, &blk)
|
|
387
458
|
results = []
|
|
@@ -426,6 +497,7 @@ module Parallel
|
|
|
426
497
|
end
|
|
427
498
|
end
|
|
428
499
|
end
|
|
500
|
+
|
|
429
501
|
exception || results
|
|
430
502
|
end
|
|
431
503
|
|
|
@@ -521,12 +593,20 @@ module Parallel
|
|
|
521
593
|
end
|
|
522
594
|
|
|
523
595
|
def with_instrumentation(item, index, options)
|
|
524
|
-
|
|
525
|
-
on_finish = options[:finish]
|
|
526
|
-
options[:mutex].synchronize { on_start.call(item, index) } if on_start
|
|
596
|
+
instrument_start(item, index, options)
|
|
527
597
|
result = yield
|
|
528
|
-
|
|
598
|
+
instrument_finish(item, index, result, options)
|
|
529
599
|
result unless options[:preserve_results] == false
|
|
530
600
|
end
|
|
601
|
+
|
|
602
|
+
def instrument_finish(item, index, result, options)
|
|
603
|
+
return unless on_finish = options[:finish]
|
|
604
|
+
options[:mutex].synchronize { on_finish.call(item, index, result) }
|
|
605
|
+
end
|
|
606
|
+
|
|
607
|
+
def instrument_start(item, index, options)
|
|
608
|
+
return unless on_start = options[:start]
|
|
609
|
+
options[:mutex].synchronize { on_start.call(item, index) }
|
|
610
|
+
end
|
|
531
611
|
end
|
|
532
612
|
end
|
|
@@ -1,3 +1,31 @@
|
|
|
1
|
+
=== 3.19.1 / 2022-04-05
|
|
2
|
+
|
|
3
|
+
* 2 bug fixes:
|
|
4
|
+
|
|
5
|
+
* Added comments to endless defn and defs. (mvz)
|
|
6
|
+
* Fixed endless method bug handling attrset names.
|
|
7
|
+
|
|
8
|
+
=== 3.19.0 / 2022-03-29
|
|
9
|
+
|
|
10
|
+
* 1 major enhancement:
|
|
11
|
+
|
|
12
|
+
* Added tentative 3.1 support.
|
|
13
|
+
|
|
14
|
+
* 7 minor enhancements:
|
|
15
|
+
|
|
16
|
+
* 3.1: bare RHS assoc: { y: } => s(:hash, s(:lit, :y), nil)
|
|
17
|
+
* 3.1: calls w/ unnamed block args (bare &)
|
|
18
|
+
* 3.1: endless defn/defs w/ paren-less calls (aka commands)
|
|
19
|
+
* 3.1: pattern capture to nonlocal vars, eg: ^@a, ^$b, ^@@c
|
|
20
|
+
* 3.1: pattern: ^(expr) => expr
|
|
21
|
+
* Improved steps for adding new versions.
|
|
22
|
+
* Improved steps for running gauntlets.
|
|
23
|
+
|
|
24
|
+
* 2 bug fixes:
|
|
25
|
+
|
|
26
|
+
* Bumped 2.6+ cached versions for rake compare.
|
|
27
|
+
* Skip test_regexp_esc_C_slash on ruby 3.1.0 because of MRI bug.
|
|
28
|
+
|
|
1
29
|
=== 3.18.1 / 2021-11-10
|
|
2
30
|
|
|
3
31
|
* 1 minor enhancement:
|
|
@@ -33,6 +33,9 @@ Tested against 801,039 files from the latest of all rubygems (as of 2013-05):
|
|
|
33
33
|
* 1.9 parser is at 99.9940% accuracy, 4.013 sigma
|
|
34
34
|
* 2.0 parser is at 99.9939% accuracy, 4.008 sigma
|
|
35
35
|
* 2.6 parser is at 99.9972% accuracy, 4.191 sigma
|
|
36
|
+
* 3.0 parser has a 100% parse rate.
|
|
37
|
+
* Tested against 2,672,412 unique ruby files across 167k gems.
|
|
38
|
+
* As do all the others now, basically.
|
|
36
39
|
|
|
37
40
|
== FEATURES/PROBLEMS:
|
|
38
41
|
|
|
@@ -62,15 +65,14 @@ You can also use Ruby19Parser, Ruby18Parser, or RubyParser.for_current_ruby:
|
|
|
62
65
|
|
|
63
66
|
To add a new version:
|
|
64
67
|
|
|
65
|
-
* New parser should be generated from lib/
|
|
66
|
-
* Extend lib/
|
|
67
|
-
* Add new version number to V2 in Rakefile for rule creation.
|
|
68
|
+
* New parser should be generated from lib/ruby[3]_parser.yy.
|
|
69
|
+
* Extend lib/ruby[3]_parser.yy with new class name.
|
|
70
|
+
* Add new version number to V2/V3 in Rakefile for rule creation.
|
|
71
|
+
* Add new (full) version to `ruby_parse` section of Rakefile for rake compare
|
|
68
72
|
* Require generated parser in lib/ruby_parser.rb.
|
|
69
73
|
* Add empty TestRubyParserShared##Plus module and TestRubyParserV## to test/test_ruby_parser.rb.
|
|
70
74
|
* Extend Manifest.txt with generated file names.
|
|
71
|
-
*
|
|
72
|
-
* add_19tests needs to have the version added
|
|
73
|
-
* VER_RE needs to have the regexp expanded
|
|
75
|
+
* Add new version number to sexp_processor's pt_testcase.rb in all_versions
|
|
74
76
|
|
|
75
77
|
Until all of these are done, you won't have a clean test run.
|
|
76
78
|
|
|
File without changes
|
|
File without changes
|
|
@@ -19,10 +19,10 @@ an external disk. Here is the config:
|
|
|
19
19
|
And I update using rake:
|
|
20
20
|
|
|
21
21
|
```
|
|
22
|
-
% cd
|
|
22
|
+
% cd GIT/rubygems/rubygems-mirror
|
|
23
23
|
% git down
|
|
24
24
|
% rake mirror:latest
|
|
25
|
-
% /Volumes/StuffA/gauntlet/bin/cleanup.rb
|
|
25
|
+
% /Volumes/StuffA/gauntlet/bin/cleanup.rb -y -v
|
|
26
26
|
```
|
|
27
27
|
|
|
28
28
|
This rather quickly updates my mirror to the latest versions of
|
|
@@ -34,22 +34,23 @@ bit, but it is pretty minimal (currently ~20 bad gems).
|
|
|
34
34
|
## Curating an Archive of Ruby Files
|
|
35
35
|
|
|
36
36
|
Next, I process the gem mirror into a much more digestable structure
|
|
37
|
-
using `
|
|
37
|
+
using `unpack_gems.rb`.
|
|
38
38
|
|
|
39
39
|
```
|
|
40
|
-
% cd RP
|
|
41
|
-
% /Volumes/StuffA/gauntlet/bin/unpack_gems.rb
|
|
40
|
+
% cd RP/gauntlet
|
|
41
|
+
% time caffeinate /Volumes/StuffA/gauntlet/bin/unpack_gems.rb -v [-a] ; say done
|
|
42
42
|
... waaaait ...
|
|
43
|
-
%
|
|
44
|
-
%
|
|
45
|
-
%
|
|
43
|
+
% DIR=gauntlet.$(today).(all|new).noindex
|
|
44
|
+
% mv hashed.noindex $DIR
|
|
45
|
+
% tar vc -T <(fd -tf . $DIR | sort) | zstd -5 -T0 --long > archives/$DIR.tar.zst ; say done
|
|
46
|
+
% ./bin/sync.sh
|
|
46
47
|
```
|
|
47
48
|
|
|
48
|
-
This script filters all the newer gems (
|
|
49
|
-
finds all the files that look like they're valid
|
|
50
|
-
valid ruby (using the current version of ruby to
|
|
51
|
-
then moves them into a SHA dir structure that looks
|
|
52
|
-
this:
|
|
49
|
+
This script filters all the newer (< 1 year old) gems (unless `-a` is
|
|
50
|
+
used), unpacks them, finds all the files that look like they're valid
|
|
51
|
+
ruby, ensures they're valid ruby (using the current version of ruby to
|
|
52
|
+
compile them), and then moves them into a SHA dir structure that looks
|
|
53
|
+
something like this:
|
|
53
54
|
|
|
54
55
|
```
|
|
55
56
|
hashed.noindex/a/b/c/<full_file_sha>.rb
|
|
@@ -64,8 +65,8 @@ Unpacking, validating, SHA'ing everything is disk and CPU intensive.
|
|
|
64
65
|
The `.noindex` extension stops spotlight from indexing the continous
|
|
65
66
|
churn of files being unpacked and moved and saves time.
|
|
66
67
|
|
|
67
|
-
Finally, I rename and archive it all up (currently using
|
|
68
|
-
|
|
68
|
+
Finally, I rename and archive it all up (currently using zstd to
|
|
69
|
+
compress).
|
|
69
70
|
|
|
70
71
|
### Stats
|
|
71
72
|
|
|
@@ -73,7 +74,7 @@ I'm not in love with it).
|
|
|
73
74
|
9696 % find gauntlet.$(today).noindex -type f | lc
|
|
74
75
|
561270
|
|
75
76
|
3.5G gauntlet.2021-08-06.noindex
|
|
76
|
-
239M gauntlet.2021-08-06.noindex.tar.
|
|
77
|
+
239M gauntlet.2021-08-06.noindex.tar.zst
|
|
77
78
|
```
|
|
78
79
|
|
|
79
80
|
So I wind up with a little over half a million unique ruby files to
|
|
@@ -84,7 +85,7 @@ parse. It's about 3.5g but compresses very nicely down to 240m
|
|
|
84
85
|
Assuming you're starting from scratch, unpack the archive once:
|
|
85
86
|
|
|
86
87
|
```
|
|
87
|
-
%
|
|
88
|
+
% zstdcat gauntlet.$(today).noindex.tar.zst | tar x
|
|
88
89
|
```
|
|
89
90
|
|
|
90
91
|
Then, either run a single process (easier to read):
|
|
@@ -96,7 +97,7 @@ Then, either run a single process (easier to read):
|
|
|
96
97
|
Or max out your machine using xargs (note the `-P 16` and choose accordingly):
|
|
97
98
|
|
|
98
99
|
```
|
|
99
|
-
% ls -d gauntlet/*.noindex/?/? | xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
|
|
100
|
+
% ls -d gauntlet/*.noindex/?/? | time xargs -n 1 -P 16 ./gauntlet/bin/gauntlet.rb
|
|
100
101
|
```
|
|
101
102
|
|
|
102
103
|
In another terminal I usually monitor the progress like so:
|
|
File without changes
|
data/bundle/ruby/2.7.0/gems/{ruby_parser-3.18.1 → ruby_parser-3.19.1}/lib/rp_stringscanner.rb
RENAMED
|
File without changes
|