rgot 1.2.0 → 1.3.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 368b98f9969eee4d2608df53e668eaaed9f06320ecd5b26096ff05c0f81dc16e
4
- data.tar.gz: b045a87ffcb3f18338ce0286e2f22b71921b5bab011e4049c203164b9ab7a077
3
+ metadata.gz: e2f0075689755b0a44db0674a5069769d1ee4c2567774d7bb7422ba5b27c8960
4
+ data.tar.gz: 9ddd64091ca0daeff256fcc5c305aad378a92472f4522b875512fb35915d74aa
5
5
  SHA512:
6
- metadata.gz: 7a3d443886aa26702be0720d8bdf8abf38dd938e3707d595ac1bf21b1edd0f3cb52f64f29e4ca11996357d8671bfaaa4edc94500560510b67c393244202a2db3
7
- data.tar.gz: 13d8fcdf048c4c82da186f96dc7bfef3eb78a35fb5df04a2f6f17badda63100587df8cd175dc0618d823d613526b9e60aeec8e18cc09075c8bf3c8c218d58841
6
+ metadata.gz: 2e24662ccc2f6f4644eec95f9764975b86d2ea40f49c3c8ae569f428809ca08df79e6517d2bbcf0ec7d08e610867972133a1a9ab5df245cf36896afb8ace7e63
7
+ data.tar.gz: ac5c041deb124812d911d0af697a06a398ced2c79bc54f6058ec4cb2499af9dd814007f3e0c8951c50448892ba0fbc0a6846dfd879c3497601b4aebf5d00116e
@@ -17,6 +17,7 @@ jobs:
17
17
  - '2.7.7'
18
18
  - '3.0.5'
19
19
  - '3.1.3'
20
+ - '3.2.0'
20
21
 
21
22
  steps:
22
23
  - uses: actions/checkout@v3
data/.gitignore CHANGED
@@ -1,2 +1,3 @@
1
1
  pkg
2
2
  tmp
3
+ /.gem_rbs_collection
data/Gemfile CHANGED
@@ -1,3 +1,7 @@
1
1
  source 'https://rubygems.org'
2
2
 
3
3
  gemspec
4
+
5
+ gem "rbs"
6
+ gem "steep"
7
+ gem "debug"
data/Gemfile.lock CHANGED
@@ -1,20 +1,73 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- rgot (1.2.0)
4
+ rgot (1.3.0)
5
5
 
6
6
  GEM
7
7
  remote: https://rubygems.org/
8
8
  specs:
9
+ activesupport (7.0.4)
10
+ concurrent-ruby (~> 1.0, >= 1.0.2)
11
+ i18n (>= 1.6, < 2)
12
+ minitest (>= 5.1)
13
+ tzinfo (~> 2.0)
14
+ ast (2.4.2)
15
+ concurrent-ruby (1.1.10)
16
+ csv (3.2.6)
17
+ debug (1.7.1)
18
+ ffi (1.15.5)
19
+ fileutils (1.7.0)
20
+ i18n (1.12.0)
21
+ concurrent-ruby (~> 1.0)
22
+ json (2.6.3)
23
+ language_server-protocol (3.17.0.2)
24
+ listen (3.7.1)
25
+ rb-fsevent (~> 0.10, >= 0.10.3)
26
+ rb-inotify (~> 0.9, >= 0.9.10)
27
+ logger (1.5.3)
28
+ minitest (5.17.0)
29
+ parallel (1.22.1)
30
+ parser (3.2.0.0)
31
+ ast (~> 2.4.1)
32
+ rainbow (3.1.1)
9
33
  rake (13.0.6)
34
+ rb-fsevent (0.11.2)
35
+ rb-inotify (0.10.1)
36
+ ffi (~> 1.0)
37
+ rbs (2.8.2)
38
+ securerandom (0.2.2)
39
+ steep (1.3.0)
40
+ activesupport (>= 5.1)
41
+ csv (>= 3.0.9)
42
+ fileutils (>= 1.1.0)
43
+ json (>= 2.1.0)
44
+ language_server-protocol (>= 3.15, < 4.0)
45
+ listen (~> 3.0)
46
+ logger (>= 1.3.0)
47
+ parallel (>= 1.0.0)
48
+ parser (>= 3.1)
49
+ rainbow (>= 2.2.2, < 4.0)
50
+ rbs (>= 2.8.0)
51
+ securerandom (>= 0.1)
52
+ strscan (>= 1.0.0)
53
+ terminal-table (>= 2, < 4)
54
+ strscan (3.0.5)
55
+ terminal-table (3.0.2)
56
+ unicode-display_width (>= 1.1.1, < 3)
57
+ tzinfo (2.0.5)
58
+ concurrent-ruby (~> 1.0)
59
+ unicode-display_width (2.4.2)
10
60
 
11
61
  PLATFORMS
12
62
  ruby
13
63
 
14
64
  DEPENDENCIES
15
65
  bundler
66
+ debug
16
67
  rake
68
+ rbs
17
69
  rgot!
70
+ steep
18
71
 
19
72
  BUNDLED WITH
20
- 2.3.26
73
+ 2.4.2
data/README.md CHANGED
@@ -93,6 +93,33 @@ ok FooTest 2.782s
93
93
 
94
94
  `b.n` is automatically adjusted.
95
95
 
96
+ ## Fuzzing
97
+
98
+ ```
99
+ $ rgot target_file_test.rb --fuzz . --fuzztime 1
100
+ ```
101
+
102
+ Fuzzing tests are also supported.
103
+ Please refer to the gloang documentation for details.
104
+
105
+ https://go.dev/security/fuzz/
106
+
107
+ ```ruby
108
+ module FooTest
109
+ # To enable fuzzing, the method name
110
+ # should be prefixed with `fuzz`.
111
+ def fuzz_any_func(f)
112
+ f.add(5, "hello")
113
+ f.fuzz do |t, i, s|
114
+ out, err = foo(i, s)
115
+ if err != nil && out != ""
116
+ t.errorf("%s, %s", out, err)
117
+ end
118
+ end
119
+ end
120
+ end
121
+ ```
122
+
96
123
  ## Example
97
124
 
98
125
  Rgot's example feature is the best and if you want to write the sample code of your library.
@@ -189,6 +216,8 @@ Method name should be set `test_*` for testing.
189
216
 
190
217
  And benchmark method should be set `benchmark_*`.
191
218
 
219
+ And fuzz method should be set `fuzz_*`.
220
+
192
221
  And example method should be set `example_*`.
193
222
 
194
223
  ```ruby
@@ -199,6 +228,9 @@ module XxxTest
199
228
  def benchmark_any_name(b)
200
229
  end
201
230
 
231
+ def fuzz_any_name(f)
232
+ end
233
+
202
234
  def example_any_name
203
235
  end
204
236
  end
@@ -218,6 +250,8 @@ Usage: rgot [options]
218
250
  --thread [count,...] set thread counts of comma split
219
251
  --require [path] load some code before running
220
252
  --load-path [path] Specify $LOAD_PATH directory
253
+ --fuzz [regexp] run the fuzz test matching `regexp`
254
+ --fuzztime [sec] time to spend fuzzing; default is to run indefinitely
221
255
  ```
222
256
 
223
257
  ## Basic
@@ -515,3 +549,39 @@ def benchmark_foo(b)
515
549
  end
516
550
  end
517
551
  ```
552
+
553
+ ## Rgot::F (Fuzzing)
554
+
555
+ ### Rgot::F#add
556
+
557
+ Set the sample value with `#add`. This value is also used as a test. It guesses the type from the value and generates a random value.
558
+
559
+ ### Rgot::F#fuzz
560
+
561
+ Generate the random value generated by `#fuzz` and execute the code.
562
+ The `t` becomes an instance of `Rgot::T` and the test can be run as usual.
563
+
564
+ ```ruby
565
+ def fuzz_foo(f)
566
+ f.add(100, "hello")
567
+ f.fuzz do |t, i, s|
568
+ i #=> 100, 84, 17, 9, 66, ...
569
+ s #=> "hello", "Y\xD5\xAB\xBA\x8E", "r\x95D\xA5\xF7", "\xCEj=\x9C\xBD", ...
570
+ if !foo(i, s)
571
+ t.error("fail with i=#{i}, s=#{s}")
572
+ end
573
+ end
574
+ end
575
+ ```
576
+
577
+ # TODO
578
+
579
+ - [ ] Support to save and load fuzzing data
580
+
581
+ ## v2
582
+
583
+ - [ ] Support sub testing
584
+ - [ ] Fix duration argument unit
585
+ - [ ] Refactoring
586
+ - [ ] Fix M#initialize argument
587
+ - [ ] Fix internal class API
data/Rakefile CHANGED
@@ -7,6 +7,7 @@ task :test do |t|
7
7
  "test/rgot_test.rb",
8
8
  "test/rgot_benchmark_test.rb",
9
9
  "test/rgot_example_test.rb",
10
+ "test/rgot_fuzzing_test.rb",
10
11
  ]
11
12
  ruby "bin/rgot -v #{targets.join(' ')}"
12
13
  end
data/Steepfile ADDED
@@ -0,0 +1,4 @@
1
+ target :lib do
2
+ check "lib"
3
+ signature "sig"
4
+ end
data/lib/rgot/b.rb CHANGED
@@ -2,13 +2,15 @@
2
2
 
3
3
  module Rgot
4
4
  class B < Common
5
- Options = Struct.new(
5
+ Options = _ = Struct.new(
6
6
  :procs,
7
7
  :threads,
8
8
  :benchtime,
9
9
  )
10
10
 
11
+ # @dynamic n, n=
11
12
  attr_accessor :n
13
+
12
14
  def initialize(benchmark_module, name, opts = Options.new)
13
15
  super()
14
16
  @n = 1
@@ -16,7 +18,7 @@ module Rgot
16
18
  @name = name
17
19
  @opts = opts
18
20
  @timer_on = false
19
- @duration = 0
21
+ @duration = 0.0
20
22
  @module.extend @module if @module
21
23
  end
22
24
 
@@ -38,34 +40,34 @@ module Rgot
38
40
  if @timer_on
39
41
  @start = Rgot.now
40
42
  end
41
- @duration = 0
43
+ @duration = 0.0
42
44
  end
43
45
 
44
46
  def run(&block)
45
47
  n = 1
46
48
  benchtime = (@opts.benchtime || 1).to_f
47
49
  catch(:skip) do
48
- run_n(n.to_i, block)
49
- while !failed? && @duration < benchtime && @n < 1e9
50
+ run_n(n, block)
51
+ while !failed? && @duration < benchtime && n < 1e9
50
52
  if @duration < (benchtime / 100.0)
51
- @n *= 100
53
+ n *= 100
52
54
  elsif @duration < (benchtime / 10.0)
53
- @n *= 10
55
+ n *= 10
54
56
  elsif @duration < (benchtime / 5.0)
55
- @n *= 5
57
+ n *= 5
56
58
  elsif @duration < (benchtime / 2.0)
57
- @n *= 2
59
+ n *= 2
58
60
  else
59
- if @n.to_i == 1
61
+ if n == 1
60
62
  break
61
63
  end
62
- @n *= 1.2
64
+ n = [(n * 1.2).to_i, n + 1].max || raise
63
65
  end
64
- run_n(@n.to_i, block)
66
+ run_n(n, block)
65
67
  end
66
68
  end
67
69
 
68
- BenchmarkResult.new(n: @n, t: @duration)
70
+ BenchmarkResult.new(n: n, t: @duration)
69
71
  end
70
72
 
71
73
  def run_parallel
@@ -83,7 +85,6 @@ module Rgot
83
85
  }.each(&:join)
84
86
  end
85
87
  end
86
- @n *= procs * threads
87
88
  Process.waitall
88
89
  end
89
90
 
@@ -2,6 +2,10 @@
2
2
 
3
3
  module Rgot
4
4
  class BenchmarkResult
5
+ # @dynamic n, t
6
+ attr_reader :n
7
+ attr_reader :t
8
+
5
9
  def initialize(n:, t:)
6
10
  @n = n
7
11
  @t = t
data/lib/rgot/cli.rb CHANGED
@@ -52,6 +52,15 @@ module Rgot
52
52
  o.on '--load-path [path]', "Specify $LOAD_PATH directory" do |arg|
53
53
  $LOAD_PATH.unshift(arg)
54
54
  end
55
+ o.on '--fuzz [regexp]', "run the fuzz test matching `regexp`" do |arg|
56
+ unless arg
57
+ raise Rgot::OptionError, "missing argument for flag --fuzz"
58
+ end
59
+ opts.fuzz = arg
60
+ end
61
+ o.on '--fuzztime [sec]', "time to spend fuzzing; default is to run indefinitely" do |arg|
62
+ opts.fuzztime = arg
63
+ end
55
64
  end
56
65
  parser.parse!(@argv)
57
66
 
@@ -93,12 +102,18 @@ module Rgot
93
102
  node = RubyVM::AbstractSyntaxTree.parse_file(testing_file).children[2]
94
103
  test_module_name = find_toplevel_name(node)
95
104
 
105
+ if opts.fuzz
106
+ # fuzzing observes changes in coverage.
107
+ require 'coverage'
108
+ Coverage.start(oneshot_lines: true)
109
+ end
96
110
  load testing_file
97
111
 
98
112
  test_module = Object.const_get(test_module_name)
99
113
  tests = []
100
114
  benchmarks = []
101
115
  examples = []
116
+ fuzz_targets = []
102
117
  main = nil
103
118
  methods = test_module.public_instance_methods
104
119
  methods.grep(/\Atest_/).each do |m|
@@ -117,7 +132,18 @@ module Rgot
117
132
  examples << Rgot::InternalExample.new(test_module, m)
118
133
  end
119
134
 
120
- m = Rgot::M.new(test_module: test_module, tests: tests, benchmarks: benchmarks, examples: examples, opts: opts)
135
+ methods.grep(/\Afuzz_/).each do |m|
136
+ fuzz_targets << Rgot::InternalFuzzTarget.new(test_module, m)
137
+ end
138
+
139
+ m = Rgot::M.new(
140
+ test_module: test_module,
141
+ tests: tests,
142
+ benchmarks: benchmarks,
143
+ examples: examples,
144
+ fuzz_targets: fuzz_targets,
145
+ opts: opts
146
+ )
121
147
  if main
122
148
  main.module.extend main.module
123
149
  main.module.instance_method(:test_main).bind(main.module).call(m)
@@ -126,8 +152,6 @@ module Rgot
126
152
  end
127
153
  end
128
154
 
129
- private
130
-
131
155
  def find_toplevel_name(node)
132
156
  case node.type
133
157
  when :MODULE
data/lib/rgot/common.rb CHANGED
@@ -5,6 +5,7 @@ require 'pathname'
5
5
 
6
6
  module Rgot
7
7
  class Common
8
+ # @dynamic output, output=
8
9
  attr_accessor :output
9
10
 
10
11
  def initialize
@@ -13,7 +14,7 @@ module Rgot
13
14
  @skipped = false
14
15
  @finished = false
15
16
  @start = Rgot.now
16
- @mutex = Mutex.new
17
+ @mutex = Thread::Mutex.new
17
18
  end
18
19
 
19
20
  def failed?
@@ -100,7 +101,7 @@ module Rgot
100
101
  # internal_log -> synchronize -> internal_log -> other log -> running method
101
102
  c = caller[4]
102
103
  path = c.sub(/:.*/, '')
103
- line = c.match(/:(\d+?):/)[1]
104
+ line = c.match(/:(\d+?):/)&.[](1)
104
105
  relative_path = Pathname.new(path).relative_path_from(Pathname.new(Dir.pwd)).to_s
105
106
  # Every line is indented at least 4 spaces.
106
107
  " #{relative_path}:#{line}: #{str}\n"
@@ -4,7 +4,10 @@ require 'ripper'
4
4
 
5
5
  module Rgot
6
6
  class ExampleParser < Ripper
7
+
8
+ # @dynamic examples, examples=
7
9
  attr_accessor :examples
10
+
8
11
  def initialize(code)
9
12
  super
10
13
  @examples = []
data/lib/rgot/f.rb ADDED
@@ -0,0 +1,230 @@
1
+ module Rgot
2
+ # def fuzz_foo(f)
3
+ # f.add(5, "hello")
4
+ # f.fuzz do |t, i, s|
5
+ # ...
6
+ # end
7
+ class F < Common
8
+ class Options
9
+ # @dynamic fuzz, fuzz=, fuzztime, fuzztime=
10
+ attr_accessor :fuzz
11
+ attr_accessor :fuzztime
12
+ def initialize(fuzz:, fuzztime:)
13
+ @fuzz = fuzz
14
+ @fuzztime = fuzztime
15
+ end
16
+ end
17
+
18
+ class CorpusEntry
19
+ # @dynamic values, values=, is_seed, is_seed=, path, path=
20
+ attr_accessor :values
21
+ attr_accessor :is_seed
22
+ attr_accessor :path
23
+ def initialize(values:, is_seed:, path:)
24
+ @values = values
25
+ @is_seed = is_seed
26
+ @path = path
27
+ end
28
+
29
+ def mutate_values
30
+ @values.map do |value|
31
+ if generator = SUPPORTED_TYPES[value.class]
32
+ generator.call(value)
33
+ else
34
+ raise "unsupported type #{value.class}"
35
+ end
36
+ end
37
+ end
38
+ end
39
+
40
+ class Coordinator
41
+ # @dynamic count, count=, interesting_count, interesting_count=
42
+ attr_accessor :count
43
+ attr_accessor :interesting_count
44
+
45
+ def initialize(warmup_input_count:)
46
+ @warmup_input_count = warmup_input_count
47
+ @before_cov = 0
48
+ @start_time = Rgot.now
49
+ @count = 0
50
+ @interesting_count = 0
51
+ @count_last_log = 0
52
+ @time_last_log = 0.0
53
+ end
54
+
55
+ def start_logger
56
+ Thread.new do
57
+ loop do
58
+ log_stats
59
+ sleep 3
60
+ end
61
+ end
62
+ end
63
+
64
+ def diff_coverage
65
+ current_cov = Coverage.peek_result.sum do |path, hash|
66
+ hash.map do |_, covs|
67
+ covs.length
68
+ end.sum
69
+ end
70
+ (current_cov - @before_cov).tap { @before_cov = current_cov }
71
+ end
72
+
73
+ def log_stats
74
+ rate = Float(count - @count_last_log) / (Rgot.now - @time_last_log)
75
+ total = @warmup_input_count + interesting_count
76
+ printf "fuzz: elapsed: %ds, execs: %d (%d/sec), new interesting: %d (total: %d)\n",
77
+ elapsed, count, rate, interesting_count, total
78
+
79
+ duration = Rgot.now - @time_last_log
80
+ @count_last_log = count
81
+ @time_last_log = Rgot.now
82
+ end
83
+
84
+ private
85
+
86
+ def elapsed
87
+ (Rgot.now - @start_time).round
88
+ end
89
+ end
90
+
91
+ SUPPORTED_TYPES = {
92
+ TrueClass => ->(v) { [true, false].sample },
93
+ FalseClass => ->(v) { [true, false].sample },
94
+ Integer => ->(v) { Random.rand(v) },
95
+ Float => ->(v) { Random.rand(v) },
96
+ String => ->(v) { Random.bytes(v.length) },
97
+ }
98
+
99
+ # @dynamic name
100
+ attr_reader :name
101
+
102
+ def initialize(fuzz_target:, opts:)
103
+ super()
104
+ @opts = opts
105
+ @fuzz_target = fuzz_target
106
+ @fuzz_block = nil
107
+ @module = fuzz_target.module
108
+ @name = fuzz_target.name
109
+ @corpus = []
110
+ end
111
+
112
+ # TODO: DRY with T
113
+ def run
114
+ catch(:skip) { call }
115
+ finish!
116
+ rescue => e
117
+ fail!
118
+ raise e
119
+ end
120
+
121
+ def run_testing
122
+ run
123
+ report if !fuzz? || failed?
124
+ end
125
+
126
+ def run_fuzzing
127
+ return unless fuzz?
128
+ raise("must call after #fuzz") unless @fuzz_block
129
+
130
+ coordinator = Coordinator.new(
131
+ warmup_input_count: @corpus.length
132
+ )
133
+ coordinator.start_logger
134
+
135
+ t = T.new(@fuzz_target.module, @fuzz_target.name)
136
+
137
+ begin
138
+ Timeout.timeout(@opts.fuzztime.to_f) do
139
+ loop do
140
+ @corpus.each do |entry|
141
+ values = entry.mutate_values
142
+
143
+ @fuzz_block.call(t, *values)
144
+
145
+ if 0 < coordinator.diff_coverage
146
+ coordinator.interesting_count += 1
147
+ end
148
+ coordinator.count += 1
149
+ fail! if t.failed?
150
+ end
151
+ end
152
+ end
153
+ rescue Timeout::Error, Interrupt
154
+ coordinator.log_stats
155
+ end
156
+
157
+ report
158
+ end
159
+
160
+ def add(*args)
161
+ args.each do |arg|
162
+ unless SUPPORTED_TYPES.key?(arg.class)
163
+ raise "unsupported type to Add #{arg.class}"
164
+ end
165
+ end
166
+ entry = CorpusEntry.new(
167
+ values: args.dup,
168
+ is_seed: true,
169
+ path: "seed##{@corpus.length}"
170
+ )
171
+ @corpus.push(entry)
172
+ end
173
+
174
+ def fuzz(&block)
175
+ unless block
176
+ raise LocalJumpError, "must set block"
177
+ end
178
+ unless 2 <= block.arity
179
+ raise "fuzz target must receive at least two arguments"
180
+ end
181
+
182
+ t = T.new(@fuzz_target.module, @fuzz_target.name)
183
+
184
+ @corpus.each do |entry|
185
+ unless entry.values.length == (block.arity - 1)
186
+ raise "wrong number of values in corpus entry: #{entry.values.length}, want #{block.arity - 1}"
187
+ end
188
+ block.call(t, *entry.values.dup)
189
+ fail! if t.failed?
190
+ end
191
+
192
+ @fuzz_block = block
193
+
194
+ nil
195
+ end
196
+
197
+ def fuzz?
198
+ return false unless @opts.fuzz
199
+ return false unless Regexp.new(@opts.fuzz.to_s).match?(@fuzz_target.name)
200
+ true
201
+ end
202
+
203
+ def report
204
+ puts @output if Rgot.verbose? && !@output.empty?
205
+ duration = Rgot.now - @start
206
+ template = "--- \e[%sm%s\e[m: %s (%.2fs)\n"
207
+ if failed?
208
+ printf template, [41, 1].join(';'), "FAIL", @name, duration
209
+ elsif Rgot.verbose?
210
+ if skipped?
211
+ printf template, [44, 1].join(';'), "SKIP", @name, duration
212
+ else
213
+ printf template, [42, 1].join(';'), "PASS", @name, duration
214
+ end
215
+ end
216
+ end
217
+
218
+ private
219
+
220
+ def call
221
+ test_method = @module.instance_method(@name).bind(@module)
222
+ if test_method.arity == 0
223
+ path, line = test_method.source_location
224
+ warn "#{path}:#{line} `#{test_method.name}' is not running. It's a testing method name, But not have argument"
225
+ else
226
+ test_method.call(self)
227
+ end
228
+ end
229
+ end
230
+ end
data/lib/rgot/m.rb CHANGED
@@ -12,9 +12,15 @@ module Rgot
12
12
  :timeout,
13
13
  :cpu,
14
14
  :thread,
15
+ :fuzz,
16
+ :fuzztime,
15
17
  ); end
16
18
 
17
- def initialize(tests:, benchmarks:, examples:, test_module: nil, opts: Options.new)
19
+ def initialize(tests:, benchmarks:, examples:, fuzz_targets: nil, test_module: nil, opts: Options.new)
20
+ unless fuzz_targets
21
+ raise "Require `fuzz_targets` keyword" if Gem::Version.new("2.0") <= Gem::Version.new(Rgot::VERSION)
22
+ warn "`Rgot::M#initialize` will require the `fuzz_targets` keyword in the next major version."
23
+ end
18
24
  unless test_module
19
25
  raise "Require `test_module` keyword" if Gem::Version.new("2.0") <= Gem::Version.new(Rgot::VERSION)
20
26
  warn "`Rgot::M#initialize` will require the `test_module` keyword in the next major version."
@@ -23,18 +29,30 @@ module Rgot
23
29
  @tests = tests
24
30
  @benchmarks = benchmarks
25
31
  @examples = examples
32
+ @fuzz_targets = fuzz_targets || []
26
33
  @test_module = test_module
27
34
  @opts = opts
28
- @cpu_list = nil
29
- @thread_list = nil
35
+
36
+ @cpu_list = []
37
+ @thread_list = []
38
+ @fs = @fuzz_targets.map do |fuzz_target|
39
+ F.new(
40
+ fuzz_target: fuzz_target,
41
+ opts: F::Options.new(
42
+ fuzz: opts.fuzz,
43
+ fuzztime: opts.fuzztime,
44
+ )
45
+ )
46
+ end
30
47
  end
31
48
 
32
49
  def run
33
50
  duration = Rgot.now
34
51
  test_ok = false
52
+ fuzz_targets_ok = false
35
53
  example_ok = false
36
54
 
37
- if @tests.empty? && @benchmarks.empty? && @examples.empty?
55
+ if @tests.empty? && @benchmarks.empty? && @examples.empty? && @fuzz_targets.empty?
38
56
  warn "rgot: warning: no tests to run"
39
57
  end
40
58
 
@@ -47,22 +65,29 @@ module Rgot
47
65
 
48
66
  Timeout.timeout(@opts.timeout.to_f) do
49
67
  test_ok = run_tests
68
+ fuzz_targets_ok = run_fuzz_tests
50
69
  example_ok = run_examples
51
70
  end
52
71
 
53
- if !test_ok || !example_ok
72
+ if !test_ok || !example_ok || !fuzz_targets_ok
54
73
  puts "FAIL"
55
74
  puts "exit status 1"
56
75
  puts sprintf("%s\t%s\t%.3fs", "FAIL", @test_module, Rgot.now - duration)
76
+ return 1
77
+ end
57
78
 
58
- 1
59
- else
60
- puts "PASS"
61
- run_benchmarks
62
- puts sprintf("%s\t%s\t%.3fs", "ok ", @test_module, Rgot.now - duration)
63
-
64
- 0
79
+ if !run_fuzzing()
80
+ puts "FAIL"
81
+ puts "exit status 1"
82
+ puts sprintf("%s\t%s\t%.3fs", "FAIL", @test_module, Rgot.now - duration)
83
+ return 1
65
84
  end
85
+
86
+ puts "PASS"
87
+ run_benchmarks
88
+ puts sprintf("%s\t%s\t%.3fs", "ok ", @test_module, Rgot.now - duration)
89
+
90
+ 0
66
91
  end
67
92
 
68
93
  private
@@ -90,9 +115,7 @@ module Rgot
90
115
  puts "=== RUN #{test.name}"
91
116
  end
92
117
  t.run
93
- if t.failed?
94
- ok = false
95
- end
118
+ ok = ok && !t.failed?
96
119
  end
97
120
  ok
98
121
  end
@@ -130,6 +153,50 @@ module Rgot
130
153
  ok
131
154
  end
132
155
 
156
+ def run_fuzz_tests
157
+ ok = true
158
+ @fs.each do |f|
159
+ if Rgot.verbose?
160
+ if f.fuzz?
161
+ puts "=== FUZZ #{f.name}"
162
+ else
163
+ puts "=== RUN #{f.name}"
164
+ end
165
+ end
166
+ f.run_testing
167
+ ok = ok && !f.failed?
168
+ end
169
+ ok
170
+ end
171
+
172
+ def run_fuzzing
173
+ if @fuzz_targets.empty? || @opts.fuzz.nil?
174
+ return true
175
+ end
176
+
177
+ fuzzing_fs = @fs.select(&:fuzz?)
178
+
179
+ if fuzzing_fs.empty?
180
+ puts "rgot: warning: no fuzz tests to fuzz"
181
+ return true
182
+ end
183
+
184
+ if fuzzing_fs.length > 1
185
+ names = fuzzing_fs.map(&:name)
186
+ puts "rgot: will not fuzz, --fuzz matches more than one fuzz test: #{names.inspect}"
187
+ return false
188
+ end
189
+
190
+ ok = true
191
+
192
+ fuzzing_fs.each do |f|
193
+ f.run_fuzzing
194
+ ok = ok && !f.failed?
195
+ end
196
+
197
+ ok
198
+ end
199
+
133
200
  def run_examples
134
201
  ok = true
135
202
  @examples.each do |example|
@@ -143,13 +210,13 @@ module Rgot
143
210
  out, _ = capture do
144
211
  method.call
145
212
  end
146
- file = method.source_location[0]
147
- r = ExampleParser.new(File.read(file))
148
- r.parse
149
- e = r.examples.find { |re| re.name == example.name }
213
+ file = method.source_location&.[](0) or raise("bug")
214
+ example_parser = ExampleParser.new(File.read(file))
215
+ example_parser.parse
216
+ e = example_parser.examples.find { |er| er.name == example.name } or raise("bug")
150
217
 
151
218
  duration = Rgot.now - start
152
- if e && e.output.strip != out.strip
219
+ if e.output.strip != out.strip
153
220
  printf("--- FAIL: %s (%.2fs)\n", e.name, duration)
154
221
  ok = false
155
222
  puts "got:"
data/lib/rgot/pb.rb CHANGED
@@ -2,8 +2,6 @@
2
2
 
3
3
  module Rgot
4
4
  class PB
5
- attr_accessor :bn
6
-
7
5
  def initialize(bn:)
8
6
  @bn = bn
9
7
  end
data/lib/rgot/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Rgot
4
- VERSION = "1.2.0"
4
+ VERSION = "1.3.0"
5
5
  end
data/lib/rgot.rb CHANGED
@@ -8,12 +8,14 @@ module Rgot
8
8
  autoload :B, 'rgot/b'
9
9
  autoload :PB, 'rgot/pb'
10
10
  autoload :BenchmarkResult, 'rgot/benchmark_result'
11
+ autoload :F, 'rgot/f'
11
12
  autoload :ExampleParser, 'rgot/example_parser'
12
13
 
13
14
  OptionError = Class.new(StandardError)
14
15
  InternalTest = Struct.new(:module, :name)
15
16
  InternalBenchmark = Struct.new(:module, :name)
16
17
  InternalExample = Struct.new(:module, :name)
18
+ InternalFuzzTarget = Struct.new(:module, :name)
17
19
  ExampleOutput = Struct.new(:name, :output)
18
20
 
19
21
  class << self
@@ -0,0 +1,49 @@
1
+ ---
2
+ sources:
3
+ - name: ruby/gem_rbs_collection
4
+ remote: https://github.com/ruby/gem_rbs_collection.git
5
+ revision: main
6
+ repo_dir: gems
7
+ path: ".gem_rbs_collection"
8
+ gems:
9
+ - name: pathname
10
+ version: '0'
11
+ source:
12
+ type: stdlib
13
+ - name: optparse
14
+ version: '0'
15
+ source:
16
+ type: stdlib
17
+ - name: timeout
18
+ version: '0'
19
+ source:
20
+ type: stdlib
21
+ - name: etc
22
+ version: '0'
23
+ source:
24
+ type: stdlib
25
+ - name: coverage
26
+ version: '0'
27
+ source:
28
+ type: stdlib
29
+ - name: concurrent-ruby
30
+ version: '1.1'
31
+ source:
32
+ type: git
33
+ name: ruby/gem_rbs_collection
34
+ revision: 9576ce5b109170f1ba8a42671bfafb64ab95bd23
35
+ remote: https://github.com/ruby/gem_rbs_collection.git
36
+ repo_dir: gems
37
+ - name: io-console
38
+ version: '0'
39
+ source:
40
+ type: stdlib
41
+ - name: logger
42
+ version: '0'
43
+ source:
44
+ type: stdlib
45
+ - name: monitor
46
+ version: '0'
47
+ source:
48
+ type: stdlib
49
+ gemfile_lock_path: Gemfile.lock
@@ -0,0 +1,53 @@
1
+ # Download sources
2
+ sources:
3
+ - name: ruby/gem_rbs_collection
4
+ remote: https://github.com/ruby/gem_rbs_collection.git
5
+ revision: main
6
+ repo_dir: gems
7
+
8
+ # A directory to install the downloaded RBSs
9
+ path: .gem_rbs_collection
10
+
11
+ gems:
12
+ - name: pathname
13
+ - name: optparse
14
+ - name: timeout
15
+ - name: etc
16
+ - name: coverage
17
+
18
+ # Skip loading rbs gem's RBS.
19
+ # It's unnecessary if you don't use rbs as a library.
20
+ - name: rbs
21
+ ignore: true
22
+ - name: steep
23
+ ignore: true
24
+ - name: activesupport
25
+ ignore: true
26
+ - name: ast
27
+ ignore: true
28
+ - name: csv
29
+ ignore: true
30
+ - name: i18n
31
+ ignore: true
32
+ - name: json
33
+ ignore: true
34
+ - name: listen
35
+ ignore: true
36
+ - name: fileutils
37
+ ignore: true
38
+ - name: minitest
39
+ ignore: true
40
+ - name: parallel
41
+ ignore: true
42
+ - name: rainbow
43
+ ignore: true
44
+ - name: rgot
45
+ ignore: true
46
+ - name: securerandom
47
+ ignore: true
48
+ - name: strscan
49
+ ignore: true
50
+ - name: forwardable
51
+ ignore: true
52
+ - name: mutex_m
53
+ ignore: true
data/sig/patch.rbs ADDED
@@ -0,0 +1,3 @@
1
+ # patch
2
+ class Ripper
3
+ end
data/sig/rgot.rbs ADDED
@@ -0,0 +1,232 @@
1
+ module Rgot
2
+ VERSION: "1.2.0"
3
+
4
+ def self.now: () -> Float
5
+ def self.benchmark: (?Hash[Symbol, String] opts_hash) { (B) -> void } -> BenchmarkResult
6
+ def self.verbose?: () -> bool
7
+
8
+ class Cli
9
+ @argv: Array[String]
10
+
11
+ def initialize: (untyped argv) -> void
12
+ def run: () -> void
13
+
14
+ private
15
+
16
+ def parse_option: (Rgot::M::Options opts) -> void
17
+ def main_process: (Rgot::M::Options opts) -> void
18
+ def testing_files: () -> Array[String]
19
+ def child_process: (Rgot::M::Options opts, String testing_file) -> Integer
20
+ # `node` is RubyVM::AbstractSyntaxTree::Node
21
+ def find_toplevel_name: (untyped node) -> Symbol
22
+ end
23
+
24
+ class Common
25
+ @failed: bool
26
+ @skipped: bool
27
+ @finished: bool
28
+ @start: Float
29
+ @mutex: Thread::Mutex
30
+
31
+ attr_accessor output: String
32
+
33
+ def initialize: () -> void
34
+
35
+ def failed?: () -> bool
36
+ def skipped?: () -> bool
37
+ def finished?: () -> bool
38
+ def fail!: () -> void
39
+ def skip!: () -> void
40
+ def finish!: () -> void
41
+ def log: (*untyped) -> nil
42
+ def logf: (*untyped) -> nil
43
+ def error: (*untyped) -> nil
44
+ def errorf: (*untyped) -> nil
45
+ def fatal: (*untyped) -> bot
46
+ def fatalf: (*untyped) -> bot
47
+ def skip: (*untyped) -> bot
48
+ def skipf: (*untyped) -> bot
49
+ def skip_now: () -> bot
50
+ def fail_now: () -> bot
51
+
52
+ private
53
+
54
+ def decorate: (String) -> String
55
+ def internal_log: (String msg) -> void
56
+ end
57
+
58
+ class M
59
+ class Options
60
+ attr_accessor bench: String?
61
+ attr_accessor benchtime: String?
62
+ attr_accessor timeout: String?
63
+ attr_accessor cpu: String?
64
+ attr_accessor thread: String?
65
+ attr_accessor fuzz: String?
66
+ attr_accessor fuzztime: String?
67
+ end
68
+
69
+ @tests: Array[InternalTest]
70
+ @benchmarks: Array[InternalBenchmark]
71
+ @examples: Array[InternalExample]
72
+ @fuzz_targets: Array[InternalFuzzTarget]
73
+ @fs: Array[F]
74
+ @test_module: Module?
75
+ @opts: M::Options
76
+ @cpu_list: Array[Integer]
77
+ @thread_list: Array[Integer]
78
+
79
+ def initialize: (
80
+ tests: Array[InternalTest],
81
+ benchmarks: Array[InternalBenchmark],
82
+ examples: Array[InternalExample],
83
+ fuzz_targets: Array[InternalFuzzTarget],
84
+ ?test_module: Module?,
85
+ ?opts: Options
86
+ ) -> void
87
+ def run: () -> Integer
88
+
89
+ private
90
+
91
+ def run_tests: () -> bool
92
+ def run_benchmarks: () -> bool
93
+ def run_fuzz_tests: () -> bool
94
+ def run_fuzzing: () -> bool
95
+ def run_examples: () -> bool
96
+ end
97
+
98
+ class T < Common
99
+ @module: Module
100
+ @name: Symbol
101
+
102
+ def initialize: (Module test_module, Symbol name) -> void
103
+ def run: () -> void
104
+ def report: () -> void
105
+ def call: () -> void
106
+ end
107
+
108
+ class B < Common
109
+ class Options
110
+ attr_accessor procs: Integer
111
+ attr_accessor threads: Integer
112
+ attr_accessor benchtime: String?
113
+ end
114
+
115
+ @module: Module?
116
+ @name: Symbol?
117
+ @opts: B::Options
118
+ @timer_on: bool
119
+ @duration: Float
120
+
121
+ attr_accessor n: Integer
122
+
123
+ def initialize: (Module? benchmark_module, Symbol? name, ?B::Options opts) -> void
124
+ def start_timer: () -> void
125
+ def stop_timer: () -> void
126
+ def reset_timer: () -> void
127
+ def run: () ?{ (B) -> void } -> BenchmarkResult
128
+ def run_parallel: () { (PB) -> void } -> void
129
+
130
+ private
131
+
132
+ def run_n: (Integer n, ?Proc? block) -> void
133
+ end
134
+
135
+ class F < Common
136
+ class Options
137
+ attr_accessor fuzz: String?
138
+ attr_accessor fuzztime: String?
139
+ def initialize: (fuzz: String?, fuzztime: String?) -> void
140
+ end
141
+ class CorpusEntry
142
+ attr_accessor values: Array[untyped]
143
+ attr_accessor is_seed: bool
144
+ attr_accessor path: String
145
+ def initialize: (values: Array[untyped], is_seed: bool, path: String) -> void
146
+ def mutate_values: () -> Array[untyped]
147
+ end
148
+ class Coordinator
149
+ attr_accessor count: Integer
150
+ attr_accessor interesting_count: Integer
151
+ def initialize: (warmup_input_count: Integer) -> void
152
+ @warmup_input_count: Integer
153
+ @before_cov: Integer
154
+ @start_time: Float
155
+ @count: Integer
156
+ @interesting_count: Integer
157
+ @count_last_log: Integer
158
+ @time_last_log: Float
159
+ def start_logger: () -> void
160
+ def diff_coverage: () -> Integer
161
+ def log_stats: () -> void
162
+ private
163
+ def elapsed: () -> Integer
164
+ end
165
+ SUPPORTED_TYPES: Hash[untyped, ^(untyped) -> untyped]
166
+ @fuzz_target: InternalFuzzTarget
167
+ @fuzz_block: Proc?
168
+ @opts: F::Options
169
+ @corpus: Array[CorpusEntry]
170
+ @module: Module
171
+ attr_reader name: Symbol
172
+ def initialize: (fuzz_target: InternalFuzzTarget, opts: F::Options) -> void
173
+ def run: () -> void
174
+ def run_testing: () -> void
175
+ def run_fuzzing: () -> void
176
+ def add: (*untyped) -> void
177
+ def fuzz: () { (*untyped) -> void } -> void
178
+ def fuzz?: () -> bool
179
+ def report: () -> void
180
+
181
+ private
182
+
183
+ def call: () -> void
184
+ end
185
+
186
+ class PB
187
+ @bn: Integer
188
+ def initialize: (bn: Integer) -> void
189
+ def next: () -> bool
190
+ end
191
+
192
+ class BenchmarkResult
193
+ attr_reader n: Integer # int // The number of iterations.
194
+ attr_reader t: Float # time.Duration // The total time taken.
195
+ def initialize: (n: Integer, t: Float) -> void
196
+ end
197
+
198
+ class ExampleParser < Ripper
199
+ @in_def: bool
200
+ @has_output: bool
201
+ @output: String
202
+ attr_accessor examples: Array[ExampleOutput]
203
+ end
204
+
205
+ class OptionError < StandardError
206
+ end
207
+
208
+ class InternalTest
209
+ attr_accessor module: Module
210
+ attr_accessor name: Symbol
211
+ end
212
+
213
+ class InternalBenchmark
214
+ attr_accessor module: Module
215
+ attr_accessor name: Symbol
216
+ end
217
+
218
+ class InternalExample
219
+ attr_accessor module: Module
220
+ attr_accessor name: Symbol
221
+ end
222
+
223
+ class InternalFuzzTarget
224
+ attr_accessor module: Module
225
+ attr_accessor name: Symbol
226
+ end
227
+
228
+ class ExampleOutput
229
+ attr_accessor output: String
230
+ attr_accessor name: Symbol
231
+ end
232
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rgot
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.0
4
+ version: 1.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - ksss
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-12-27 00:00:00.000000000 Z
11
+ date: 2023-01-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -53,6 +53,7 @@ files:
53
53
  - LICENSE.txt
54
54
  - README.md
55
55
  - Rakefile
56
+ - Steepfile
56
57
  - bin/rgot
57
58
  - lib/rgot.rb
58
59
  - lib/rgot/b.rb
@@ -60,11 +61,16 @@ files:
60
61
  - lib/rgot/cli.rb
61
62
  - lib/rgot/common.rb
62
63
  - lib/rgot/example_parser.rb
64
+ - lib/rgot/f.rb
63
65
  - lib/rgot/m.rb
64
66
  - lib/rgot/pb.rb
65
67
  - lib/rgot/t.rb
66
68
  - lib/rgot/version.rb
69
+ - rbs_collection.lock.yaml
70
+ - rbs_collection.yaml
67
71
  - rgot.gemspec
72
+ - sig/patch.rbs
73
+ - sig/rgot.rbs
68
74
  homepage: https://github.com/ksss/rgot
69
75
  licenses:
70
76
  - MIT
@@ -84,7 +90,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
84
90
  - !ruby/object:Gem::Version
85
91
  version: '0'
86
92
  requirements: []
87
- rubygems_version: 3.3.26
93
+ rubygems_version: 3.4.1
88
94
  signing_key:
89
95
  specification_version: 4
90
96
  summary: Ruby + Golang Testing = Rgot