rgot 1.2.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 368b98f9969eee4d2608df53e668eaaed9f06320ecd5b26096ff05c0f81dc16e
4
- data.tar.gz: b045a87ffcb3f18338ce0286e2f22b71921b5bab011e4049c203164b9ab7a077
3
+ metadata.gz: e2f0075689755b0a44db0674a5069769d1ee4c2567774d7bb7422ba5b27c8960
4
+ data.tar.gz: 9ddd64091ca0daeff256fcc5c305aad378a92472f4522b875512fb35915d74aa
5
5
  SHA512:
6
- metadata.gz: 7a3d443886aa26702be0720d8bdf8abf38dd938e3707d595ac1bf21b1edd0f3cb52f64f29e4ca11996357d8671bfaaa4edc94500560510b67c393244202a2db3
7
- data.tar.gz: 13d8fcdf048c4c82da186f96dc7bfef3eb78a35fb5df04a2f6f17badda63100587df8cd175dc0618d823d613526b9e60aeec8e18cc09075c8bf3c8c218d58841
6
+ metadata.gz: 2e24662ccc2f6f4644eec95f9764975b86d2ea40f49c3c8ae569f428809ca08df79e6517d2bbcf0ec7d08e610867972133a1a9ab5df245cf36896afb8ace7e63
7
+ data.tar.gz: ac5c041deb124812d911d0af697a06a398ced2c79bc54f6058ec4cb2499af9dd814007f3e0c8951c50448892ba0fbc0a6846dfd879c3497601b4aebf5d00116e
@@ -17,6 +17,7 @@ jobs:
17
17
  - '2.7.7'
18
18
  - '3.0.5'
19
19
  - '3.1.3'
20
+ - '3.2.0'
20
21
 
21
22
  steps:
22
23
  - uses: actions/checkout@v3
data/.gitignore CHANGED
@@ -1,2 +1,3 @@
1
1
  pkg
2
2
  tmp
3
+ /.gem_rbs_collection
data/Gemfile CHANGED
@@ -1,3 +1,7 @@
1
1
  source 'https://rubygems.org'
2
2
 
3
3
  gemspec
4
+
5
+ gem "rbs"
6
+ gem "steep"
7
+ gem "debug"
data/Gemfile.lock CHANGED
@@ -1,20 +1,73 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- rgot (1.2.0)
4
+ rgot (1.3.0)
5
5
 
6
6
  GEM
7
7
  remote: https://rubygems.org/
8
8
  specs:
9
+ activesupport (7.0.4)
10
+ concurrent-ruby (~> 1.0, >= 1.0.2)
11
+ i18n (>= 1.6, < 2)
12
+ minitest (>= 5.1)
13
+ tzinfo (~> 2.0)
14
+ ast (2.4.2)
15
+ concurrent-ruby (1.1.10)
16
+ csv (3.2.6)
17
+ debug (1.7.1)
18
+ ffi (1.15.5)
19
+ fileutils (1.7.0)
20
+ i18n (1.12.0)
21
+ concurrent-ruby (~> 1.0)
22
+ json (2.6.3)
23
+ language_server-protocol (3.17.0.2)
24
+ listen (3.7.1)
25
+ rb-fsevent (~> 0.10, >= 0.10.3)
26
+ rb-inotify (~> 0.9, >= 0.9.10)
27
+ logger (1.5.3)
28
+ minitest (5.17.0)
29
+ parallel (1.22.1)
30
+ parser (3.2.0.0)
31
+ ast (~> 2.4.1)
32
+ rainbow (3.1.1)
9
33
  rake (13.0.6)
34
+ rb-fsevent (0.11.2)
35
+ rb-inotify (0.10.1)
36
+ ffi (~> 1.0)
37
+ rbs (2.8.2)
38
+ securerandom (0.2.2)
39
+ steep (1.3.0)
40
+ activesupport (>= 5.1)
41
+ csv (>= 3.0.9)
42
+ fileutils (>= 1.1.0)
43
+ json (>= 2.1.0)
44
+ language_server-protocol (>= 3.15, < 4.0)
45
+ listen (~> 3.0)
46
+ logger (>= 1.3.0)
47
+ parallel (>= 1.0.0)
48
+ parser (>= 3.1)
49
+ rainbow (>= 2.2.2, < 4.0)
50
+ rbs (>= 2.8.0)
51
+ securerandom (>= 0.1)
52
+ strscan (>= 1.0.0)
53
+ terminal-table (>= 2, < 4)
54
+ strscan (3.0.5)
55
+ terminal-table (3.0.2)
56
+ unicode-display_width (>= 1.1.1, < 3)
57
+ tzinfo (2.0.5)
58
+ concurrent-ruby (~> 1.0)
59
+ unicode-display_width (2.4.2)
10
60
 
11
61
  PLATFORMS
12
62
  ruby
13
63
 
14
64
  DEPENDENCIES
15
65
  bundler
66
+ debug
16
67
  rake
68
+ rbs
17
69
  rgot!
70
+ steep
18
71
 
19
72
  BUNDLED WITH
20
- 2.3.26
73
+ 2.4.2
data/README.md CHANGED
@@ -93,6 +93,33 @@ ok FooTest 2.782s
93
93
 
94
94
  `b.n` is automatically adjusted.
95
95
 
96
+ ## Fuzzing
97
+
98
+ ```
99
+ $ rgot target_file_test.rb --fuzz . --fuzztime 1
100
+ ```
101
+
102
+ Fuzzing tests are also supported.
103
+ Please refer to the gloang documentation for details.
104
+
105
+ https://go.dev/security/fuzz/
106
+
107
+ ```ruby
108
+ module FooTest
109
+ # To enable fuzzing, the method name
110
+ # should be prefixed with `fuzz`.
111
+ def fuzz_any_func(f)
112
+ f.add(5, "hello")
113
+ f.fuzz do |t, i, s|
114
+ out, err = foo(i, s)
115
+ if err != nil && out != ""
116
+ t.errorf("%s, %s", out, err)
117
+ end
118
+ end
119
+ end
120
+ end
121
+ ```
122
+
96
123
  ## Example
97
124
 
98
125
  Rgot's example feature is the best and if you want to write the sample code of your library.
@@ -189,6 +216,8 @@ Method name should be set `test_*` for testing.
189
216
 
190
217
  And benchmark method should be set `benchmark_*`.
191
218
 
219
+ And fuzz method should be set `fuzz_*`.
220
+
192
221
  And example method should be set `example_*`.
193
222
 
194
223
  ```ruby
@@ -199,6 +228,9 @@ module XxxTest
199
228
  def benchmark_any_name(b)
200
229
  end
201
230
 
231
+ def fuzz_any_name(f)
232
+ end
233
+
202
234
  def example_any_name
203
235
  end
204
236
  end
@@ -218,6 +250,8 @@ Usage: rgot [options]
218
250
  --thread [count,...] set thread counts of comma split
219
251
  --require [path] load some code before running
220
252
  --load-path [path] Specify $LOAD_PATH directory
253
+ --fuzz [regexp] run the fuzz test matching `regexp`
254
+ --fuzztime [sec] time to spend fuzzing; default is to run indefinitely
221
255
  ```
222
256
 
223
257
  ## Basic
@@ -515,3 +549,39 @@ def benchmark_foo(b)
515
549
  end
516
550
  end
517
551
  ```
552
+
553
+ ## Rgot::F (Fuzzing)
554
+
555
+ ### Rgot::F#add
556
+
557
+ Set the sample value with `#add`. This value is also used as a test. It guesses the type from the value and generates a random value.
558
+
559
+ ### Rgot::F#fuzz
560
+
561
+ Generate the random value generated by `#fuzz` and execute the code.
562
+ The `t` becomes an instance of `Rgot::T` and the test can be run as usual.
563
+
564
+ ```ruby
565
+ def fuzz_foo(f)
566
+ f.add(100, "hello")
567
+ f.fuzz do |t, i, s|
568
+ i #=> 100, 84, 17, 9, 66, ...
569
+ s #=> "hello", "Y\xD5\xAB\xBA\x8E", "r\x95D\xA5\xF7", "\xCEj=\x9C\xBD", ...
570
+ if !foo(i, s)
571
+ t.error("fail with i=#{i}, s=#{s}")
572
+ end
573
+ end
574
+ end
575
+ ```
576
+
577
+ # TODO
578
+
579
+ - [ ] Support to save and load fuzzing data
580
+
581
+ ## v2
582
+
583
+ - [ ] Support sub testing
584
+ - [ ] Fix duration argument unit
585
+ - [ ] Refactoring
586
+ - [ ] Fix M#initialize argument
587
+ - [ ] Fix internal class API
data/Rakefile CHANGED
@@ -7,6 +7,7 @@ task :test do |t|
7
7
  "test/rgot_test.rb",
8
8
  "test/rgot_benchmark_test.rb",
9
9
  "test/rgot_example_test.rb",
10
+ "test/rgot_fuzzing_test.rb",
10
11
  ]
11
12
  ruby "bin/rgot -v #{targets.join(' ')}"
12
13
  end
data/Steepfile ADDED
@@ -0,0 +1,4 @@
1
+ target :lib do
2
+ check "lib"
3
+ signature "sig"
4
+ end
data/lib/rgot/b.rb CHANGED
@@ -2,13 +2,15 @@
2
2
 
3
3
  module Rgot
4
4
  class B < Common
5
- Options = Struct.new(
5
+ Options = _ = Struct.new(
6
6
  :procs,
7
7
  :threads,
8
8
  :benchtime,
9
9
  )
10
10
 
11
+ # @dynamic n, n=
11
12
  attr_accessor :n
13
+
12
14
  def initialize(benchmark_module, name, opts = Options.new)
13
15
  super()
14
16
  @n = 1
@@ -16,7 +18,7 @@ module Rgot
16
18
  @name = name
17
19
  @opts = opts
18
20
  @timer_on = false
19
- @duration = 0
21
+ @duration = 0.0
20
22
  @module.extend @module if @module
21
23
  end
22
24
 
@@ -38,34 +40,34 @@ module Rgot
38
40
  if @timer_on
39
41
  @start = Rgot.now
40
42
  end
41
- @duration = 0
43
+ @duration = 0.0
42
44
  end
43
45
 
44
46
  def run(&block)
45
47
  n = 1
46
48
  benchtime = (@opts.benchtime || 1).to_f
47
49
  catch(:skip) do
48
- run_n(n.to_i, block)
49
- while !failed? && @duration < benchtime && @n < 1e9
50
+ run_n(n, block)
51
+ while !failed? && @duration < benchtime && n < 1e9
50
52
  if @duration < (benchtime / 100.0)
51
- @n *= 100
53
+ n *= 100
52
54
  elsif @duration < (benchtime / 10.0)
53
- @n *= 10
55
+ n *= 10
54
56
  elsif @duration < (benchtime / 5.0)
55
- @n *= 5
57
+ n *= 5
56
58
  elsif @duration < (benchtime / 2.0)
57
- @n *= 2
59
+ n *= 2
58
60
  else
59
- if @n.to_i == 1
61
+ if n == 1
60
62
  break
61
63
  end
62
- @n *= 1.2
64
+ n = [(n * 1.2).to_i, n + 1].max || raise
63
65
  end
64
- run_n(@n.to_i, block)
66
+ run_n(n, block)
65
67
  end
66
68
  end
67
69
 
68
- BenchmarkResult.new(n: @n, t: @duration)
70
+ BenchmarkResult.new(n: n, t: @duration)
69
71
  end
70
72
 
71
73
  def run_parallel
@@ -83,7 +85,6 @@ module Rgot
83
85
  }.each(&:join)
84
86
  end
85
87
  end
86
- @n *= procs * threads
87
88
  Process.waitall
88
89
  end
89
90
 
@@ -2,6 +2,10 @@
2
2
 
3
3
  module Rgot
4
4
  class BenchmarkResult
5
+ # @dynamic n, t
6
+ attr_reader :n
7
+ attr_reader :t
8
+
5
9
  def initialize(n:, t:)
6
10
  @n = n
7
11
  @t = t
data/lib/rgot/cli.rb CHANGED
@@ -52,6 +52,15 @@ module Rgot
52
52
  o.on '--load-path [path]', "Specify $LOAD_PATH directory" do |arg|
53
53
  $LOAD_PATH.unshift(arg)
54
54
  end
55
+ o.on '--fuzz [regexp]', "run the fuzz test matching `regexp`" do |arg|
56
+ unless arg
57
+ raise Rgot::OptionError, "missing argument for flag --fuzz"
58
+ end
59
+ opts.fuzz = arg
60
+ end
61
+ o.on '--fuzztime [sec]', "time to spend fuzzing; default is to run indefinitely" do |arg|
62
+ opts.fuzztime = arg
63
+ end
55
64
  end
56
65
  parser.parse!(@argv)
57
66
 
@@ -93,12 +102,18 @@ module Rgot
93
102
  node = RubyVM::AbstractSyntaxTree.parse_file(testing_file).children[2]
94
103
  test_module_name = find_toplevel_name(node)
95
104
 
105
+ if opts.fuzz
106
+ # fuzzing observes changes in coverage.
107
+ require 'coverage'
108
+ Coverage.start(oneshot_lines: true)
109
+ end
96
110
  load testing_file
97
111
 
98
112
  test_module = Object.const_get(test_module_name)
99
113
  tests = []
100
114
  benchmarks = []
101
115
  examples = []
116
+ fuzz_targets = []
102
117
  main = nil
103
118
  methods = test_module.public_instance_methods
104
119
  methods.grep(/\Atest_/).each do |m|
@@ -117,7 +132,18 @@ module Rgot
117
132
  examples << Rgot::InternalExample.new(test_module, m)
118
133
  end
119
134
 
120
- m = Rgot::M.new(test_module: test_module, tests: tests, benchmarks: benchmarks, examples: examples, opts: opts)
135
+ methods.grep(/\Afuzz_/).each do |m|
136
+ fuzz_targets << Rgot::InternalFuzzTarget.new(test_module, m)
137
+ end
138
+
139
+ m = Rgot::M.new(
140
+ test_module: test_module,
141
+ tests: tests,
142
+ benchmarks: benchmarks,
143
+ examples: examples,
144
+ fuzz_targets: fuzz_targets,
145
+ opts: opts
146
+ )
121
147
  if main
122
148
  main.module.extend main.module
123
149
  main.module.instance_method(:test_main).bind(main.module).call(m)
@@ -126,8 +152,6 @@ module Rgot
126
152
  end
127
153
  end
128
154
 
129
- private
130
-
131
155
  def find_toplevel_name(node)
132
156
  case node.type
133
157
  when :MODULE
data/lib/rgot/common.rb CHANGED
@@ -5,6 +5,7 @@ require 'pathname'
5
5
 
6
6
  module Rgot
7
7
  class Common
8
+ # @dynamic output, output=
8
9
  attr_accessor :output
9
10
 
10
11
  def initialize
@@ -13,7 +14,7 @@ module Rgot
13
14
  @skipped = false
14
15
  @finished = false
15
16
  @start = Rgot.now
16
- @mutex = Mutex.new
17
+ @mutex = Thread::Mutex.new
17
18
  end
18
19
 
19
20
  def failed?
@@ -100,7 +101,7 @@ module Rgot
100
101
  # internal_log -> synchronize -> internal_log -> other log -> running method
101
102
  c = caller[4]
102
103
  path = c.sub(/:.*/, '')
103
- line = c.match(/:(\d+?):/)[1]
104
+ line = c.match(/:(\d+?):/)&.[](1)
104
105
  relative_path = Pathname.new(path).relative_path_from(Pathname.new(Dir.pwd)).to_s
105
106
  # Every line is indented at least 4 spaces.
106
107
  " #{relative_path}:#{line}: #{str}\n"
@@ -4,7 +4,10 @@ require 'ripper'
4
4
 
5
5
  module Rgot
6
6
  class ExampleParser < Ripper
7
+
8
+ # @dynamic examples, examples=
7
9
  attr_accessor :examples
10
+
8
11
  def initialize(code)
9
12
  super
10
13
  @examples = []
data/lib/rgot/f.rb ADDED
@@ -0,0 +1,230 @@
1
+ module Rgot
2
+ # def fuzz_foo(f)
3
+ # f.add(5, "hello")
4
+ # f.fuzz do |t, i, s|
5
+ # ...
6
+ # end
7
+ class F < Common
8
+ class Options
9
+ # @dynamic fuzz, fuzz=, fuzztime, fuzztime=
10
+ attr_accessor :fuzz
11
+ attr_accessor :fuzztime
12
+ def initialize(fuzz:, fuzztime:)
13
+ @fuzz = fuzz
14
+ @fuzztime = fuzztime
15
+ end
16
+ end
17
+
18
+ class CorpusEntry
19
+ # @dynamic values, values=, is_seed, is_seed=, path, path=
20
+ attr_accessor :values
21
+ attr_accessor :is_seed
22
+ attr_accessor :path
23
+ def initialize(values:, is_seed:, path:)
24
+ @values = values
25
+ @is_seed = is_seed
26
+ @path = path
27
+ end
28
+
29
+ def mutate_values
30
+ @values.map do |value|
31
+ if generator = SUPPORTED_TYPES[value.class]
32
+ generator.call(value)
33
+ else
34
+ raise "unsupported type #{value.class}"
35
+ end
36
+ end
37
+ end
38
+ end
39
+
40
+ class Coordinator
41
+ # @dynamic count, count=, interesting_count, interesting_count=
42
+ attr_accessor :count
43
+ attr_accessor :interesting_count
44
+
45
+ def initialize(warmup_input_count:)
46
+ @warmup_input_count = warmup_input_count
47
+ @before_cov = 0
48
+ @start_time = Rgot.now
49
+ @count = 0
50
+ @interesting_count = 0
51
+ @count_last_log = 0
52
+ @time_last_log = 0.0
53
+ end
54
+
55
+ def start_logger
56
+ Thread.new do
57
+ loop do
58
+ log_stats
59
+ sleep 3
60
+ end
61
+ end
62
+ end
63
+
64
+ def diff_coverage
65
+ current_cov = Coverage.peek_result.sum do |path, hash|
66
+ hash.map do |_, covs|
67
+ covs.length
68
+ end.sum
69
+ end
70
+ (current_cov - @before_cov).tap { @before_cov = current_cov }
71
+ end
72
+
73
+ def log_stats
74
+ rate = Float(count - @count_last_log) / (Rgot.now - @time_last_log)
75
+ total = @warmup_input_count + interesting_count
76
+ printf "fuzz: elapsed: %ds, execs: %d (%d/sec), new interesting: %d (total: %d)\n",
77
+ elapsed, count, rate, interesting_count, total
78
+
79
+ duration = Rgot.now - @time_last_log
80
+ @count_last_log = count
81
+ @time_last_log = Rgot.now
82
+ end
83
+
84
+ private
85
+
86
+ def elapsed
87
+ (Rgot.now - @start_time).round
88
+ end
89
+ end
90
+
91
+ SUPPORTED_TYPES = {
92
+ TrueClass => ->(v) { [true, false].sample },
93
+ FalseClass => ->(v) { [true, false].sample },
94
+ Integer => ->(v) { Random.rand(v) },
95
+ Float => ->(v) { Random.rand(v) },
96
+ String => ->(v) { Random.bytes(v.length) },
97
+ }
98
+
99
+ # @dynamic name
100
+ attr_reader :name
101
+
102
+ def initialize(fuzz_target:, opts:)
103
+ super()
104
+ @opts = opts
105
+ @fuzz_target = fuzz_target
106
+ @fuzz_block = nil
107
+ @module = fuzz_target.module
108
+ @name = fuzz_target.name
109
+ @corpus = []
110
+ end
111
+
112
+ # TODO: DRY with T
113
+ def run
114
+ catch(:skip) { call }
115
+ finish!
116
+ rescue => e
117
+ fail!
118
+ raise e
119
+ end
120
+
121
+ def run_testing
122
+ run
123
+ report if !fuzz? || failed?
124
+ end
125
+
126
+ def run_fuzzing
127
+ return unless fuzz?
128
+ raise("must call after #fuzz") unless @fuzz_block
129
+
130
+ coordinator = Coordinator.new(
131
+ warmup_input_count: @corpus.length
132
+ )
133
+ coordinator.start_logger
134
+
135
+ t = T.new(@fuzz_target.module, @fuzz_target.name)
136
+
137
+ begin
138
+ Timeout.timeout(@opts.fuzztime.to_f) do
139
+ loop do
140
+ @corpus.each do |entry|
141
+ values = entry.mutate_values
142
+
143
+ @fuzz_block.call(t, *values)
144
+
145
+ if 0 < coordinator.diff_coverage
146
+ coordinator.interesting_count += 1
147
+ end
148
+ coordinator.count += 1
149
+ fail! if t.failed?
150
+ end
151
+ end
152
+ end
153
+ rescue Timeout::Error, Interrupt
154
+ coordinator.log_stats
155
+ end
156
+
157
+ report
158
+ end
159
+
160
+ def add(*args)
161
+ args.each do |arg|
162
+ unless SUPPORTED_TYPES.key?(arg.class)
163
+ raise "unsupported type to Add #{arg.class}"
164
+ end
165
+ end
166
+ entry = CorpusEntry.new(
167
+ values: args.dup,
168
+ is_seed: true,
169
+ path: "seed##{@corpus.length}"
170
+ )
171
+ @corpus.push(entry)
172
+ end
173
+
174
+ def fuzz(&block)
175
+ unless block
176
+ raise LocalJumpError, "must set block"
177
+ end
178
+ unless 2 <= block.arity
179
+ raise "fuzz target must receive at least two arguments"
180
+ end
181
+
182
+ t = T.new(@fuzz_target.module, @fuzz_target.name)
183
+
184
+ @corpus.each do |entry|
185
+ unless entry.values.length == (block.arity - 1)
186
+ raise "wrong number of values in corpus entry: #{entry.values.length}, want #{block.arity - 1}"
187
+ end
188
+ block.call(t, *entry.values.dup)
189
+ fail! if t.failed?
190
+ end
191
+
192
+ @fuzz_block = block
193
+
194
+ nil
195
+ end
196
+
197
+ def fuzz?
198
+ return false unless @opts.fuzz
199
+ return false unless Regexp.new(@opts.fuzz.to_s).match?(@fuzz_target.name)
200
+ true
201
+ end
202
+
203
+ def report
204
+ puts @output if Rgot.verbose? && !@output.empty?
205
+ duration = Rgot.now - @start
206
+ template = "--- \e[%sm%s\e[m: %s (%.2fs)\n"
207
+ if failed?
208
+ printf template, [41, 1].join(';'), "FAIL", @name, duration
209
+ elsif Rgot.verbose?
210
+ if skipped?
211
+ printf template, [44, 1].join(';'), "SKIP", @name, duration
212
+ else
213
+ printf template, [42, 1].join(';'), "PASS", @name, duration
214
+ end
215
+ end
216
+ end
217
+
218
+ private
219
+
220
+ def call
221
+ test_method = @module.instance_method(@name).bind(@module)
222
+ if test_method.arity == 0
223
+ path, line = test_method.source_location
224
+ warn "#{path}:#{line} `#{test_method.name}' is not running. It's a testing method name, But not have argument"
225
+ else
226
+ test_method.call(self)
227
+ end
228
+ end
229
+ end
230
+ end
data/lib/rgot/m.rb CHANGED
@@ -12,9 +12,15 @@ module Rgot
12
12
  :timeout,
13
13
  :cpu,
14
14
  :thread,
15
+ :fuzz,
16
+ :fuzztime,
15
17
  ); end
16
18
 
17
- def initialize(tests:, benchmarks:, examples:, test_module: nil, opts: Options.new)
19
+ def initialize(tests:, benchmarks:, examples:, fuzz_targets: nil, test_module: nil, opts: Options.new)
20
+ unless fuzz_targets
21
+ raise "Require `fuzz_targets` keyword" if Gem::Version.new("2.0") <= Gem::Version.new(Rgot::VERSION)
22
+ warn "`Rgot::M#initialize` will require the `fuzz_targets` keyword in the next major version."
23
+ end
18
24
  unless test_module
19
25
  raise "Require `test_module` keyword" if Gem::Version.new("2.0") <= Gem::Version.new(Rgot::VERSION)
20
26
  warn "`Rgot::M#initialize` will require the `test_module` keyword in the next major version."
@@ -23,18 +29,30 @@ module Rgot
23
29
  @tests = tests
24
30
  @benchmarks = benchmarks
25
31
  @examples = examples
32
+ @fuzz_targets = fuzz_targets || []
26
33
  @test_module = test_module
27
34
  @opts = opts
28
- @cpu_list = nil
29
- @thread_list = nil
35
+
36
+ @cpu_list = []
37
+ @thread_list = []
38
+ @fs = @fuzz_targets.map do |fuzz_target|
39
+ F.new(
40
+ fuzz_target: fuzz_target,
41
+ opts: F::Options.new(
42
+ fuzz: opts.fuzz,
43
+ fuzztime: opts.fuzztime,
44
+ )
45
+ )
46
+ end
30
47
  end
31
48
 
32
49
  def run
33
50
  duration = Rgot.now
34
51
  test_ok = false
52
+ fuzz_targets_ok = false
35
53
  example_ok = false
36
54
 
37
- if @tests.empty? && @benchmarks.empty? && @examples.empty?
55
+ if @tests.empty? && @benchmarks.empty? && @examples.empty? && @fuzz_targets.empty?
38
56
  warn "rgot: warning: no tests to run"
39
57
  end
40
58
 
@@ -47,22 +65,29 @@ module Rgot
47
65
 
48
66
  Timeout.timeout(@opts.timeout.to_f) do
49
67
  test_ok = run_tests
68
+ fuzz_targets_ok = run_fuzz_tests
50
69
  example_ok = run_examples
51
70
  end
52
71
 
53
- if !test_ok || !example_ok
72
+ if !test_ok || !example_ok || !fuzz_targets_ok
54
73
  puts "FAIL"
55
74
  puts "exit status 1"
56
75
  puts sprintf("%s\t%s\t%.3fs", "FAIL", @test_module, Rgot.now - duration)
76
+ return 1
77
+ end
57
78
 
58
- 1
59
- else
60
- puts "PASS"
61
- run_benchmarks
62
- puts sprintf("%s\t%s\t%.3fs", "ok ", @test_module, Rgot.now - duration)
63
-
64
- 0
79
+ if !run_fuzzing()
80
+ puts "FAIL"
81
+ puts "exit status 1"
82
+ puts sprintf("%s\t%s\t%.3fs", "FAIL", @test_module, Rgot.now - duration)
83
+ return 1
65
84
  end
85
+
86
+ puts "PASS"
87
+ run_benchmarks
88
+ puts sprintf("%s\t%s\t%.3fs", "ok ", @test_module, Rgot.now - duration)
89
+
90
+ 0
66
91
  end
67
92
 
68
93
  private
@@ -90,9 +115,7 @@ module Rgot
90
115
  puts "=== RUN #{test.name}"
91
116
  end
92
117
  t.run
93
- if t.failed?
94
- ok = false
95
- end
118
+ ok = ok && !t.failed?
96
119
  end
97
120
  ok
98
121
  end
@@ -130,6 +153,50 @@ module Rgot
130
153
  ok
131
154
  end
132
155
 
156
+ def run_fuzz_tests
157
+ ok = true
158
+ @fs.each do |f|
159
+ if Rgot.verbose?
160
+ if f.fuzz?
161
+ puts "=== FUZZ #{f.name}"
162
+ else
163
+ puts "=== RUN #{f.name}"
164
+ end
165
+ end
166
+ f.run_testing
167
+ ok = ok && !f.failed?
168
+ end
169
+ ok
170
+ end
171
+
172
+ def run_fuzzing
173
+ if @fuzz_targets.empty? || @opts.fuzz.nil?
174
+ return true
175
+ end
176
+
177
+ fuzzing_fs = @fs.select(&:fuzz?)
178
+
179
+ if fuzzing_fs.empty?
180
+ puts "rgot: warning: no fuzz tests to fuzz"
181
+ return true
182
+ end
183
+
184
+ if fuzzing_fs.length > 1
185
+ names = fuzzing_fs.map(&:name)
186
+ puts "rgot: will not fuzz, --fuzz matches more than one fuzz test: #{names.inspect}"
187
+ return false
188
+ end
189
+
190
+ ok = true
191
+
192
+ fuzzing_fs.each do |f|
193
+ f.run_fuzzing
194
+ ok = ok && !f.failed?
195
+ end
196
+
197
+ ok
198
+ end
199
+
133
200
  def run_examples
134
201
  ok = true
135
202
  @examples.each do |example|
@@ -143,13 +210,13 @@ module Rgot
143
210
  out, _ = capture do
144
211
  method.call
145
212
  end
146
- file = method.source_location[0]
147
- r = ExampleParser.new(File.read(file))
148
- r.parse
149
- e = r.examples.find { |re| re.name == example.name }
213
+ file = method.source_location&.[](0) or raise("bug")
214
+ example_parser = ExampleParser.new(File.read(file))
215
+ example_parser.parse
216
+ e = example_parser.examples.find { |er| er.name == example.name } or raise("bug")
150
217
 
151
218
  duration = Rgot.now - start
152
- if e && e.output.strip != out.strip
219
+ if e.output.strip != out.strip
153
220
  printf("--- FAIL: %s (%.2fs)\n", e.name, duration)
154
221
  ok = false
155
222
  puts "got:"
data/lib/rgot/pb.rb CHANGED
@@ -2,8 +2,6 @@
2
2
 
3
3
  module Rgot
4
4
  class PB
5
- attr_accessor :bn
6
-
7
5
  def initialize(bn:)
8
6
  @bn = bn
9
7
  end
data/lib/rgot/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Rgot
4
- VERSION = "1.2.0"
4
+ VERSION = "1.3.0"
5
5
  end
data/lib/rgot.rb CHANGED
@@ -8,12 +8,14 @@ module Rgot
8
8
  autoload :B, 'rgot/b'
9
9
  autoload :PB, 'rgot/pb'
10
10
  autoload :BenchmarkResult, 'rgot/benchmark_result'
11
+ autoload :F, 'rgot/f'
11
12
  autoload :ExampleParser, 'rgot/example_parser'
12
13
 
13
14
  OptionError = Class.new(StandardError)
14
15
  InternalTest = Struct.new(:module, :name)
15
16
  InternalBenchmark = Struct.new(:module, :name)
16
17
  InternalExample = Struct.new(:module, :name)
18
+ InternalFuzzTarget = Struct.new(:module, :name)
17
19
  ExampleOutput = Struct.new(:name, :output)
18
20
 
19
21
  class << self
@@ -0,0 +1,49 @@
1
+ ---
2
+ sources:
3
+ - name: ruby/gem_rbs_collection
4
+ remote: https://github.com/ruby/gem_rbs_collection.git
5
+ revision: main
6
+ repo_dir: gems
7
+ path: ".gem_rbs_collection"
8
+ gems:
9
+ - name: pathname
10
+ version: '0'
11
+ source:
12
+ type: stdlib
13
+ - name: optparse
14
+ version: '0'
15
+ source:
16
+ type: stdlib
17
+ - name: timeout
18
+ version: '0'
19
+ source:
20
+ type: stdlib
21
+ - name: etc
22
+ version: '0'
23
+ source:
24
+ type: stdlib
25
+ - name: coverage
26
+ version: '0'
27
+ source:
28
+ type: stdlib
29
+ - name: concurrent-ruby
30
+ version: '1.1'
31
+ source:
32
+ type: git
33
+ name: ruby/gem_rbs_collection
34
+ revision: 9576ce5b109170f1ba8a42671bfafb64ab95bd23
35
+ remote: https://github.com/ruby/gem_rbs_collection.git
36
+ repo_dir: gems
37
+ - name: io-console
38
+ version: '0'
39
+ source:
40
+ type: stdlib
41
+ - name: logger
42
+ version: '0'
43
+ source:
44
+ type: stdlib
45
+ - name: monitor
46
+ version: '0'
47
+ source:
48
+ type: stdlib
49
+ gemfile_lock_path: Gemfile.lock
@@ -0,0 +1,53 @@
1
+ # Download sources
2
+ sources:
3
+ - name: ruby/gem_rbs_collection
4
+ remote: https://github.com/ruby/gem_rbs_collection.git
5
+ revision: main
6
+ repo_dir: gems
7
+
8
+ # A directory to install the downloaded RBSs
9
+ path: .gem_rbs_collection
10
+
11
+ gems:
12
+ - name: pathname
13
+ - name: optparse
14
+ - name: timeout
15
+ - name: etc
16
+ - name: coverage
17
+
18
+ # Skip loading rbs gem's RBS.
19
+ # It's unnecessary if you don't use rbs as a library.
20
+ - name: rbs
21
+ ignore: true
22
+ - name: steep
23
+ ignore: true
24
+ - name: activesupport
25
+ ignore: true
26
+ - name: ast
27
+ ignore: true
28
+ - name: csv
29
+ ignore: true
30
+ - name: i18n
31
+ ignore: true
32
+ - name: json
33
+ ignore: true
34
+ - name: listen
35
+ ignore: true
36
+ - name: fileutils
37
+ ignore: true
38
+ - name: minitest
39
+ ignore: true
40
+ - name: parallel
41
+ ignore: true
42
+ - name: rainbow
43
+ ignore: true
44
+ - name: rgot
45
+ ignore: true
46
+ - name: securerandom
47
+ ignore: true
48
+ - name: strscan
49
+ ignore: true
50
+ - name: forwardable
51
+ ignore: true
52
+ - name: mutex_m
53
+ ignore: true
data/sig/patch.rbs ADDED
@@ -0,0 +1,3 @@
1
+ # patch
2
+ class Ripper
3
+ end
data/sig/rgot.rbs ADDED
@@ -0,0 +1,232 @@
1
+ module Rgot
2
+ VERSION: "1.2.0"
3
+
4
+ def self.now: () -> Float
5
+ def self.benchmark: (?Hash[Symbol, String] opts_hash) { (B) -> void } -> BenchmarkResult
6
+ def self.verbose?: () -> bool
7
+
8
+ class Cli
9
+ @argv: Array[String]
10
+
11
+ def initialize: (untyped argv) -> void
12
+ def run: () -> void
13
+
14
+ private
15
+
16
+ def parse_option: (Rgot::M::Options opts) -> void
17
+ def main_process: (Rgot::M::Options opts) -> void
18
+ def testing_files: () -> Array[String]
19
+ def child_process: (Rgot::M::Options opts, String testing_file) -> Integer
20
+ # `node` is RubyVM::AbstractSyntaxTree::Node
21
+ def find_toplevel_name: (untyped node) -> Symbol
22
+ end
23
+
24
+ class Common
25
+ @failed: bool
26
+ @skipped: bool
27
+ @finished: bool
28
+ @start: Float
29
+ @mutex: Thread::Mutex
30
+
31
+ attr_accessor output: String
32
+
33
+ def initialize: () -> void
34
+
35
+ def failed?: () -> bool
36
+ def skipped?: () -> bool
37
+ def finished?: () -> bool
38
+ def fail!: () -> void
39
+ def skip!: () -> void
40
+ def finish!: () -> void
41
+ def log: (*untyped) -> nil
42
+ def logf: (*untyped) -> nil
43
+ def error: (*untyped) -> nil
44
+ def errorf: (*untyped) -> nil
45
+ def fatal: (*untyped) -> bot
46
+ def fatalf: (*untyped) -> bot
47
+ def skip: (*untyped) -> bot
48
+ def skipf: (*untyped) -> bot
49
+ def skip_now: () -> bot
50
+ def fail_now: () -> bot
51
+
52
+ private
53
+
54
+ def decorate: (String) -> String
55
+ def internal_log: (String msg) -> void
56
+ end
57
+
58
+ class M
59
+ class Options
60
+ attr_accessor bench: String?
61
+ attr_accessor benchtime: String?
62
+ attr_accessor timeout: String?
63
+ attr_accessor cpu: String?
64
+ attr_accessor thread: String?
65
+ attr_accessor fuzz: String?
66
+ attr_accessor fuzztime: String?
67
+ end
68
+
69
+ @tests: Array[InternalTest]
70
+ @benchmarks: Array[InternalBenchmark]
71
+ @examples: Array[InternalExample]
72
+ @fuzz_targets: Array[InternalFuzzTarget]
73
+ @fs: Array[F]
74
+ @test_module: Module?
75
+ @opts: M::Options
76
+ @cpu_list: Array[Integer]
77
+ @thread_list: Array[Integer]
78
+
79
+ def initialize: (
80
+ tests: Array[InternalTest],
81
+ benchmarks: Array[InternalBenchmark],
82
+ examples: Array[InternalExample],
83
+ fuzz_targets: Array[InternalFuzzTarget],
84
+ ?test_module: Module?,
85
+ ?opts: Options
86
+ ) -> void
87
+ def run: () -> Integer
88
+
89
+ private
90
+
91
+ def run_tests: () -> bool
92
+ def run_benchmarks: () -> bool
93
+ def run_fuzz_tests: () -> bool
94
+ def run_fuzzing: () -> bool
95
+ def run_examples: () -> bool
96
+ end
97
+
98
+ class T < Common
99
+ @module: Module
100
+ @name: Symbol
101
+
102
+ def initialize: (Module test_module, Symbol name) -> void
103
+ def run: () -> void
104
+ def report: () -> void
105
+ def call: () -> void
106
+ end
107
+
108
+ class B < Common
109
+ class Options
110
+ attr_accessor procs: Integer
111
+ attr_accessor threads: Integer
112
+ attr_accessor benchtime: String?
113
+ end
114
+
115
+ @module: Module?
116
+ @name: Symbol?
117
+ @opts: B::Options
118
+ @timer_on: bool
119
+ @duration: Float
120
+
121
+ attr_accessor n: Integer
122
+
123
+ def initialize: (Module? benchmark_module, Symbol? name, ?B::Options opts) -> void
124
+ def start_timer: () -> void
125
+ def stop_timer: () -> void
126
+ def reset_timer: () -> void
127
+ def run: () ?{ (B) -> void } -> BenchmarkResult
128
+ def run_parallel: () { (PB) -> void } -> void
129
+
130
+ private
131
+
132
+ def run_n: (Integer n, ?Proc? block) -> void
133
+ end
134
+
135
+ class F < Common
136
+ class Options
137
+ attr_accessor fuzz: String?
138
+ attr_accessor fuzztime: String?
139
+ def initialize: (fuzz: String?, fuzztime: String?) -> void
140
+ end
141
+ class CorpusEntry
142
+ attr_accessor values: Array[untyped]
143
+ attr_accessor is_seed: bool
144
+ attr_accessor path: String
145
+ def initialize: (values: Array[untyped], is_seed: bool, path: String) -> void
146
+ def mutate_values: () -> Array[untyped]
147
+ end
148
+ class Coordinator
149
+ attr_accessor count: Integer
150
+ attr_accessor interesting_count: Integer
151
+ def initialize: (warmup_input_count: Integer) -> void
152
+ @warmup_input_count: Integer
153
+ @before_cov: Integer
154
+ @start_time: Float
155
+ @count: Integer
156
+ @interesting_count: Integer
157
+ @count_last_log: Integer
158
+ @time_last_log: Float
159
+ def start_logger: () -> void
160
+ def diff_coverage: () -> Integer
161
+ def log_stats: () -> void
162
+ private
163
+ def elapsed: () -> Integer
164
+ end
165
+ SUPPORTED_TYPES: Hash[untyped, ^(untyped) -> untyped]
166
+ @fuzz_target: InternalFuzzTarget
167
+ @fuzz_block: Proc?
168
+ @opts: F::Options
169
+ @corpus: Array[CorpusEntry]
170
+ @module: Module
171
+ attr_reader name: Symbol
172
+ def initialize: (fuzz_target: InternalFuzzTarget, opts: F::Options) -> void
173
+ def run: () -> void
174
+ def run_testing: () -> void
175
+ def run_fuzzing: () -> void
176
+ def add: (*untyped) -> void
177
+ def fuzz: () { (*untyped) -> void } -> void
178
+ def fuzz?: () -> bool
179
+ def report: () -> void
180
+
181
+ private
182
+
183
+ def call: () -> void
184
+ end
185
+
186
+ class PB
187
+ @bn: Integer
188
+ def initialize: (bn: Integer) -> void
189
+ def next: () -> bool
190
+ end
191
+
192
+ class BenchmarkResult
193
+ attr_reader n: Integer # int // The number of iterations.
194
+ attr_reader t: Float # time.Duration // The total time taken.
195
+ def initialize: (n: Integer, t: Float) -> void
196
+ end
197
+
198
+ class ExampleParser < Ripper
199
+ @in_def: bool
200
+ @has_output: bool
201
+ @output: String
202
+ attr_accessor examples: Array[ExampleOutput]
203
+ end
204
+
205
+ class OptionError < StandardError
206
+ end
207
+
208
+ class InternalTest
209
+ attr_accessor module: Module
210
+ attr_accessor name: Symbol
211
+ end
212
+
213
+ class InternalBenchmark
214
+ attr_accessor module: Module
215
+ attr_accessor name: Symbol
216
+ end
217
+
218
+ class InternalExample
219
+ attr_accessor module: Module
220
+ attr_accessor name: Symbol
221
+ end
222
+
223
+ class InternalFuzzTarget
224
+ attr_accessor module: Module
225
+ attr_accessor name: Symbol
226
+ end
227
+
228
+ class ExampleOutput
229
+ attr_accessor output: String
230
+ attr_accessor name: Symbol
231
+ end
232
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rgot
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.0
4
+ version: 1.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - ksss
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-12-27 00:00:00.000000000 Z
11
+ date: 2023-01-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -53,6 +53,7 @@ files:
53
53
  - LICENSE.txt
54
54
  - README.md
55
55
  - Rakefile
56
+ - Steepfile
56
57
  - bin/rgot
57
58
  - lib/rgot.rb
58
59
  - lib/rgot/b.rb
@@ -60,11 +61,16 @@ files:
60
61
  - lib/rgot/cli.rb
61
62
  - lib/rgot/common.rb
62
63
  - lib/rgot/example_parser.rb
64
+ - lib/rgot/f.rb
63
65
  - lib/rgot/m.rb
64
66
  - lib/rgot/pb.rb
65
67
  - lib/rgot/t.rb
66
68
  - lib/rgot/version.rb
69
+ - rbs_collection.lock.yaml
70
+ - rbs_collection.yaml
67
71
  - rgot.gemspec
72
+ - sig/patch.rbs
73
+ - sig/rgot.rbs
68
74
  homepage: https://github.com/ksss/rgot
69
75
  licenses:
70
76
  - MIT
@@ -84,7 +90,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
84
90
  - !ruby/object:Gem::Version
85
91
  version: '0'
86
92
  requirements: []
87
- rubygems_version: 3.3.26
93
+ rubygems_version: 3.4.1
88
94
  signing_key:
89
95
  specification_version: 4
90
96
  summary: Ruby + Golang Testing = Rgot