fuzzbert 1.0.0 → 1.0.1

data/README.md

@@ -22,21 +22,7 @@ For further information on random testing, here are two excellent starting point
 ## Defining a random test
 
 FuzzBert defines an RSpec-like DSL that can be used to define different fuzzing
-scenarios. The DSL uses three words: `fuzz`, `deploy` and `data`. `fuzz` can be
-thought of as defining a new scenario, such as "fuzz this command line tool",
-"fuzz this particular URL of my web app" or "fuzz this library method taking 
-external input".
-
-Within a `fuzz` block, there must be one occurence of `deploy` and one or several
-occurences of `data`. The `deploy` block is the spot where we deliver the random
-payload that has been generated. It is agnostic about the actual target in order to
-leave you free to fuzz whatever you require in your particular case. The `data`
-blocks define the shape of the random data being generated. There can be more than
-one such block because it is often beneficial to not only shoot completely random
-data at the target - you often want to deliver more structured data as well, trying
-to find the edge cases deeper within your code. Good random test suites make use
-of both - totally random data as well as structured data - in order to cover as
-much "code surface" as possible.
+scenarios. The DSL uses three words: `fuzz`, `deploy` and `data`. 
 
 Here is a quick example that fuzzes `JSON.parse`:
 
@@ -78,6 +64,21 @@ fuzz "JSON.parse" do
 end
 ```
 
+`fuzz` can be thought of as defining a new scenario, such as "fuzz this command 
+line tool", "fuzz this particular URL of my web app" or "fuzz this library method 
+taking external input".
+
+Within a `fuzz` block, there must be one occurence of `deploy` and one or several
+occurences of `data`. The `deploy` block is the spot where we deliver the random
+payload that has been generated. It is agnostic about the actual target in order to
+leave you free to fuzz whatever you require in your particular case. The `data`
+blocks define the shape of the random data being generated. There can be more than
+one such block because it is often beneficial to not only shoot completely random
+data at the target - you often want to deliver more structured data as well, trying
+to find the edge cases deeper within your code. Good random test suites make use
+of both - totally random data as well as structured data - in order to cover as
+much "code surface" as possible.
+
 The `deploy` block takes the generated data as a parameter. The block itself is
 responsible of deploying the payload. An execution is considered successful if
 the `deploy` block passes with no uncaught error being raised. If an error slips
@@ -88,14 +89,91 @@ considered as a failure.
 choose completely custom lambdas of your own or use those predefined for you in
 `FuzzBert::Generators`.
 
+## Running a random test
+
+Once the FuzzBert files are set up, you may run your tests similar to how you
+would run unit tests:
+
+    fuzzbert "fuzz/**/fuzz_*.rb"
+
+If your FuzzBert files are already in a directory named 'fuzz' and each of them
+begins with 'fuzz_', you may omit the pattern altogether. 
+
+Each `fuzz` block defines a `TestSuite`. These are executed in a round-robin manner.
+Each individual `TestSuite` will then apply the `deploy` block with a sample of
+data generated successively by each one of the `data` blocks. Once all `data` blocks
+are used up, the next `TestSuite` will be executed etc. By default, a FuzzBert
+fuzzing session runs forever, until the process is either killed or by manually hitting
+`CTRL+C` for example. This was a deliberate design choice since random testing suites
+need to be run for quite some time to be effective. It's something you want to run over
+the weekend rather than for a couple of minutes. Still, it can make sense to explicitly
+limit the number of runs, for example when integrating FuzzBert with a CI server or
+with Travis. You can do so by passing the `--limit` parameter to the `fuzzbert`
+executable:
+
+    fuzzbert --limit 1000000 "fuzz/**/fuzz_*.rb"
+
+Every single execution of `deploy` is run in a separate process. The main reason for
+this is that we typically want to detect hard crashes when a C extension or even Ruby
+itself encounters an input it can't handle. Besides being able to cope with these cases,
+running in separate processes proves beneficial otherwise as well: by default, FuzzBert
+runs the tests in four separate processes at once, therefore utilizing your CPU's cores
+effectively. You can tweak that setting with `--pool-size` to set this number to 1 
+(for completely sequential runs) or to the exact number of cores your CPU offers.
+
+    fuzzbert --pool-size 1 my/fuzzbert/file
+
+## What happens if we encounter a bug?
+
+If a test should end up failing (either the process crashed completely or caused an
+uncaught error), FuzzBert will output the failing test on your terminal and tell you
+where it stored the data that caused this. This conveniently allows you to run FuzzBert
+over the weekend and when you return on Monday, the troubleshooters will sit there all
+lined up for you to go through and filter. By using the `--console` command line switch
+you can tell FuzzBert to not explicitly store the data, but echoing the data that
+caused the crash to the terminal instead.
+
+    fuzzbert --console "fuzz/**/fuzz_*.rb"
+
+If you don't want to litter your current working directory with the files generated
+by FuzzBert, you can also specify a specific path to where they should be saved
+instead:
+
+    fuzzbert --bug-dir bugs "fuzz/**/fuzz_*.rb"
+
+This is still not quite what you want to happen in case a test crashes? There's
+also the possibility to define a handler of your own:
+
+```ruby
+require 'fuzzbert'
+
+class MyHandler
+  def handle(error_data)
+    #create an issue in the bug tracker
+    puts error_data[:id]
+    p error_data[:data]
+    puts error_data[:pid]
+    puts error_data[:status]
+  end
+end
+
+fuzz "Define here as usual" do
+  ...
+end
+```
+
+Then tell fuzzbert by telling it about the custom handler:
+
+    fuzzbert --handler MyHandler my/fuzzbert/file
+
 ## Templates
 
 Using the approach described so far is most useful for binary protocols, but as
-soon as largely String-based data is needed it can soon become tedious. What you
-actually want in these situations is some sort of template mechanism that comes
-with mostly fixed data and only replaces a few selected parts with randomly
-generated data. This, too, is possible with FuzzBert, it comes with a minimal
-templating language:
+soon as you work with mainly String-based data this can quickly become a chore. 
+What you actually want in these situations is some sort of template mechanism 
+that comes with mostly fixed data and only replaces a few selected parts with
+randomly generated data. This, too, is possible with FuzzBert, it comes with a 
+minimal templating language:
 
 ```ruby
 require 'fuzzbert'
@@ -119,6 +197,10 @@ fuzz "My Web App" do
 end
 ```
 
+Simply specify your template variables using `${..}` and assign a callback for
+them via `set`. Of course you may escape the dollar sign with a back slash as
+usual.
+
 ## Mutators
 
 Mutation is the principle used in "Babysitting an Army of Monkeys". The basis for
@@ -141,37 +223,8 @@ fuzz "Web App" do
 end
 ```
 
-## How the tests are performed
-
-Each `fuzz` block defines a `TestSuite`. These are executed in a round-robin manner.
-Each individual `TestSuite` will then apply the `deploy` block with a sample of
-data generated successively by each one of the `data` blocks. Once all `data` blocks
-were used, the next `TestSuite` will be executed and so on... By default, a FuzzBert
-fuzzing session runs forever, until the process is either killed or by manually hitting
-`CTRL+C` for example. This was a deliberate design choice since random testing suites
-need to be run for quite some time to be effective. It's something you want to run over
-the weekend rather than for a couple of minutes. Still, it can make sense to explicitly
-limit the number of runs, for example when integrating FuzzBert with a CI server or
-with Travis. You can do so by passing the `--limit` parameter to the `fuzzbert`
-executable.
-
-Every single execution of `deploy` is run in a separate process. The main reason for
-this is that we typically want to detect hard crashes when a C extension or even Ruby
-itself encounters an input it can't handle. Besides being able to cope with these cases,
-running in separate processes proves beneficial otherwise as well: by default, FuzzBert
-runs the tests in four separate processes at once, therefore utilizing your CPU's cores
-effectively. You can tweak that setting with `--pool-size` to set this number to 1 
-(for completely sequential runs) or to the exact number of cores your CPU offers.
-
-## What happens if we encounter a bug?
-
-If a test should end up failing (either the process crashed completely or caused an
-uncaught error), FuzzBert will output the failing test on your terminal and tell you
-where it stored the data that caused this. This conveniently allows you to run FuzzBert
-over the weekend and when you return on Monday, the troubleshooters will sit there all
-lined up for you to go through and filter. By using the `--console` command line switch
-you can tell FuzzBert to not explicitly store the data, but echoing the data that
-caused the crash to the terminal instead.
+This will take the original JSON data and modify one byte each time data is being
+generated.
 
 ## Rake integration
 

data/lib/fuzzbert/autorun.rb

@@ -17,49 +17,82 @@ module FuzzBert::AutoRun
   end
 
   def run(options=nil)
-    executor = options ? FuzzBert::Executor.new(TEST_CASES, options) : FuzzBert::Executor.new(TEST_CASES)
-    executor.run
+    raise RuntimeError.new "No test cases were found" if TEST_CASES.empty?
+    FuzzBert::Executor.new(TEST_CASES, options).run
   end
 
   private; module_function
 
-  def load_files(files)
-    files.each do |pattern|
-      Dir.glob(pattern).each { |f| load File.expand_path(f) }
+    def load_files(files)
+      files.each do |pattern|
+        Dir.glob(pattern).each { |f| load File.expand_path(f) }
+      end
     end
-  end
 
-  def process_args(args = [])
-    options = {}
-    orig_args = args.dup
+    def process_args(args = [])
+      options = {}
+      orig_args = args.dup
 
-    OptionParser.new do |opts|
-      opts.banner  = 'FuzzBert options:'
-      opts.version = FuzzBert::VERSION
+      OptionParser.new do |opts|
+        opts.banner  = 'Usage: fuzzbert [OPTIONS] PATTERN [PATTERNS]'
+        opts.separator <<-EOS
 
-      opts.on '-h', '--help', 'Display this help.' do
-        puts opts
-        exit
-      end
+Run your random tests by pointing fuzzbert to a single or many explicit files
+or by providing a pattern. The default pattern is 'fuzz/**/fuzz_*.rb, assuming
+that your FuzzBert files (files beginning with 'fuzz_') live in a directory
+named fuzz located under the current directory that you are in.
 
-      opts.on '--pool-size SIZE', Integer, "Sets the number of concurrently running processes to SIZE" do |n|
-        options[:pool_size] = n.to_i
-      end
+By default, fuzzbert will run the tests you specify forever, be sure to hit
+CTRL+C when you are done or specify a limit with '--limit'.
 
-      opts.on '--limit LIMIT', Integer, "Instead of running permanently, fuzzing will be stopped after running LIMIT of instances" do |n|
-        p n
-        options[:limit] = n.to_i
-      end
+        EOS
+
+        opts.version = FuzzBert::VERSION
+
+        opts.on '-h', '--help', 'Run ' do
+          puts opts
+          exit
+        end
+
+        opts.on '--pool-size SIZE', Integer, "Sets the number of concurrently running processes to SIZE. Default is 4." do |n|
+          options[:pool_size] = n.to_i
+        end
 
-      opts.on '--console', "Output the failing cases including data on the console instead of saving them in a file" do
-        options[:handler] = FuzzBert::Handler::Console.new
+        opts.on '--limit LIMIT', Integer, "Instead of running permanently, fuzzing will be stopped after running LIMIT of instances." do |n|
+          options[:limit] = n.to_i
+        end
+
+        opts.on '--console', "Output the failing cases including data on the console instead of saving them in a file." do
+          options[:handler] = FuzzBert::Handler::Console.new
+        end
+
+        opts.on '--sleep-delay SECONDS', Float, "Specify the number of SECONDS that the main process sleeps before checking that the limit has been reached. Default is 1." do |f|
+          options[:sleep_delay] = f.to_f
+        end
+
+        opts.on '--handler CLASS', String, "Specify the full path to a CLASS that will serve as your Handler." do |path|
+          #lazy initialization: the Handler must be defined in one of the fuzzer files
+          options[:handler] = Class.new do
+            @@path = path
+
+            def handle(id, data, pid, status)
+              @inner ||= Object.const_get(@@path).new
+              @inner.handle(id, data, pid, status)
+            end
+          end.new
+        end
+
+        opts.on '--bug-dir DIRECTORY', String, "The DIRECTORY where the resulting bug files will be stored. Default is the current directory." do |dir|
+          raise ArgumentError.new "#{dir} is not a directory" unless Dir.exists?(dir)
+          options[:handler] = FuzzBert::Handler::FileOutput.new(dir)
+        end
+
+        opts.parse! args
       end
 
-      opts.parse! args
+      raise ArgumentError.new("No file pattern was given") if args.empty?
+      [options, args]
     end
 
-    [options, args]
-  end
-
 end
 

data/lib/fuzzbert/dsl.rb

@@ -1,6 +1,8 @@
 module FuzzBert::DSL
   def fuzz(*args, &blk)
     suite = FuzzBert::TestSuite.create(*args, &blk)
+    raise RuntimeError.new "No 'deploy' block was given" unless suite.test
+    raise RuntimeError.new "No 'data' blocks were given" unless suite.generators
     FuzzBert::AutoRun.register(suite)
   end
 end

data/lib/fuzzbert/error_handler.rb

@@ -1,19 +1,67 @@
 
 module FuzzBert::Handler
+
+  module ConsoleHelper
+    def info(error_data)
+      id = error_data[:id]
+      status = error_data[:status]
+
+      crashed = status.termsig
+
+      if crashed
+        puts "The data caused a hard crash."
+      else
+        puts "The data caused an uncaught error."
+      end
+    end
+  end
+
   class FileOutput
-    def handle(id, data, pid, status)
-      prefix = status.termsig ? "crash" : "bug"
-      filename = "#{prefix}#{pid}"
+    include FuzzBert::Handler::ConsoleHelper
+
+    def initialize(dir=nil)
+      @dir = dir
+      if @dir && !@dir.end_with?("/")
+        @dir << "/"
+      end
+    end
+
+    def handle(error_data)
+      id = error_data[:id]
+      data = error_data[:data]
+      status = error_data[:status]
+      pid = error_data[:pid]
+
+      crashed = status.termsig
+      prefix = crashed ? "crash" : "bug"
+
+      filename = "#{dir_prefix}#{prefix}#{pid}"
+      while File.exists?(filename)
+        filename << ('a'..'z').to_a.sample
+      end
       File.open(filename, "wb") { |f| f.print(data) }
+
       puts "#{id} failed. Data was saved as #{filename}."
+      info(error_data)
     end
+
+    private
+
+      def dir_prefix
+        return "./" unless @dir
+        @dir
+      end
   end
 
   class Console
-    def handle(id, data, pid, status)
-      puts "#{id} failed. Data: #{data.inspect}"
+    include FuzzBert::Handler::ConsoleHelper
+
+    def handle(error_data)
+      puts "#{error_data[:id]} failed. Data: #{error_data[:data].inspect}"
+      info(error_data)
     end
   end
+  
 end
 
 

data/lib/fuzzbert/executor.rb

@@ -1,19 +1,27 @@
 class FuzzBert::Executor
 
-  attr_reader :pool_size, :limit, :handler
+  attr_reader :pool_size, :limit, :handler, :sleep_delay
 
   DEFAULT_POOL_SIZE = 4
   DEFAULT_LIMIT = -1
   DEFAULT_HANDLER = FuzzBert::Handler::FileOutput
+  DEFAULT_SLEEP_DELAY = 1
 
-  def initialize(suites, args = { 
+  DEFAULT_ARGS = {
     pool_size: DEFAULT_POOL_SIZE,
     limit: DEFAULT_LIMIT,
-    handler: DEFAULT_HANDLER.new
-  })
+    handler: DEFAULT_HANDLER.new,
+    sleep_delay: DEFAULT_SLEEP_DELAY
+  }
+
+  def initialize(suites, args = DEFAULT_ARGS)
+    raise ArgumentError.new("No test cases were passed") unless suites
+
+    args ||= DEFAULT_ARGS
     @pool_size = args[:pool_size] || DEFAULT_POOL_SIZE
     @limit = args[:limit] || DEFAULT_LIMIT
     @handler = args[:handler] || DEFAULT_HANDLER.new
+    @sleep_delay = args[:sleep_delay] || DEFAULT_SLEEP_DELAY
     @data_cache = {}
     @n = 0
     @exiting = false
@@ -31,123 +39,129 @@ class FuzzBert::Executor
 
   private
 
-  def run_instance(description, test, generator)
-    data = generator.to_data
-    pid = fork do
-      begin
-        test.run(data)
-      rescue StandardError
-        abort
+    def run_instance(description, test, generator)
+      data = generator.to_data
+      pid = fork do
+        begin
+          test.run(data)
+        rescue StandardError
+          abort
+        end
       end
+      id = "#{description}/#{generator.description}"
+      @data_cache[pid] = [id, data]
     end
-    id = "#{description}/#{generator.description}"
-    @data_cache[pid] = [id, data]
-  end
 
-  def trap_child_exit
-    trap(:CHLD) do 
-      begin
-        while exitval = Process.wait2(-1, Process::WNOHANG)
-          pid = exitval[0]
-          status = exitval[1]
-          data_ary = @data_cache.delete(pid)
-          unless status.success?
-            handle(data_ary[0], data_ary[1], pid, status) unless interrupted(status)
-          end
-          @n += 1
-          if @limit == -1 || @n < @limit
-            run_instance(*@producer.next) 
-          else
-            @running = false
+    def trap_child_exit
+      trap(:CHLD) do 
+        begin
+          while exitval = Process.wait2(-1, Process::WNOHANG)
+            pid = exitval[0]
+            status = exitval[1]
+            data_ary = @data_cache.delete(pid)
+            unless status.success?
+              handle({ 
+                id: data_ary[0], 
+                data: data_ary[1], 
+                pid: pid, 
+                status: status
+              }) unless interrupted(status)
+            end
+            @n += 1
+            if @limit == -1 || @n < @limit
+              run_instance(*@producer.next) 
+            else
+              @running = false
+            end
           end
+        rescue Errno::ECHILD
         end
-      rescue Errno::ECHILD
       end
     end
-  end
 
-  def trap_interrupt
-    trap(:INT) do
-      exit! (1) if @exiting
-      @exiting = true
-      graceful_exit
+    def trap_interrupt
+      trap(:INT) do
+        exit! (1) if @exiting
+        @exiting = true
+        graceful_exit
+      end
     end
-  end
 
-  def graceful_exit
-    puts "\nExiting...Interrupt again to exit immediately"
-    begin
-      while Process.wait; end
-    rescue Errno::ECHILD
+    def graceful_exit
+      puts "\nExiting...Interrupt again to exit immediately"
+      begin
+        while Process.wait; end
+      rescue Errno::ECHILD
+      end
+      exit 0
     end
-    exit 0
-  end
-
-  def handle(id, data, pid, status)
-    @handler.handle(id, data, pid, status)
-  end
 
-  def interrupted(status)
-    return false if status.exited?
-    return true if status.termsig == nil || status.termsig == 2
-  end
-
-  def conditional_sleep
-    sleep 0.1 until @running == false
-  end
+    def handle(error_data)
+      @handler.handle(error_data)
+    end
 
-  class DataProducer
-    def initialize(suites)
-      @ring = Ring.new(suites)
-      update
+    def interrupted(status)
+      return false if status.exited?
+      return true if status.termsig == nil || status.termsig == 2
     end
 
-    def update
-      @suite = @ring.next
-      @gen_iter = ProcessSafeEnumerator.new(@suite.generators)
+    def conditional_sleep
+      sleep @sleep_delay until @running == false
     end
 
-    def next
-      gen = nil
-      until gen
-        begin
-          gen = @gen_iter.next
-        rescue StopIteration
-          update
-        end
+    class DataProducer
+      def initialize(suites)
+        @ring = Ring.new(suites)
+        update
       end
-      [@suite.description, @suite.test, gen]
-    end
-      
-    class Ring
-      def initialize(objs)
-        @i = 0
-        objs = [objs] unless objs.respond_to?(:each)
-        @objs = objs.to_a
+
+      def update
+        @suite = @ring.next
+        @gen_iter = ProcessSafeEnumerator.new(@suite.generators)
       end
 
       def next
-        obj = @objs[@i]
-        @i = (@i + 1) % @objs.size
-        obj
+        gen = nil
+        until gen
+          begin
+            gen = @gen_iter.next
+          rescue StopIteration
+            update
+          end
+        end
+        [@suite.description, @suite.test, gen]
       end
-    end
+        
+      class Ring
+        def initialize(objs)
+          @i = 0
+          objs = [objs] unless objs.respond_to?(:each)
+          @objs = objs.to_a
+          raise ArgumentError.new("No test cases found") if @objs.empty?
+        end
 
-    #needed because the Fiber used for normal Enumerators has race conditions
-    class ProcessSafeEnumerator
-      def initialize(ary)
-        @i = 0
-        @ary = ary.to_a
+        def next
+          obj = @objs[@i]
+          @i = (@i + 1) % @objs.size
+          obj
+        end
       end
 
-      def next
-        obj = @ary[@i]
-        raise StopIteration unless obj
-        @i += 1
-        obj
+      #needed because the Fiber used for normal Enumerators has race conditions
+      class ProcessSafeEnumerator
+        def initialize(ary)
+          @i = 0
+          @ary = ary.to_a
+        end
+
+        def next
+          obj = @ary[@i]
+          raise StopIteration unless obj
+          @i += 1
+          obj
+        end
       end
     end
-  end
 
 end