fuzzbert 1.0.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Martin Boßlet
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,204 @@
+# FuzzBert [![Build Status](https://secure.travis-ci.org/krypt/FuzzBert.png?branch=master)](http://travis-ci.org/krypt/FuzzBert)
+
+A random testing/fuzzer framework for Ruby.
+
+Random Testing (or "fuzzing") is not really new, it has been around for quite
+some time. Yet it still hasn't found widespread adoption in everyday coding
+practices, much too often it is only used for the purpose of finding exploits
+for existing applications or libraries. FuzzBert wants to improve this situation.
+It's a simple fuzzing framework with an RSpec-like DSL that will allow you to 
+integrate random tests in your project with minimal effort.
+
+For further information on random testing, here are two excellent starting points:
+
+* [Udacity CS258](http://www.udacity.com/overview/Course/cs258/)
+* [Babysitting an Army of Monkeys](http://fuzzinginfo.files.wordpress.com/2012/05/cmiller-csw-2010.pdf)
+
+## Installation
+
+    gem install fuzzbert
+    fuzzbert --help
+
+## Defining a random test
+
+FuzzBert defines an RSpec-like DSL that can be used to define different fuzzing
+scenarios. The DSL uses three words: `fuzz`, `deploy` and `data`. `fuzz` can be
+thought of as defining a new scenario, such as "fuzz this command line tool",
+"fuzz this particular URL of my web app" or "fuzz this library method taking 
+external input".
+
+Within a `fuzz` block, there must be one occurence of `deploy` and one or several
+occurences of `data`. The `deploy` block is the spot where we deliver the random
+payload that has been generated. It is agnostic about the actual target in order to
+leave you free to fuzz whatever you require in your particular case. The `data`
+blocks define the shape of the random data being generated. There can be more than
+one such block because it is often beneficial to not only shoot completely random
+data at the target - you often want to deliver more structured data as well, trying
+to find the edge cases deeper within your code. Good random test suites make use
+of both - totally random data as well as structured data - in order to cover as
+much "code surface" as possible.
+
+Here is a quick example that fuzzes `JSON.parse`:
+
+```ruby
+require 'json'
+require 'fuzzbert'
+
+fuzz "JSON.parse" do
+
+  deploy do |data|
+    begin
+      JSON.parse data
+    rescue StandardError
+      #fine, we just want to capture crashes
+    end
+  end
+
+  data "completely random" do
+    FuzzBert::Generators.random
+  end
+
+  data "enclosing curly braces" do
+    c = FuzzBert::Container.new
+    c << FuzzBert::Generators.fixed("{")
+    c << FuzzBert::Generators.random
+    c << FuzzBert::Generators.fixed("}")
+    c.generator
+  end
+
+  data "my custom generator" do
+    prng = Random.new
+    lambda do
+      buf = '{ user: { '
+      buf << prng.bytes(100)
+      buf << ' } }'
+    end
+  end
+
+end
+```
+
+The `deploy` block takes the generated data as a parameter. The block itself is
+responsible of deploying the payload. An execution is considered successful if
+the `deploy` block passes with no uncaught error being raised. If an error slips
+through or if the Ruby process crashes altogether, the execution is of course
+considered as a failure. 
+
+`data` blocks must return a lambda or proc that takes no argument. You can either
+choose completely custom lambdas of your own or use those predefined for you in
+`FuzzBert::Generators`.
+
+## Templates
+
+Using the approach described so far is most useful for binary protocols, but as
+soon as largely String-based data is needed it can soon become tedious. What you
+actually want in these situations is some sort of template mechanism that comes
+with mostly fixed data and only replaces a few selected parts with randomly
+generated data. This, too, is possible with FuzzBert, it comes with a minimal
+templating language:
+
+```ruby
+require 'fuzzbert'
+
+fuzz "My Web App" do
+
+  deploy do |data|
+    # Send the data to your web app with httpclient or similar.
+    # You define the "error conditions": if a response to some
+    # data is not as expected, you could simply raise an error
+    # here.
+  end
+
+  data "JSON generated from a template" do
+    t = FuzzBert::Template.new '{ user: { id: ${id}, name: "${name}" } }'
+    t.set(:id) { FuzzBert::Generators.cycle(1..10000) }
+    t.set(:name) { FuzzBert::Generators.random }
+    t.generator
+  end
+
+end
+```
+
+## Mutators
+
+Mutation is the principle used in "Babysitting an Army of Monkeys". The basis for
+the mutation tests is a valid sample of input that is then modified in exactly one
+position in each test instance. You can apply this principle as follows:
+
+```ruby
+require 'fuzzbert'
+
+fuzz "Web App" do
+  deploy do |data|
+    #send JSON data via HTTP
+  end
+
+  data "mutated data" do
+    m = FuzzBert::Mutator.new '{ user: { id: 42, name: "FuzzBert" }'
+    m.generator
+  end
+
+end
+```
+
+## How the tests are performed
+
+Each `fuzz` block defines a `TestSuite`. These are executed in a round-robin manner.
+Each individual `TestSuite` will then apply the `deploy` block with a sample of
+data generated successively by each one of the `data` blocks. Once all `data` blocks
+were used, the next `TestSuite` will be executed and so on... By default, a FuzzBert
+fuzzing session runs forever, until the process is either killed or by manually hitting
+`CTRL+C` for example. This was a deliberate design choice since random testing suites
+need to be run for quite some time to be effective. It's something you want to run over
+the weekend rather than for a couple of minutes. Still, it can make sense to explicitly
+limit the number of runs, for example when integrating FuzzBert with a CI server or
+with Travis. You can do so by passing the `--limit` parameter to the `fuzzbert`
+executable.
+
+Every single execution of `deploy` is run in a separate process. The main reason for
+this is that we typically want to detect hard crashes when a C extension or even Ruby
+itself encounters an input it can't handle. Besides being able to cope with these cases,
+running in separate processes proves beneficial otherwise as well: by default, FuzzBert
+runs the tests in four separate processes at once, therefore utilizing your CPU's cores
+effectively. You can tweak that setting with `--pool-size` to set this number to 1 
+(for completely sequential runs) or to the exact number of cores your CPU offers.
+
+## What happens if we encounter a bug?
+
+If a test should end up failing (either the process crashed completely or caused an
+uncaught error), FuzzBert will output the failing test on your terminal and tell you
+where it stored the data that caused this. This conveniently allows you to run FuzzBert
+over the weekend and when you return on Monday, the troubleshooters will sit there all
+lined up for you to go through and filter. By using the `--console` command line switch
+you can tell FuzzBert to not explicitly store the data, but echoing the data that
+caused the crash to the terminal instead.
+
+## Rake integration
+
+You may integrate Rake tasks for FuzzBert similar to how you would include a task for
+Rspec:
+
+```ruby
+require 'rake'
+require 'fuzzbert/rake_task'
+
+FuzzBert::RakeTask.new(:fuzz) do |spec|
+  spec.fuzzbert_opts = ['--limit 10000000', '--console']
+  spec.pattern = 'fuzz/**/fuzz_*.rb'
+end
+```
+
+## Supported versions
+
+FuzzBert has been confirmed to run on CRuby 1.9.3 and Rubinius 2.0.0dev. Since
+it heavily relies on forking, it does not run on JRuby so far, but support is planned and
+on its way.
+
+You may also use FuzzBert for fuzzing arbitrary applications or libraries that aren't
+connected to Ruby at all - have a look in the examples that ship with FuzzBert.
+ 
+## License
+
+Copyright (c) 2012 Martin Boßlet. Distributed under the MIT License. See LICENSE for 
+further details.
+

data/bin/fuzzbert

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+begin
+  require 'fuzzbert'
+  FuzzBert::AutoRun.autorun
+rescue LoadError
+  $stderr.puts "Could not find fuzzbert."
+  exit(1)
+end

data/lib/fuzzbert.rb

@@ -0,0 +1,36 @@
+=begin
+
+= Info
+
+FuzzBert - Random Testing / Fuzzing in Ruby
+
+Copyright (C) 2012
+Martin Bosslet <martin.bosslet@googlemail.com>
+All rights reserved.
+
+= License
+
+See the file 'LICENSE' for further details.
+
+=end
+
+module FuzzBert
+
+  PRNG = Random.new
+
+end
+
+require_relative 'fuzzbert/version'
+require_relative 'fuzzbert/generation'
+require_relative 'fuzzbert/generators'
+require_relative 'fuzzbert/generator'
+require_relative 'fuzzbert/mutator'
+require_relative 'fuzzbert/template'
+require_relative 'fuzzbert/container'
+require_relative 'fuzzbert/test'
+require_relative 'fuzzbert/error_handler'
+require_relative 'fuzzbert/test_suite'
+require_relative 'fuzzbert/executor'
+require_relative 'fuzzbert/autorun'
+require_relative 'fuzzbert/dsl'
+

data/lib/fuzzbert/autorun.rb

@@ -0,0 +1,65 @@
+require 'optparse'
+
+module FuzzBert::AutoRun
+
+  TEST_CASES = []
+
+  module_function
+
+  def register(suite)
+    TEST_CASES << suite
+  end
+
+  def autorun
+    options, files = process_args(ARGV)
+    load_files(files)
+    run(options)
+  end
+
+  def run(options=nil)
+    executor = options ? FuzzBert::Executor.new(TEST_CASES, options) : FuzzBert::Executor.new(TEST_CASES)
+    executor.run
+  end
+
+  private; module_function
+
+  def load_files(files)
+    files.each do |pattern|
+      Dir.glob(pattern).each { |f| load File.expand_path(f) }
+    end
+  end
+
+  def process_args(args = [])
+    options = {}
+    orig_args = args.dup
+
+    OptionParser.new do |opts|
+      opts.banner  = 'FuzzBert options:'
+      opts.version = FuzzBert::VERSION
+
+      opts.on '-h', '--help', 'Display this help.' do
+        puts opts
+        exit
+      end
+
+      opts.on '--pool-size SIZE', Integer, "Sets the number of concurrently running processes to SIZE" do |n|
+        options[:pool_size] = n.to_i
+      end
+
+      opts.on '--limit LIMIT', Integer, "Instead of running permanently, fuzzing will be stopped after running LIMIT of instances" do |n|
+        p n
+        options[:limit] = n.to_i
+      end
+
+      opts.on '--console', "Output the failing cases including data on the console instead of saving them in a file" do
+        options[:handler] = FuzzBert::Handler::Console.new
+      end
+
+      opts.parse! args
+    end
+
+    [options, args]
+  end
+
+end
+

data/lib/fuzzbert/autorun.rb~

@@ -0,0 +1,33 @@
+
+module FuzzBert::AutoRun
+
+  TEST_CASES = []
+
+  module_function
+
+  def register(suite)
+    TEST_CASES << suite
+  end
+
+  def autorun
+    at_exit do
+        next if $! # don't run if there was an exception
+
+        load_files(ARGV)
+        run
+
+    end unless @@installed_at_exit
+    @@installed_at_exit = true
+  end
+
+  def run
+    FuzzBert::Executor.new(TEST_CASES).run
+  end
+
+  private
+
+  def load_files(args)
+  end
+
+end
+

data/lib/fuzzbert/container.rb

@@ -0,0 +1,21 @@
+
+class FuzzBert::Container
+  include FuzzBert::Generation
+
+  def initialize(generators=[])
+    @generators = generators
+  end
+
+  def <<(generator)
+    @generators << generator
+  end
+
+  def to_data
+    "".tap do |buf|
+      @generators.each { |gen| buf << gen.call }
+    end
+  end
+
+end
+
+

data/lib/fuzzbert/dsl.rb

@@ -0,0 +1,9 @@
+module FuzzBert::DSL
+  def fuzz(*args, &blk)
+    suite = FuzzBert::TestSuite.create(*args, &blk)
+    FuzzBert::AutoRun.register(suite)
+  end
+end
+
+extend FuzzBert::DSL
+Module.send(:include, FuzzBert::DSL)

data/lib/fuzzbert/error_handler.rb

@@ -0,0 +1,19 @@
+
+module FuzzBert::Handler
+  class FileOutput
+    def handle(id, data, pid, status)
+      prefix = status.termsig ? "crash" : "bug"
+      filename = "#{prefix}#{pid}"
+      File.open(filename, "wb") { |f| f.print(data) }
+      puts "#{id} failed. Data was saved as #{filename}."
+    end
+  end
+
+  class Console
+    def handle(id, data, pid, status)
+      puts "#{id} failed. Data: #{data.inspect}"
+    end
+  end
+end
+
+

data/lib/fuzzbert/executor.rb

@@ -0,0 +1,153 @@
+class FuzzBert::Executor
+
+  attr_reader :pool_size, :limit, :handler
+
+  DEFAULT_POOL_SIZE = 4
+  DEFAULT_LIMIT = -1
+  DEFAULT_HANDLER = FuzzBert::Handler::FileOutput
+
+  def initialize(suites, args = { 
+    pool_size: DEFAULT_POOL_SIZE,
+    limit: DEFAULT_LIMIT,
+    handler: DEFAULT_HANDLER.new
+  })
+    @pool_size = args[:pool_size] || DEFAULT_POOL_SIZE
+    @limit = args[:limit] || DEFAULT_LIMIT
+    @handler = args[:handler] || DEFAULT_HANDLER.new
+    @data_cache = {}
+    @n = 0
+    @exiting = false
+    @producer = DataProducer.new(suites)
+  end
+
+  def run
+    trap_child_exit
+    trap_interrupt
+
+    @pool_size.times { run_instance(*@producer.next) }
+    @running = true
+    @limit == -1 ? sleep : conditional_sleep
+  end
+
+  private
+
+  def run_instance(description, test, generator)
+    data = generator.to_data
+    pid = fork do
+      begin
+        test.run(data)
+      rescue StandardError
+        abort
+      end
+    end
+    id = "#{description}/#{generator.description}"
+    @data_cache[pid] = [id, data]
+  end
+
+  def trap_child_exit
+    trap(:CHLD) do 
+      begin
+        while exitval = Process.wait2(-1, Process::WNOHANG)
+          pid = exitval[0]
+          status = exitval[1]
+          data_ary = @data_cache.delete(pid)
+          unless status.success?
+            handle(data_ary[0], data_ary[1], pid, status) unless interrupted(status)
+          end
+          @n += 1
+          if @limit == -1 || @n < @limit
+            run_instance(*@producer.next) 
+          else
+            @running = false
+          end
+        end
+      rescue Errno::ECHILD
+      end
+    end
+  end
+
+  def trap_interrupt
+    trap(:INT) do
+      exit! (1) if @exiting
+      @exiting = true
+      graceful_exit
+    end
+  end
+
+  def graceful_exit
+    puts "\nExiting...Interrupt again to exit immediately"
+    begin
+      while Process.wait; end
+    rescue Errno::ECHILD
+    end
+    exit 0
+  end
+
+  def handle(id, data, pid, status)
+    @handler.handle(id, data, pid, status)
+  end
+
+  def interrupted(status)
+    return false if status.exited?
+    return true if status.termsig == nil || status.termsig == 2
+  end
+
+  def conditional_sleep
+    sleep 0.1 until @running == false
+  end
+
+  class DataProducer
+    def initialize(suites)
+      @ring = Ring.new(suites)
+      update
+    end
+
+    def update
+      @suite = @ring.next
+      @gen_iter = ProcessSafeEnumerator.new(@suite.generators)
+    end
+
+    def next
+      gen = nil
+      until gen
+        begin
+          gen = @gen_iter.next
+        rescue StopIteration
+          update
+        end
+      end
+      [@suite.description, @suite.test, gen]
+    end
+      
+    class Ring
+      def initialize(objs)
+        @i = 0
+        objs = [objs] unless objs.respond_to?(:each)
+        @objs = objs.to_a
+      end
+
+      def next
+        obj = @objs[@i]
+        @i = (@i + 1) % @objs.size
+        obj
+      end
+    end
+
+    #needed because the Fiber used for normal Enumerators has race conditions
+    class ProcessSafeEnumerator
+      def initialize(ary)
+        @i = 0
+        @ary = ary.to_a
+      end
+
+      def next
+        obj = @ary[@i]
+        raise StopIteration unless obj
+        @i += 1
+        obj
+      end
+    end
+  end
+
+end
+