timing_attack 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d03f591c320a5983d1a95d594e0ac067f79f9019
-  data.tar.gz: 587b251ddbe7ee8698fb9f1f6309ce2c0934b915
+  metadata.gz: 5b6f971fe9e7eb6b8aa282dffbd7c0eea3a2f009
+  data.tar.gz: e05deb06b3c31be39f128c7906ef3534abad6d09
 SHA512:
-  metadata.gz: 599c51ebac9e5b3ca49fec6d3364682427de3d6c85eda144b981c7138393cb658b2a88ac11f5a1eecef7b768569c3eb212a151634a34012758af18267fa05a26
-  data.tar.gz: c61b9cf40bcb33a9beed350d1f39b79a7f93c67430100c9fb021ca6bd23a72defc7a8814b7e7e5e78db61b2bbefe74ea02d3a5fa3a8eaf40e54433b6ba4e39bd
+  metadata.gz: c0e1f43f47fafd477678072023fcba97c6db2dbe12f6d09f6c1face92feed5fe6a17f2a16e79bb692e0db1b4e912f7136f5cd47d3f07e66dabafbf52ab293df3
+  data.tar.gz: 8977b5ddbea3da6e3511c1e9b0c9921d194a99e1cfabe7cbe104882579a949a562458fcc17adf72ded2ddb8cbc08e51cf4950349fe30046f9e342f85d5743e73

data/README.md

@@ -12,24 +12,21 @@ discrepancies in the application's response time.
 ## Usage
 
 ```
-timing_attack [options] -a <input> -b <input> -u <target> <inputs>
+timing_attack [options] -u <target> <inputs>
     -u, --url URL                    URL of endpoint to profile
-    -a, --a-example A_EXAMPLE        Known test case that belongs to Group A
-    -b, --b-example B_EXAMPLE        Known test case that belongs to Group B
     -n, --number NUM                 Requests per input
-        --a-name A_NAME              Name of Group A
-        --b-name B_NAME              Name of Group B
+    -t, --threshold NUM              Minimum threshold, in seconds, for meaningfulness (default: 0.05)
     -p, --post                       Use POST, not GET
     -q, --quiet                      Quiet mode (don't display progress bars)
+        --mean                       Use mean for calculations
+        --median                     Use median for calculations
+        --percentile N               Use Nth percentile for calculations
     -v, --version                    Print version information
     -h, --help                       Display this screen
 ```
 
-**NB**: If the provided examples are invalid, discvery will fail.  Always check
-your results!  If very similar inputs are being sorted differently, you may have
-used bad training data.
-
 ### An example
+
 Consider that we we want to gather information from a Rails server running
 locally at `http://localhost:3000`.  Let's say that we know the following:
 * `charles@poodles.com` exists in the database
@@ -41,34 +38,39 @@ the database.
 We execute (using `-q` to suppress the progress bar)
 ```bash
 % timing_attack -q -u http://localhost:3000/login \
-                -a charles@poodles.com -b invalid@fake.fake \
-                candidate@address.com other@address.com
+                candidate@address.com other@address.com \
+                charles@poodles.com invalid@fake.fake
 ```
 ```
-Group A:
-  candidate@address.com         ~0.1969s
-Group B:
-  other@address.com             ~0.1096s
+Short tests:
+  other@address.com             0.0926
+  invalid@fake.fake             0.0947
+Long tests:
+  candidate@address.com         0.1708
+  charles@poodles.com           0.1823
 ```
-`candidate@address.com` is in the same group as `charles@poodles.com` (Group A),
-while `other@address.com` is in Group B with `invalid@fake.fake`
-Thus we know that `candidate@address.com` exists in the database, and that
-`other@example.com` does not.
 
-To make things a bit friendlier, we can rename groups with the `--a-name` and
-`--b-name` options:
-```bash
-% timing_attack -q -u http://localhost:3000/login \
-                -a charles@poodles.com -b invalid@fake.fake \
-                --a-name 'Valid logins' --b-name 'Invalid logins' \
-                candidate@address.com other@address.com
-```
-```
-Valid logins:
-  candidate@address.com         ~0.1988s
-Invalid logins:
-  other@address.com             ~0.1065s
-```
+Note that you don't need to know anything about the database when attacking.  It
+is, however, nice to have a bit of information as a sanity check.
+
+## How it works
+
+The various inputs are each thrown at the endpoint `--number` times.  The
+`--percentile`th percentile of each input's results is considered the
+representative result for that input.  Inputs are then sorted according to
+their representative results and the largest spike in their graph is found.
+Results then split into short and long groups according to this spike.
+
+The `--mean` flag uses the average of results for a particular input as its
+representative result.  The `--median` flag simply uses the 50th percentile.
+According to [Crosby, Wallach, and
+Reidi](https://www.cs.rice.edu/~dwallach/pub/crosby-timing2009.pdf), results
+with percentiles above ~15, median, and mean are all quite noisy, so you should
+probably keep `--percentile` low.
+
+I was very surprised to find that I get correct results against remote targets
+with `--num` around 20.  Default is 5, as that has been sufficient in my tests
+for LAN and local targets.
 
 ## Contributing
 
@@ -76,14 +78,14 @@ Bug reports and pull requests are welcome [here](https://github.com/ffleming/tim
 
 ## Disclaimer
 
-TimingAttack is quick and dirty.  It is meant to exploit *known* timing attacks based
-upon two known values.  TimingAttack is *not* for discovering the existence of
-timing-based vulnerabilities.
+TimingAttack is quick and dirty.
 
 Also, don't use TimingAttack against machines that aren't yours.
 
 ## Todo
 * Tests
-* Better heuristic than naïve mean comparison
-* Auto-discovering heuristic that doesn't require example test cases
+* More intelligent filtering than nth-percentile + spike detection
+  * CW&R's box test
 * Customizable query parameters
+* Threading for requests?
+  * Custom or just use Typhoeus

data/exe/timing_attack

@@ -5,15 +5,17 @@ require 'optparse'
 options = {}
 opt_parser = OptionParser.new do |opts|
   opts.program_name = File.basename(__FILE__)
-  opts.banner = "#{opts.program_name} [options] -a <input> -b <input> -u <target> <inputs>"
+  opts.banner = "#{opts.program_name} [options] -u <target> <inputs>"
   opts.on("-u URL", "--url URL", "URL of endpoint to profile") { |str| options[:url] = str }
-  opts.on("-a A_EXAMPLE", "--a-example A_EXAMPLE", "Known test case that belongs to Group A") { |str| options[:a_example] = str }
-  opts.on("-b B_EXAMPLE", "--b-example B_EXAMPLE", "Known test case that belongs to Group B") { |str| options[:b_example] = str }
-  opts.on("-n NUM", "--number NUM", "Requests per input") { |num| options[:iterations] = num.to_i }
-  opts.on("--a-name A_NAME", "Name of Group A") { |str| options[:a_name] = str }
-  opts.on("--b-name B_NAME", "Name of Group B") { |str| options[:b_name] = str }
+  opts.on("-n NUM", "--number NUM", "Requests per input (default: 0.025)") { |num| options[:iterations] = num.to_i }
+  opts.on("-t NUM", "--threshold NUM", "Minimum threshold, in seconds, for meaningfulness (default: 0.025)") do |num|
+    options[:threshold] = num.to_f
+  end
   opts.on("-p", "--post", "Use POST, not GET") { |bool| options[:method] = bool ? :post : :get }
   opts.on("-q", "--quiet", "Quiet mode (don't display progress bars)") { |bool| options[:verbose] = !bool }
+  opts.on("--mean", "Use mean for calculations") { |bool| options[:mean] = bool }
+  opts.on("--median", "Use median for calculations") { |bool| options[:median] = bool }
+  opts.on("--percentile N", "Use Nth percentile for calculations (default: 10)") { |num| options[:percentile] = num.to_i }
   opts.on_tail("-v", "--version", "Print version information") do
     gem = Gem::Specification.find_by_name('timing_attack')
     puts "#{gem.name} #{gem.version}"
@@ -29,11 +31,18 @@ rescue OptionParser::InvalidOption => e
   puts opt_parser
   exit
 end
+options[:verbose] = true if options[:verbose].nil?
+if options[:percentile]
+  options.delete(:mean)
+elsif options[:median]
+  options[:percentile] = 50
+elsif options[:mean]
+  options.delete(:percentile)
+end
 
 begin
-  atk = TimingAttack::Attacker.new(inputs: ARGV, options: options)
+  atk = TimingAttack::CliAttacker.new(inputs: ARGV, options: options)
   atk.run!
-  puts atk
 rescue ArgumentError => e
   STDERR.puts e.message
   puts opt_parser

data/lib/timing_attack/cli_attacker.rb

@@ -0,0 +1,106 @@
+module TimingAttack
+  class CliAttacker
+    def initialize(inputs: [], options: {})
+      @options = DEFAULT_OPTIONS.merge(options)
+      raise ArgumentError.new("url is a required argument") unless options.has_key? :url
+      raise ArgumentError.new("Need at least 2 inputs") if inputs.count < 2
+      raise ArgumentError.new("Iterations can't be < 3") if iterations < 3
+      unless @options.has_key? :width
+        @options[:width] = inputs.dup.map(&:length).push(30).sort.last
+      end
+      @attacks = inputs.map { |input| TestCase.new(input: input, options: @options) }
+    end
+
+    def run!
+      puts "Target: #{url}" if verbose?
+      warmup!
+      attack!
+      puts report
+    end
+
+    private
+
+    attr_reader :attacks, :options, :grouper
+
+    def report
+      ret = ''
+      hsh = grouper.serialize
+      if hsh[:spike_delta] < threshold
+        ret << "\n* Spike delta of #{sprintf('%.4f', hsh[:spike_delta])} is less than #{sprintf('%.4f', threshold)} * \n\n"
+      end
+      [:short, :long].each do |sym|
+        ret << "#{sym.to_s.capitalize} tests:\n"
+        hsh.fetch(sym).each do |input, time|
+          ret << "  #{input.ljust(width)}"
+          ret << sprintf('%.4f', time) << "\n"
+        end
+      end
+      ret
+    end
+
+    def warmup!
+      2.times do
+        warmup = TestCase.new(input: attacks.sample.input, options: options)
+        warmup.test!
+      end
+    end
+
+    def attack!
+      iterations.times do
+        attacks.each do |attack|
+          attack.test!
+          attack_bar.increment
+        end
+      end
+    end
+
+    def grouper
+      return @grouper unless @grouper.nil?
+      group_by = if options.fetch(:mean, false)
+                    :mean
+                  else
+                    { percentile: options.fetch(:percentile) }
+                  end
+      @grouper = Grouper.new(attacks: attacks, group_by: group_by)
+    end
+
+    def attack_bar
+      return null_bar unless verbose?
+      @attack_bar ||= ProgressBar.create(title: "  Attacking".ljust(15),
+                                         total: iterations * attacks.length,
+                                         format: bar_format
+                                        )
+    end
+
+    def benchmark_bar
+      return null_bar unless verbose?
+      @benchmark_bar ||= ProgressBar.create(title: "  Benchmarking".ljust(15),
+                                            total: iterations * 2,
+                                            format: bar_format
+                                           )
+    end
+
+    def bar_format
+      @bar_format ||= "%t (%E) |%B|"
+    end
+
+    def null_bar
+      @null_bar_klass ||= Struct.new('NullProgressBar', :increment)
+      @null_bar ||= @null_bar_klass.new
+    end
+
+    %i(iterations url verbose width method mean percentile threshold).each do |sym|
+      define_method(sym) { options.fetch sym }
+    end
+    alias_method :verbose?, :verbose
+
+    DEFAULT_OPTIONS = {
+      verbose: false,
+      method: :get,
+      iterations: 5,
+      mean: false,
+      threshold: 0.025,
+      percentile: 10
+    }.freeze
+  end
+end

data/lib/timing_attack/grouper.rb

@@ -0,0 +1,99 @@
+module TimingAttack
+  class Grouper
+    attr_reader :short_tests, :long_tests
+    def initialize(attacks: , group_by: {})
+      @attacks = attacks
+      setup_grouping_opts!(group_by)
+      @short_tests = []
+      @long_tests = []
+      group_attacks
+      serialize
+      freeze
+    end
+
+    def serialize
+      @serialize ||= {}.tap do |h|
+        h[:attack_method] = test_method
+        h[:attack_args]   = test_args
+        h[:short]         = serialize_tests(short_tests)
+        h[:long]          = serialize_tests(long_tests)
+        h[:spike_delta]   = spike_delta
+      end
+    end
+
+    private
+
+    ALLOWED_TEST_SYMBOLS = %i(mean median percentile).freeze
+
+    attr_reader :test_method, :test_args, :attacks, :test_hash, :spike_delta
+
+    def setup_grouping_opts!(group_by)
+      case group_by
+      when Symbol
+        setup_symbol_opts!(group_by)
+      when Hash
+        setup_hash_opts!(group_by)
+      else
+        raise ArgumentError.new("Don't know what to do with #{group_by.class} #{group_by}")
+      end
+    end
+
+    def setup_symbol_opts!(symbol)
+      case symbol
+      when :mean
+        @test_method = :mean
+        @test_args = []
+      when :median
+        @test_method = :percentile
+        @test_args = [50]
+      when :percentile
+        @test_method = :percentile
+        @test_args = [10]
+      else
+        raise ArgumentError.new("Allowed symbols are #{ALLOWED_TEST_SYMBOLS.join(', ')}")
+      end
+    end
+
+    def setup_hash_opts!(hash)
+      raise ArgumentError.new("Must provide configuration to Grouper") if hash.empty?
+      key, value = hash.first
+      unless ALLOWED_TEST_SYMBOLS.include? key
+        raise ArgumentError.new("Allowed keys are #{ALLOWED_TEST_SYMBOLS.join(', ')}")
+      end
+      @test_method = key
+      @test_args = value.is_a?(Array) ? value : [value]
+    end
+
+    def value_from_test(test)
+      test.public_send(test_method, *test_args)
+    end
+
+    def serialize_tests(test_arr)
+      test_arr.each_with_object({}) do |test, ret|
+        ret[test.input] = value_from_test(test)
+      end
+    end
+
+    def group_attacks
+      spike = decorated_attacks.max { |a,b| a[:delta] <=> b[:delta] }
+      index = decorated_attacks.index(spike)
+      stripped = decorated_attacks.map {|a| a[:attack] }
+      @short_tests = stripped[0..(index-1)]
+      @long_tests = stripped[index..-1]
+      @spike_delta = spike[:delta]
+    end
+
+    def decorated_attacks
+      return @decorated_attacks unless @decorated_attacks.nil?
+      sorted = attacks.sort { |a,b| value_from_test(a) <=> value_from_test(b) }
+      @decorated_attacks = sorted.each_with_object([]).with_index do |(attack, memo), index|
+        delta = if index == 0
+                  0.0
+                else
+                  value_from_test(attack) - value_from_test(sorted[index-1])
+                end
+        memo << { attack: attack, delta: delta }
+      end
+    end
+  end
+end

data/lib/timing_attack/test_case.rb

@@ -5,6 +5,7 @@ module TimingAttack
       @input = input
       @options = options
       @times = []
+      @percentiles = []
     end
 
     def test!
@@ -21,36 +22,22 @@ module TimingAttack
       times.push(diff)
     end
 
-    def to_s
-      "#{input.ljust(options.fetch(:width))}~#{sprintf('%.4f', mean_time)}s"
+    def mean
+      times.reduce(:+) / times.size.to_f
     end
 
-    def derive_group_from(a_test: , b_test: )
-      unless a_test.is_a?(TestCase) && b_test.is_a?(TestCase)
-        raise ArgumentError.new("a_test and b_test must be TestCases")
+    def percentile(n)
+      raise ArgumentError.new("Can't have a percentile > 100") if n > 100
+      if percentiles[n].nil?
+        position = ((times.length - 1) * (n/100.0)).to_i
+        percentiles[n] = times.sort[position]
+      else
+        percentiles[n]
       end
-      d_a = (mean_time - a_test.mean_time).abs
-      d_b = (mean_time - b_test.mean_time).abs
-      @group_a = (d_a < d_b)
-    end
-
-    def group_a
-      raise ArgumentError.new("Have not yet determined group membership") if @group_a.nil?
-      @group_a
-    end
-    alias_method :group_a?, :group_a
-
-    def group_b
-      !group_a
-    end
-    alias_method :group_b?, :group_b
-
-    def mean_time
-      times.reduce(:+) / times.size.to_f
     end
 
     private
 
-    attr_reader :times, :options
+    attr_reader :times, :options, :percentiles
   end
 end

data/lib/timing_attack/version.rb

@@ -1,3 +1,3 @@
 module TimingAttack
-  VERSION = "0.1.0"
+  VERSION = "0.2.1"
 end

data/lib/timing_attack.rb

@@ -1,8 +1,9 @@
 require 'httparty'
 require 'ruby-progressbar'
 require "timing_attack/version"
+require "timing_attack/grouper"
 require "timing_attack/test_case"
-require "timing_attack/attacker"
+require "timing_attack/cli_attacker"
 
 module TimingAttack
 end

data/timing_attack.gemspec

@@ -9,7 +9,7 @@ Gem::Specification.new do |spec|
   spec.authors       = ["Forrest Fleming"]
   spec.email         = ["ffleming@gmail.com"]
 
-  spec.summary       = "Performtiming attacks against web applications"
+  spec.summary       = "Perform timing attacks against web applications"
   spec.description   = "Profile web applications by noting differences in response times based on input values"
   spec.homepage      = "https://www.github.com/ffleming/timing_attack"
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: timing_attack
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.1
 platform: ruby
 authors:
 - Forrest Fleming
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-07-16 00:00:00.000000000 Z
+date: 2016-07-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ruby-progressbar
@@ -99,7 +99,8 @@ files:
 - bin/setup
 - exe/timing_attack
 - lib/timing_attack.rb
-- lib/timing_attack/attacker.rb
+- lib/timing_attack/cli_attacker.rb
+- lib/timing_attack/grouper.rb
 - lib/timing_attack/test_case.rb
 - lib/timing_attack/version.rb
 - timing_attack.gemspec
@@ -126,5 +127,5 @@ rubyforge_project:
 rubygems_version: 2.4.8
 signing_key: 
 specification_version: 4
-summary: Performtiming attacks against web applications
+summary: Perform timing attacks against web applications
 test_files: []

data/lib/timing_attack/attacker.rb

@@ -1,130 +0,0 @@
-module TimingAttack
-  class Attacker
-    def initialize(inputs: [], options: {})
-      @options = DEFAULT_OPTIONS.merge(options)
-      unless @options.has_key? :width
-        @options[:width] = [a_name, b_name, *inputs].map(&:length).push(30).sort.last
-      end
-      %i(a_example b_example url).each do |arg|
-        raise ArgumentError.new("#{arg} is a required argument") unless options.has_key? arg
-      end
-      @attacks = inputs.map { |input| TestCase.new(input: input, options: @options) }
-    end
-
-    def run!
-      puts "Target: #{url}" if verbose?
-      warmup!
-      benchmark!
-      attack!
-    end
-
-    def to_s
-      ret = ""
-      if verbose?
-        ret << "Benchmark results\n"
-        ret << "  #{a_name.ljust(width)}~#{sprintf('%.4f', a_benchmark.mean_time,)}s\n"
-        ret << "  #{b_name.ljust(width)}~#{sprintf('%.4f', b_benchmark.mean_time)}s\n"
-      end
-      ret << attack_string
-    end
-
-    private
-
-    attr_reader :attacks, :options
-
-    def warmup!
-      @warmup_test ||= TestCase.new(input: a_example, options: options)
-      2.times { @warmup_test.test! }
-    end
-
-    def benchmark!
-      iterations.times do
-        [a_benchmark, b_benchmark].each do |test_case|
-          test_case.test!
-          benchmark_bar.increment
-        end
-      end
-    end
-
-    def attack!
-      iterations.times do
-        attacks.each do |attack|
-          attack.test!
-          attack_bar.increment
-        end
-      end
-      attacks.each { |attack| attack.derive_group_from(a_test: a_benchmark, b_test: b_benchmark) }
-    end
-
-    def a_benchmark
-      @a_benchmark ||= TestCase.new(input: a_example, options: options)
-    end
-
-    def b_benchmark
-      @b_benchmark ||= TestCase.new(input: b_example, options: options)
-    end
-
-    def indent(str)
-      "  #{str.ljust(width)}"
-    end
-
-    def a_attacks
-      attacks.select { |a| a.group_a? }
-    end
-
-    def b_attacks
-      attacks.select { |a| a.group_b? }
-    end
-
-    def attack_string
-      ret = ""
-      unless a_attacks.empty?
-        ret << "#{a_name}:\n"
-        ret << a_attacks.map {|a| indent(a.to_s)}.join("\n")
-      end
-      unless b_attacks.empty?
-        ret << "\n#{b_name}:\n"
-        ret << b_attacks.map {|a| indent(a.to_s)}.join("\n")
-      end
-      "#{ret}\n"
-    end
-
-    def attack_bar
-      return null_bar unless verbose?
-      @attack_bar ||= ProgressBar.create(title: "  Attacking".ljust(15),
-                                         total: iterations * attacks.length,
-                                         format: bar_format
-                                        )
-    end
-
-    def benchmark_bar
-      return null_bar unless verbose?
-      @benchmark_bar ||= ProgressBar.create(title: "  Benchmarking".ljust(15),
-                                            total: iterations * 2,
-                                            format: bar_format
-                                           )
-    end
-
-    def bar_format
-      @bar_format ||= "%t (%E) |%B|"
-    end
-
-    def null_bar
-      @null_bar_klass ||= Struct.new('NullProgressBar', :increment)
-      @null_bar ||= @null_bar_klass.new
-    end
-
-    %i(iterations url verbose a_name b_name a_example b_example width method).each do |sym|
-      define_method(sym) { options.fetch sym }
-    end
-    alias_method :verbose?, :verbose
-
-    DEFAULT_OPTIONS = {
-      verbose: true,
-      a_name: "Group A",
-      b_name: "Group B",
-      method: :get,
-      iterations: 5
-    }.freeze
-  end
-end