benchmark-trend 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3396e746a6b1b03c60abcfce59c39bf7ad59eb182092b6900bd5cef6d9a21b8d
-  data.tar.gz: b6dc71b8aa5aa02dd45d6cfe4e3c423006c0aa153573a84f1f476d13dbee5b57
+  metadata.gz: 2684a8a9c5ed41c53e7510602d352dccb111150e7ce757567f3e1340a679eb5f
+  data.tar.gz: 8b7363a4ca90e53557d1a4397e30a5a37a93515976a42773c5225d1715e00ae3
 SHA512:
-  metadata.gz: 03466650b2858047f192477c9c69e9d4f41b27ca6138acfed84f6ef2024346672b3b7d087aabd85d8259c24d858410e6b8efaa05d4669a607a70d824136295c7
-  data.tar.gz: 8036b5662f6392c6b13a9d8d69a03b700294e909f1496de46c8582a95fbdcbcf6d62b61e901d6af1b659e2056d5e724d185ab2c061dc3750d32c7857de9c9843
+  metadata.gz: 4d27391b226536bb79d6dbe5e75ba3d66cf2362993b9d6afcb5409ea2a8a999deef276d7708a6c0bfa9b28eeeaffd9f31d8b2238e9f3daff21e3357baa80397d
+  data.tar.gz: bdae380f8bdaf37432cda9f03820421fb9f7fb00f9633d0c8a9a0b5b575f566ef905f9ec70ab83c4d0eaeb64162c2a43d7fe659722c90576db40ad621a553ed6

data/CHANGELOG.md

@@ -1,7 +1,23 @@
 # Change log
 
+## [v0.2.0] - 2018-09-30
+
+### Added
+* Add ability to measure monotonic time
+* Add ability to repeat measurements to increase stability of execution times
+
+### Changed
+* Change to prefer simpler complexity for similar measurements
+* Change to use monotonic clock
+* Change to differentiate linear vs logarithmic complexity for small values
+* Change to differentiate linear vs constant complexity for small values
+
+## Fixed
+* Fix fit_power to correctly calculate slope and intercept
+
 ## [v0.1.0] - 2018-09-08
 
 * Inital implementation and release
 
+[v0.2.0]: https://github.com/piotrmurach/benchmark-trend/compare/v0.1.0...v0.2.0
 [v0.1.0]: https://github.com/piotrmurach/benchmark-trend/compare/v0.1.0

data/README.md

@@ -14,7 +14,7 @@
 [coverage]: https://coveralls.io/github/piotrmurach/benchmark-trend?branch=master
 [inchpages]: http://inch-ci.org/github/piotrmurach/benchmark-trend
 
-> Measure pefromance trends of Ruby code based on the input size distribution.
+> Measure performance trends of Ruby code based on the input size distribution.
 
 **Benchmark::Trend** will help you estimate the computational complexity of Ruby code by running it on inputs increasing in size, measuring their execution times, and then fitting these observations into a model that best predicts how a given Ruby code will scale as a function of growing workload.
 
@@ -43,11 +43,14 @@ Or install it yourself as:
 ## Contents
 
 * [1. Usage](#1-usage)
-* [2. API](#2--api)
+* [2. API](#2-api)
   * [2.1 range](#21-range)
   * [2.2 infer_trend](#22-infer_trend)
+    * [2.2.1 repeat](#221-repeat)
   * [2.3 fit](#23-fit)
   * [2.4 fit_at](#24-fit_at)
+* [3. Examples](#3-examples)
+  * [3.1 Ruby array max](#31-ruby-array-max)
 
 ## 1. Usage
 
@@ -59,12 +62,12 @@ def fibonacci(n)
 end
 ```
 
-To measure the actual complexity of above function, we will use `infer_tren` method and pass it as a first argument an array of integer sizes and a block to execute the method:
+To measure the actual complexity of above function, we will use `infer_trend` method and pass it as a first argument an array of integer sizes and a block to execute the method:
 
 ```ruby
 numbers = Benchmark::Trend.range(1, 28, ratio: 2)
 
-trend, trends = Benchmark::Trend.infer_trend(numbers) do |n|
+trend, trends = Benchmark::Trend.infer_trend(numbers) do |n, i|
   fibonacci(n)
 end
 ```
@@ -134,7 +137,7 @@ Benchmark::Trend.range(8, 8 << 10, ratio: 2)
 
 ### 2.2 infer_trend
 
-To calculate an asymptotic behaviour of Rub code by inferring its computational complexity use `infer_trend`. This method takes as an argument an array of inputs which can be generated using [range](#21-range). The code to measure needs to be provided inside a block.
+To calculate an asymptotic behaviour of Ruby code by inferring its computational complexity use `infer_trend`. This method takes as an argument an array of inputs which can be generated using [range](#21-range). The code to measure needs to be provided inside a block. Two parameters are always yielded to a block, first, the actual data input and second the current index matching the input.
 
 For example, let's assume you would like to find out asymptotic behaviour of a Fibonacci algorithm:
 
@@ -154,7 +157,7 @@ numbers = Benchmark::Trend.range(1, 32, ratio: 2)
 Then measure the performance of the Fibonacci algorithm for each of the data points and fit the observations into a model to predict behaviour as a function of input size:
 
 ```ruby
-trend, trends = Benchmark::Trend.infer_trend(numbers) do |n|
+trend, trends = Benchmark::Trend.infer_trend(numbers) do |n, i|
   fibonacci(n)
 end
 ```
@@ -204,6 +207,23 @@ print trends[trend]
 #  :residual=>0.9052392775178072}
 ```
 
+### 2.2.1 repeat
+
+To increase stability of you tests consider repeating all time execution measurements using `:repeat` keyword.
+
+Start by generating a range of inputs for your algorithm:
+
+```ruby
+numbers = Benchmark::Trend.range(1, 32, ratio: 2)
+# => [1, 2, 4, 8, 16, 32]
+```
+
+and then run your algorithm for each input repeating measurements `100` times:
+
+```ruby
+Benchmark::Trend.infer_trend(numbers, repeat: 100) { |n, i| ... }
+```
+
 ### 2.3 fit
 
 Use `fit` method if you wish to fit arbitrary data into a model with a slope and intercept parameters that minimize the error.
@@ -264,6 +284,51 @@ Benchamrk::Trend.fit_at(:exponential, slope: 1.382889711685203, intercept: 3.822
 
 This means Fibonacci recursive algorithm will take about 1.45 year to complete!
 
+## 3. Examples
+
+### 3.1 Ruby array max
+
+Suppose you wish to find an asymptotic behaviour of Ruby built Array `max` method.
+
+You could start with generating a [range](#21-range) of inputs:
+
+```ruby
+array_sizes = Benchmark::Trend.range(1, 100_000)
+# => [1, 8, 64, 512, 4096, 32768, 100000]
+```
+
+Next, based on the generated ranges create arrays containing randomly generated integers:
+
+```ruby
+number_arrays = array_sizes.map { |n| Array.new(n) { rand(n) } }
+```
+
+Then feed this information to infer a trend:
+
+```ruby
+trend, trends = Benchmark::Trend.infer_trend(array_sizes) do |n, i|
+  number_arrays[i].max
+end
+```
+
+Unsuprisingly, we discover that Ruby's `max` call scales linearily with the input size:
+
+```ruby
+print trend
+# => linear
+```
+
+We can also see from the residual value that this is a near perfect fit:
+
+```ruby
+print trends[trend]
+# =>
+# {:trend=>"0.00 + 0.00*x",
+#  :slope=>5.873536409841244e-09,
+#  :intercept=>3.028647045635842e-05,
+#  :residual=>0.9986764704492359}
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/examples/fib_constant.rb

@@ -0,0 +1,16 @@
+require_relative '../lib/benchmark-trend'
+
+# constant
+def fib_const(n)
+  phi = (1 + Math.sqrt(5))/2
+  (phi ** n / Math.sqrt(5)).round
+end
+
+numbers = Benchmark::Trend.range(1, 1400, ratio: 2)
+trend, trends = Benchmark::Trend.infer_trend(numbers, repeat: 100) do |n|
+  fib_const(n)
+end
+
+puts "Trend: #{trend}"
+puts "Trend data:"
+pp trends

data/examples/fib_linear.rb

@@ -0,0 +1,17 @@
+require_relative '../lib/benchmark-trend'
+
+# linear
+def fib_iter(n)
+  a, b = 0, 1
+  n.times { a, b = b, a + b}
+  a
+end
+
+numbers = Benchmark::Trend.range(1, 20_000)
+trend, trends = Benchmark::Trend.infer_trend(numbers) do |n|
+  fib_iter(n)
+end
+
+puts "Trend: #{trend}"
+puts "Trend data:"
+pp trends

data/lib/benchmark/trend.rb

@@ -14,6 +14,31 @@ module Benchmark
       private_class_method(method)
     end
 
+    if defined?(Process::CLOCK_MONOTONIC)
+      # Object representing current time
+      def time_now
+        Process.clock_gettime Process::CLOCK_MONOTONIC
+      end
+      module_function :time_now
+    else
+      # Object represeting current time
+      def time_now
+        Time.now
+      end
+      module_function :time_now
+    end
+
+    # Measure time elapsed with a monotonic clock
+    #
+    # @public
+    def clock_time
+      before = time_now
+      yield
+      after = time_now
+      after - before
+    end
+    module_function :clock_time
+
     # Generate a range of inputs spaced by powers.
     #
     # The default range is generated in the multiples of 8.
@@ -66,18 +91,25 @@ module Benchmark
     # @param [Array[Numeric]] data
     #   the data to run measurements for
     #
+    # @param [Integer] repeat
+    #   nubmer of times work is called to compute execution time
+    #
     # @return [Array[Array, Array]]
     #
     # @api public
-    def measure_execution_time(data = nil, &work)
+    def measure_execution_time(data = nil, repeat: 1, &work)
       inputs = data || range(1, 10_000)
       times  = []
 
-      inputs.each do |input|
+      inputs.each_with_index do |input, i|
         GC.start
-        times << ::Benchmark.realtime do
-          work.(input)
+        measurements = []
+
+        repeat.times do
+          measurements << clock_time { work.(input, i) }
         end
+
+        times << measurements.reduce(&:+).to_f / measurements.size
       end
       [inputs, times]
     end
@@ -126,7 +158,7 @@ module Benchmark
 
     # Finds a line of best fit that approxmimates power function
     #
-    # Function form: y = ax^b
+    # Function form: y = bx^a
     #
     # @return [Numeric, Numeric, Numeric]
     #   returns a, b, and rr values
@@ -136,7 +168,7 @@ module Benchmark
       a, b, rr = fit(xs, ys, tran_x: ->(x) { Math.log(x) },
                              tran_y: ->(y) { Math.log(y) })
 
-      [Math.exp(b), a, rr]
+      [a, Math.exp(b), rr]
     end
     module_function :fit_power
 
@@ -172,7 +204,7 @@ module Benchmark
     #
     # @api public
     def fit(xs, ys, tran_x: ->(x) { x }, tran_y: ->(y) { y })
-      eps    = 0.000001
+      eps    = (10 ** -10)
       n      = 0
       sum_x  = 0.0
       sum_x2 = 0.0
@@ -193,9 +225,11 @@ module Benchmark
       tx  = n * sum_x2 - sum_x ** 2
       ty  = n * sum_y2 - sum_y ** 2
 
+      is_linear = tran_x.(Math::E) * tran_y.(Math::E) == Math::E ** 2
+
       if tx.abs < eps # no variation in xs
         raise ArgumentError, "No variation in data #{xs}"
-      elsif ty.abs < eps # no variation in ys - constant fit
+      elsif ty.abs < eps && is_linear # no variation in ys - constant fit
         slope = 0
         intercept = sum_y / n
         residual_sq = 1 # doesn't exist
@@ -249,7 +283,7 @@ module Benchmark
       case type
       when :logarithmic, :log
         "%.2f + %.2f*ln(x)"
-      when :linear
+      when :linear, :constant
         "%.2f + %.2f*x"
       when :power
         "%.2f * x^%.2f"
@@ -268,14 +302,18 @@ module Benchmark
     #
     # Fits the executiom times for each range to several fit models.
     #
+    # @param [Integer] repeat
+    #   nubmer of times work is called to compute execution time
+    #
     # @yieldparam work
+    #   the block of which the complexity is measured
     #
     # @return [Array[Symbol, Hash]]
     #   the best fitting and all the trends
     #
     # @api public
-    def infer_trend(data, &work)
-      ns, times = *measure_execution_time(data, &work)
+    def infer_trend(data, repeat: 1, &work)
+      ns, times = *measure_execution_time(data, repeat: repeat, &work)
       best_fit = :none
       best_residual = 0
       fitted = {}
@@ -287,9 +325,12 @@ module Benchmark
         a, b, rr = *send(:"fit_#{fit}", ns, times)
         # goodness of model
         aic = n * (Math.log(Math::PI) + 1) + n * Math.log(rr / n)
+        if a == 0 && fit == :linear
+          fit = :constant
+        end
         fitted[fit] = { trend: format_fit(fit) % [a, b],
                         slope: a, intercept: b, residual: rr }
-        if rr > best_residual && aic > best_aic
+        if rr >= best_residual && aic >= best_aic
           best_residual = rr
           best_fit = fit
           best_aic = aic

data/lib/benchmark/trend/version.rb

@@ -2,6 +2,6 @@
 
 module Benchmark
   module Trend
-    VERSION = "0.1.0"
+    VERSION = "0.2.0"
   end # Trend
 end # Benchmark

data/spec/unit/fit_power_spec.rb

@@ -3,12 +3,12 @@
 RSpec.describe Benchmark::Trend, '#fit_power' do
   it 'calculates perfect power fit' do
     xs = [1, 2, 3, 4, 5]
-    ys = xs.map { |x| 1.5*(x ** 2) }
+    ys = xs.map { |x| 1.5 * (x ** 2) }
 
     a, b, rr = Benchmark::Trend.fit_power(xs, ys)
 
-    expect(a).to be_within(0.001).of(1.5)
-    expect(b).to be_within(0.001).of(2.0)
+    expect(a).to be_within(0.001).of(2.0)
+    expect(b).to be_within(0.001).of(1.5)
     expect(rr).to be_within(0.001).of(1.0)
   end
 
@@ -19,8 +19,8 @@ RSpec.describe Benchmark::Trend, '#fit_power' do
 
     a, b, rr = Benchmark::Trend.fit_power(xs, ys)
 
-    expect(a).to be_within(0.001).of(1.0)
-    expect(b).to be_within(0.001).of(1.5)
+    expect(a).to be_within(0.001).of(1.5)
+    expect(b).to be_within(0.001).of(1.0)
     expect(rr).to be_within(0.001).of(0.999)
   end
 end

data/spec/unit/infer_trend_spec.rb

@@ -23,15 +23,23 @@ RSpec.describe Benchmark::Trend, '#infer_trend' do
     a
   end
 
-  # constant
+  # logarithmic
   def fib_const(n)
     phi = (1 + Math.sqrt(5))/2
     (phi ** n / Math.sqrt(5)).round
   end
 
+  it "infers constant trend" do
+    numbers = Benchmark::Trend.range(1, 100_000)
+    trend, = Benchmark::Trend.infer_trend(numbers, repeat: 100) do |n|
+      n
+    end
+
+    expect(trend).to eq(:constant)
+  end
+
   it "infers fibonacci classic algorithm trend to be exponential" do
-    numbers = Benchmark::Trend.range(1, 28, ratio: 2)
-    trend, trends = Benchmark::Trend.infer_trend(numbers) do |n|
+    trend, trends = Benchmark::Trend.infer_trend((1..20), repeat: 10) do |n|
       fibonacci(n)
     end
 
@@ -52,29 +60,38 @@ RSpec.describe Benchmark::Trend, '#infer_trend' do
     expect(trend).to eq(:linear)
   end
 
-  it "infers fibonacci constant algorithm trend to be linear" do
-    numbers = Benchmark::Trend.range(1, 500)
-    trend, trends = Benchmark::Trend.infer_trend(numbers) do |n|
+  it "infers fibonacci constant algorithm trend to be constant" do
+    # exponetiation by squaring has logarithmic complexity
+    numbers = Benchmark::Trend.range(1, 1400, ratio: 2)
+    trend, trends = Benchmark::Trend.infer_trend(numbers, repeat: 100) do |n|
       fib_const(n)
     end
 
-    expect(trend).to eq(:linear)
-    expect(trends[trend][:slope]).to eq(0)
+    expect(trend).to eq(:constant)
+    expect(trends[trend][:slope]).to be_within(0.0001).of(0)
   end
 
   it "infers finding maximum value trend to be linear" do
     array_sizes = Benchmark::Trend.range(1, 100_000)
-    number_arrays = array_sizes.map { |n| Array.new(n) { rand(n) } }.each
+    numbers = array_sizes.map { |n| Array.new(n) { rand(n) } }
 
-    trend, trends = Benchmark::Trend.infer_trend(array_sizes) do
-      number_arrays.next.max
+    trend, trends = Benchmark::Trend.infer_trend(array_sizes, repeat: 10) do |n, i|
+      numbers[i].max
     end
 
     expect(trend).to eq(:linear)
-    expect(trends).to match(
-      hash_including(:exponential, :power, :linear, :logarithmic))
-    expect(trends[:exponential]).to match(
-      hash_including(:trend, :slope, :intercept, :residual)
-    )
+    expect(trends[trend][:slope]).to be_within(0.0001).of(0)
+  end
+
+  it "infers binary search trend to be constant" do
+    range = Benchmark::Trend.range(10, 8 << 10, ratio: 2)
+    numbers = range.reduce([]) { |acc, n| acc << (1..n).to_a; acc }
+
+    trend, trends = Benchmark::Trend.infer_trend(range, repeat: 100) do |n, i|
+      numbers[i].bsearch { |x| x == n/2 }
+    end
+
+    expect(trend).to eq(:constant)
+    expect(trends[trend][:slope]).to be_within(0.0001).of(0)
   end
 end

data/spec/unit/measure_execution_time_spec.rb

@@ -2,7 +2,7 @@
 #
 RSpec.describe Benchmark::Trend, "#measure_execution_time" do
   it "measures performance times" do
-    func = -> (x) { x ** 2 }
+    func = -> (x, i) { x ** 2 }
 
     data = Benchmark::Trend.measure_execution_time(&func)
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: benchmark-trend
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Piotr Murach
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-09-08 00:00:00.000000000 Z
+date: 2018-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -70,6 +70,8 @@ files:
 - benchmark-trend.gemspec
 - bin/console
 - bin/setup
+- examples/fib_constant.rb
+- examples/fib_linear.rb
 - exe/bench-trend
 - lib/benchmark-trend.rb
 - lib/benchmark/trend.rb