viiite 0.1.0 → 0.2.0

data/CHANGELOG.md

@@ -1,3 +1,73 @@
+# 0.2.0
+
+## Awesome new features
+
+* Viiite is now able to work on complete benchmark suites (located in
+  './benchmarks' and looking for '**/*.rb' files by default). Subcommands
+  automatically resolve benchmark names passed as arguments. For example:
+
+    ./benchmarks/
+      bench_iteration.rb     -> viiite run bench_iteration
+      Array/
+        bench_sort.rb        -> viiite run Array/bench_sort
+
+* Default behavior remains compatible with 0.1.0:
+  * 'run', 'plot' and 'report' accept a benchmark file as first argument and
+    bypass the benchmark suite in this case.
+  * when invoked without any argument, 'plot' and 'report' assume a .rash stream
+    on standard input
+
+* For each benchmark, the results of the last run are saved in a cache. 'plot'
+  and 'report' will use data in the cache if available or automatically run the
+  benchmark.
+
+* The default behavior may be controlled through the following main options:
+  --suite=[FOLDER], --pattern=[GLOB], --[no-]cache=[FOLDER], --append, --write. Example:
+
+    # Run all benchmarks on all rubies, 10 times each, appending all results
+    # in cache
+    rvm exec viiite --append run --runs=10
+
+## Enhancements to individual commands
+
+* 'viiite run'
+
+  * Added a --runs=NB option to run the same benchmark NB times. This allows
+    keeping benchmarks clean; no NB.times{ ... } or range_over(1..NB, :run)
+    for obtaining a representative sample are needed in the benchmark itself.
+  * The previous option comes hand-in-hand with --run-key=KEY that allows
+    specifying the attribute name for the run number (defaults to :run)
+  * Without argument, runs all the benchmarks of the suite
+
+* 'viiite plot'
+
+  * Added --highcharts options; outputs a json array containing one hash for
+    each chart, to be used as options Highcharts.Chart(...) (cfr. highcharts.js)
+  * Added -d option, to debug; outputs the query result that would be used by
+    the concrete --xxx formatter.
+
+* 'viiite report'
+
+  * output has been improved: user, system, total and real time are explicitely
+    shown (different attributes/columns instead of one tms measure).
+  * added a '--ff=FORMAT' option for float format of the different measures
+    (defaults to %.6f)
+  * added a '--stddev=[MEASURE]' option for displaying standard deviation over
+    all regrouped runs (on a specific measure defaulting to tms.total)
+
+## Other improvements
+
+  * The garbage collector is started before running report{ ... } blocks
+  * Viiite does not depend on benchmark (stdlib) anymore
+  * Improved Viiite.which_ruby; it should always output something like
+    '#{rubyname} #{version}#{patchlevel}' from now on
+
+## Bug fixes
+
+* Fixed a bug when using 'viiite report --regroup=x,y,z --hierarchy' with more
+  than 2 regrouping attributes.
+
+
 # 0.1.0 / 2011-08-20
 
 * Enhancements

data/Gemfile

@@ -11,15 +11,8 @@ group :release do
   gem "wlang", "~> 0.10.2"
 end
 
-group :doc do
-  gem "yard", "~> 0.7.2"
-  gem "bluecloth", "~> 2.1.0"
-  gem "jekyll", "~> 0.11.0"
-end
-
 group :runtime do
-  gem "alf", "~> 0.10.0"
-  gem "quickl", "~> 0.4.0"
+  gem "alf", "~> 0.10.1"
   gem "gnuplot", "~> 2.3.6"
+  gem "json", ">= 0"
 end
-

data/Gemfile.lock

@@ -1,32 +1,16 @@
 GEM
   remote: http://rubygems.org/
   specs:
-    albino (1.3.3)
-      posix-spawn (>= 0.3.6)
-    alf (0.10.0)
-      myrrha (~> 1.2.0)
-      quickl (~> 0.4.0)
-    bluecloth (2.1.0)
-    classifier (1.3.3)
-      fast-stemmer (>= 1.0.0)
-    diff-lcs (1.1.2)
-    directory_watcher (1.4.0)
-    fast-stemmer (1.0.0)
+    alf (0.10.1)
+      highline (~> 1.6.2)
+      myrrha (~> 1.2.1)
+      quickl (~> 0.4.1)
+    diff-lcs (1.1.3)
     gnuplot (2.3.6)
-    jekyll (0.11.0)
-      albino (>= 1.3.2)
-      classifier (>= 1.3.1)
-      directory_watcher (>= 1.1.1)
-      kramdown (>= 0.13.2)
-      liquid (>= 1.9.0)
-      maruku (>= 0.5.9)
-    kramdown (0.13.3)
-    liquid (2.2.2)
-    maruku (0.6.0)
-      syntax (>= 1.0.0)
-    myrrha (1.2.0)
-    posix-spawn (0.3.6)
-    quickl (0.4.0)
+    highline (1.6.2)
+    json (1.5.3)
+    myrrha (1.2.1)
+    quickl (0.4.1)
     rake (0.9.2)
     rspec (2.6.0)
       rspec-core (~> 2.6.0)
@@ -36,20 +20,15 @@ GEM
     rspec-expectations (2.6.0)
       diff-lcs (~> 1.1.2)
     rspec-mocks (2.6.0)
-    syntax (1.0.0)
     wlang (0.10.2)
-    yard (0.7.2)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  alf (~> 0.10.0)
-  bluecloth (~> 2.1.0)
+  alf (~> 0.10.1)
   gnuplot (~> 2.3.6)
-  jekyll (~> 0.11.0)
-  quickl (~> 0.4.0)
+  json
   rake (~> 0.9.2)
   rspec (~> 2.6.0)
   wlang (~> 0.10.2)
-  yard (~> 0.7.2)

data/README.md

@@ -1,16 +1,16 @@
 # Viiite
 
-Viiite brings tools to benchmark and analyze the complexity of your algorithms. 
-It has been designed as an alternative to Benchmark that let your benchmarks 
+Viiite brings tools to benchmark and analyze the complexity of your algorithms.
+It has been designed as an alternative to Benchmark that let your benchmarks
 evolve smoothly from simple measures to more complex infrastructures.
 
     [sudo] gem install viiite
 
 Viiite uses [semantic versionning](http://semver.org) and has not yet reached the
-public API required for 1.0.0. The safe way to require viiite for now is as 
+public API required for 1.0.0. The safe way to require viiite for now is as
 follows:
 
-    gem "viiite", "~> 0.1.0"
+    gem "viiite", "~> 0.2.0"
 
 Learn more on the [github-pages of this project](http://blambeau.github.com/viiite)!
 
@@ -22,9 +22,8 @@ Learn more on the [github-pages of this project](http://blambeau.github.com/viii
 ## From simple measures ...
 
 ```ruby
-require 'viiite'
-n = 15000
-Viiite.bm do |r|
+Viiite.bench do |r|
+  n = 15000
   r.report(:for)   { for i in 1..n; a = "1"; end }
   r.report(:times) { n.times do   ; a = "1"; end }
   r.report(:upto)  { 1.upto(n) do ; a = "1"; end }
@@ -33,22 +32,21 @@ end
 
 ```terminal
 $ viiite report bench_iteration.rb
-+--------+-----------------------------------------------+
-| :bench | :measure                                      |
-+--------+-----------------------------------------------+
-| :for   |   0.010000   0.000000   0.010000 (  0.010454) |
-| :times |   0.000000   0.000000   0.000000 (  0.002448) |
-| :upto  |   0.000000   0.000000   0.000000 (  0.002451) |
-+--------+-----------------------------------------------+
++--------+----------+----------+----------+----------+
+| :bench | :user    | :system  | :total   | :real    |
++--------+----------+----------+----------+----------+
+| :for   | 0.010000 | 0.000000 | 0.010000 | 0.013039 |
+| :times | 0.000000 | 0.000000 | 0.000000 | 0.003753 |
+| :upto  | 0.010000 | 0.000000 | 0.010000 | 0.003803 |
++--------+----------+----------+----------+----------+
 ```
 
 ## To more complex ones ...
 
 ```ruby
-require 'viiite'
-n = 15000
-Viiite.bm do |r|
+Viiite.bench do |r|
   r.variation_point :ruby, Viiite.which_ruby
+  n = 15000
   r.report(:for)   { for i in 1..n; a = "1"; end }
   r.report(:times) { n.times do   ; a = "1"; end }
   r.report(:upto)  { 1.upto(n) do ; a = "1"; end }
@@ -56,96 +54,43 @@ end
 ```
 
 ```terminal
-$ rvm exec viiite run bench_iteration.rb | viiite report --hierarchy --regroup=viiite,ruby
-+--------+--------------------------------------------------------------------+
-| :bench | :measure                                                           |
-+--------+--------------------------------------------------------------------+
-| :for   | +----------------+-----------------------------------------------+ |
-|        | | :ruby          | :measure                                      | |
-|        | +----------------+-----------------------------------------------+ |
-|        | | ruby 1.8.7     |   0.010000   0.000000   0.010000 (  0.009677) | |
-|        | | ruby 1.9.3dev  |   0.005000   0.000000   0.005000 (  0.005175) | |
-|        | | jruby 1.6.3    |   0.071000   0.000000   0.071000 (  0.071000) | |
-|        | +----------------+-----------------------------------------------+ |
-| :times | +----------------+-----------------------------------------------+ |
-|        | | :ruby          | :measure                                      | |
-|        | +----------------+-----------------------------------------------+ |
-|        | | ruby 1.8.7     |   0.000000   0.000000   0.000000 (  0.003275) | |
-|        | | ruby 1.9.3dev  |   0.005000   0.000000   0.005000 (  0.002634) | |
-|        | | jruby 1.6.3    |   0.028000   0.000000   0.028000 (  0.028000) | |
-|        | +----------------+-----------------------------------------------+ |
-| ...    | ...                                                                |
+$ rvm exec viiite run bench_iteration.rb | viiite report --hierarchy --regroup=bench,ruby
++--------+-------------------------------------------------------+
+| :bench | :measure                                              |
++--------+-------------------------------------------------------+
+| :for   | +----------------+-------+---------+--------+-------+ |
+|        | | :ruby          | :user | :system | :total | :real | |
+|        | +----------------+-------+---------+--------+-------+ |
+|        | | ruby 1.8.7     | 0.000 |   0.000 |  0.000 | 0.004 | |
+|        | | ruby 1.9.3dev  | 0.010 |   0.000 |  0.010 | 0.014 | |
+|        | +----------------+-------+---------+--------+-------+ |
+| :times | +----------------+-------+---------+--------+-------+ |
+|        | | :ruby          | :user | :system | :total | :real | |
+|        | +----------------+-------+---------+--------+-------+ |
+|        | | ruby 1.8.7     | 0.020 |   0.000 |  0.020 | 0.018 | |
+|        | | ruby 1.9.3dev  | 0.010 |   0.000 |  0.010 | 0.004 | |
+|        | +----------------+-------+---------+--------+-------+ |
+| :upto  | +----------------+-------+---------+--------+-------+ |
+|        | | :ruby          | :user | :system | :total | :real | |
+|        | +----------------+-------+---------+--------+-------+ |
+|        | | ruby 1.8.7     | 0.010 |   0.000 |  0.010 | 0.004 | |
+|        | | ruby 1.9.3dev  | 0.010 |   0.000 |  0.010 | 0.011 | |
+|        | +----------------+-------+---------+--------+-------+ |
++--------+-------------------------------------------------------+
 ```
 
 ## To awesomeness ...
 
 ```ruby
-require 'viiite'
-Viiite.bm do |b|
+Viiite.bench do |b|
   b.variation_point :ruby, Viiite.which_ruby
   b.range_over([100, 200, 300, 400, 500], :size) do |size|
-    b.range_over(1..5, :i) do
-      bench_case = Array.random(size)
-      b.report(:quicksort) { bench_case.quicksort }
-      b.report(:bubblesort){ bench_case.bubblesort }
-    end
+    bench_case = Array.new(size){ rand }
+    b.report(:bubblesort){ bench_case.bubblesort }
   end
 end
 ```
 
-```terminal
-$ viiite plot bench_sort.rb -x size -y tms.total --graph=ruby --series=viiite --gnuplot | gnuplot
-
-                                    ruby 1.8.7
-
-  0.35 ++-------+-------+--------+-------+--------+--------+-------+-------++
-       +        +       +        +       +        +       bubblesort **A*** +
-       |                                                   quicksort ##B###*A
-   0.3 ++                                                               ***++
-       |                                                             ***    |
-  0.25 ++                                                          **      ++
-       |                                                        ***         |
-       |                                                     ***            |
-   0.2 ++                                                 *A*              ++
-       |                                              ****                  |
-       |                                          ****                      |
-  0.15 ++                                      ***                         ++
-       |                                   ****                             |
-       |                               **A*                                 |
-   0.1 ++                        ******                                    ++
-       |                   ******                                           |
-  0.05 ++           ****A**                                                ++
-       |    ********                                                        |
-       A****    +       +        +       +        +        +       +########B
-     0 B################B################B#################B########-------++
-      100      150     200      250     300      350      400     450      500
-
-
-                                  ruby 1.9.3dev
-
-   0.1 ++-------+-------+--------+-------+--------+--------+-------+-------*A
-       +        +       +        +       +        +       bubblesort **A*** +
-       |                                                   quicksort **B### |
-       |                                                           **       |
-  0.08 ++                                                       ***        ++
-       |                                                     ***            |
-       |                                                  *A*               |
-       |                                               ***                  |
-  0.06 ++                                           ***                    ++
-       |                                         ***                        |
-       |                                      ***                           |
-  0.04 ++                                  ***                             ++
-       |                               **A*                                 |
-       |                           ****                                     |
-       |                       ****                                         |
-  0.02 ++                  ****                                            ++
-       |              **A**                                                 |
-       |        ******                                                      |
-       +  ******+       +        +       +        +        +       +########B
-     0 A**##############B################B#################B########-------++
-      100      150     200      250     300      350      400     450      500
-```
-
 ```terminal
 $ viiite plot bench_sort.rb -x size -y tms.total --graph=viiite --series=ruby --gnuplot | gnuplot
                                     bubblesort
@@ -171,34 +116,8 @@ $ viiite plot bench_sort.rb -x size -y tms.total --graph=viiite --series=ruby --
        A****    ########B########+       +        +        +       +        +
      0 B########+-------+--------+-------+--------+--------+-------+-------++
       100      150     200      250     300      350      400     450      500
-
-
-                                      quicksort
-
-  0.012 ++-------+-------+--------+-------+--------+-------+--------+------++
-        +        +       +        +       +        +      ruby 1.8.7+**A*** +
-        |                                              ruby 1.9.3dev ##B### |
-   0.01 ++                                                                **A
-        |                                                           ******  |
-        |                                                     ******        |
-  0.008 ++                                                 A**             ++
-        |                                                **                 |
-        |                                              **                   |
-        |                                             *                     B
-  0.006 ++                                          **                    ##+
-        |                                         **                    ##  |
-        |                                       **                     #    |
-  0.004 ++                                     *                     ##    ++
-        |                                    **                    ##       |
-        |                                  **                    ##         |
-  0.002 ++             ##B##############**A########             #          ++
-        |        ######           ******           ######     ##            |
-        +  ######+       +  ******+       +        +     ##+##      +       +
-      0 A****************A**------+-------+--------+-------B--------+------++
-       100      150     200      250     300      350     400      450     500
-
 ```
 
 ## On the devel side
 
-Fork the project on github ... and so on. 
+Fork the project on github ... and so on.

data/examples/bench_attr_vs_method.rb

@@ -0,0 +1,18 @@
+class Foo
+  attr_reader :bench_reader
+  def initialize
+    @bench_reader = @bench_method = 10
+  end
+  def bench_method
+    @bench_method
+  end
+end
+
+Viiite.bench do |b|
+  foo = Foo.new
+  b.variation_point :ruby, Viiite.which_ruby
+  b.range_over([1, 100_000, 1_000_000], :runs) do |runs|
+    b.report(:bench_reader){ runs.times{ foo.bench_reader } }
+    b.report(:bench_method){ runs.times{ foo.bench_method } }
+  end
+end

data/examples/bench_iteration.rb

@@ -1,9 +1,7 @@
-require 'viiite'
-n = 15000
-Viiite.bm do |r|
+Viiite.bench do |r|
+  n = 15000
   r.variation_point :ruby, Viiite.which_ruby
   r.report(:for)   { for i in 1..n; a = "1"; end }
   r.report(:times) { n.times do   ; a = "1"; end }
   r.report(:upto)  { 1.upto(n) do ; a = "1"; end }
 end
-

data/examples/bench_sort.rb

@@ -1,9 +1,5 @@
 class Array
 
-  def self.random(size)
-    Array.new(size){ Kernel.rand }
-  end
-
   def quicksort(list = self)
      return list if list.nil? or list.size <= 1
      less, more = list[1..-1].partition { |i| i < list[0] }
@@ -25,14 +21,13 @@ class Array
 end
 
 require 'viiite'
-Viiite.bm do |b|
+Viiite.bench do |b|
   b.variation_point :ruby, Viiite.which_ruby
   b.range_over([100, 200, 300, 400, 500], :size) do |size|
     b.range_over(1..5, :i) do
-      bench_case = Array.random(size)
+      bench_case = Array.new(size) { rand }
       b.report(:quicksort) { bench_case.quicksort }
       b.report(:bubblesort){ bench_case.bubblesort }
     end
   end
 end
-

data/lib/viiite.rb

@@ -1,31 +1,35 @@
+require 'fileutils'
+require "delegate"
+
 require "viiite/version"
 require "viiite/loader"
+require "viiite/errors"
 require "viiite/tms"
-require "viiite/formatter"
-require "viiite/runner"
+require "viiite/benchmark"
 require "viiite/command"
-require "viiite/viiite_file"
-require "benchmark"
+require "viiite/bdb"
 
 #
 # Benchmarking and complexity analyzer utility
 #
 module Viiite
 
-  # Builds a Tms object
-  def self.Tms(*args)
-    Viiite::Tms.coerce(args)
-  end
-
-  def self.measure(&block)
-    Viiite::Tms.coerce Benchmark.measure(&block)
+  def self.measure
+    t0, r0 = Process.times, Time.now
+    yield
+    t1, r1 = Process.times, Time.now
+    Tms.new(t1.utime  - t0.utime,
+            t1.stime  - t0.stime,
+            t1.cutime - t0.cutime,
+            t1.cstime - t0.cstime,
+            r1 - r0)
   end
 
   # Builds a runner instance via the DSL definition given by the block.
   #
   # Example
   #
-  #  Viiite.runner do |b|
+  #  Viiite.bench do |b|
   #    b.variation_point :ruby_version, Viiite.which_ruby
   #    b.range_over([100, 1000, 10000, 100000], :runs) do |runs|
   #      b.variation_point :test, :via_reader do
@@ -36,20 +40,38 @@ module Viiite
   #      end
   #    end
   #  end
-  # 
+  #
+  def self.bench(&block)
+    Benchmark.new(block)
+  end
+
+  # Alias of Viiite.bench for compatibility
   def self.bm(&block)
-    Runner.new(block)
+    warn "Viiite.bm is deprecated, use Viiite.bench instead #{caller[0]}"
+    bench(&block)
   end
 
   #
   # Returns a short string with a ruby interpreter description
-  # 
+  #
   def self.which_ruby
     if Object.const_defined?(:RUBY_DESCRIPTION)
-      RUBY_DESCRIPTION =~ /^([^\s]+\s*[^\s]+)/
-      $1
+      short_ruby_description(RUBY_DESCRIPTION)
+    else
+      "ruby #{RUBY_VERSION}"
+    end
+  end
+
+  def self.short_ruby_description(description)
+    case description
+    when /Ruby Enterprise Edition (\d{4}\.\d{2})/
+      "ree #{$1}"
+    when /^(\w+ \d\.\d\.\d) .+ patchlevel (\d+)/
+      "#{$1}p#{$2}"
+    when /^\w+ \S+/
+      $&
     else
-      "ruby #{RUBY_VERSION} (#{RUBY_PLATFORM})"
+      raise "Unknown ruby interpreter: #{description}"
     end
   end