bench9000 0.1

data/lib/bench9000/stats.rb

@@ -0,0 +1,41 @@
+# Copyright (c) 2014 Oracle and/or its affiliates. All rights reserved. This
+# code is released under a tri EPL/GPL/LGPL license. You can use it,
+# redistribute it and/or modify it under the terms of the:
+# 
+# Eclipse Public License version 1.0
+# GNU General Public License version 2
+# GNU Lesser General Public License version 2.1
+
+module Bench
+
+  module Stats
+
+    def self.sum(samples)
+      samples.inject(0, :+)
+    end
+
+    def self.mean(samples)
+      sum(samples) / samples.size
+    end
+
+    def self.sample_variance(samples)
+      m = mean(samples)
+      sum = samples.inject(0) { |a, i| a + (i - m)**2 }
+      sum / (samples.length - 1).to_f
+    end
+
+    def self.standard_deviation(samples)
+      Math.sqrt(sample_variance(samples))
+    end
+
+    def self.range(samples)
+      samples.max - samples.min
+    end
+
+    def self.format_percent(p)
+      "#{(p * 100).round(2)}%"
+    end
+
+  end
+
+end

data/readme.md

@@ -0,0 +1,205 @@
+## Philosophy
+
+We hope to provide a range of benchmarking tools for Ruby implementers that is suited to a range of different use cases. A common set of benchmarks is coupled with a harness that provides multiple commands.
+
+Questions that we aim to answer include 'have my last few commits affected performance?', 'how much faster is this implementation of Ruby compared to this one?', 'how long does this implementation take to warm up compared to this one?'. We also include functionality for graphing results.
+
+We focus on macro benchmarks (at least a screen-full of code and hopefully much more) and kernels isolated from production code. We also focus primarily on peak temporal performance (how fast it runs when it's warmed up), but we do include measurements of how long warmup takes.
+
+We include common synthetic benchmarks as well as benchmarks taken from unmodified off-the-shelf gems in use in production today.
+
+Another focus is being gentle on early implementations of Ruby. Benchmarks are run in a subprocess with minimal language requirements for the harness.
+
+Detailed notes on the methodology we have used can be found in the final section of this document.
+
+## Scoring benchmarks
+
+To get an abstract score that represents performance with higher is better:
+
+    bench9000 score 2.2.3 classic
+
+<!-- -->
+
+    classic-mandelbrot 2.2.3 2000
+    classic-nbody 2.2.3 2000
+    ...
+
+The score is `1 / time * arbitrary constant`, with the arbitrary constant so that JRuby on a typical system achieves around the order of 1000.
+
+## Detailed timing results
+
+There are several formats which you can print detailed information in. In all formats the detailed information includes, in order where relevant:
+
+* Total warmup time (s)
+* Mean sample time (s)
+* Abstract score (no unit)
+
+`--value-per-line` lists each value on its own line with keys, perhaps for reading by another program:
+
+    bench9000 detail 2.2.3 rbx-2.5.8 classic --value-per-line
+
+<!-- -->
+
+    classic-mandelbrot 2.2.3 warmup 20
+    classic-mandelbrot 2.2.3 sample 10
+    classic-mandelbrot 2.2.3 score 2000
+    classic-mandelbrot rbx-2.5.8 warmup 20
+    classic-mandelbrot rbx-2.5.8 sample 10
+    classic-mandelbrot rbx-2.5.8 score 2000
+    ...
+
+`--benchmark-per-line` lists one benchmark per line, with clusters of values for each implementation and no keys, perhaps for pasting into a spreadsheet:
+
+    bench9000 detail 2.2.3 rbx-2.5.8 classic --benchmark-per-line
+
+<!-- -->
+
+    classic-mandelbrot 20 10 2000 20 10 2000
+    ...
+
+`--json` outputs all the data in JSON:
+
+    bench9000 detail 2.2.3 rbx-2.5.8 classic --json
+
+<!-- -->
+
+    {
+        "benchmarks": [...],
+        "implementations": [...],
+        "measurements": [
+            {
+                "benchmark": ...,
+                "implementation": ...,
+                "warmup_time": ...,
+                "sample_mean": ...,
+                "sample_error": ...,
+                "score": ...,
+                "score_error": ...,
+                "warmup_samples": [...],
+                "samples": [...]
+            },
+            ...
+        ]
+    }
+
+## Comparing two implementations
+
+The performance of two or more implementations can be compared against the first:
+
+    bench9000 compare 2.2.3 rbx-2.5.8 jruby-1.7.18 classic
+
+<!-- -->
+
+    classic-mandelbrot 150% 200%
+    classic-nbody 150% 200%
+    ...
+
+The percentage shows performance of each implementation beyond the first, relative to the first.
+
+## Comparing performance against a reference
+
+Set a reference point with some Ruby:
+
+    bench9000 compare-reference 2.2.3 classic
+
+<!-- -->
+
+    classic-mandelbrot 2000
+    classic-nbody 2000
+    ...
+
+This writes data to `reference.txt`.
+
+You can then compare against this reference point multiple times:
+
+    bench9000 compare-reference jruby-1.7.18-noindy jruby-1.7.18-indy
+
+<!-- -->
+
+    classic-mandelbrot 150% 200%
+    classic-nbody 150% 200%
+    ...
+
+This reads data from `reference.txt`.
+
+## Detailed Reports
+
+To generate a detailed interactive report with charts and tables:
+
+    bench9000 report 2.2.3 rbx-2.5.8 classic
+
+The output will go into report.html. You can set `--baseline implementation` to configure which implementation to use as the default baseline. You can set `--notes notes.html` to include extra content in the notes tab.
+
+## Separating measurement and report generation
+
+All commands accept a `--data` flag. If this is set measurements will be read from this file and used instead of making new measurements. Existing measurements, plus any new measurements will also be written back to the file.
+
+You can use this with your commands as normal:
+
+    bench9000 report 2.2.3 rbx-2.5.8 classic --data data.txt
+
+This allows you to interrupt and resume measurements, and to keep measurements from one report to use in another, such as implementations that have not changed.
+
+The `remove` command allows measurements to be removed from the file. Here the listed implementations and benchmarks are those to remove, rather than those to measure:
+
+    bench9000 remove 2.2.3 --data data.txt
+
+A common use of this is to keep a file with measurements from implementations that are stable, and to remove development implementations to re-measure after changes.
+
+## Other options
+
+* `--show-commands` print the implementation commands as they're executed
+* `--show-samples` print individual samples as they're measured
+
+If you want to run a benchmark manually, you can use `--show-commands` to get the Ruby command being run. This command normally expects to be driven by the harness over a pipe to produce more results or to stop. If you want to use the benchmark without the harness you can use the `--loop` command to just keep running forever, printing iteration times, or the `--loopn a b` command to run `a` iterations and `b` micro-iterations if the benchmark is using the micro harness (just set `b` to one if you're not using the micro harness).
+
+## Configuration
+
+Configuration files specify the available Ruby implementations, benchmarks and benchmark groups. A configuration file named `default.config.rb`is automatically loaded. It's being looked up in `{.,bench,benchmark,benchmarks}` directories. You can load additional configuration files for any command using `--config file` (Using same directories for lookup. Suffix `.config.rb` can be omitted.). 
+
+## Implementations
+
+The default configuration includes common Ruby implementations which are expected to be found in `rbenv`.
+
+* `1.8.7-p375`
+* `1.9.3-p551`
+* `2.0.0-p647`
+* `2.1.7`
+* `2.2.3`
+* `2.3.0-dev`
+* `jruby-9.0.4.0-int`
+* `jruby-9.0.4.0-noindy`
+* `jruby-9.0.4.0-indy`
+* `rbx-2.5.8-int`
+* `rbx-2.5.8`
+* `topaz-dev`
+
+Development versions of implementations are enabled if you set environment variables to find them. You should have a built version of the implementation in this path, which should be the root directory of the repository.
+
+* `jruby-dev-(int|noindy|indy)` - set `JRUBY_DEV_DIR`
+* `jruby-dev-truffle-nograal` - set `JRUBY_DEV_DIR`
+* `jruby-dev-truffle-graal` - set `JRUBY_DEV_DIR` and `GRAAL_BIN`
+
+You can add your own implementations by writing your own config file, or use the `custom` implementation by setting `RUBY_BIN` and `RUBY_ARGS`.
+
+## Methodology
+
+### Warmup
+
+Before we consider if an implementation is warmed up, we run for at least T0 seconds.
+
+We then consider an implementation to be warmed up when the last N samples have a range relative to the mean of less than E. When a run of N samples have a range less than that, they are considered to be the first N samples where the implementation is warmed up. If the benchmark does not reach this state within W samples or within T1 seconds, we give the user a warning and will start measuring samples anyway. Some benchmarks just don't warm up, or don't warm up in reasonable time. It's fine to ignore this warning in normal usage.
+
+We then take S samples (starting with the last N that passed our warmup test) for our measurements.
+
+If you are going to publish results based on these benchmarks, you should manually verify warmup using lag or auto-correlation plots.
+
+We have chosen T0 to be 30, N to be 20, E to be 0.1, W to be 100, T1 to be 240, and S to be 10. These are arbitrary, but by comparing with the lag plots of our data they do seem to do the right thing.
+
+### Error
+
+We currently use the standard deviation as our error.
+
+## Rubygem
+
+This tool is also released as a gem for use in other projects. See [example usage](https://github.com/ruby-concurrency/concurrent-ruby/tree/master/benchmarks). 

data/vendor/chunky_png/BENCHMARKS.rdoc

@@ -0,0 +1,31 @@
+= ChunkyPNG benchmark suite
+
+This small benchmark suite is intended to test the speed of PNG decoding 
+and encoding against different ruby interpreters. You can set the number 
+of runs by passing the N environment variable. You can execute them as
+rake task or as standalone script.
+
+    rake benchmark:encoding
+    rake benchmark:decoding
+    
+    rake benchmark N=10           # Run all of them, using n=10
+
+Use rvm to simply run the tests against different interpreters. Of course, 
+make sure that the chunky_png is installed for all your interpreters.
+
+    rvm 1.8.7,1.9.3,rbx rake benchmark N=10
+
+== Results
+
+Some of my benchmark (N=50) result on my 2007 iMac can be found in this gist:
+http://gist.github.com/495323
+
+== Why is this relevant?
+
+ChunkyPNG is a pure Ruby library to handle PNG files. Decoding a PNG requires
+a lot of integer math and bitwise operations, and moderate use of the unpack
+method to read binary data. Encoding is a good test for Array#pack, and
+depending on the encoding options, also requires a lot of calculations.
+Therefore, the library is a good benchmark candidate for these methods and
+algorithms. It has been used to improve the Array#pack and String#unpack
+methods in Rubinius.

data/vendor/chunky_png/Gemfile

@@ -0,0 +1,10 @@
+source 'https://rubygems.org'
+gemspec
+
+platforms :jruby do
+  gem 'jruby-openssl'
+end
+
+platform :rbx do
+  gem 'rubysl'
+end

data/vendor/chunky_png/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010-2013 Willem van Bergen
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/vendor/chunky_png/README.rdoc

@@ -0,0 +1,84 @@
+= Chunky PNG {<img src="https://travis-ci.org/wvanbergen/chunky_png.png?branch=master" alt="Build Status" />}[https://travis-ci.org/wvanbergen/chunky_png]
+
+This library can read and write PNG files. It is written in pure Ruby for
+maximum portability. Let me rephrase: it does NOT require RMagick or any other
+memory leaking image library.
+
+Source code:: http://github.com/wvanbergen/chunky_png/tree
+RDoc:: http://rdoc.info/gems/chunky_png/frames
+Wiki:: http://github.com/wvanbergen/chunky_png/wiki
+Issue tracker:: http://github.com/wvanbergen/chunky_png/issues
+
+== Features
+
+* Decodes any image that the PNG standard allows. This includes all standard
+  color modes, all bit depths and all transparency, interlacing and filtering options.
+* Encodes images supports all color modes (true color, grayscale and indexed)
+  and transparency for all these color modes. The best color mode will be
+  chosen automatically, based on the amount of used colors.
+* R/W access to the image's pixels.
+* R/W access to all image metadata that is stored in chunks.
+* Memory efficient (uses a Fixnum, i.e. 4 or 8 bytes of memory per pixel, depending
+  on the hardware)
+* Reasonably fast for Ruby standards, by only using integer math and a highly
+  optimized saving routine.
+* Interoperability with RMagick if you really have to.
+
+Also, have a look at OilyPNG at http://github.com/wvanbergen/oily_png. OilyPNG is a 
+mixin module that implements some of the ChunkyPNG algorithms in C, which 
+provides a massive speed boost to encoding and decoding.
+
+== Usage
+
+  require 'chunky_png'
+
+  # Creating an image from scratch, save as an interlaced PNG
+  png = ChunkyPNG::Image.new(16, 16, ChunkyPNG::Color::TRANSPARENT)
+  png[1,1] = ChunkyPNG::Color.rgba(10, 20, 30, 128)
+  png[2,1] = ChunkyPNG::Color('black @ 0.5')
+  png.save('filename.png', :interlace => true)
+
+  # Compose images using alpha blending.
+  avatar = ChunkyPNG::Image.from_file('avatar.png')
+  badge  = ChunkyPNG::Image.from_file('no_ie_badge.png')
+  avatar.compose!(badge, 10, 10)
+  avatar.save('composited.png', :fast_rgba) # Force the fast saving routine.
+  
+  # Accessing metadata
+  image = ChunkyPNG::Image.from_file('with_metadata.png')
+  puts image.metadata['Title']
+  image.metadata['Author'] = 'Willem van Bergen'
+  image.save('with_metadata.png') # Overwrite file
+  
+  # Low level access to PNG chunks
+  png_stream = ChunkyPNG::Datastream.from_file('filename.png')
+  png_stream.each_chunk { |chunk| p chunk.type }
+
+Also check out the following screencast by John Davison, which illustrates 
+basic usage of the library: http://devcasts.co/ruby-chunky_png-a-great-gem-for-on-the-fly-png-manipulation.
+
+For more information, see the project wiki on http://github.com/wvanbergen/chunky_png/wiki
+or the RDOC documentation on http://rdoc.info/gems/chunky_png/frames
+
+== Security warning
+
+ChunkyPNG is vulnerable to decompression bombs, which means that ChunkyPNG is vulnerable to 
+DOS attacks by running out of memory when loading a specifically crafted PNG file. Because 
+of the pure-Ruby nature of the library it is very hard to fix this problem in the library 
+itself.
+
+In order to safely deal with untrusted images, you should make sure to do the image 
+processing using ChunkyPNG in a separate process, e.g. by using fork or a background
+processing library.
+
+== About
+
+The library is written by Willem van Bergen for Floorplanner.com, and released
+under the MIT license (see LICENSE). Please contact me for questions or
+remarks. Patches are greatly appreciated!
+
+Please check out the changelog on https://github.com/wvanbergen/chunky_png/wiki/Changelog
+to see what changed in all versions.
+
+P.S.: The name of this library is intentionally similar to Chunky Bacon and 
+Chunky GIF. Use Google if you want to know _why. :-)

data/vendor/chunky_png/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+Dir['tasks/*.rake'].each { |file| load(file) }
+
+RSpec::Core::RakeTask.new(:spec) do |task|
+  task.pattern = "./spec/**/*_spec.rb"
+  task.rspec_opts = ['--color']
+end
+
+task :default => [:spec]

data/vendor/chunky_png/benchmarks/decoding_benchmark.rb

@@ -0,0 +1,36 @@
+require 'rubygems'
+require 'bundler/setup'
+
+require 'benchmark'
+require 'chunky_png'
+
+def image_file(name)
+  File.join(File.dirname(__FILE__), '..', 'spec', 'resources', name)
+end
+
+def image_data(name)
+  data = nil
+  File.open(image_file(name), 'rb') { |f| data = f.read }
+  data
+end
+
+no_filtering_stream    = image_data('pixelstream_fast_rgba.png')
+up_filtering_stream    = image_data('pixelstream_reference.png')
+paeth_filtering_stream = image_data('pixelstream_best_compression.png')
+rgba_pixelstream       = image_data('pixelstream.rgba')
+rgb_pixelstream        = image_data('pixelstream.rgb')
+
+n = (ENV['N'] || '5').to_i
+
+puts "---------------------------------------------"
+puts "ChunkyPNG (#{ChunkyPNG::VERSION}) decoding benchmark (n=#{n})"
+puts "---------------------------------------------"
+puts
+
+Benchmark.bmbm do |x|
+  x.report('PNG - no filtering')    { n.times { ChunkyPNG::Image.from_blob(no_filtering_stream) } }
+  x.report('PNG - UP filtering')    { n.times { ChunkyPNG::Image.from_blob(up_filtering_stream) } }
+  x.report('PNG - PAETH filtering') { n.times { ChunkyPNG::Image.from_blob(paeth_filtering_stream) } }
+  x.report('From RGBA pixelstream') { n.times { ChunkyPNG::Image.from_rgba_stream(240, 180, rgba_pixelstream) } }
+  x.report('From RGB pixelstream')  { n.times { ChunkyPNG::Image.from_rgb_stream(240, 180, rgb_pixelstream) } }
+end